Search is changing shape. Instead of a ranked list of links, users increasingly get a single generated answer — from a chatbot or an AI Overview — and act on it directly. Traditional link click-through rates have slid from roughly 15% to 8%. If the answer is the new surface, the answer is also the new ad slot. Generative Engine Marketing (GEM) asks: how do you place an ad inside a generated response without ruining the response?

What GEM-Bench provides

GEM-Bench is the first benchmark for the satisfaction–engagement trade-off in ad-injected responses. It covers two surfaces (chatbot and AI Overview) with three datasets: MT-Human (open-ended humanities questions), LM-Market (1,701 real-user travel/recipe/software queries), and CA-Prod (120 keyword queries over 2,215 human-labeled products). Responses are scored by a quantitative-plus-qualitative judge along user satisfaction (accuracy, naturalness, personality, trust) and engagement (ad notice, click intent).

Prompt-first vs. answer-first

The benchmark contrasts two families of pipelines. Ad-Chat is prompt-first: tell the LLM up front to work an ad into its answer. Ad-LLM is answer-first: generate the best raw answer, retrieve the most relevant ad, then find the insertion point that least disturbs the response flow before rewriting. The case studies are telling — prompt-first injection can distort facts (recommending an intercity coach for a 10-minute tram ride), while answer-first injection preserves the answer's utility and attaches the ad where it fits.

Findings

The best Ad-LLM variant beats Ad-Chat by +14.8–15.6% relative satisfaction across all three datasets, consistently across 4 judge LLMs × 3 datasets (12/12). A human study with 118 participants and 3,140 ratings agrees: 5 of 6 criteria favor Ad-LLM, and human–LLM judgments correlate at Spearman ρ = 0.73. An ad-density ablation shows satisfaction dropping steadily as more ads are injected, with predicted clicks peaking at just 2 ads per answer — more ads is not more engagement.

The benchmark ships as a production CLI and agent skill — bring your own ad database, models, and pricing, and reproduce or extend all three datasets.