# Engine agreement study: methodology

**Question:** How often do ChatGPT, Perplexity and Claude recommend the same brands for the same buying question?

**Date run:** 2026-11-04 (all 90 answers collected the same day, within about 10 minutes).

## Sample

- 30 neutral, unbranded English buyer-intent questions written for a US buyer, 3 per category across 10 categories: CRM for small business, project management software, email marketing, accounting software for freelancers, running shoes, mattresses, password managers, website builders, VPNs, online payroll. No question names a brand. Full list in `questions.json`.
- Each question asked once per engine: 30 x 3 = 90 answers. 0 API errors.

## Engines and settings (mirrors TrendlyAI's engine calls)

| Engine | API | Model | Search settings |
|---|---|---|---|
| ChatGPT | OpenAI Responses API | gpt-4o-mini | tools: `web_search_preview`, user_location approximate, country US. The model decides whether to search. |
| Perplexity | Chat Completions | sonar | `web_search_options.user_location.country = US` (search context size reported as "low") |
| Claude | Anthropic Messages | claude-haiku-4-5 | tool `web_search_20250305`, max_uses 3, user_location approximate, country US. The model decides whether to search. max_tokens 2048. |

No system prompt was used. The user message was the question text only. Concurrency was 6 parallel requests with up to 4 retries (exponential backoff).

Search usage that resulted: ChatGPT searched on 11 of 30 questions, Claude on 23 of 30, and Perplexity on all 30 (it always searches).

## Brand extraction

- Each answer was passed to claude-haiku-4-5 (temperature 0) with a strict JSON extraction prompt (see `extract.mjs`). It returns the ordered list of brands the answer recommends or presents as options, normalized to brand level ("HubSpot CRM" becomes "HubSpot", "Brooks Ghost 16" becomes "Brooks"). Review sites, publishers, brands mentioned only as integrations, and brands named only as something to avoid are excluded.
- A second normalization pass in `analyze.mjs` merges remaining variants: Zoho Books / Zoho Invoice / Zoho Campaigns -> Zoho; QuickBooks Online -> QuickBooks; Monday.com / monday -> monday.com; Hoka One One / Hoka -> HOKA; Sendinblue -> Brevo and ConvertKit -> Kit (both renamed brands); Freshsales -> Freshworks; Keeper Security -> Keeper; ProtonVPN -> Proton VPN; Patriot Software -> Patriot; WinkBeds -> WinkBed; Square Online -> Square; Jira Software -> Jira; WordPress.com -> WordPress; Shopify Email -> Shopify; Apple Passwords -> Apple; Hostinger Reach -> Hostinger. Duplicates within one answer are kept at first position only.
- **Spot check (10 answers checked by hand against the raw text):** pm-2|claude, crm-2|chatgpt, email-3|perplexity, web-1|chatgpt, crm-1|chatgpt, pw-3|perplexity, matt-1|perplexity, email-2|perplexity, matt-2|claude, shoe-1|claude. 9 of 10 were correct. pm-2|claude wrongly included HubSpot, Salesforce and Adobe, which the answer named only as integrations. The prompt was tightened to exclude integrations, and all 90 extractions were rerun (first pass kept as `raw/extracted.v1.json`). The rerun fixed pm-2|claude. The two empty Claude extractions (matt-2, shoe-1) are correct: in both, Claude answered from general principles without searching and named no brands.
- Judgment calls: brands that belong to a parent (Titan by Brooklyn Bedding, Freshsales by Freshworks) are kept at the level the answer used, apart from the aliases above. "Trello with a CRM power-up" in a CRM answer counts as Trello.

## Metrics

Per question (see `data/per-question-metrics.csv`):
- **Top-1 agreement:** all 3 engines share the same first brand ("all3"), exactly 2 do ("two"), or none do ("none"). An engine that named no brands has no top pick and can't match.
- **Top-5 overlap:** pairwise Jaccard similarity of each engine's first 5 brands (intersection / union), and the count of brands that appear in all three top-5 lists.
- **Unique brands:** size of the union of all brands named by the three engines (full lists).
- **Brands named by one engine only:** brands in that union named by exactly one engine.
- **Brands per answer** and **sources cited per answer** per engine. Sources = unique URLs: ChatGPT `url_citation` annotations, Perplexity's `citations` list, Claude's text citations (`web_search_result_location`). Perplexity lists every source it retrieved, while ChatGPT and Claude count only inline citations, so source counts are not strictly like for like.

Aggregates overall and by category are in `data/metrics-summary.json` and `data/by-category-metrics.csv`. Brand frequency is in `data/brand-frequency.csv`.

## Headline results

- Top-1 agreement: all three engines matched on 7 of 30 questions (23%), exactly two matched on 16 (53%), and none matched on 7 (23%).
- Pairwise top-1 match: ChatGPT and Perplexity 11/30, ChatGPT and Claude 13/30, Perplexity and Claude 13/30.
- Mean top-5 Jaccard: 0.48 (ChatGPT-Perplexity 0.53, ChatGPT-Claude 0.47, Perplexity-Claude 0.43).
- Brands in all three top-5 lists: 2.23 on average.
- Unique brands per question: 8.23 on average. 127 distinct brands overall. 114 of 247 question-brand pairs (46%) were named by only one engine.
- Brands per answer: ChatGPT 5.50, Perplexity 4.77, Claude 4.83.
- Sources per answer: Perplexity 10.0 (all 30 answers), ChatGPT 3.0 across all answers (8.2 on the 11 answers where it searched), Claude 3.33 across all answers (4.3 on the 23 where it searched).

## Caveats

- **One sample per engine per question.** AI answers vary from run to run, so a rerun would give different lists. These are snapshot rates, not stable estimates. With 30 questions, one question moves a percentage by about 3 points.
- **API answers, not consumer apps.** The ChatGPT, Perplexity and Claude consumer apps use different (usually larger) models, system prompts, memory and personalization. gpt-4o-mini, sonar and claude-haiku-4-5 are the low-cost API models TrendlyAI uses.
- **Search is the model's choice** for ChatGPT and Claude. Some differences come from one engine searching while another didn't. ChatGPT-Perplexity top-5 overlap was 0.64 on the 11 questions where ChatGPT searched vs 0.46 on the 19 where it didn't (small samples, treat as a hint).
- **Extraction is model-based.** It was spot-checked on 10 answers with one fix applied, but it is not perfect. Brand-level normalization treats different products from the same brand (e.g. two Brooks shoes) as agreement.
- **US, English, consumer and SMB categories only.** Results may differ in other markets, languages and B2B enterprise categories.

## Cost

Estimated total API spend: about US$1.28. That breaks down as ChatGPT $0.29, Perplexity $0.14 (from Perplexity's reported per-call cost), Claude $0.58, and extraction $0.28 (both extraction passes). Figures use list prices: gpt-4o-mini $0.15/$0.60 per M tokens plus $25 per 1k web search calls, and Haiku 4.5 $1/$5 per M tokens plus $10 per 1k searches.

## Files

- `questions.json`: the 30 questions
- `run.mjs`, `extract.mjs`, `analyze.mjs`: collection, extraction and analysis scripts (Node 22, no dependencies)
- `raw/answers.json`: full raw answers, sources and usage per engine and question
- `raw/extracted.json`: brand extraction output (`raw/extracted.v1.json` is the first pass)
- `data/answers-brands.json|csv`: normalized brand lists per answer
- `data/per-question-metrics.csv`, `data/by-category-metrics.csv`, `data/brand-frequency.csv`, `data/metrics-summary.json`