Last updated: 8 October 2026 We asked ChatGPT, Claude and Perplexity the same 16 questions about prop firms, with web search switched on, and checked all 48 answers against the firms’ own pages. 36 were fully correct. Six were wrong, five partly correct and one outdated. The misses clustered around prices and pass rates, and most traced back to third-party sites rather than the firm.
36 of 48 answers fully correct (75%). By assistant: Perplexity 14 of 16, ChatGPT 11 of 16, Claude 11 of 16.
All three gave a wrong price for Goat Funded Trader’s 100K account, each citing third-party listings.
Pass-rate answers went wrong when a model used a third-party simulation or an old Reddit post instead of the firm’s own disclosure.
How we tested
Each assistant got the same 16 questions on 2 October 2026 between about 1:27 and 1:29 AM ET, with web search enabled and no system prompt. The models were gpt-5.4-mini (ChatGPT), claude-haiku-4.5 (Claude) and Perplexity Sonar. Gemini was rate-limited on every attempt and is not included. One grader marked each answer against an answer key built from the firm’s own page, with the exact quote saved.
| Assistant | Correct | Partly correct | Outdated | Wrong |
|---|---|---|---|---|
| Perplexity (Sonar) | 14 | 1 | 0 | 1 |
| ChatGPT (gpt-5.4-mini) | 11 | 3 | 1 | 1 |
| Claude (haiku-4.5) | 11 | 1 | 0 | 4 |
| Total | 36 | 5 | 1 | 6 |
Where the answers went wrong
- Goat Funded Trader price. ChatGPT said $458, and Claude and Perplexity said $398 list ($298.50 with 25% off). Goat’s own pricing data said $478 list with a 45% code on the test date. All three cited third-party sites.
- Tradeify pass rate. Claude said Tradeify publishes no pass rate and offered a third-party simulation (33.5%). Tradeify’s homepage publishes 17.2%.
- MyFundedFutures pass rate. Claude gave 29.2% from a third-party simulation. The firm publishes 19.97%.
- Topstep pass rate. ChatGPT gave Topstep’s 2024 figures (12.4%) from a Reddit thread. The 2025 figure (16.8%) was on topstep.com.
- Tradeify price. Claude said $359, which is not the Growth 50K price. Tradeify’s checkout data said $145.
Limits of this test
- A small sample: 16 questions, one run each, lighter model versions, one grader.
- Prices change. Goat’s prices and codes have moved since 2 October, so this is a dated snapshot, not a ranking of assistants.
- We plan to re-run the same questions each quarter with refreshed answer keys.
Data: every question, answer, citation and grade is at propdatalab.com/ai-answers-prop-firms-audit/ (CSV on the page). The answer key draws on the same sources as our open rules dataset (DOI 10.5281/zenodo.23108163).
By PropDataLab · All PropDataLab datasets