Sign up and get 3 free requests with Start plan accessSign up →

Research finding

7 of 7 model lines were overconfident on average in the latest weekly window

Do frontier models know how often they are right?

Updated 18 Nov 2026 · sources: Weekly Calibration #15, Weekly Calibration #14, Monthly Calibration #3, Weekly Calibration #13, Weekly Calibration #12, Weekly Calibration #11, Weekly Calibration #10, Weekly Calibration #9, Weekly Calibration #8, Weekly Calibration #7, Weekly Calibration #6, Weekly Calibration #5, Weekly Calibration #4, Weekly Calibration #3, Weekly Calibration #2, Weekly Calibration #1

All findingsThe benchmarkResearch reports

Answer

In Weekly Calibration #15 (Nov 9-15, 2026 UTC), the field stated 60.9% mean confidence and hit 48.5% on 5,669 scored calls — a gap of +12.4pp at a Brier score of 0.2677. The field sits 12.4pp above its own accuracy, and that is not one model's problem: 7 of 7 model lines carry a positive gap in this window, and 7 of 7 score a Brier above 0.25 — the uninformative baseline the issues name, the score a predictor that answers 50% to everything would get. Claude Opus 5.5 recorded the lowest Brier score in this window, at 0.2555, with a mean confidence–accuracy gap of +7.2pp.

Hit rate follows the benchmark's TP/SL/expiry outcome rules; it is not simply the asset's price direction at the end of the forecast horizon.

Over the wider window, Monthly Calibration #3 put the field gap at +13.2pp and the field Brier at 0.2701 on 22,579 scored calls, with a field hit rate of 48.6%. Claude Opus 5.5 recorded the lowest Brier score over the month, at 0.2600. That issue also publishes a per-line drift verdict on the weeks inside the month: 4 narrowing, 1 widening, 2 flat.

Across the 15 published weekly windows the field gap has narrowed — from +20.4pp in Weekly Calibration #1 to +12.4pp in Weekly Calibration #15, a move of -8.0pp. The reports do not read that as a trend, and neither does this page: five weekly points cannot separate drift from market regime.

Evidence

Stated confidence against realised accuracy — Weekly Calibration #15

ModelnMean stated confidenceHit rateGapBrier95% Wilson
Claude Opus 5.561355.8%48.6%+7.2pp0.2555[44.7%, 52.6%]
Claude Fable 570559.2%48.5%+10.7pp0.2622[44.8%, 52.2%]
Qwen 3.8 Max72558.4%45.7%+12.8pp0.2650[42.1%, 49.3%]
Grok 4.796359.6%47.6%+12.1pp0.2658[44.4%, 50.7%]
DeepSeek V4 Pro83960.2%47.4%+12.8pp0.2679[44.1%, 50.8%]
GPT-6.1 Sol1,03464.6%51.1%+13.5pp0.2719[48.0%, 54.1%]
Gemini 3.1 Pro79066.2%49.8%+16.4pp0.2811[46.3%, 53.2%]
Field (all models)5,66960.9%48.5%+12.4pp0.2677—

Weekly Calibration #15 · Nov 9-15, 2026 UTC · cutoff Mon 16 Nov 2026 16:00 UTC

Gap = mean stated confidence minus hit rate, in percentage points; a positive gap is overconfidence.

Stated confidence against realised accuracy — Monthly Calibration #3

Model linenMean stated confidenceHit rateGapBrier95% Wilson
Claude Opus 5.52,61258.6%49.6%+9.0pp0.2600[47.7%, 51.5%]
Grok 4.73,06960.0%48.6%+11.3pp0.2631[46.9%, 50.4%]
Qwen 3.8 Max3,34459.3%48.4%+10.9pp0.2631[46.7%, 50.0%]
DeepSeek V4 Pro3,48560.4%48.5%+11.9pp0.2653[46.9%, 50.2%]
Claude Fable 53,00360.5%48.2%+12.3pp0.2661[46.4%, 50.0%]
GPT-6.1 Sol3,62765.8%48.9%+16.9pp0.2816[47.3%, 50.5%]
Gemini 3.1 Pro3,43966.9%48.4%+18.5pp0.2872[46.7%, 50.1%]
Field (all models)22,57961.8%48.6%+13.2pp0.2701—

Monthly Calibration #3 · Oct 1-31, 2026 UTC · cutoff Sun 1 Nov 2026 16:00 UTC

Gap = mean stated confidence minus hit rate, in percentage points; a positive gap is overconfidence. Rows represent model lines. Where a version changed during the window, the row includes forecasts from the versions active on each date (Claude Opus 5 → Claude Opus 5.5 on 17 Oct 2026; Grok 4.6 → Grok 4.7 on 17 Oct 2026; ChatGPT 5.6 Sol → GPT-6.1 Sol on 17 Oct 2026). The current model name identifies the line, not every historical forecast.

Across issues

Field gap and Brier, issue by issue

IssueWindownMean stated confidenceHit rateGapBrier
Weekly Calibration #1Aug 3-9, 2026 UTC4,04262.8%42.4%+20.4pp0.2908
Weekly Calibration #2Aug 10-16, 2026 UTC3,67062.0%43.8%+18.3pp0.2836
Weekly Calibration #3Aug 17-23, 2026 UTC5,13062.7%54.0%+8.6pp0.2559
Weekly Calibration #4Aug 24-30, 2026 UTC4,57662.6%42.9%+19.6pp0.2884
Weekly Calibration #5Aug 31-Sep 6, 2026 UTC4,52062.4%42.8%+19.6pp0.2874
Weekly Calibration #6Sep 7-13, 2026 UTC4,46862.1%39.6%+22.5pp0.2948
Weekly Calibration #7Sep 14-20, 2026 UTC5,59862.8%50.7%+12.1pp0.2669
Weekly Calibration #8Sep 21-27, 2026 UTC4,95863.0%46.8%+16.1pp0.2751
Weekly Calibration #9Sep 28-Oct 4, 2026 UTC4,33162.5%49.6%+12.9pp0.2698
Weekly Calibration #10Oct 5-11, 2026 UTC4,60161.9%43.8%+18.2pp0.2822
Weekly Calibration #11Oct 12-18, 2026 UTC5,08461.9%46.6%+15.4pp0.2759
Weekly Calibration #12Oct 19-25, 2026 UTC5,85661.5%57.0%+4.5pp0.2484
Weekly Calibration #13Oct 26-Nov 1, 2026 UTC5,09461.2%41.6%+19.6pp0.2858
Weekly Calibration #14Nov 2-8, 2026 UTC4,86561.1%44.8%+16.3pp0.2772
Weekly Calibration #15Nov 9-15, 2026 UTC5,66960.9%48.5%+12.4pp0.2677
Monthly Calibration #3Oct 1-31, 2026 UTC22,57961.8%48.6%+13.2pp0.2701

Weekly rows are the field line of each published issue at its own frozen cutoff; the monthly row is one pull at the monthly cutoff, so the two do not have to agree to the last decimal. First to last weekly window, the gap narrowed by 8.0pp.

Caveats

The issue's own caveats, in its words:

Prediction metrics (this report) and trading metrics are kept in separate sections per methodology -- they are never combined into a single score.
95% Wilson CIs shown are descriptive, not inferential: observations inside one window are dependent, so read them as a range, not a formal coverage guarantee.
All 5,669 scored calls sit inside one market regime. This window had trend on 5 of 7 days (issue #14: 4 of 7); every comparison with issue #14 is a comparison across regimes as well as across weeks. Fifteen week-over-week points cannot separate drift from regime; no durability claim is made.
Confidence buckets start at 50. Calls stated below that floor are counted in a table of their own — 5 of them in this window, against 5,669 scored calls in the main table.
Calibration is not accuracy. A line can be well calibrated and still be wrong more often than not, and a line can be accurate while overstating how sure it was; the two are reported side by side, apart.

Definitions

  • Hit — direction: exit tp1/tp2 -> hit, sl -> miss, expiry -> sign of gross pnl.
  • The confidence–accuracy gap measures average overconfidence; it does not describe calibration at every confidence level. A lower Brier score is not by itself evidence of better calibration.

How the issue says to read these tables, in its words:

Gap pp = mean stated confidence minus hit-rate, in percentage points; positive means overconfident. Brier is the mean squared error of the stated probability against the realised outcome, so lower is better and 0.25 is what an uninformative always-50% predictor scores. Coverage = calls scored divided by the mature calls available to that model. Confidence buckets are the models' own natural breakpoints, not an arbitrary binning; cells below N=10 are marked insufficient and never used to rank anything. Prediction and trading metrics live in separate sections and are never combined.

Reports and data

IssuePublishedFiles
Weekly Calibration #1518 Nov 2026PDFMDJSON
Weekly Calibration #1411 Nov 2026PDFMDJSON
Monthly Calibration #35 Nov 2026PDFMDJSON
Weekly Calibration #134 Nov 2026PDFMDJSON
Weekly Calibration #1228 Oct 2026PDFMDJSON
Weekly Calibration #1121 Oct 2026PDFMDJSON
Weekly Calibration #1014 Oct 2026PDFMDJSON
Weekly Calibration #97 Oct 2026PDFMDJSON
Weekly Calibration #830 Sept 2026PDFMDJSON
Weekly Calibration #722 Sept 2026PDFMDJSON
Weekly Calibration #615 Sept 2026PDFMDJSON
Weekly Calibration #58 Sept 2026PDFMDJSON
Weekly Calibration #42 Sept 2026PDFMDJSON
Weekly Calibration #326 Aug 2026PDFMDJSON
Weekly Calibration #219 Aug 2026PDFMDJSON
Weekly Calibration #110 Aug 2026PDFMDJSON

Dataset cardPublic APIDaily snapshots

Cite

MarketMania Research (2026). Research reports (weekly, monthly, quarterly), PDF/MD/JSON. https://marketmania.ai/research — dataset card: https://marketmania.ai/research/dataset

To cite one reading instead, name the issue it came from and the date it was published: an issue is frozen, so a citation to one is stable.

Research benchmark — not financial or investment advice; paper trading only.