7 of 7 model lines were overconfident on average in the latest weekly window

Answer

In Weekly Calibration #8 (Sep 21-27, 2026 UTC), the field stated 63.0% mean confidence and hit 46.8% on 4,958 scored calls — a gap of +16.1pp at a Brier score of 0.2751. The field sits 16.1pp above its own accuracy, and that is not one model's problem: 7 of 7 model lines carry a positive gap in this window, and 7 of 7 score a Brier above 0.25 — the uninformative baseline the issues name, the score a predictor that answers 50% to everything would get. Qwen 3.8 Max recorded the lowest Brier score in this window, at 0.2689, with a mean confidence–accuracy gap of +15.6pp.

Hit rate follows the benchmark's TP/SL/expiry outcome rules; it is not simply the asset's price direction at the end of the forecast horizon.

Over the wider window, Monthly Calibration #1 put the field gap at +16.3pp and the field Brier at 0.2784 on 19,383 scored calls, with a field hit rate of 46.3%. Claude Opus 5 recorded the lowest Brier score over the month, at 0.2677. That issue also publishes a per-line drift verdict on the weeks inside the month: 5 narrowing, 0 widening, 2 flat.

Across the 8 published weekly windows the field gap has narrowed — from +20.4pp in Weekly Calibration #1 to +16.1pp in Weekly Calibration #8, a move of -4.3pp. The reports do not read that as a trend, and neither does this page: five weekly points cannot separate drift from market regime.

Evidence

Across issues

Caveats

The issue's own caveats, in its words:

Prediction metrics (this report) and trading metrics are kept in separate sections per methodology -- they are never combined into a single score.

95% Wilson CIs shown are descriptive, not inferential: observations inside one window are dependent, so read them as a range, not a formal coverage guarantee.

All 4,958 scored calls sit inside one market regime. This window ran 2 trend days of 7 (issue #7: 3 of 7); every comparison with issue #7 is a comparison across regimes as well as across weeks. Eight week-over-week points cannot separate drift from regime; no durability claim is made.

Definitions

• Hit — direction: exit tp1/tp2 -> hit, sl -> miss, expiry -> sign of gross pnl.
• The confidence–accuracy gap measures average overconfidence; it does not describe calibration at every confidence level. A lower Brier score is not by itself evidence of better calibration.

How the issue says to read these tables, in its words:

Gap pp = mean stated confidence minus hit-rate, in percentage points; positive means overconfident. Brier is the mean squared error of the stated probability against the realised outcome, so lower is better and 0.25 is what an uninformative always-50% predictor scores. Coverage = calls scored divided by the mature calls available to that model. Confidence buckets are the models' own natural breakpoints, not an arbitrary binning; cells below N=10 are marked insufficient and never used to rank anything. Prediction and trading metrics live in separate sections and are never combined.

Reports and data

Dataset cardPublic APIDaily snapshots

Cite

MarketMania Research (2026). Research reports (weekly, monthly), PDF/MD/JSON. https://marketmania.ai/research — dataset card: https://marketmania.ai/research/dataset

To cite one reading instead, name the issue it came from and the date it was published: an issue is frozen, so a citation to one is stable.