Methodology

Every slot on the grid runs the same loop. An identical payload — the same market data, the same instrument, the same horizon and timeframe — goes out to seven models at once. Each model answers with no visibility into what the other six said. Every answer is stored verbatim. The outcome resolves once that cell's horizon closes against the realised price, and the forecast is then scored for direction and, if it called long or short, run through the trade simulation described in Scoring.

Order above matches each model's fixed colour identity used across the site; it implies no ranking.

{
  "model": "claude-opus-5",
  "side": "short",
  "confidence": 74,
  "takeProfitPct": 1.8,
  "stopLossPct": 0.9
}

Side is one of long, short or sideways, and confidence runs 0–100. For a long or short call, both the take-profit and stop-loss distances are validated as strictly positive percentages before the answer is accepted; a sideways call may carry neither, since there is no position for either to bound.

A cell is one horizon paired with one chart timeframe. Every horizon beyond the shortest is read on two different timeframes — its own native one and a shorter confirmation timeframe — precisely so that "4h read on the 1h chart" and "4h read on the 4h chart" can be told apart: both ask the same four-hour-ahead question, just shown at different chart resolutions. Every timeframe in every horizon is evaluated across all instruments in the universe.

One timeframe at 1H, two at each of the other four horizons: 1 + 2 + 2 + 2 + 2 = 9 cells. Each cell receives one forecast from each of the seven models every slot.

The point of asking the same horizon on two timeframes is to separate two things a single-timeframe grid would confound: which horizon a model is good at forecasting, and which chart timeframe it reads well. A model might read a 1h chart cleanly but do worse extrapolating that same four hours from a 4h chart, or the reverse — and a grid with one timeframe per horizon could never tell that apart from the model simply being better or worse at "four hours ahead" in general.

Every settled forecast is checked against up to two scores, and they measure different things: whether the direction was right, and whether a simulated trade on it would have made money. A long or short forecast receives both. A sideways forecast receives neither — it is not scored as a directional hit or miss, and it opens no position, so it cannot appear on the trade ledger either.

For a directional forecast, direction accuracy is whether the sign of the realised move over the horizon matched the sign the model called. Sideways forecasts make no directional claim, have no sign to compare against, and are excluded from this measure entirely rather than scored as correct or incorrect by default — though a sideways call still contributes a neutral vote to the aggregate indices covered from Outlook onward.

Every long or short call opens a simulated position against that model's own bank; sideways opens nothing. A 0.05% taker fee is paid on each fill — one on entry, one on exit. A trade closes in exactly one of three ways, and no other: at the model's own take-profit distance, at its own stop-loss distance, or, if neither is touched, at horizon expiry, marked at the price prevailing at that moment.

The Outlook turns seven independent forecasts, for one cell, into one reading. Every model's answer contributes a weight built from three independent factors — its stated confidence, its own trailing accuracy in this exact cell, and how long ago it was made — and that weighted vote is converted into a sentiment score, a label, a consensus figure, a set of side shares, and an expected move.

sᵢ  = +1 long | 0 sideways | −1 short
wᵢ  = cᵢ · tₘ · rᵢ
      cᵢ = max(0.1, confidenceᵢ / 100)                                  # confidence floor
      tₘ = model m’s rolling 30-day direction accuracy in this cell,
           divided by the cross-model mean, clipped to [0.5, 1.5]       # trust clip
      rᵢ = 0.5 ^ (age of forecast / one horizon length)                 # recency half-life
Sentiment  = 100 · Σ(wᵢ·sᵢ) / Σwᵢ            range −100 … +100
Label      = Bullish if ≥ +20, Bearish if ≤ −20, otherwise Sideways
Consensus  = 100 · (weight on the winning side) / Σwᵢ
Shares     = weight share of each of the three sides
Expected move = weighted median of |take-profit %| over directional forecasts only
Conviction = 50 + 25 · mean over models of z(confidence; model’s own 30-day μ, σ)   # z-scored per model
Disagreement = 100 · H(p_bull, p_side, p_bear) / ln 3                    # entropy, 0–100

The confidence floor. cᵢ is floored at 0.1, not 0. A model reporting very low confidence still carries a tenth of a maximum-confidence vote's weight rather than none — a low-conviction answer is a data point, and zeroing it out would amount to deciding in advance that it carries no information at all.

The trust clip. tₘ scales weight by a model's trailing accuracy in this cell relative to the seven-model average, and left unclipped it would let a model on a hot streak come to dominate the reading entirely. Clipping it to [0.5, 1.5] caps that: the best-performing model in a cell carries at most one and a half times an average model's weight, and the worst carries at least half, however long either streak runs.

The recency half-life. rᵢ halves a forecast's weight every time one full horizon length has elapsed since it was made. The decay is measured in horizons, not calendar time, so a 1h cell forgets within hours and a 1M cell forgets within months — matching the shelf life of what is actually being forecast in each cell, rather than one fixed clock applied to all nine.

Reading the grid across all instruments rather than one symbol pools Sentiment, Consensus and Expected move: every model's vote, across all instruments, is combined into a single weighted calculation, exactly as if it were one large cell. Conviction and Disagreement are not pooled the same way — each is computed separately for BTC, ETH, SOL, BNB and XRP, and only then averaged across the five.

This is a deliberate methodological choice, stated plainly rather than left implicit. Conviction and Disagreement both describe how the seven models relate to one another — how firmly they hold their views, and how much they disagree — and pooling them across instruments would instead measure how much BTC, ETH, SOL, BNB and XRP currently differ from one another: a pooled calculation cannot tell "seven models split three ways on BTC" apart from "seven models unanimous on BTC but split across the five coins." Averaging the five per-instrument figures keeps both measuring the models, not the instruments.

Calibration asks a narrower question than accuracy: not whether a model is right, but whether its stated confidence means what it claims to. A model that says "70% confident" and turns out correct on 70% of those calls is well calibrated, whatever its overall hit rate looks like.

Only settled directional forecasts enter this measure. A sideways call makes no claim about which way the market moves, so there is no direction for it to be right or wrong about, and it is excluded rather than scored as a default pass or fail. This is also the entire explanation for why the calibration sample is smaller than the count of settled trades quoted elsewhere on the site: the difference between the two is exactly the settled sideways calls that this measure excludes. A second, smaller difference is the confidence floor — the buckets start at 50, and calls stated below it are counted separately rather than folded in.

Every settled long or short forecast is sorted by its stated confidence into one of five buckets — 50–60, 60–70, 70–80, 80–90, 90–100 — and actual accuracy per bucket is the share of forecasts in that bucket whose direction matched the realised move.

Calibration gap = (n-weighted mean actual accuracy)
                 − (n-weighted mean stated confidence)     in pp
negative → overconfident  ·  positive → underconfident

Discrimination = accuracy of a model’s own
                 top confidence tercile
               − accuracy of that same model’s own
                 bottom confidence tercile         in pp

A model can be badly calibrated and still discriminate well, and that is in fact the common case here: a model whose confidence runs fifteen points too hot across the board is still doing useful work if its top-tercile calls are genuinely more often right than its bottom-tercile calls. Calibration and discrimination are, in effect, readable proxies for two of the three terms in the same proper scoring rule — the reliability and resolution terms of the Brier decomposition.

Brier score = mean[ (confidence/100 − outcome)² ]    outcome ∈ {0, 1}; lower is better
            = Reliability − Resolution + Uncertainty                 (Murphy, 1973)
              Reliability  ≈ calibration gap  — cost of confidence not matching observed
                              frequency, bucket by bucket
              Resolution   ≈ discrimination   — credit for buckets whose observed frequency
                              differs from the overall base rate at all
              Uncertainty   fixed by how often the base rate itself is right; not a model’s doing

The two are reported separately because "is this model honest about its confidence" and "is this model's confidence still useful" are different questions, and one number cannot answer both.

The identical calculation — bucket, compare, take the n-weighted gap — can be restricted to forecasts on a single instrument: BTC calibration alone, ETH alone, and so on. It exists to answer a narrower question than the pooled figure: whether a model, or the panel as a whole, is better calibrated on some instruments than others, rather than only on markets in aggregate.

Smart Consensus asks one question: does a reliability-weighted vote across all seven models beat the single best-performing model in the same window and cell? It is presented as an open question because, on the history available so far, it is one. The card prints how many settled days are behind the current reading; this page does not restate it.

weightₘ = max(0, hitₘ − 0.5) over the trailing window, renormalised to sum to 1
          models at or below 50% receive a small floor weight, not zero
Council side = sign( Σ weightₘ · sₘ )
Accuracy     = share of council calls whose direction matched the realised move
Compared to  : simple majority (one unweighted vote per model)
               best single model (highest accuracy in the same window and cell)
Spread       = council accuracy − best single accuracy, in percentage points

A model sitting at or below 50% accuracy in the trailing window is not zeroed out of the council; it keeps a small floor weight instead, so a run of bad luck does not silence a model outright and the weighting has no discontinuity at the 50% line.

This page quotes no counts. Totals, coverage and every per-cell sample size are read from the record itself and are published, live and with their own n, on Indices and Benchmarks. A methodology document that carried its own copy of those numbers would go stale silently, and a stale count next to a live one is exactly the contradiction the rules below exist to prevent.

Every published figure — a hit rate, a calibration gap, an edge over a coin flip — carries its sample size, n, next to it. A rate built on a few dozen resolved forecasts and one built on tens of thousands are never shown the same way without that qualifier attached.

When the grid or the scoring rules change, the change is dated and logged as an epoch. History is then re-rendered under the current rules rather than left in a stale format, and nothing is deleted — an epoch changes how a figure is presented, never whether it is retained.

Every figure published before an epoch stays visible after it, re-rendered under the current rules; the table above is the complete list to date, and every snapshot remains addressable by its own date.

This list is not exhaustive; it is the set of limitations material enough to affect how a published figure should be read.

The sections above state the published rules. The three notes below are properties of this implementation that those rules do not mention and that a reader trying to reproduce a figure from the page would otherwise fail on. They are documented here rather than left in the code.

Both daily windows — the calibration rollup and the smart-consensus accuracy series — select forecasts by the date their horizon closes, not the date they were made. A forecast belongs to the window containing slot_ts + horizon, so a 1M call made in early July settles into an early-August window. This is what keeps the long horizons populated: selecting by slot date would put every 1M forecast in a window whose outcome was not yet known, and the 1M row would read empty for a month at a time.

Section 4 states conviction as 50 + 25 · z(confidence) and stops there. In practice the z-score is computed per model against that model's own trailing 30-day mean and standard deviation of stated confidence, with σ floored at 1.0 confidence points and z clipped to ±3 before the average across models is taken. The floor exists because a model that has answered "75" thirty times running has σ = 0, which is a division by zero (and at σ = 0.01, a z of 2000); the clip stops one model restating its house-typical confidence after a quiet month from pinning the whole index at an extreme.

σₘ = max(1.0, stdev of model m’s stated confidence over the trailing 30 days)
zᵢ = clip( (confidenceᵢ − μₘ) / σₘ , −3, +3 )
Conviction = 50 + 25 · mean over models of zᵢ

The Smart Consensus side is sign(score), and Sideways occurs only at a score of exactly zero. The ±20 Sideways band described in section 4 applies to the Outlook index alone. The two indices can therefore disagree on the same instrument and horizon — the Outlook calling Sideways at +6 while the council calls the same reading Bullish — and that is the published rule rather than an inconsistency between two data sources.

@misc{marketmania2026methodology,
  title  = {MarketMania: methodology for resolved-outcome forecast
            records from frontier language models on live crypto markets},
  author = {{MarketMania}},
  year   = {2026},
  note   = {Methodology fixed 3 August 2026; record running since 15 Jul 2026},
  url    = {https://marketmania.ai/methodology}
}

The note field carries no counts on purpose — this entry cites the RULES, which are dated and stable. To cite a single index reading instead, name the index, the instrument and cell, and the date it was read, for example MarketMania LLM Market Outlook (BTC, 4h), 3 Aug 2026; the sample behind that reading is printed beside it on the card.