Benchmark methodology

Seven frontier models forecast the same five coins on a fixed schedule. Every model sees the same inputs at the same time, so the leaderboard compares reasoning, not data access. Each answer is validated for structure, then paper-traded on Binance USDT-M futures with a fixed notional and a fixed starting bank. Two layers are reported and kept separate: accuracy scoring (did the model answer well and directionally right) and the trading simulation (what a mechanical trader would have earned following the answer).

The public benchmark launched on 15 July 2026 with a full statistics reset. All seven lineup slots have run since day one; a slot upgraded to a newer model version mid-season keeps its full history under the successor's name, marked in the table below.

The benchmark covers the five largest coins — BTC, ETH, SOL, BNB, XRP — forecast on every horizon. A wider set of higher-beta names ran on the weekly and monthly cadence until 27 July 2026; their history is preserved unchanged (see the changelog below), and the universe is reserved for future expansion.

For each coin the model answers across five forecast horizons (FH). Each horizon is queried once per available data timeframe (TF) — a separate request per timeframe, so the model never mixes resolutions inside one answer:

Every request carries the same, fully disclosed payload for the single timeframe being asked:

• — OHLCV candles of that one timeframe;
• — ten indicator sets: ohlcv, rsi, macd, sma, ema, bbands, atr, adx, obv, vwap;
• — four futures metrics: open interest, mark price, liquidations, funding rate;
• — a curated news digest: per-period story buckets (crypto for the coin plus world context), with the freshness window scaled to the horizon — 24 hours for 1H/4H up to 30 days for 1M.

A single forecast is a structured answer — a side (long / short / sideways), a take-profit %, a stop-loss %, and a confidence score — never free text.

The payload is the raw market data exactly as the Analyze product sends it — no model-specific tuning of any kind. One shared payload per slot means every model is answering the same question.

A strict validator checks every answer against the required schema. If an answer is malformed or incomplete, the model is asked again with the identical payload, up to two retries. If it still fails, the answer is marked invalid and counted against the model — a model is rewarded for answering clearly and consistently, not just for being directionally right.

Slot coverage is tracked continuously and shown live in the leaderboard-page counters; incomplete slots are never backfilled silently — every gap stays visible in the data.

Valid forecasts are paper-traded under one mechanical rule set — no discretion, no overrides:

• — venue: Binance USDT-M futures, simulated on 1-minute candles;
• — entry = the open of the first 1-minute candle after the model answers;
• — fixed $100 notional per trade; every model starts from the same $1,000 bank;
• — a 0.05% taker fee is applied per fill; PnL is reported both gross and net of fees;
• — when a single 1-minute candle touches both the stop and the target, the simulation is conservative — stop-loss first.

From these settled trades we derive win rate, gross and net PnL, fees paid, max drawdown, and a running equity curve for each model.

The leaderboard is ranked under two frozen configs. Neither can be changed in the sandbox.

Per-forecast mode treats each forecast as an independent trade opened at entry and closed at its own TP, SL or horizon expiry.

Position mode maintains a single running position per coin and reconciles each new forecast against it:

• — opposite side → close the position and reverse;
• — same side → keep the position and update the TP/SL levels;
• — sideways → keep the position unchanged;
• — anchored weekly / monthly: the position has a fixed expiry at the anchor boundary; revisions move TP/SL relative to the revision price, and an opposite revision closes the position at market.

Both official configs trade the model's own take-profit and stop-loss as a single leg — a trade closes at the model's TP, its SL, or horizon expiry, exactly as forecast. Stepped take-profits and break-even exist only as sandbox experiment knobs and never affect the official rankings.

Accuracy scoring (validity and directional correctness) and the trading simulation (equity from mechanical fills) are computed and displayed independently. A model can score well on accuracy and still trade poorly, or vice-versa; keeping the layers separate is deliberate.

• — 2026-07-15 — Benchmark launch. Statistics reset to zero and the TF-split request axis introduced. Grok 4.5 added to the lineup.
• — 2026-07-22 — Grid growth: FH 4H gained TF 4H, and the 1H coin set expanded from BTC/ETH to the top-5 (23 Jul).
• — 2026-07-25 — Data-pipeline fix: weekly/monthly candles are now aggregated from dailies. The 1W-timeframe cell was unavailable 15–25 Jul; the gap is kept as-is — history is never backfilled.
• — 2026-07-26 — The parallel compressed-payload track (V2) concluded after a paired-quality test showed parity with the raw payload. The benchmark runs on a single raw payload; historical V2 data is preserved.
• — 2026-07-27 — Coin universe unified to the top-5 on every horizon; the five extra weekly/monthly names were frozen in history (reserved for a future expansion).
• — 2026-07-30 — Official simulations switched to the model's own TP/SL: the house TP-ladder and break-even were removed from the official configs, the full history was re-simulated under the honest rules and the leaderboard re-ranked. Same day, Claude Opus 4.8 was succeeded by Claude Opus 5 — one lineup slot, history preserved.

The benchmark is a research and transparency tool. It is not financial advice. All fills are simulated — there is no real order flow, and real-world slippage, funding and liquidity are not fully modelled. Past simulated performance does not predict future results.