{
  "report": "stability-index-run-2",
  "title": "Stability Index — Run 2",
  "date": "2026-08-10",
  "series": "quarterly",
  "series_name": "Stability Index",
  "methodology": "v1.1",
  "url": "https://marketmania.ai/research",
  "snapshot": {
    "calls": 945,
    "models": 7,
    "sets_per_model": 27,
    "repeats_per_set": 5,
    "tickers": 3,
    "fh_tf_cells": 9,
    "runtime": "1h43m",
    "harness": "identical to Jul 31 baseline"
  },
  "tldr": [
    "Frontier LLMs flip direction only when they are unsure. Across 1,890 identical-prompt repeats in two runs, we observed 7 hard flips (long↔short on a byte-identical payload) — and zero of them happened at confidence ≥ 70. Every flip involved a low-confidence answer.",
    "claude-opus-5 and gpt-5.6-sol lead Run 2 at 88.9% unanimity each; claude-fable-5 — the July leader — dropped to 70.4%, with its instability concentrated entirely in the 1-month horizon (4/6 → 1/6 unanimous sets).",
    "The August 10 market slice was quiet, and that inflates raw unanimity. Agreeing on \"sideways\" is easier than agreeing on a direction. Decomposed: opus-5's 24 unanimous sets are 6 directional + 18 sideways (July: 19 + 2). Cross-run ranking comparisons must respect this — that is exactly why this is a longitudinal series.",
    "qwen-3.8-max debuts more stable than its predecessor: 66.7% unanimity vs qwen-3.7-max's 51.9% on July 31, hard flips 2 → 0, invalid answers 3.0% → 0.0%. It is also far more cautious: sideways share 17.6% → 64.4%.",
    "Format quality field-wide is excellent: invalid rate 0.0-1.5% in Run 2 (July: up to 5.2%), zero sign-rule violations in either run."
  ],
  "results_run2": [
    {
      "model": "claude-opus-5",
      "unanimity_pct": 88.9,
      "modal_pct": 97.0,
      "hard_flips": 0,
      "hc_hard_flips": 0,
      "soft_flips": 3,
      "invalid_pct": 0.0,
      "sideways_pct": 71.9,
      "conf_sd": 0.0
    },
    {
      "model": "gpt-5.6-sol",
      "unanimity_pct": 88.9,
      "modal_pct": 96.3,
      "hard_flips": 0,
      "hc_hard_flips": 0,
      "soft_flips": 3,
      "invalid_pct": 0.0,
      "sideways_pct": 62.2,
      "conf_sd": 1.2
    },
    {
      "model": "claude-fable-5",
      "unanimity_pct": 70.4,
      "modal_pct": 91.1,
      "hard_flips": 0,
      "hc_hard_flips": 0,
      "soft_flips": 8,
      "invalid_pct": 0.0,
      "sideways_pct": 52.6,
      "conf_sd": 1.2
    },
    {
      "model": "grok-4.5",
      "unanimity_pct": 66.7,
      "modal_pct": 89.6,
      "hard_flips": 0,
      "hc_hard_flips": 0,
      "soft_flips": 9,
      "invalid_pct": 0.0,
      "sideways_pct": 61.5,
      "conf_sd": 1.8
    },
    {
      "model": "qwen-3.8-max",
      "unanimity_pct": 66.7,
      "modal_pct": 88.9,
      "hard_flips": 0,
      "hc_hard_flips": 0,
      "soft_flips": 9,
      "invalid_pct": 0.0,
      "sideways_pct": 64.4,
      "conf_sd": 1.8
    },
    {
      "model": "gemini-3.1-pro",
      "unanimity_pct": 59.3,
      "modal_pct": 88.5,
      "hard_flips": 1,
      "hc_hard_flips": 0,
      "soft_flips": 9,
      "invalid_pct": 1.5,
      "sideways_pct": 54.1,
      "conf_sd": 3.2
    },
    {
      "model": "deepseek-v4-pro",
      "unanimity_pct": 48.1,
      "modal_pct": 85.2,
      "hard_flips": 1,
      "hc_hard_flips": 0,
      "soft_flips": 13,
      "invalid_pct": 0.0,
      "sideways_pct": 68.9,
      "conf_sd": 3.4
    }
  ],
  "results_baseline_jul31": [
    {
      "model": "claude-fable-5",
      "unanimity_pct": 81.5,
      "modal_pct": 94.8,
      "hard_flips": 0,
      "soft_flips": 5,
      "invalid_pct": 0.0,
      "sideways_pct": 19.3,
      "conf_sd": 1.2
    },
    {
      "model": "claude-opus-5",
      "unanimity_pct": 77.8,
      "modal_pct": 93.3,
      "hard_flips": 0,
      "soft_flips": 6,
      "invalid_pct": 0.0,
      "sideways_pct": 19.3,
      "conf_sd": 1.2
    },
    {
      "model": "gpt-5.6-sol",
      "unanimity_pct": 70.4,
      "modal_pct": 89.6,
      "hard_flips": 0,
      "soft_flips": 8,
      "invalid_pct": 0.0,
      "sideways_pct": 15.6,
      "conf_sd": 2.3
    },
    {
      "model": "grok-4.5",
      "unanimity_pct": 70.4,
      "modal_pct": 88.9,
      "hard_flips": 3,
      "soft_flips": 5,
      "invalid_pct": 0.0,
      "sideways_pct": 13.3,
      "conf_sd": 1.4
    },
    {
      "model": "qwen-3.7-max",
      "unanimity_pct": 51.9,
      "modal_pct": 85.2,
      "hard_flips": 2,
      "soft_flips": 10,
      "invalid_pct": 3.0,
      "sideways_pct": 17.6,
      "conf_sd": 3.6
    },
    {
      "model": "gemini-3.1-pro",
      "unanimity_pct": 33.3,
      "modal_pct": 83.1,
      "hard_flips": 0,
      "soft_flips": 15,
      "invalid_pct": 5.2,
      "sideways_pct": 46.9,
      "conf_sd": 3.5
    },
    {
      "model": "deepseek-v4-pro",
      "unanimity_pct": 29.6,
      "modal_pct": 75.0,
      "hard_flips": 0,
      "soft_flips": 19,
      "invalid_pct": 1.5,
      "sideways_pct": 38.3,
      "conf_sd": 5.1
    }
  ],
  "unanimous_sets_split": [
    {
      "model": "claude-opus-5",
      "jul31": {
        "directional": 19,
        "sideways": 2
      },
      "aug10": {
        "directional": 6,
        "sideways": 18
      }
    },
    {
      "model": "gpt-5.6-sol",
      "jul31": {
        "directional": 19,
        "sideways": 0
      },
      "aug10": {
        "directional": 9,
        "sideways": 15
      }
    },
    {
      "model": "claude-fable-5",
      "jul31": {
        "directional": 19,
        "sideways": 3
      },
      "aug10": {
        "directional": 8,
        "sideways": 11
      }
    },
    {
      "model": "grok-4.5",
      "jul31": {
        "directional": 18,
        "sideways": 1
      },
      "aug10": {
        "directional": 6,
        "sideways": 12
      }
    },
    {
      "model": "qwen (3.7 → 3.8)",
      "jul31": {
        "directional": 14,
        "sideways": 0
      },
      "aug10": {
        "directional": 5,
        "sideways": 13
      }
    },
    {
      "model": "gemini-3.1-pro",
      "jul31": {
        "directional": 6,
        "sideways": 3
      },
      "aug10": {
        "directional": 8,
        "sideways": 8
      }
    },
    {
      "model": "deepseek-v4-pro",
      "jul31": {
        "directional": 6,
        "sideways": 2
      },
      "aug10": {
        "directional": 2,
        "sideways": 11
      }
    }
  ],
  "hard_flips_finding": {
    "total_flips_two_runs": 7,
    "hc_flips_conf_ge_70": 0,
    "repeats_total": 1890,
    "detail": "July: grok-4.5 x3, qwen-3.7-max x2; August: gemini-3.1-pro x1, deepseek-v4-pro x1"
  },
  "generational_qwen": {
    "qwen_3_7_max_jul31": {
      "unanimity_pct": 51.9,
      "modal_pct": 85.2,
      "hard_flips": 2,
      "invalid_pct": 3.0,
      "sideways_pct": 17.6
    },
    "qwen_3_8_max_aug10": {
      "unanimity_pct": 66.7,
      "modal_pct": 88.9,
      "hard_flips": 0,
      "invalid_pct": 0.0,
      "sideways_pct": 64.4
    }
  },
  "per_fh_unanimity": [
    {
      "model": "claude-fable-5",
      "1h": {
        "jul": "9/9",
        "aug": "9/9"
      },
      "4h": {
        "jul": "9/9",
        "aug": "9/9"
      },
      "1m": {
        "jul": "4/6",
        "aug": "1/6"
      }
    },
    {
      "model": "claude-opus-5",
      "1w": {
        "jul": "3/6",
        "aug": "6/6"
      },
      "1d": {
        "jul": "5/6",
        "aug": "6/6"
      }
    },
    {
      "model": "gpt-5.6-sol",
      "1m": {
        "jul": "2/6",
        "aug": "5/6"
      },
      "1w": {
        "jul": "3/6",
        "aug": "5/6"
      }
    },
    {
      "model": "gemini-3.1-pro",
      "1h": {
        "jul": "0/3",
        "aug": "2/3"
      },
      "1w": {
        "jul": "1/6",
        "aug": "4/6"
      }
    },
    {
      "model": "grok-4.5",
      "4h": {
        "jul": "6/6",
        "aug": "4/6"
      },
      "1d": {
        "jul": "5/6",
        "aug": "4/6"
      },
      "1w": {
        "jul": "4/6",
        "aug": "3/6"
      }
    }
  ],
  "practical_implications": [
    "Treat sub-70 confidence as a caution flag. Every flip we have observed involved at least one low-confidence answer; the models' own uncertainty is a usable filter.",
    "Do not over-interpret single-run rankings. With N=5 repeats, adjacent models sit within noise; only large gaps (and the accumulated series) rank reliably.",
    "Quiet markets inflate raw unanimity. Compare like slices, or read the directional/sideways decomposition (Section 3) before concluding a model \"got steadier\"."
  ],
  "limitations": [
    "N=5 repeats per set → unanimity moves in 3.7 pp steps; adjacent models are within noise, ranking is reliable only across large gaps.",
    "One market slice per run; slices differ between runs by design (see §3). Longitudinal conclusions require the series, not a pair.",
    "Invalid answers count against unanimity (production rules); transport-level failures are tracked separately and were zero in Run 2.",
    "Confidence is model-self-reported; the ≥70 threshold for \"high confidence\" is a fixed convention of methodology v1.1."
  ],
  "reproducibility": {
    "runs": [
      "si_baseline_20260731-150302",
      "si_baseline_20260810-131022"
    ],
    "results_format": "jsonl",
    "payload_hashes": "sha-256",
    "note": "metrics recomputed from raw results"
  },
  "citation_bibtex": "@misc{marketmania2026si2,\n  title  = {Stability Index --- Run 2: repeat-stability of frontier LLM market forecasts},\n  author = {{MarketMania Research}},\n  year   = {2026},\n  month  = {August},\n  day    = {10},\n  url    = {https://marketmania.ai/research},\n  note   = {Methodology v1.1; runs si_baseline_20260731-150302 and si_baseline_20260810-131022}\n}"
}