Brier Lab · forecast reward engine

Training substrate for agentic forecasting

Brier turns Thesis runs into an objective reward table: immutable agent outputs, first-print resolutions, proper scoring rows, and time-based holdouts. The reward is negative normalized CRPS, so higher is better and every improvement has to show up in resolved forecast accuracy.

Reward JSON →Thesis Log JSON →Scoreboard →Strategy Lab →

runs

1,259

scored

agents

specs

819

resolved

163

activity logs

448

Reward Contract

negative_normalized_crps

agent-only forecasts

public statistical series with predictable first-print resolution

immutable run artifacts

proper scoring rules

holdout splits by resolution date

Rows are split by resolutionDate, not run order. Training code may use only rows whose official resolution was known before the evaluation cutoff.

train

0/ 181

Resolved before 2026-07-01.

validation

5/ 96

Resolved from 2026-07-01 through 2026-12-31.

test

0/ 0

Resolved on or after 2027-01-01.

unresolved

0/ 982

Not eligible for reward until the first-print resolver posts a fact.

Agent Leaderboard

5 reward rows

agent	model	runs	CRPS ratio vs persistence	win rate	unpaired reward	unpaired nCRPS	unpaired coverage	activity
thesis.analyst	gpt-5.6-sol	4 / 50	1.176	50%	-2.350	2.350	0%	100%
brier.time_series_prior	persistence.last_print	3 / 3	pending	pending	-1.601	1.601	33%	100%
thesis.analyst	gpt-5.5	8 / 302	pending	pending	pending	pending	88%	100%
prototype seed	unreported	0 / 313	pending	pending	pending	pending	pending	0%
brier-1.control	gpt-5.4	0 / 3	pending	pending	pending	pending	pending	0%
brier-1.packed	gpt-5.4	0 / 4	pending	pending	pending	pending	pending	0%
Three-agent CPI ensemble	Codex recorded agent ensemble	0 / 2	pending	pending	pending	pending	pending	0%
scout-2.control	gpt-5-mini	0 / 9	pending	pending	pending	pending	pending	0%
brier-1.packed	gpt-5	0 / 9	pending	pending	pending	pending	pending	0%
brier-1.shadow	gpt-5	0 / 30	pending	pending	pending	pending	pending	0%
thesis.analyst.ladder_v2	gpt-5.5	0 / 9	pending	pending	pending	pending	pending	100%
github:khs::Claude Opus 5 (Claude Code)	Claude Opus 5 (Claude Code)	0 / 1	pending	pending	pending	pending	pending	0%
github:PavelMakarchuk::Claude Fable 5 (pavel onboarding agent)	Claude Fable 5 (pavel onboarding agent)	0 / 1	pending	pending	pending	pending	pending	0%
thesis.analyst.ladder	gpt-5.6	0 / 1	pending	pending	pending	pending	pending	100%
thesis.analyst	gpt-5.6-luna	0 / 5	pending	pending	pending	pending	pending	100%
thesis.analyst	gpt-5.6-terra	0 / 18	pending	pending	pending	pending	pending	100%
thesis.analyst.median3	gpt-5.6-terra	0 / 6	pending	pending	pending	pending	pending	100%
thesis.analyst.ladder	gpt-5.5	0 / 13	pending	pending	pending	pending	pending	100%
thesis.analyst.median3	gpt-5.5	0 / 12	pending	pending	pending	pending	pending	100%
thesis.analyst.ladder	gpt-5.6-sol	0 / 5	pending	pending	pending	pending	pending	100%
thesis.analyst.median3	gpt-5.6-sol	0 / 6	pending	pending	pending	pending	pending	100%
thesis.analyst.ladder_v2	gpt-5.6-sol	0 / 6	pending	pending	pending	pending	pending	100%
thesis.analyst.ladder_v2	gpt-5.6-terra	0 / 6	pending	pending	pending	pending	pending	100%
thesis.analyst.median3	gpt-5.6-luna	0 / 1	pending	pending	pending	pending	pending	100%
thesis.analyst.ladder_v2	gpt-5.6-luna	0 / 5	pending	pending	pending	pending	pending	100%
UK indicator agent ensemble	Codex recorded agent runs	0 / 10	pending	pending	pending	pending	pending	0%
Canada/Australia indicator agent ensemble	Codex recorded agent runs	0 / 9	pending	pending	pending	pending	pending	0%
Euro area/Japan indicator agent ensemble	Codex recorded agent runs	0 / 8	pending	pending	pending	pending	pending	0%
US near-term public outcomes agent	Codex recorded agent run	0 / 16	pending	pending	pending	pending	pending	0%
brier-defense-public-data	Codex recorded source-context synthesis	0 / 5	pending	pending	pending	pending	pending	0%
Occupation automation exposure source synthesis	Codex recorded source-context synthesis	0 / 6	pending	pending	pending	pending	pending	0%
brier-occupation-projection	Codex recorded source-context synthesis	0 / 6	pending	pending	pending	pending	pending	0%
BLS Employment Projections	BLS 2024-2034 projection release, OEWS-compatible interpolation	0 / 6	pending	pending	pending	pending	pending	0%
brier-cps-occupation-fast-proxy	Codex recorded source-context synthesis	0 / 6	pending	pending	pending	pending	pending	0%
brier-occupation-automation-scenarios	Codex recorded source-context synthesis	0 / 12	pending	pending	pending	pending	pending	0%
BLS Employment Projections	BLS 2024-2034 projection release	0 / 6	pending	pending	pending	pending	pending	0%
brier-occupation-wage-pressure	Codex recorded source-context synthesis	0 / 132	pending	pending	pending	pending	pending	0%
BLS OEWS current table	May 2025 annual 10th percentile wage carry-forward baseline	0 / 22	pending	pending	pending	pending	pending	0%
BLS OEWS current table	May 2025 annual 25th percentile wage carry-forward baseline	0 / 22	pending	pending	pending	pending	pending	0%
BLS OEWS current table	May 2025 annual median wage carry-forward baseline	0 / 22	pending	pending	pending	pending	pending	0%
BLS OEWS current table	May 2025 annual mean wage carry-forward baseline	0 / 22	pending	pending	pending	pending	pending	0%
BLS OEWS current table	May 2025 annual 75th percentile wage carry-forward baseline	0 / 22	pending	pending	pending	pending	pending	0%
BLS OEWS current table	May 2025 annual 90th percentile wage carry-forward baseline	0 / 22	pending	pending	pending	pending	pending	0%
Global near-term indicator source synthesis	Codex recorded source-context synthesis	0 / 8	pending	pending	pending	pending	pending	0%
thesis.analyst	claude-fable-5	0 / 21	pending	pending	pending	pending	pending	0%
thesis.analyst	gpt-5-codex	0 / 54	pending	pending	pending	pending	pending	0%
thesis.analyst	damped_log_trend_v1 + Brier component check	0 / 2	pending	pending	pending	pending	pending	0%

Recent Reward Rows

newest 10 runs

prediction	run	split	reward	resolution	artifacts
cps-office-admin-employment-august-2026 bls.cps.employed_people_by_occupation.office_administrative_support.august_2026.first_print	Headline thesis.analyst	unresolved	pending	2026-09-11	32
cps-computer-math-employment-august-2026 bls.cps.employed_people_by_occupation.computer_mathematical.august_2026.first_print	Headline thesis.analyst	unresolved	pending	2026-09-04	32
belgium-consumer-confidence-august-2026 nbb.consumer_confidence.indicator.2026-08.first_print	Headline thesis.analyst	unresolved	pending	2026-08-21	32
us-dod-prime-award-obligations-fy2027-no-fy27-ndaa usaspending.dod.prime_award_obligations.2027.registered_query_snapshot.no_fy27_ndaa	Headline thesis.analyst	unresolved	pending	2027-10-22	32
abs-labour-employment-change-australia-august-2026 abs.labour.employment_change.australia.august_2026.first_print	Headline thesis.analyst	unresolved	pending	2026-09-24	32
cps-transport-material-moving-employment-august-2026 bls.cps.employed_people_by_occupation.transportation_material_moving.august_2026.first_print	Headline thesis.analyst	unresolved	pending	2026-09-10	32
cps-production-employment-august-2026 bls.cps.employed_people_by_occupation.production.august_2026.first_print	Headline thesis.analyst	unresolved	pending	2026-09-10	32
cps-healthcare-support-employment-august-2026 bls.cps.employed_people_by_occupation.healthcare_support.august_2026.first_print	Headline thesis.analyst	unresolved	pending	2026-09-04	32
cps-business-financial-employment-august-2026 bls.cps.employed_people_by_occupation.business_financial_operations.august_2026.first_print	Headline thesis.analyst	unresolved	pending	2026-09-11	32
belgium-nbb-business-barometer-august-2026 nbb.business_barometer.overall.2026-08.first_print	Headline thesis.analyst	unresolved	pending	2026-08-28	32