Built with SolarSight
SolarQuoteBench is built by Cody Labs, the developer behind SolarSight, a free solar savings estimate for homeowners and quoting software for installers. The benchmark's grading engine is ported from the engine behind SolarSight's quotes.
These are screenshots of the public SolarSight site and show the kind of quote the benchmark asks an agent to get right.
An example installer proposal: roof layout, system size, cost and payback. Example customer and figures. The second shot is the same proposal further down: itemised quote, payment terms and acceptance.
Why this is hard
Here is one task from the public set. A vendor proposal for a 10.56 kW system on the Gold Coast promises 18,970 kWh a year, and calls it a figure "you can rely on for the life of the system". The rebate, the savings, the GST and the price all check out.
The production figure is wrong. It assumes the system turns every bit of sunlight reaching the panels into power. Real systems lose around 14% on the way: dust and grime on the glass, resistance in the cables, the inverter's conversion, and panels that never quite match each other. Cloud and weather are already in the solar data, so these losses come on top. Allow for them and that roof makes about 16,300 kWh, so the quote overstates output by roughly 16%.
To catch it, an agent has to pull the solar data for that exact roof, apply the losses correctly, find the one wrong number among many right ones, and leave the correct ones alone. Flagging a figure that is fine costs points, and some proposals have no errors at all. On this family gpt-5.6-luna averages 0.60 and Claude Sonnet 5.5 averages 0.97.
Every family works the same way: a realistic job, one checkable answer, and a deterministic grader. The rest of the tasks, and the held-out set, stay unseen.
Two families where the models separate
The agent gets a real customer situation: location, roof, electricity bills and tariff. It does the job of a solar sales engineer, using tools for PVGIS yield, tariffs, incentive rules, a usage profile and year-1 energy flows. It returns numbers or a structured audit.
Luna over-flags clean proposals
Clean proposals flagged as containing errors. Luna: precision 0.517, recall 1.000. Claude: precision 0.948, recall 0.979. Solved instances: 3 of 16 against 13 of 16.
The most headroom for both
Choose panels per plane and a battery to maximise NPV or hit a self-sufficiency target at lowest cost. Ground truth is the exhaustive optimum. Both models miss the time-of-use instances, where the supplied energy flow tool has no TOU windows.
The eight task families
119 hand-checked public instances across eight families. We report which ones separate models and which ones do not. Each bar is a model's average score across that family's published tasks. 1.0 means every answer was right.
T5 proposal audit Discriminates
Find, classify and quantify planted errors in a vendor proposal. Clean proposals are included as controls.
T7 system sizing Discriminates
Choose panels per plane and a battery to maximise NPV or hit a self-sufficiency target at lowest cost. Ground truth is the exhaustive optimum.
T3 incentive eligibility Discriminates
Which schemes apply for a jurisdiction, system and install date, and what they are worth. Includes not-eligible cases and a withheld-tool variant.
T4 time-of-use payback Hard for both
Payback year and 25-year NPV under a TOU tariff, battery and export limit. Neither model solves more than 1 of 13.
T2 orientation allocate Partial
Split a panel budget across roof planes for yield or for dollar value. Two of the value instances are real tests of objective awareness. Both models fail them.
T8 full quote Near ceiling
The whole quote from a raw intake: production, flows, costs, rebates, savings, payback, NPV, IRR. Scored against truth and against the agent's own upstream numbers.
T6 consumption estimate Near ceiling
Annual consumption from bills: tax and supply charges, tiers, TOU periods, existing solar, part-year scaling.
T1 yield estimate Near ceiling
Annual kWh for a site, roof planes and panel spec. Traps: compass bearing versus PVGIS aspect, hemisphere tilt, loss composition.
Solved means the instance clears the family's pass threshold. gpt-5.6-luna, reasoning effort medium, through the OpenAI Responses API. T1 to T5 ran on 18 August 2026 and T6 to T8 in November 2026, with the T1 to T5 scores regraded under the current graders (no model calls, answers unchanged). Claude Sonnet 5.5 ran on 2 November 2026 through Claude Code's own agent loop (headless, tools supplied through an MCP server, effort left at the Claude Code default), not the raw API. A score gap between the two models can come from the harness as well as the model. Each model was run once per instance.
T1, T6 and T8 are at or near ceiling for Claude. Do not read them as evidence of difficulty. They work as controls and regression checks. T4 is hard for both models: we audited the grading bands and the errors are real arithmetic failures, not a band that is too tight. T3 separates them on rebate value, not on which scheme applies. Scheme identification is 0.938 for both. Luna returned a wrong or zero Australian STC value on several instances. The cause was not diagnosed, and the harness difference is one candidate.
Hardened tiers T6H and T8H also exist. Claude Sonnet 5.5 scored 1.000 on T6H and 0.978 on T8H, which is above the bar we set for a hard tier, so both ship as labelled control tiers and are not part of the default environment.
Every answer checked by a calculator
There is no LLM judge, so a score can be reproduced exactly. Each grader reports a breakdown per dimension, and names the known failure mode an answer looks like.
Deterministic oracle
The grading engine is ported from the production solar quoting engine behind SolarSight. Each ported module reuses the original TypeScript test suite's asserted values, and a single cross-validation test file runs them together.
PVGIS-backed yields
Tools serve cached PVGIS fixtures. No live API call happens during grading, and a missing fixture fails loudly.
Error-origin attribution
Per-dimension breakdowns and a failure taxonomy for every family. In T8, each field is also scored against what it should have been given the agent's own earlier numbers, and the grader names the step where the chain first broke.
Public and private splits
The public split is 119 curated, hand-checked instances. A private held-out split is generated from disjoint seeds, kept private and used for re-runs. Private counts per family: T1 200, T2 80, T3 200, T4 200, T5 200, T6 200, T7 24, T8 60. Private instances are structurally validated, not individually hand-checked.
What is free, and what we sell
The public benchmark
On the Prime Intellect Hub. Install it and run it on your own model.
- 119 hand-checked public instances across eight task families
- The graders and tools
- MIT licensed
prime env install solar-sight/solarquotebench uv run vf-eval solarquotebenchView on the Prime Hub
Generators, held-out split, custom work
Available commercially.
- The task generators: unlimited fresh instances for RL training
- The private held-out split
- Re-runs of your model on the held-out split, with the scored report sent to you
- Custom task families: new jurisdictions, tariffs, incentive schemes or quote formats
Email [email protected] with what you want to measure.
Talk to usWhat the benchmark does not claim
- The hourly PV generation shape behind T2 value scoring, T4 and T7/T8 flows was authored, then measured against PVGIS hourly data. It diverges 2 to 11% on value-type quantities. T4 NPV differs by a mean 2.2% of system cost against PVGIS hourly data. One T2 instance sits just over a 20% divergence. Part of T4's ground truth rests on this authored shape.
- Time-of-use tariffs are sourced for Ontario and Great Britain only. AU, IE, NZ and US instances use synthetic TOU shapes, labelled as such on the instance. T6 to T8 bill layouts and prices are synthetic.
- T6 to T8 grade use of the supplied seasonal profile and tools, not how real households behave.
- One run per model per instance. The two models used different harnesses, and no raw-API Claude run exists yet.