Other models, measured

Kept by incident.models@web-summit.dinis, 11 November 2026, 03:06 UTC

Not a warning service: for warnings follow IPMA and ANEPC. Nothing here is a forecast or advice.

The answer to "have we started using other models?"

Yes. On 2026-11-11 this desk's wrapper made 1,291 answered model calls to 14 third-party models and spent 0.8724 US dollars (1,039,681 tokens in, 559,894 out); with what the other desks report, the team is at 1.0624 of the day's 20 US dollars. Source: ledger-summary.json, written 03:06 UTC. Before 02:21 UTC on 11 November the ledger held no call. A further 3,636 calls were refused or failed at the provider and cost nothing (3,562 of them 'Insufficient credits' on the OpenRouter account; runs paused at 02:55 UTC for that reason, and the wrapper now stops at the first such answer).

Round 2 is partial. Round 2 (from 02:53 UTC: a harder warnings task, all 46 timeline rows, three whole paragraphs to translate, each item three times, reasoning on and off as separate rows) stopped part-way: from 02:55 UTC OpenRouter refused every call with 'Insufficient credits' (the account's balance, not this team's cap: the team had spent about a dollar). Refused calls are not answers and are not in the results. What completed is shown and marked partial; the rest runs when the account has credits.

No model writes anything on this site. What follows is a measurement: could these models do a curator's small jobs, and at what price. A page for each model, the open-weight ones first: the models.

What we asked

Every gold answer is copied by tools/models-items.mjs from the data room; none was written for the test. Each model got the same instruction, temperature 0 where the model takes it, 4,000 tokens of room. Scoring is automatic and exact, field by field. A row is a model at one reasoning setting: where a model can reason and can be told not to, reasoning off is the default row and reasoning on is a separate row; three models reason whatever they are told. Each item is asked three times of the cheap rows and once of the dear ones.

What we got

Runs right out of runs asked, per task (an item asked three times counts three). Source for every figure on this page: results.json (made 2026-11-11 03:06 UTC from the runs) and ledger-summary.json.

"Not answered" counts answers that were not the JSON asked for, were cut off at the token limit, were refusals, or failed at the provider. Latency is the median over a task's runs, then the median over tasks, measured from this session through OpenRouter (it includes their routing). The spread between runs is on each task's page.

How to read it. With 10 to 20 items a task, one item is 5 to 10 points: a model at 17 of 20 and one at 16 of 20 are not told apart by this. What round 1 does show: the copy-a-field task is too easy to separate anything (every model, every item); the smallest models fall away as soon as a rule has to be applied; and the price for the same answers differs by a factor of about a hundred, mostly because some models reason at length by default. Round 2 is partial. Round 2 (from 02:53 UTC: a harder warnings task, all 46 timeline rows, three whole paragraphs to translate, each item three times, reasoning on and off as separate rows) stopped part-way: from 02:55 UTC OpenRouter refused every call with 'Insufficient credits' (the account's balance, not this team's cap: the team had spent about a dollar). Refused calls are not answers and are not in the results. What completed is shown and marked partial; the rest runs when the account has credits. The one new task that ran far enough to read, warnings in force at an hour, does separate the models: see its page.

Which model for which job here

For the other agents: which model to call for which job on this site, from round 1. Our choice, a hypothesis from small samples; the figures beside each are that model's on the task named. Every call through tools/models-ask.mjs or tools/llm.mjs; reasoning off unless the row says otherwise; a model's output never reaches a page without the check in the last column. As data: which.json.

What 100 timeline rows cost. Labelling and decision extraction, per 100 items, at the prices OpenRouter reported (from results.json, usd_per_100_items):

So a second reader on every row of a 100-row timeline costs under one US cent on the two large open-weight models that answer without reasoning (Gemma 4 31B, Llama 4 Maverick), and 30 to 60 cents on the two that reason at length by default. Reasoning off is the default row; what it costs when on is the row beside it.

What we would use each for

Our reading of the table: a hypothesis, from small samples, to be re-run as the item sets grow.

• Turning a structured official message (CAP, IPMA's JSON) into our fields: a small open-weight model (google/gemma-3-4b-it, or google/gemma-4-31b-it for a margin), or no model at all: the cheapest got 17 of 17, and a parser does this for nothing and cannot be wrong in new ways. A model earns its place only when the text is prose.
• Extracting who decided and for which hours from official prose: a large open-weight model (google/gemma-4-31b-it, meta-llama/llama-4-maverick, deepseek/deepseek-v4.1-flash) or a cheap commercial one (google/gemini-3.5-flash-lite, anthropic/claude-haiku-5.5): each got 10 of 10 at 0.0001 to 0.0005 US dollars an item. Not the models under about 10 billion parameters: they invent hours. Whatever the model, a null window stays null until a person or the source fills it.
• A first translation of IPMA's warning text for the English gloss the Weather face asks for: google/gemma-4-31b-it or meta-llama/llama-4-maverick, with the figures check run on every output and the Portuguese always shown beside it. IPMA's text is the warning; the English is a reading aid and says so.
• Labelling fact, evidence, hypothesis: none on its own yet: the best agree with our labels on 17 of 20 and the disagreements are informative. Useful now as a second reader that flags rows where several models disagree with the lead's label, for the lead to look at; never as the labeller.
• Anything in a loop, every few minutes: not the models that reason by default (qwen/qwen3.5-397b-a17b, mistralai/mistral-large-4-0, qwen/qwen3.5-9b) at their default settings: for the same answers they took 10 to 30 seconds an item and the two large ones cost more than the other eleven models together. Asked not to reason (the rows marked #reasoning-off), the two Qwen models cost a fifteenth to a seventieth as much. The small one did as well on translation (20 of 20, having stopped running out of tokens) and worse where a rule is weighed (labels 12 to 10 of 20, decisions 9 to 6 of 10). The large one did as well everywhere it answered: decisions 10 of 10, translation 17 of 17, labels 12 of 15 against 16 of 20 with reasoning. A correction: at 02:39 UTC this page said the large one fell from 16 to 12 of 20 on labels; five of those 20 calls had been refused by the provider for lack of credits and were counted as wrong answers. They are now left out. Mistral Large 4, GPT-OSS 20B and Gemini 3.5 Flash Lite reason whatever they are told (the provider refuses the setting).
• What other sources and answer engines say (the shadow report): an answer engine with search (perplexity/sonar, about half a cent a question in one test call), through the same wrapper, its answer kept as evidence about the engine, with the pages it cites. Measured by incident.shadow@, not here: its first round is at shadow/.

What the models tell us about our own data

The label task doubles as a check of the timeline, and it found a gap in our labels rather than a wrong label: on three rows most models disagree with the lead's label, and all three are things we counted or observed ourselves (s-1110T2358, s-1111T0015, s-1110T2153). The lead keeps the labels; the question it raises for Dinis, with each row and each model's answer, is on the label page.

The spend

From the ledger's summary (ledger-summary.json, 2026-11-11 03:06 UTC). Costs are what OpenRouter reported for each call. The cap: 20 US dollars a day for the whole team (Dinis, 11 November ~02:50 UTC, BRIEF.md; 5 before), kept as shares (budgets.json): a ledger lives in the session that makes the calls, so each agent's calls are refused once its own share is spent.

The budget. The team's cap is 20 US dollars a day from 02:50 UTC on 11 November (5 before); this desk's share is 6 and it has spent about 0.87. Round 2 in full (109 items, 20 rows, three runs of the cheap rows) is about 2 to 3 dollars. It needs credits on the OpenRouter account first: the account's balance, which is not ours to top up, ran out at 02:55 UTC.

How it is kept

• Every call goes through tools/llm.mjs: one ledger line per call, the raw request and response in a store outside git (an encrypted vault), a daily cap kept as shares.
• The items: models/evals/<task>/items.json. Each model's parsed answers: models/evals/<task>/results/<model>.json (each with the sha256 of its raw record). The roster: models.json; the licences: licences.json.
• Run again: node tools/models-items.mjs && node tools/models-run.mjs && node tools/models-report.mjs.
• For agents: the table is results.json; which model for which job is which.json; the spend is ledger-summary.json.
• Asking a model from another desk: agents/models/ask.md.