The cheaper model call made each successful task cost more
A candidate model is cheaper per request. In a small replay of the same tasks, it completes fewer of them. The invoice still drops, so you have to divide by the work that actually got done before deciding to switch.
Here’s a fictional fixture you can run in Node.js. It comes from a question we work with at Lumina: does a cheaper model still look cheaper once you include the tool calls and retries it takes to finish the task?
Compare task runs, including failures
Both versions get the same 40 task IDs. Twenty are classified as tool tasks before replay. Each row carries the total cost in cents and elapsed time for one task, including retries. A successful task meets the same outcome check in both versions. In a live replay I would use the recorded outcomes and billing data, rather than generating rows like this.
Save the code as replay-cost.cjs and run node replay-cost.cjs.
const assert = require('node:assert/strict');
const tasks = Array.from({ length: 40 }, (_, id) => ({
id,
usesTool: id >= 20,
}));
function fixture({ cents, passes, toolMs }) {
return tasks.map(({ id, usesTool }) => ({
id,
usesTool,
cents,
passed: id % 20 < passes,
ms: (usesTool ? toolMs : 600) + (id % 20) * 30,
}));
}
function report(runs) {
const passed = runs.filter((r) => r.passed).length;
const cents = runs.reduce((total, r) => total + r.cents, 0);
const toolTimes = runs.filter((r) => r.usesTool).map((r) => r.ms)
.sort((a, b) => a - b);
const p95 = toolTimes[Math.ceil(toolTimes.length * 0.95) - 1];
return {
passed,
totalCents: cents,
centsPerPass: passed ? Number((cents / passed).toFixed(2)) : null,
toolP95Ms: p95 ?? null,
};
}
const current = fixture({ cents: 6, passes: 18, toolMs: 1500 });
const candidate = fixture({ cents: 5, passes: 14, toolMs: 2250 });
assert.deepEqual(current.map((r) => r.id), candidate.map((r) => r.id));
assert.deepEqual(report(current), {
passed: 36, totalCents: 240, centsPerPass: 6.67, toolP95Ms: 2040,
});
assert.deepEqual(report(candidate), {
passed: 28, totalCents: 200, centsPerPass: 7.14, toolP95Ms: 2790,
});
console.table({ current: report(current), candidate: report(candidate) });
The candidate costs 200 cents instead of 240 for the replay. It also finishes eight fewer tasks. Cost per successful task goes from 6.67 to 7.14 cents, and the tool cohort’s observed p95 moves from 2.04 to 2.79 seconds. This p95 describes twenty fixture runs; it is not an estimate of a production tail.
Pick the release rule before seeing the bill
For a real replay, I’d save the task IDs and expected outcomes, count failures in the cost numerator, and report timeouts alongside the tool cohort’s latency. A task that switches from a tool path to a text-only answer should still be judged on whether it actually achieved the intended result.
Agree on acceptable success, latency and cost before comparing models. The example gives you the three numbers you need to start that conversation.
I build the products behind the Lumina courses. Our Cost & Latency bundle includes 16 modules and 15 Python labs on measuring trade-offs like these. The fixture uses invented numbers; the script was verified locally. Drafted with AI assistance.
Comments
Join the discussionNo comments yet. Be the first to comment.