Benchmark AI_TRUECOST_

real AI cost, not vanity metrics

Follow on X updates & details

claude-fable-5

Running claude-fable-5 inside cursor.

FastAPI: Add POST /api/v1/projects/{project_id}/tasks endpoint Plan → Review plan → Execute → Review & fix → Open PR → Verify PR → Verify e2e

All steps run as one workflow, inside a single Docker container.

Swipe to view more

5 of the 10 steps are agent sessions, each started fresh: only the repository working tree and the previous step's output carry over. Host steps orchestrate and verify, and cost nothing.
# Step Kind What happens
1 pullIssue host Fetches the GitHub issue and creates a branch
2 plan agent Drafts a plan from the ticket, including tests
3 reviewPlan agent Always runs; returns the full final plan
4 execute agent Implements the landing page; may run repeated preview captures; no harness step timeout
5 reviewAndCorrect agent Visual review against the brief and reference render; fixes layout and copy; no harness step timeout
6 openPullRequest agent Commits, pushes, opens a pull request into main
7 verifyPullRequest host Checks that the exact pull request is open
8 verifyPullRequestCheck host Waits for the required end-to-end check to pass
9 summarizeUsage host Totals tokens and cost
10 score host Records the final result
Speed
Reasoning
Run 5 2026-09-02 · standard · high $4.99 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $1.16 1m 17s 14 4,475 23
reviewPlan succeeded $0.73 0m 53s 12 3,794 18
execute succeeded $1.51 1m 36s 24 6,236 23
reviewAndCorrect succeeded $0.72 1m 05s 12 2,494 14
openPullRequest succeeded $0.87 1m 18s 24 3,599 14
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 33s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$4.99
Agent time
6m 09s
Input tokens
86
Output tokens
20,598
Cache-read tokens
1,870,705
Tool calls
92

Tool call mix

read
53
shell
26
edit
7
glob
4
createPlan
2
Run 4 2026-09-01 · standard · high $4.97 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $1.18 1m 16s 14 4,443 23
reviewPlan succeeded $0.73 0m 54s 12 3,726 18
execute succeeded $1.48 1m 34s 23 6,084 23
reviewAndCorrect succeeded $0.71 1m 04s 12 2,441 14
openPullRequest succeeded $0.87 1m 17s 24 3,687 14
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 36s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$4.97
Agent time
6m 03s
Input tokens
85
Output tokens
20,381
Cache-read tokens
1,863,179
Tool calls
92

Tool call mix

read
52
shell
26
edit
7
glob
5
createPlan
2
Run 3 2026-09-01 · standard · high $4.99 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $1.17 1m 18s 14 4,402 23
reviewPlan succeeded $0.73 0m 53s 12 3,707 18
execute succeeded $1.53 1m 34s 23 6,295 23
reviewAndCorrect succeeded $0.71 1m 03s 12 2,521 14
openPullRequest succeeded $0.86 1m 19s 23 3,583 14
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 34s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$4.99
Agent time
6m 07s
Input tokens
84
Output tokens
20,508
Cache-read tokens
1,855,394
Tool calls
92

Tool call mix

read
54
shell
26
edit
6
glob
4
createPlan
2
Run 2 2026-09-01 · standard · high $4.97 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $1.15 1m 19s 14 4,523 23
reviewPlan succeeded $0.73 0m 53s 12 3,727 18
execute succeeded $1.51 1m 34s 24 6,310 23
reviewAndCorrect succeeded $0.71 1m 06s 12 2,441 14
openPullRequest succeeded $0.87 1m 17s 24 3,686 14
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 32s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$4.97
Agent time
6m 10s
Input tokens
86
Output tokens
20,687
Cache-read tokens
1,849,226
Tool calls
92

Tool call mix

read
54
shell
26
edit
6
glob
5
createPlan
1
Run 1 2026-09-01 · standard · high $5.02 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $1.16 1m 15s 14 4,461 23
reviewPlan succeeded $0.73 0m 54s 12 3,827 18
execute succeeded $1.52 1m 34s 23 6,273 23
reviewAndCorrect succeeded $0.72 1m 05s 12 2,496 14
openPullRequest succeeded $0.88 1m 20s 24 3,648 14
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 30s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$5.02
Agent time
6m 08s
Input tokens
85
Output tokens
20,705
Cache-read tokens
1,875,929
Tool calls
92

Tool call mix

read
55
shell
25
edit
7
glob
4
createPlan
1