Benchmark AI_TRUECOST_

real AI cost, not vanity metrics

Follow on X updates & details

claude-haiku-5.5

Running claude-haiku-5.5 inside cursor.

FastAPI: Add POST /api/v1/projects/{project_id}/tasks endpoint Plan → Review plan → Execute → Review & fix → Open PR → Verify PR → Verify e2e

All steps run as one workflow, inside a single Docker container.

Swipe to view more

5 of the 10 steps are agent sessions, each started fresh: only the repository working tree and the previous step's output carry over. Host steps orchestrate and verify, and cost nothing.
# Step Kind What happens
1 pullIssue host Fetches the GitHub issue and creates a branch
2 plan agent Drafts a plan from the ticket, including tests
3 reviewPlan agent Always runs; returns the full final plan
4 execute agent Implements the landing page; may run repeated preview captures; no harness step timeout
5 reviewAndCorrect agent Visual review against the brief and reference render; fixes layout and copy; no harness step timeout
6 openPullRequest agent Commits, pushes, opens a pull request into main
7 verifyPullRequest host Checks that the exact pull request is open
8 verifyPullRequestCheck host Waits for the required end-to-end check to pass
9 summarizeUsage host Totals tokens and cost
10 score host Records the final result
Speed
Reasoning
Run 5 2026-10-07 · standard · high $0.07 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.01 1m 30s 16 12,610 29
reviewPlan succeeded $0.01 1m 07s 16 12,545 28
execute succeeded $0.02 1m 35s 38 13,948 36
reviewAndCorrect succeeded $0.02 1m 28s 42 12,062 25
openPullRequest succeeded $0.01 0m 42s 20 3,567 8
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 22s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.07
Agent time
6m 23s
Input tokens
132
Output tokens
54,732
Cache-read tokens
2,165,804
Tool calls
126

Tool call mix

read
65
shell
32
edit
19
glob
6
createPlan
2
grep
2
Run 4 2026-10-07 · standard · high $0.07 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.01 1m 32s 16 12,634 29
reviewPlan succeeded $0.01 1m 09s 16 12,814 28
execute succeeded $0.02 1m 34s 39 13,986 36
reviewAndCorrect succeeded $0.02 1m 27s 42 12,159 25
openPullRequest succeeded $0.01 0m 42s 20 3,524 8
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 22s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.07
Agent time
6m 23s
Input tokens
133
Output tokens
55,117
Cache-read tokens
2,163,471
Tool calls
126

Tool call mix

read
68
shell
30
edit
18
glob
7
createPlan
2
grep
1
Run 3 2026-10-07 · standard · high $0.07 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.01 1m 29s 16 12,421 29
reviewPlan succeeded $0.01 1m 08s 16 12,510 28
execute succeeded $0.02 1m 37s 38 14,120 36
reviewAndCorrect succeeded $0.02 1m 28s 42 12,176 25
openPullRequest succeeded $0.01 0m 41s 20 3,606 8
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 20s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.07
Agent time
6m 23s
Input tokens
132
Output tokens
54,833
Cache-read tokens
2,176,479
Tool calls
126

Tool call mix

read
64
shell
30
edit
22
glob
5
createPlan
3
grep
2
Run 2 2026-10-07 · standard · high $0.07 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.01 1m 28s 16 12,802 29
reviewPlan succeeded $0.01 1m 07s 16 12,542 28
execute succeeded $0.02 1m 33s 38 13,673 36
reviewAndCorrect succeeded $0.02 1m 28s 41 12,223 25
openPullRequest succeeded $0.01 0m 43s 20 3,595 8
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 23s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.07
Agent time
6m 19s
Input tokens
131
Output tokens
54,835
Cache-read tokens
2,149,719
Tool calls
126

Tool call mix

read
67
shell
32
edit
17
glob
5
createPlan
4
grep
1
Run 1 2026-10-07 · standard · high $0.07 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.01 1m 32s 16 12,509 29
reviewPlan succeeded $0.01 1m 07s 16 12,627 28
execute succeeded $0.02 1m 36s 38 14,036 36
reviewAndCorrect succeeded $0.02 1m 27s 42 12,231 25
openPullRequest succeeded $0.01 0m 43s 20 3,595 8
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 19s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.07
Agent time
6m 23s
Input tokens
132
Output tokens
54,998
Cache-read tokens
2,169,085
Tool calls
126

Tool call mix

read
64
shell
32
edit
19
glob
7
createPlan
2
grep
2