Benchmark AI_TRUECOST_

real AI cost, not vanity metrics

Follow on X updates & details

claude-sonnet-5.5

Running claude-sonnet-5.5 inside cursor.

FastAPI: Add POST /api/v1/projects/{project_id}/tasks endpoint Plan → Review plan → Execute → Review & fix → Open PR → Verify PR → Verify e2e

All steps run as one workflow, inside a single Docker container.

Swipe to view more

5 of the 10 steps are agent sessions, each started fresh: only the repository working tree and the previous step's output carry over. Host steps orchestrate and verify, and cost nothing.
# Step Kind What happens
1 pullIssue host Fetches the GitHub issue and creates a branch
2 plan agent Drafts a plan from the ticket, including tests
3 reviewPlan agent Always runs; returns the full final plan
4 execute agent Implements the landing page; may run repeated preview captures; no harness step timeout
5 reviewAndCorrect agent Visual review against the brief and reference render; fixes layout and copy; no harness step timeout
6 openPullRequest agent Commits, pushes, opens a pull request into main
7 verifyPullRequest host Checks that the exact pull request is open
8 verifyPullRequestCheck host Waits for the required end-to-end check to pass
9 summarizeUsage host Totals tokens and cost
10 score host Records the final result
Speed
Reasoning
Run 5 2026-10-08 · standard · high $0.62 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.16 0m 57s 12 5,402 22
reviewPlan succeeded $0.13 0m 55s 14 7,357 22
execute succeeded $0.19 1m 13s 22 7,516 16
reviewAndCorrect succeeded $0.09 0m 20s 8 1,298 3
openPullRequest succeeded $0.05 0m 28s 18 1,530 8
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 32s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.62
Agent time
3m 53s
Input tokens
74
Output tokens
23,103
Cache-read tokens
884,264
Tool calls
71

Tool call mix

read
37
shell
26
edit
4
glob
2
createPlan
1
grep
1
Run 4 2026-10-07 · standard · high $0.61 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.15 0m 57s 12 5,341 22
reviewPlan succeeded $0.13 0m 55s 14 7,259 22
execute succeeded $0.19 1m 11s 21 7,369 16
reviewAndCorrect succeeded $0.09 0m 21s 8 1,323 3
openPullRequest succeeded $0.05 0m 29s 18 1,549 8
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 29s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.61
Agent time
3m 53s
Input tokens
73
Output tokens
22,841
Cache-read tokens
888,816
Tool calls
71

Tool call mix

read
36
shell
29
glob
3
edit
2
createPlan
1
grep
0
Run 3 2026-10-07 · standard · high $0.61 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.16 0m 58s 12 5,536 22
reviewPlan succeeded $0.13 0m 55s 14 7,458 22
execute succeeded $0.18 1m 14s 21 7,389 16
reviewAndCorrect succeeded $0.09 0m 20s 8 1,269 3
openPullRequest succeeded $0.05 0m 28s 18 1,520 8
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 34s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.61
Agent time
3m 56s
Input tokens
73
Output tokens
23,172
Cache-read tokens
876,560
Tool calls
71

Tool call mix

read
37
shell
24
edit
5
glob
2
grep
2
createPlan
1
Run 2 2026-10-07 · standard · high $0.62 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.16 0m 56s 12 5,536 22
reviewPlan succeeded $0.13 0m 55s 14 7,219 22
execute succeeded $0.19 1m 13s 22 7,482 16
reviewAndCorrect succeeded $0.09 0m 20s 8 1,297 3
openPullRequest succeeded $0.05 0m 28s 18 1,543 8
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 34s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.62
Agent time
3m 53s
Input tokens
74
Output tokens
23,077
Cache-read tokens
885,812
Tool calls
71

Tool call mix

read
39
shell
25
edit
4
glob
2
grep
1
createPlan
0
Run 1 2026-10-07 · standard · high $0.62 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.16 0m 58s 12 5,499 22
reviewPlan succeeded $0.13 0m 56s 14 7,522 22
execute succeeded $0.19 1m 15s 22 7,358 16
reviewAndCorrect succeeded $0.09 0m 20s 8 1,309 3
openPullRequest succeeded $0.05 0m 28s 18 1,522 8
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 33s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.62
Agent time
3m 56s
Input tokens
74
Output tokens
23,210
Cache-read tokens
891,681
Tool calls
71

Tool call mix

read
38
shell
27
edit
3
createPlan
1
glob
1
grep
1