Benchmark AI_TRUECOST_

real AI cost, not vanity metrics

Follow on X updates & details

composer-2.5

Running composer-2.5 inside cursor.

FastAPI: Add POST /api/v1/projects/{project_id}/tasks endpoint Plan → Review plan → Execute → Review & fix → Open PR → Verify PR → Verify e2e

All steps run as one workflow, inside a single Docker container.

Swipe to view more

5 of the 10 steps are agent sessions, each started fresh: only the repository working tree and the previous step's output carry over. Host steps orchestrate and verify, and cost nothing.
# Step Kind What happens
1 pullIssue host Fetches the GitHub issue and creates a branch
2 plan agent Drafts a plan from the ticket, including tests
3 reviewPlan agent Always runs; returns the full final plan
4 execute agent Implements code and tests from the reviewed plan
5 reviewAndCorrect agent Runs checks, reviews against the plan, fixes, re-runs checks
6 openPullRequest agent Commits, pushes, opens a pull request into main
7 verifyPullRequest host Checks that the exact pull request is open
8 verifyPullRequestCheck host Waits for the required end-to-end check to pass
9 summarizeUsage host Totals tokens and cost
10 score host Records the final result
Speed
Reasoning
Run 5 2026-08-21 · standard · default $0.33 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.06 2m 57s 25,769 5,263 23
reviewPlan succeeded $0.05 0m 55s 21,338 5,470 23
execute succeeded $0.07 0m 54s 21,047 4,203 31
reviewAndCorrect succeeded $0.09 1m 26s 25,040 6,329 38
openPullRequest succeeded $0.06 0m 46s 16,853 1,696 18
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 32s 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$0.33
Input tokens
110,047
Output tokens
22,961
Cache-read tokens
1,082,016
Tool calls
133

Tool call mix

read
72
shell
30
edit
8
glob
8
grep
8
task
3
createPlan
2
readLints
2
Run 4 2026-08-20 · standard · default $0.31 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.05 2m 34s 29,582 5,223 21
reviewPlan succeeded $0.05 0m 55s 21,796 4,377 23
execute succeeded $0.07 2m 44s 19,602 3,767 29
reviewAndCorrect succeeded $0.08 1m 32s 27,669 6,060 32
openPullRequest succeeded $0.06 0m 45s 18,299 2,077 19
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 22s 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$0.31
Input tokens
116,948
Output tokens
21,504
Cache-read tokens
998,233
Tool calls
124

Tool call mix

read
69
shell
29
glob
8
edit
7
grep
7
createPlan
2
task
2
Run 3 2026-08-19 · standard · default $0.31 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.05 3m 03s 26,317 6,336 22
reviewPlan succeeded $0.04 1m 38s 20,840 4,566 22
execute succeeded $0.05 1m 13s 21,480 3,658 27
reviewAndCorrect succeeded $0.08 1m 29s 26,253 3,730 30
openPullRequest succeeded $0.08 1m 03s 18,491 2,037 23
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 22s 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$0.31
Input tokens
113,381
Output tokens
20,327
Cache-read tokens
993,322
Tool calls
124

Tool call mix

read
68
shell
34
edit
8
glob
6
grep
4
createPlan
2
task
2
Run 2 2026-08-19 · standard · default $0.28 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.05 2m 40s 26,363 5,874 24
reviewPlan succeeded $0.05 1m 17s 21,379 4,897 24
execute succeeded $0.05 1m 14s 21,089 3,733 27
reviewAndCorrect succeeded $0.06 0m 59s 24,022 2,118 25
openPullRequest succeeded $0.07 0m 51s 18,625 1,978 23
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 21s 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$0.28
Input tokens
111,478
Output tokens
18,600
Cache-read tokens
891,828
Tool calls
123

Tool call mix

read
66
shell
33
edit
8
glob
6
grep
5
createPlan
2
task
2
readLints
1
Run 1 2026-08-17 · standard · default $0.28 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.05 1m 09s 20,190 5,224 25
reviewPlan succeeded $0.05 0m 44s 21,564 5,122 26
execute succeeded $0.05 3m 16s 21,369 4,199 26
reviewAndCorrect succeeded $0.06 0m 39s 26,075 1,630 23
openPullRequest succeeded $0.07 0m 48s 19,257 1,984 22
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$0.28
Input tokens
108,455
Output tokens
18,159
Cache-read tokens
918,294
Tool calls
122

Tool call mix

read
67
shell
31
edit
9
glob
7
grep
6
createPlan
2