Benchmark AI_TRUECOST_

real AI cost, not vanity metrics

Follow on X updates & details

grok-4.6

Running grok-4.6 inside cursor.

FastAPI: Add POST /api/v1/projects/{project_id}/tasks endpoint Plan → Review plan → Execute → Review & fix → Open PR → Verify PR → Verify e2e

All steps run as one workflow, inside a single Docker container.

Swipe to view more

5 of the 10 steps are agent sessions, each started fresh: only the repository working tree and the previous step's output carry over. Host steps orchestrate and verify, and cost nothing.
# Step Kind What happens
1 pullIssue host Fetches the GitHub issue and creates a branch
2 plan agent Drafts a plan from the ticket, including tests
3 reviewPlan agent Always runs; returns the full final plan
4 execute agent Implements code and tests from the reviewed plan
5 reviewAndCorrect agent Runs checks, reviews against the plan, fixes, re-runs checks
6 openPullRequest agent Commits, pushes, opens a pull request into main
7 verifyPullRequest host Checks that the exact pull request is open
8 verifyPullRequestCheck host Waits for the required end-to-end check to pass
9 summarizeUsage host Totals tokens and cost
10 score host Records the final result
Speed
Reasoning
Run 5 2026-08-21 · standard · high $1.15 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.16 1m 31s 32,520 7,102 31
reviewPlan succeeded $0.15 0m 57s 31,390 6,904 27
execute succeeded $0.21 1m 01s 35,774 7,359 31
reviewAndCorrect succeeded $0.40 2m 38s 60,531 19,856 35
openPullRequest succeeded $0.22 0m 59s 31,600 3,748 19
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 22s 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$1.15
Input tokens
191,815
Output tokens
44,969
Cache-read tokens
986,304
Tool calls
143

Tool call mix

read
94
shell
21
glob
13
edit
8
grep
5
createPlan
2
Run 4 2026-08-19 · standard · high $1.26 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.16 1m 04s 33,123 5,876 34
reviewPlan succeeded $0.16 2m 56s 36,198 5,853 25
execute succeeded $0.23 1m 39s 39,817 8,977 32
reviewAndCorrect succeeded $0.47 2m 28s 62,705 17,053 38
openPullRequest succeeded $0.23 1m 05s 32,418 3,804 21
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 22s 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$1.26
Input tokens
204,261
Output tokens
41,563
Cache-read tokens
1,203,008
Tool calls
150

Tool call mix

read
94
shell
23
glob
16
edit
7
grep
6
createPlan
2
askQuestion
1
task
1
Run 3 2026-08-19 · standard · high $1.22 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.15 0m 58s 34,986 5,507 28
reviewPlan succeeded $0.16 1m 03s 33,595 6,297 26
execute succeeded $0.22 1m 21s 36,208 6,589 31
reviewAndCorrect succeeded $0.45 3m 15s 57,412 18,268 37
openPullRequest succeeded $0.25 1m 00s 32,458 4,247 22
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 21s 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$1.22
Input tokens
194,659
Output tokens
40,908
Cache-read tokens
1,173,056
Tool calls
144

Tool call mix

read
91
shell
22
edit
12
glob
12
grep
4
createPlan
2
askQuestion
1
Run 2 2026-08-19 · standard · high $1.40 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.17 1m 13s 33,786 6,183 32
reviewPlan succeeded $0.17 2m 24s 42,571 6,623 21
execute succeeded $0.24 1m 36s 38,450 7,662 34
reviewAndCorrect succeeded $0.52 3m 03s 64,740 19,943 41
openPullRequest succeeded $0.30 1m 04s 33,600 4,831 21
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 22s 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$1.40
Input tokens
213,147
Output tokens
45,242
Cache-read tokens
1,395,520
Tool calls
149

Tool call mix

read
93
shell
24
edit
11
glob
11
grep
5
createPlan
2
task
2
askQuestion
1
Run 1 2026-08-17 · standard · high $1.40 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.15 2m 14s 37,874 5,887 23
reviewPlan succeeded $0.18 1m 54s 39,083 5,953 21
execute succeeded $0.22 1m 53s 37,807 6,952 34
reviewAndCorrect succeeded $0.59 3m 21s 72,639 24,189 44
openPullRequest succeeded $0.25 0m 49s 31,948 3,952 23
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$1.40
Input tokens
219,351
Output tokens
46,933
Cache-read tokens
1,354,816
Tool calls
145

Tool call mix

read
80
shell
25
glob
15
edit
10
grep
8
task
4
createPlan
2
getMcpTools
1