Benchmark AI_TRUECOST_

real AI cost, not vanity metrics

Follow on X updates & details

grok-4.7

Running grok-4.7 inside cursor.

FastAPI: Add POST /api/v1/projects/{project_id}/tasks endpoint Plan → Review plan → Execute → Review & fix → Open PR → Verify PR → Verify e2e

All steps run as one workflow, inside a single Docker container.

Swipe to view more

5 of the 10 steps are agent sessions, each started fresh: only the repository working tree and the previous step's output carry over. Host steps orchestrate and verify, and cost nothing.
# Step Kind What happens
1 pullIssue host Fetches the GitHub issue and creates a branch
2 plan agent Drafts a plan from the ticket, including tests
3 reviewPlan agent Always runs; returns the full final plan
4 execute agent Implements the landing page; may run repeated preview captures; no harness step timeout
5 reviewAndCorrect agent Visual review against the brief and reference render; fixes layout and copy; no harness step timeout
6 openPullRequest agent Commits, pushes, opens a pull request into main
7 verifyPullRequest host Checks that the exact pull request is open
8 verifyPullRequestCheck host Waits for the required end-to-end check to pass
9 summarizeUsage host Totals tokens and cost
10 score host Records the final result
Speed
Reasoning
Run 5 2026-09-21 · standard · high $2.01 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.22 2m 50s 63,973 10,489 27
reviewPlan succeeded $0.17 2m 03s 42,307 8,237 25
execute succeeded $0.39 2m 57s 62,375 11,757 42
reviewAndCorrect succeeded $1.06 8m 36s 207,235 38,049 48
openPullRequest succeeded $0.16 0m 42s 30,462 1,712 11
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 21s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.01
Agent time
17m 07s
Input tokens
406,352
Output tokens
70,244
Cache-read tokens
1,546,368
Tool calls
153

Tool call mix

read
93
shell
21
edit
14
glob
12
grep
9
createPlan
2
getMcpTools
2
Run 4 2026-09-21 · standard · high $2.02 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.22 2m 54s 64,735 10,723 27
reviewPlan succeeded $0.17 2m 05s 42,379 8,268 25
execute succeeded $0.39 2m 59s 63,014 11,841 42
reviewAndCorrect succeeded $1.07 8m 24s 211,466 38,713 48
openPullRequest succeeded $0.16 0m 41s 29,707 1,725 11
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 22s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.02
Agent time
17m 04s
Input tokens
411,301
Output tokens
71,270
Cache-read tokens
1,531,538
Tool calls
153

Tool call mix

read
92
shell
20
glob
16
edit
14
grep
9
createPlan
2
getMcpTools
0
Run 3 2026-09-21 · standard · high $2.01 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.22 2m 49s 62,601 10,671 27
reviewPlan succeeded $0.17 2m 02s 42,550 8,388 25
execute succeeded $0.39 2m 59s 61,926 11,674 42
reviewAndCorrect succeeded $1.07 8m 41s 206,918 37,617 48
openPullRequest succeeded $0.16 0m 42s 29,753 1,750 11
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 24s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.01
Agent time
17m 13s
Input tokens
403,748
Output tokens
70,100
Cache-read tokens
1,566,955
Tool calls
153

Tool call mix

read
92
shell
20
edit
16
glob
14
grep
7
getMcpTools
3
createPlan
1
Run 2 2026-09-21 · standard · high $2.00 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.22 2m 54s 62,463 10,557 27
reviewPlan succeeded $0.17 2m 03s 42,177 8,319 25
execute succeeded $0.40 2m 54s 63,262 11,537 42
reviewAndCorrect succeeded $1.06 8m 37s 202,121 37,499 48
openPullRequest succeeded $0.16 0m 43s 29,805 1,714 11
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 20s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.00
Agent time
17m 11s
Input tokens
399,828
Output tokens
69,626
Cache-read tokens
1,569,082
Tool calls
153

Tool call mix

read
91
shell
18
edit
14
glob
14
grep
13
getMcpTools
2
createPlan
1
Run 1 2026-09-21 · standard · high $2.02 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.22 2m 54s 64,882 10,687 27
reviewPlan succeeded $0.17 2m 01s 42,125 8,386 25
execute succeeded $0.40 2m 58s 62,941 11,731 42
reviewAndCorrect succeeded $1.07 8m 29s 208,213 38,791 48
openPullRequest succeeded $0.16 0m 43s 30,984 1,678 11
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 19s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.02
Agent time
17m 04s
Input tokens
409,145
Output tokens
71,273
Cache-read tokens
1,547,410
Tool calls
153

Tool call mix

read
92
shell
21
glob
14
edit
12
grep
10
createPlan
3
getMcpTools
1