Benchmark AI_TRUECOST_

real AI cost, not vanity metrics

Follow on X updates & details

gpt-5.6-luna

Running gpt-5.6-luna inside cursor.

FastAPI: Add POST /api/v1/projects/{project_id}/tasks endpoint Plan → Review plan → Execute → Review & fix → Open PR → Verify PR → Verify e2e

All steps run as one workflow, inside a single Docker container.

Swipe to view more

5 of the 10 steps are agent sessions, each started fresh: only the repository working tree and the previous step's output carry over. Host steps orchestrate and verify, and cost nothing.
# Step Kind What happens
1 pullIssue host Fetches the GitHub issue and creates a branch
2 plan agent Drafts a plan from the ticket, including tests
3 reviewPlan agent Always runs; returns the full final plan
4 execute agent Implements code and tests from the reviewed plan
5 reviewAndCorrect agent Runs checks, reviews against the plan, fixes, re-runs checks
6 openPullRequest agent Commits, pushes, opens a pull request into main
7 verifyPullRequest host Checks that the exact pull request is open
8 verifyPullRequestCheck host Waits for the required end-to-end check to pass
9 summarizeUsage host Totals tokens and cost
10 score host Records the final result
Speed
Reasoning
Run 7 2026-08-21 · standard · high $0.11 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.01 0m 47s 18 3,212 31
reviewPlan succeeded $0.01 0m 28s 12 2,243 24
execute succeeded $0.04 1m 40s 72 6,711 53
reviewAndCorrect succeeded $0.02 0m 58s 39 3,166 35
openPullRequest succeeded $0.02 1m 06s 48 3,831 26
verifyPullRequest succeeded $0.00 0m 00s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 11s 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$0.11
Input tokens
189
Output tokens
19,163
Cache-read tokens
1,866,789
Tool calls
169

Tool call mix

read
84
shell
54
glob
15
edit
12
createPlan
2
grep
2
Run 6 2026-08-21 · standard · high $0.09 FAIL

Note

Committed and pushed the branch, but no PR was opened because `gh` was not installed (`gh: command not found`).

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.02 0m 56s 21 3,633 28
reviewPlan succeeded $0.01 0m 25s 12 1,852 23
execute succeeded $0.02 1m 07s 45 4,516 36
reviewAndCorrect succeeded $0.02 0m 50s 27 2,666 30
openPullRequest succeeded $0.02 0m 56s 42 2,859 26
verifyPullRequest succeeded $0.00 0m 00s 0 0 0
verifyPullRequestCheck succeeded $0.00 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$0.09
Input tokens
147
Output tokens
15,526
Cache-read tokens
1,252,005
Tool calls
143

Tool call mix

read
73
shell
37
glob
19
edit
7
createPlan
2
task
2
getMcpTools
1
grep
1
webFetch
1
Run 5 2026-08-21 · standard · high $0.09 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.01 0m 45s 18 2,761 30
reviewPlan succeeded $0.01 0m 26s 12 2,253 22
execute succeeded $0.03 1m 19s 51 4,931 36
reviewAndCorrect succeeded $0.02 0m 42s 21 2,137 26
openPullRequest succeeded $0.02 0m 53s 42 2,484 22
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 21s 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$0.09
Input tokens
144
Output tokens
14,566
Cache-read tokens
1,257,130
Tool calls
136

Tool call mix

read
73
shell
37
glob
15
edit
8
createPlan
2
grep
1
Run 4 2026-08-21 · standard · high $0.09 FAIL

Note

Committed and pushed the branch, but no PR was opened because `gh` was not installed (`gh: command not found`).

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.01 0m 45s 15 2,826 29
reviewPlan succeeded $0.01 0m 26s 12 2,055 20
execute succeeded $0.03 1m 25s 57 5,505 45
reviewAndCorrect succeeded $0.02 0m 50s 30 2,809 36
openPullRequest succeeded $0.01 0m 41s 27 1,727 15
verifyPullRequest succeeded $0.00 0m 00s 0 0 0
verifyPullRequestCheck succeeded $0.00 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$0.09
Input tokens
141
Output tokens
14,922
Cache-read tokens
1,254,232
Tool calls
145

Tool call mix

read
78
shell
39
glob
13
edit
11
createPlan
2
grep
2
Run 3 2026-08-21 · standard · high $0.10 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.01 0m 41s 15 2,449 33
reviewPlan succeeded $0.01 0m 30s 12 2,361 22
execute succeeded $0.03 1m 35s 66 5,813 43
reviewAndCorrect succeeded $0.02 1m 04s 30 4,147 43
openPullRequest succeeded $0.02 0m 57s 39 3,330 22
verifyPullRequest succeeded $0.00 0m 00s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 21s 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$0.10
Input tokens
162
Output tokens
18,100
Cache-read tokens
1,531,536
Tool calls
163

Tool call mix

read
78
shell
50
glob
18
edit
9
grep
4
await
2
createPlan
2
Run 2 2026-08-21 · standard · high $0.09 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.01 0m 43s 15 2,537 31
reviewPlan succeeded $0.01 0m 27s 12 2,296 22
execute succeeded $0.03 1m 16s 57 5,119 39
reviewAndCorrect succeeded $0.02 0m 43s 21 2,517 25
openPullRequest succeeded $0.02 1m 03s 48 3,373 26
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 21s 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$0.09
Input tokens
153
Output tokens
15,842
Cache-read tokens
1,351,567
Tool calls
143

Tool call mix

read
75
shell
39
glob
13
edit
9
grep
3
await
2
createPlan
2
Run 1 2026-08-21 · standard · high $0.09 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.01 0m 45s 15 2,721 29
reviewPlan succeeded $0.01 0m 23s 12 2,088 22
execute succeeded $0.03 1m 05s 63 5,288 37
reviewAndCorrect succeeded $0.02 0m 43s 24 2,706 24
openPullRequest succeeded $0.02 1m 03s 51 3,341 29
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 21s 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$0.09
Input tokens
165
Output tokens
16,144
Cache-read tokens
1,482,115
Tool calls
141

Tool call mix

read
75
shell
37
glob
13
await
6
edit
6
createPlan
2
grep
2