Benchmark AI_TRUECOST_

real AI cost, not vanity metrics

Follow on X updates & details

← Home

gpt-6-astra

Running gpt-6-astra inside codex.

Workflow

FastAPI: Add POST /api/v1/projects/{project_id}/tasks endpoint Plan → Review plan → Execute → Review & fix → Open PR → Verify PR → Verify e2e

Every run uses the same steps in the same order. Full workflow definition →

All steps run as one workflow, inside a single Docker container.

Swipe to view more

5 of the 10 steps are agent sessions, each started fresh: only the repository working tree and the previous step's output carry over. Host steps orchestrate and verify, and cost nothing.
# Step Kind What happens
1 pullIssue host Fetches the GitHub issue and creates a branch
2 plan agent Drafts a plan from the ticket, including tests
3 reviewPlan agent Always runs; returns the full final plan
4 execute agent Implements the landing page; may run repeated preview captures; no harness step timeout
5 reviewAndCorrect agent Visual review against the brief and reference render; fixes layout and copy; no harness step timeout
6 openPullRequest agent Commits, pushes, opens a pull request into main
7 verifyPullRequest host Checks that the exact pull request is open
8 verifyPullRequestCheck host Waits for the required end-to-end check to pass
9 summarizeUsage host Totals tokens and cost
10 score host Records the final result

Runs

Speed
Reasoning
Run 5 2026-09-07 · standard · low $2.05 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.27 0m 56s 16,715 1,340 1
reviewPlan succeeded $0.24 1m 16s 9,351 2,053 2
execute succeeded $0.59 3m 04s 18,522 4,778 10
reviewAndCorrect succeeded $0.57 1m 28s 33,752 1,633 12
openPullRequest succeeded $0.37 1m 11s 12,939 1,340 8
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 21s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.05
Agent time
7m 56s
Input tokens
91,279
Output tokens
11,144
Cache-read tokens
575,488
Tool calls
33

Tool call mix

command_execution
31
file_change
2
Run 4 2026-09-06 · standard · low $2.03 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.27 0m 54s 16,487 1,328 1
reviewPlan succeeded $0.24 1m 15s 9,159 2,020 2
execute succeeded $0.59 3m 04s 18,538 4,674 10
reviewAndCorrect succeeded $0.58 1m 26s 33,511 1,661 12
openPullRequest succeeded $0.36 1m 12s 13,086 1,357 8
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 23s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.03
Agent time
7m 52s
Input tokens
90,781
Output tokens
11,040
Cache-read tokens
573,226
Tool calls
33

Tool call mix

command_execution
32
file_change
1
Run 3 2026-09-06 · standard · low $2.05 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.27 0m 55s 16,857 1,319 1
reviewPlan succeeded $0.24 1m 16s 9,193 2,056 2
execute succeeded $0.59 3m 01s 18,607 4,735 10
reviewAndCorrect succeeded $0.58 1m 29s 34,276 1,609 12
openPullRequest succeeded $0.37 1m 11s 13,041 1,316 8
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 20s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.05
Agent time
7m 51s
Input tokens
91,974
Output tokens
11,035
Cache-read tokens
574,196
Tool calls
33

Tool call mix

command_execution
30
file_change
3
Run 2 2026-09-06 · standard · low $2.04 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.27 0m 57s 16,447 1,352 1
reviewPlan succeeded $0.24 1m 16s 9,287 2,093 2
execute succeeded $0.59 3m 09s 18,108 4,849 10
reviewAndCorrect succeeded $0.57 1m 30s 33,524 1,642 12
openPullRequest succeeded $0.37 1m 11s 13,071 1,316 8
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 19s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.04
Agent time
8m 03s
Input tokens
90,437
Output tokens
11,252
Cache-read tokens
573,515
Tool calls
33

Tool call mix

command_execution
31
file_change
2
Run 1 2026-09-06 · standard · low $2.04 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.27 0m 56s 16,359 1,330 1
reviewPlan succeeded $0.24 1m 15s 9,336 2,066 2
execute succeeded $0.59 3m 07s 18,431 4,728 10
reviewAndCorrect succeeded $0.58 1m 28s 34,006 1,636 12
openPullRequest succeeded $0.37 1m 10s 13,146 1,360 8
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 22s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.04
Agent time
7m 56s
Input tokens
91,278
Output tokens
11,120
Cache-read tokens
566,954
Tool calls
33

Tool call mix

command_execution
32
file_change
1
Run 5 2026-09-05 · standard · high $2.77 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.29 1m 02s 14,935 1,592 2
reviewPlan succeeded $0.23 1m 21s 8,696 2,074 1
execute succeeded $0.99 4m 17s 39,851 7,539 20
reviewAndCorrect succeeded $0.70 1m 42s 32,980 2,420 33
openPullRequest succeeded $0.55 1m 26s 19,996 1,730 10
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 21s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.77
Agent time
9m 49s
Input tokens
116,458
Output tokens
15,355
Cache-read tokens
839,168
Tool calls
66

Tool call mix

command_execution
61
file_change
5
Run 4 2026-09-04 · standard · high $2.77 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.29 1m 02s 14,731 1,589 2
reviewPlan succeeded $0.23 1m 20s 8,750 2,080 1
execute succeeded $0.99 4m 15s 39,376 7,575 20
reviewAndCorrect succeeded $0.70 1m 41s 32,452 2,419 33
openPullRequest succeeded $0.55 1m 28s 20,325 1,700 10
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 21s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.77
Agent time
9m 46s
Input tokens
115,634
Output tokens
15,363
Cache-read tokens
846,650
Tool calls
66

Tool call mix

command_execution
61
file_change
5
Run 3 2026-09-04 · standard · high $2.78 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.30 1m 01s 15,106 1,618 2
reviewPlan succeeded $0.23 1m 22s 8,671 2,089 1
execute succeeded $0.99 4m 11s 40,690 7,425 20
reviewAndCorrect succeeded $0.71 1m 41s 33,329 2,442 33
openPullRequest succeeded $0.55 1m 27s 20,491 1,754 10
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 22s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.78
Agent time
9m 43s
Input tokens
118,287
Output tokens
15,328
Cache-read tokens
834,490
Tool calls
66

Tool call mix

command_execution
58
file_change
8
Run 2 2026-09-04 · standard · high $2.76 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.30 1m 01s 15,081 1,600 2
reviewPlan succeeded $0.23 1m 22s 8,518 2,031 1
execute succeeded $0.99 4m 16s 39,457 7,590 20
reviewAndCorrect succeeded $0.69 1m 42s 32,164 2,428 33
openPullRequest succeeded $0.55 1m 28s 20,369 1,734 10
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 20s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.76
Agent time
9m 49s
Input tokens
115,589
Output tokens
15,383
Cache-read tokens
832,133
Tool calls
66

Tool call mix

command_execution
59
file_change
7
Run 1 2026-09-04 · standard · high $2.79 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.29 1m 03s 14,589 1,573 2
reviewPlan succeeded $0.23 1m 21s 8,764 2,078 1
execute succeeded $1.00 4m 13s 39,492 7,723 20
reviewAndCorrect succeeded $0.71 1m 43s 33,737 2,405 33
openPullRequest succeeded $0.56 1m 26s 20,371 1,760 10
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 20s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.79
Agent time
9m 45s
Input tokens
116,953
Output tokens
15,539
Cache-read tokens
841,568
Tool calls
66

Tool call mix

command_execution
58
file_change
8