Benchmark AI_TRUECOST_

real AI cost, not vanity metrics

Follow on X updates & details

deepseek-v4.1-flash

Running deepseek-v4.1-flash inside devin.

FastAPI: Add POST /api/v1/projects/{project_id}/tasks endpoint Plan → Review plan → Execute → Review & fix → Open PR → Verify PR → Verify e2e

All steps run as one workflow, inside a single Docker container.

Swipe to view more

5 of the 10 steps are agent sessions, each started fresh: only the repository working tree and the previous step's output carry over. Host steps orchestrate and verify, and cost nothing.
# Step Kind What happens
1 pullIssue host Fetches the GitHub issue and creates a branch
2 plan agent Drafts a plan from the ticket, including tests
3 reviewPlan agent Always runs; returns the full final plan
4 execute agent Implements the landing page; may run repeated preview captures; no harness step timeout
5 reviewAndCorrect agent Visual review against the brief and reference render; fixes layout and copy; no harness step timeout
6 openPullRequest agent Commits, pushes, opens a pull request into main
7 verifyPullRequest host Checks that the exact pull request is open
8 verifyPullRequestCheck host Waits for the required end-to-end check to pass
9 summarizeUsage host Totals tokens and cost
10 score host Records the final result
Speed
Reasoning
Run 1 2026-09-14 · standard · high $0.08 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.01 0m 49s 47,694 5,551 31
reviewPlan succeeded $0.01 0m 37s 16,973 6,069 22
execute succeeded $0.03 1m 27s 95,822 6,589 33
reviewAndCorrect succeeded $0.02 0m 47s 65,840 5,392 23
openPullRequest succeeded $0.01 0m 37s 28,028 2,413 16
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 21s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.08
Agent time
4m 16s
Input tokens
254,357
Output tokens
26,014
Cache-read tokens
1,091,608
Tool calls
125

Tool call mix

read
77
exec
36
edit
7
grep
2
todo_write
2
find_file_by_name
1