Design decisions
Departures from the spec’s SHOULD defaults, and choices the spec leaves open. Newest last.
M0: repository skeleton
Section titled “M0: repository skeleton”- Version in its own file.
Harness::VERSIONlives inlib/harness/version.rb(required bylib/harness.rb) so the gemspec can load it without loading the whole library. - Minimal
Harness::Execalready in M0.bin/checkmust runruby -wcand the tests as subprocesses, and the only allowed way to do that isHarness::Exec. M0 ships argv-only running, a scrubbed environment, a timeout (process-group kill on POSIX,taskkill /T /Fon Windows) and an output cap. WindowsPATHEXT/.cmdresolution and the full test set for timeouts and caps follow in M1. [cmd, argv0]form.ExeccallsOpen3.popen3(env, [argv[0], argv[0]], *rest, ...). With a plain one-element argv, Ruby may hand the string to a shell; the pair form never does.- Reader threads, not
IO.select.IO.selectdoes not work on pipes on Windows, so stdout and stderr are drained by one thread each. - I1 matches at a word start. The denylist is matched case-insensitively, but not after a letter or digit. Otherwise the provider name “ollama” (section 11.2) would count as a model family.
- I3 scan scope and method. The scan covers
lib/,exe/andbin/. Comments and simple string literals are blanked first, so a chat role named “system” is not a hit. The pipe forms ofopen/IO.readare matched on the raw line. - I13 secret patterns need a body.
sk-must not follow a letter or digit and needs 16+ key characters, sotask-1ordisk-usagedo not match;ghp_,github_pat_andAKIAlikewise need their key body. The file set is what git would commit (git ls-files --cached --others --exclude-standard), with a directory walk as a fallback when git is missing. - Scanners live in
test/support/guard_scan.rb. The denylist and shell patterns must not appear inlib/. Every scanner has a positive self-test, so an empty result means “clean”, not “scanner broken”. - No
stringioin tests. It is not on the I14 allowlist, so tests use a smallTestSupport::Sink. - CLI.
harness --versionand--helpwork; every subcommand reports “not implemented yet” with exit code 2 until its milestone lands. - Spec and agent notes stay outside the repository. The design spec and the agent working rules live one level above this checkout and are not committed.
M1: config, Exec, model layer
Section titled “M1: config, Exec, model layer”- Batch files run directly, not through
cmd.exe /d /s /c. Checked on Windows: Ruby quotes each argument, so a resolved.cmdpath with spaces (e.g. a Node install under Program Files) breaks undercmd /s /c. Passing the resolved.cmdpath straight toOpen3.popen3lets Ruby wrap it incmd.exewith the right quoting. The safety rule stays: arguments to a.cmd/.batthat contain& | < > ^ % ", CR or LF are refused (UnsafeArgumentError).Exec.resolvefollowsPATHandPATHEXTand never picks an extensionless shim. - A timed-out process reports
exit_code: nilon every OS.taskkill /Fleaves exit status 1 on Windows. - Pure-Ruby SHA-256 fallback. Some Windows application-control policies block
digest.so(andopenssl.so).Harness::Sha256usesDigestwhen it loads, else a FIPS 180-4 implementation checked against the standard vectors. Without OpenSSL,httpsendpoints are unavailable on such machines; localhttpworks. - Probe order. Thinking control is probed before structured output, tools and throughput, so those requests
run with thinking off and stay short. The JSON-schema check asks a question whose natural answer is prose,
so a server that ignores
response_formatis not mistaken for one that enforces it. A thinking control counts only when it cuts the answer length by at least 20%. - Thinking-off request fields (one per control, in
Wire::THINKING_OFF):chat_template_kwargs.enable_thinking = false,reasoning_effort = "none",thinking_budget_tokens = 0. They are generic request parameters; the probe decides which one a server honours. - The schema goes into the system prompt in
json_objectmode too, not only inpromptmode:json_objectguarantees JSON syntax but not the shape. Completionhas one extra field,tool_calls, which the review tool loop (M6) needs.- Usage is summed over every request a call made (first try, shrink, retry). Records hold one file per HTTP
request with its outcome (
ok, amodel.*reason,busy:429,shrink:N,overflow,unavailable). - “At most N” shrinks once to N whenever N is below the current
max_tokens, as the spec says; if the server still refuses, it is a context overflow. complete_eachbatches ask for{"items": [...]}with exactly one entry per item and split into single calls onModelOutputErrororContextOverflowError. Each item’sCompletionshares the batch’s text and usage.- Config is strict: unknown keys are errors (keys starting with
_are comments). Env overrides:HARNESS_<PROFILE>_BASE_URLandHARNESS_<PROFILE>_MODEL, with the profile name upper-cased and other characters turned into_. - Token estimate: 3 bytes per token plus 4 per message, deliberately pessimistic.
M2: worktree, paths, edits, gates, baseline
Section titled “M2: worktree, paths, edits, gates, baseline”.gitis always protected, whateverprotectedPathssays. Glob matching addsFile::FNM_SYSCASE, so on Windows.GIT/configand.Envare caught too, and a pattern ending in/**covers every depth below it. Path segments containing:are refused (drive letters, NTFS alternate data streams).- The I2 check runs twice per write: when an edit is staged and again right before the file is written.
- Edits apply all or nothing. If any edit is rejected, nothing is written and every rejection is returned,
so the caller can give the model one retry for the rejected edits. If a write fails midway, files already
written are restored. A whole-file rewrite needs the file to be under
wholeFileRewriteMaxBytesand listed infiles.modify. Replace edits keep the file’s own line endings. - Reverting a task runs
git checkout -- .andgit clean -fd, then deletes the files the task created by name, becausegit clean -fdleaves ignored files alone. blocked.patchwithout staging:git diff HEADfor tracked files plusgit diff --no-index -- /dev/null <file>for each untracked file. The harness never stages until it commits, so a revert stays a plain checkout and clean.- Git allowlist (
Repo::Git::ALLOWED) matches whole command shapes, e.g.checkout -- .only andworktree add -b <branch> <dir> HEADonly, and refuses--exec,--output,--upload-pack,--receive-pack,--ext-diffand--textconvanywhere. - Gate env adds
COMSPECto the base list (Windows needs it to run.cmdfiles) and setsnpm_config_yes=false. WithCI=1and no TTY, npx would otherwise install a missing package without asking, which I10 forbids. Output saying the tool is missing (could not determine executable to run,is not recognized as an internal or external command, …) makes the gateskippedwith that reason. - Placeholders: an empty
{files}skips the gate (“no files to check”). An empty{tests}is dropped so the whole suite runs (baseline and integration). A path starting with-is passed as./-name. - The baseline runs every gate that does not need touched files (scope
projectandtests, both phases), not only project gates and unit tests, so integration gates have something to be compared with in M3. - Baseline comparison: diagnostics are matched as a multiset by (file, code, message), ignoring line
numbers. Unparsed output compares by its last 60 lines. Timeouts compare by status. A skipped gate is never
ok?(I6). - Parsers in M2:
tscandastro_check(both tsc formats), eslint “stylish”, and minimalvitestandplaywrightfailure lines. Anything else falls back to the last 60 lines.
M3: the per-task loop
Section titled “M3: the per-task loop”- Everything a model wrote is fenced when it goes back to a model, including the plan’s task text and the previous reply in an apply retry. The retry is one user message (“your previous reply” fenced, plus the fenced rejections), not an assistant turn, so I9 holds without exceptions.
- The untrusted fence tag is
untrusted-data(REVIEWER useduntrusted-pr-data). Injection patterns are ported unchanged. - Prompt templates carry the SPDX header as ERB comments, which render as nothing. The edit rules are shared between implement and fix through one partial template.
- Review is a placeholder until M6: the loop emits a
skippedevent with subjectreviewand moves on. - A task’s final outcome event uses subject
task(done attask.commit, blocked attask.commit, skipped attask.gather). The fold uses that subject to set task status, soskippedevents for other things (e.g. review) do not change a task. - A gate that is skipped during a task blocks it at once (
gate.skipped:<gate>): no fix can install a missing tool. A gate named in acceptance but not defined by the stack counts as skipped. - When a failing gate has no parsed new diagnostics (timeout, unparsed output), the fixer gets one synthetic diagnostic with the gate’s reason, and “the same errors twice” compares those too.
- A model outage stops the queue: the current task is blocked with
model.unavailableand reverted, the remaining tasks are skipped, integration does not run, and the exit code is 2, so the run can be resumed (M4). - Integration runs the
integrationgates and everytests-scope gate, without file arguments (the whole suite), and compares them with the baseline. A non-blocking gate never fails the run. - Gathering in M3 packs the task text, the full text of
modifyandcreatefiles (required; otherwiseTaskTooLargeError, recorded asplan.too_large_for_context), thenreadfiles and completed-task summaries if they fit. Map signatures and docs come with M5. Protected paths are never read into a prompt. harness runneeds--from-planuntil the planner lands (M4). Runs, logs and the probe cache live next to the config file.
M4: planner, plan checks, questions, resume
Section titled “M4: planner, plan checks, questions, resume”- New reason code
plan.unknown_gate: an acceptance item that names a gate the stack does not define. It would otherwise block every task as “skipped”, so the planner is told at once. - Outside-the-repo paths in a plan report as
plan.protected_pathwith the message “is outside the repository”; the spec lists no separate code for them. - A plan given with
--from-plan, or edited after--plan-only, goes through the same checks but is not replanned. Problems stop the run asplan_rejected(exit 1) with the messages printed. - After
maxReplanRoundsreplans the run stops asplan_rejected(exit 1), with the last plan saved and every problem printed, as “stop and show the user” asks. - The planner’s repository view until M5: the file list (with sizes, skipping dependency, build and binary files) plus the full text of files whose path contains a word from the request, packed into two thirds of the planner’s budget. The context map replaces this in M5.
- The request itself is not fenced; it is the user’s instruction. Everything else the planner sees (file list, files, its previous plan, check messages, and the question and answer pairs) is fenced.
- Questions: only blocking questions pause the run (status
waiting, exit 3).harness answer RUN_IDtakes one answer per blocking question, from--answers FILE(a JSON array) or typed on stdin. The answers go back to the planner, which plans again, and may ask again. - Resume rules per task state:
pending,gathering,implementingandapplyingrevert the task’s partial work and start over.applied,gating,fixing,reviewingandcommittingcontinue at the gates with the saved fix rounds and touched files. A task whose commit is already HEAD is done.doneandblockedkeep their outcome, exceptblockedbymodel.unavailable, which runs again. A task with no saved state starts from a clean worktree. - The lock lives at
runs/<runId>/harness.lock, not in the worktree, wheregit add -Awould commit it. One run is one worktree, so this is the same “one run per worktree” rule. - Run statuses in
run.json:planning,waiting,planned,plan_rejected,running,done,failed,stopped.
M5: context map and packing
Section titled “M5: context map and packing”- Files the map leaves out are the REVIEWER
skipReasonlists (binaries, lockfiles, build output, generated and minified files, lines over 1000 bytes), plus.git/,.cache/, files over 1 MB and everyprotectedPathsmatch. A protected file is never listed, described or summarised. - The map cache is
.cache/map/<sha256>.json, keyed by the file’s language and content hash, so a rescan reads every file but parses only changed ones. Summaries are cached next to it by content hash. - Import resolution: relative specifiers, the
~/and@/aliases (both read assrc/), a.jsor.jsxspecifier that names a.tsor.tsxfile, andindexfiles. Package imports andtsconfigpaths beyond those two aliases are not resolved. - A signature is the file’s path, language and size, its route,
getStaticPaths, resolved imports, export names, content collections, thePropstype text, and each export line as written. Ruby and Python files list theirclass,moduleanddeflines. - Content collections are found where
defineCollectionis assigned (const blog = defineCollection(...)). The file holding that call is where the schema lives. - Packing is strict: sections go in priority order, and once one does not fit, it and everything after it
are omitted, so the lowest priorities are always trimmed first. The one fallback is a
files.readfile that does not fit in full: its signature is tried in the same place. - Diagnostics and the diff in a fix round are counted before every optional section and never trimmed. The
loop already caps diagnostics at
maxDiagnostics, and a fix round without them is useless. - Signatures and summaries cover direct imports and importers of the task’s files (
related, depth 1), nearest first. At most 6 summaries are made per bundle. The planner overview uses depth 2 and at most 8. - Summaries come from the summarizer role and are made only when a bundle has room for them. A model failure, or a reply that reads like an instruction (I9), means no summary. Large files are cut to the summarizer’s context first.
- Docs passages (priority 5) are not packed yet. They need the REVIEWER docs retrieval, which comes with the review port in M6.
- A task that does not fit at run time is still blocked (
plan.too_large_for_context), not sent back to the planner. Plan checks already catch this before the run starts, so the run-time case means the files grew during the run.
M6: review port and the section 8 changes
Section titled “M6: review port and the section 8 changes”- Evidence is the local docs checkout only. Version detection, the BM25 index, the checkout index and the
retriever are ported. REVIEWER’s online docs, network guard and dataset are not: online retrieval needs https,
and neither is needed for the gate to stay honest, since missing docs make framework claims abstain, never
accept. The stack’s
docs.localpoints at a checkout;docs.siteUrl(optional) builds links for the report. - Framework claims are recognised with REVIEWER’s Astro markers when the stack’s framework is Astro. For other
frameworks only a verdict of
claim_type: framework-apinaming the framework counts. - Fetch tools (
read_file, at most 120 lines a call, andfind_symbol, at most 3 files) go throughRepo::Paths(I2) and come back fenced (I9). Everything they return is added to the code the gate checks quotes against.lookup_docsis offered only when docs are configured. Unknown tools get an error text. - Without native tool calls the model replies
{"lookup": {"tool", "args"}}or{"done": true}. - The verifier’s schema adds
depends_on_unseen,unseenandrequires_type_violation. An accept withdepends_on_unseenabstains (verify.unseen_code).requires_type_violationrejects (verify.requires_type_violation) when every type-checking gate (parsertscorastro_check) passed for the task; with no such gate it has no effect. - The challenge has the same fetch tools and a required
unseenfield. A non-emptyunseenabstains (verify.unseen_code) whatever the rest says; only a concrete concession survives. - Confirm by test: the fixer writes one test, named
harness-confirm-<task>-<claim>next to the project’s first test file (with its suffix), or undertests/with.test.ts. It is run with the stack’s task-phasescope: testsgate and compared with the baseline, then deleted. A regression confirms; a pass isconfirm.test_passes; no test gate, no test, or a test that could not run isconfirm.test_not_writable. - Follow-up fix tasks are
<task>R<n>, depend on the reviewed task, modify the claim’s file, and accept on the confirming test plus the parent’s gate items. They run right after their parent, are appended toplan.json(resume knows them), and get their test fromtasks/<id>/confirm-test.json. A follow-up’s own review makes no further tasks. - Review results are saved as
tasks/<id>/review.jsonandfinal_review.json(findings, questions, abstained, rejected, suspicious, failures) for the report in M7. A model outage during review is recorded and the commit goes ahead; the next model call stops the run as before. - The final review runs once over
base..HEADafter integration, with confirmation by test but no follow-up tasks. Diffs for review carry 10 lines of context. - Limits not in DESIGN.md: at most 6 diff chunks of up to 1500 lines and 60% of the reviewer’s budget, one re-evaluation round, verification batches of 3 that split to single claims on failure, 2 passages per support check. Discovery gets docs for at most 4 framework APIs, 700 characters each.
- Task titles are not in review prompts: they are planner output, so the subject is just the task id (I9).
Hardening found in review
Section titled “Hardening found in review”- Git never runs hooks. Every git command runs with
-c core.hooksPath=<.cache>/no-hooks(an empty directory, checked before each call), and commits also pass--no-verify. The allowlist only acceptscommit --no-verify -m. Hooks are repository code (or a model’s edit to.husky/) that would run outside the gate allowlist (I3). .huskyis always protected, like.git, and is in the example stack’sprotectedPaths.- Windows path aliases are refused: a path segment ending in a dot or a space (
.git.,.git) and one that looks like a short name (GIT~1) are refused before any read or write. - Gates cannot install or touch a remote: a gate whose program is
gitorgh, a package manager with an install-like subcommand (install,i,add,ci,update,upgrade,up,dlx), ornpx/pnpx/bunxwith--yesis a config error (I4, I10).
M7: report, init, status and eval
Section titled “M7: report, init, status and eval”- The report is built only from the run’s files (run.json, plan.json, task folders, review records, events,
model records) and rewritten at the end of every run and by
harness report RUN_ID. Text that came from a model is kept on one line, with table pipes and angle brackets escaped, so it cannot add headings, table cells or HTML. Integration results are saved asintegration.jsonfor it. - Questions in the report are the open and answered blocking questions, the planner’s non-blocking questions, and every review claim that went to confirmation but was not confirmed. Findings list only confirmed claims (I8).
harness statusprints the progress view of section 11.5 as text, from the event log only. With no RUN_ID it shows the newest run;--watchredraws every second until the run finishes, waits for answers, or stops after planning. “Final review” is a stage in the strip.harness initasks for the server URL, the name of the env var with an API key (never the key, I12), the model (listed fromGET /models, or typed), the context size (probeor a number) and an optional docs checkout. It validates the result withConfig.parsebefore writing. Not ported from the REVIEWER wizard: GitHub owner and repos (runs name their repository), cloning the docs (a download), the dataset, and the loopback-only URL rule (any OpenAI-compatible endpoint is allowed). Provider detection arrives with onboarding in M8.dependencyDirs(new stack key;["node_modules"]in the example astro stack): git-ignored folders that are mirrored from the repository into each worktree at intake, as hard links (copies where links fail). Without it no Astro gate can run in a worktree, because Node looks for packages only in parent folders. The user’s folder is only read, and nothing is installed (I10).- Eval tasks point at a prepared local repository through
repo.ref(a path or$VAR, then a commit orHEAD). Remote URLs are refused, since cloning is a download the harness does not do.eval/repos/astro-miniis the template; eval/README.md shows how to prepare it withnpm install. - Scoring: solved means the hidden tests pass with the stack’s task test gate (
scope: tests), run on just the hidden files after the run. “Findings confirmed” is confirmed claims / claims sent to confirmation. Regressions are integration gates whose verdict isregressed. A run that stops waiting for answers is scored unsolved. Worktrees are removed after scoring unless--keep. - Profile mappings override
rolesand may addprofileson top of the base config; with no map, one mapping namedconfiguses the config as it is. - Not verified yet: the ten hidden tests have not run against a real Astro install here (no Node packages were downloaded). They use the container API and may need small fixes on the first manual eval.
M8: onboarding and the app
Section titled “M8: onboarding and the app”- Fonts: InterVariable.woff2, InterVariable-Italic.woff2 and OFL.txt from the official Inter 4.1 release (github.com/rsms/inter). Nothing else is bundled; no CDN.
- Two guardrail tests changed, deliberately: the I3 scan skips binary files (the fonts), and still scans every
text file under
lib/,app.jsincluded (which therefore has no backticks). The I4 call-site test lists four places that start processes: the git wrapper, the gate runner, the fixed provider subcommands, and the first-start browser opener. A new I4 test pins the provider subcommand list. - Server: one thread per connection, one request per connection, headers up to 16 KB and bodies up to 1 MB. Refusals: 421 for a foreign Host, 403 for a missing token or foreign Origin, 400 for a POST without a JSON body, 405 for other methods. CSP, nosniff, no-referrer, DENY framing and same-origin policies on every response; no CORS headers.
- The token is 32 random bytes per start, in a meta tag of the page. GET requests need no token: they change nothing.
- Chat mode streams through
Model::Stream(server-sent chunks), with reasoning fields and inline<think>tags split into a Thinking block. Context fitting keeps the newest messages that fit in the probed context minus the output reserve, the system prompt and 600 tokens for the summary; older messages are summarised once by the chat’s own model and the summary is cached in the chat. - Build mode runs the pipeline in a thread of the app process (no subprocess). The run id is chosen before
intake so the card appears at once. Events stream by SSE from
after=<seq>; the page drops events it has seen, so reconnects never duplicate. Answers are one per line in the text box. - The progress view folds events in the browser the same way
Events.folddoes; reason texts come fromReasons::TEXTin each event (reasonText). - Providers: LM Studio and Ollama are probed in parallel with 2 s timeouts. CLI actions:
lms server start,ls --json,load <key>,ps;ollama list,pull <name>,ps; model names match/\A[\w.:\/@-]{1,200}\z/. The first provider set up takes every role; later ones are added as profiles.harness initruns the same detection and offers what it found. - Mirrored dependencies are protected (I11). Every
dependencyDirsfolder is added to the stack’s protected paths, andnode_modulesis protected in every stack, so no edit, test or tool result can target them. Every file the harness writes in a worktree goes throughRepo::AtomicFile(a temporary file renamed over the target), which replaces a hard link instead of writing through it. Remaining risk: a project tool run by a gate could rewrite a mirrored file in place; the gates are the user’s own commands.
Shakedown with a real model
Section titled “Shakedown with a real model”A first run with a local model on a real Astro 7 project found three bugs, each fixed with a test:
- Inspection-only tasks (empty
files.createandfiles.modify) are rejected by the plan checks (plan.no_changes), and the planner prompt says to put the reading into the task that writes. - A test the task writes (
{"kind": "test", "write": true}) is added tofiles.createwhen the plan does not list it underfiles.modify, so the checks, the file count and the edit rules all see it. The edit rules name the exact test path to use. - Unparsed gate output (no parser, only the output tail) is compared with volatile text normalised: ANSI
colours, ISO dates, clock times, durations, sizes, run ids, worktree and temp paths, and content hashes. Two
unparsed failures match when their exit status and normalised tails are equal; a changed count such as “3 tests
failed” still regresses. An
astro buildfailing identically at baseline and integration is no longer “regressed”. - The eval template used
/// <reference types="vitest" />, which Astro 7’s types reject; it now usesvitest/config, andastro checkis clean on it. Hidden test 01 passes against a reference solution on Astro 7.4.
App colours
Section titled “App colours”After the first real use, the colours were changed so they complement each other. The dark purples of DESIGN.md
11.4 are replaced by neutral blacks (--bg #0b0b0d, --surface #16161a, --border #2b2b33), the light purple is much lighter (--accent-soft #ddcbff), and corners are 4px (cards, inputs) and 3px
(buttons). The brand purple --accent #5809e0 stays the one strong colour.
Checks run locally, not in hosted CI
Section titled “Checks run locally, not in hosted CI”The repository has no GitHub Actions workflows, and pull requests should not add one. The check is
ruby bin/check: syntax and warnings, the guardrail scans and every test. It needs no model server, network or
GPU, and runs in a few minutes on a laptop.
- Contributors run
ruby bin/checkbefore opening a pull request, and paste its last three lines (syntax,guardrails,tests) with their OS and Ruby version into the pull request description. - Maintainers run it again locally on the pull request’s branch before merging, on Windows and Linux with Ruby 3.3 and 3.4 where they can, so every change is checked on more than the author’s machine.
- Model-backed checks (evals, live runs against a model server) stay manual, as before.
This departs from DESIGN.md 16, which describes a CI workflow.
Projects and workspaces in the app
Section titled “Projects and workspaces in the app”- Workspaces: every run’s worktree is
<home>/workspaces/<runId>(home is the folder ofconfig.json), never next to or inside the user’s repository.workspaces/andprojects/are git-ignored. - The project picker replaces the free-text repository path in Build mode. Projects are folders under
<home>/projects/, plus folders linked once by pasting their path; linked folders are stored by name inconfig.json("projects": [{"name", "path"}]). Labels are names; paths are shown short (projects/<name>,~/..., middle cut when long). Under the picker a line shows the short path, git, branch, clean or dirty, and the detected stack and version, or a precise error. - No folder browser: the server never lists directories. A pasted path is validated (absolute, a folder, the top of a git repository, not part of the harness itself) and described, nothing more.
- Downloads need confirmation: “Clone from a git URL” (a shallow https clone into
projects/<name>) and “New Astro project” (npm create astro@latest <name> -- --template minimal --install --no-git --yes, thengit initand a first commit) first answer 409 with the exact command; they run only when the page sends"confirm": trueafter the user accepted a dialog showing it. The git allowlist gains exactly these two shapes,clone --depth 1 -- <https URL> projects/<name>andinit -q -b main, and the I4 tests pin them. The create command is a fifth process site, also pinned. - Chat mode is a planning conversation: the system prompt asks the model to understand the goal, discuss approaches and trade-offs, sketch small steps, and suggest a request for Build mode; it never claims to change files.
- Layout: the main column is
minmax(0, 1fr)and the top bar and composer wrap, so nothing runs past the right edge at narrow widths.
Instruction-like text in task context
Section titled “Instruction-like text in task context”Before each implement or fix call, every section of the context bundle (files, signatures, summaries) is scanned
with the same patterns the review uses. A match is recorded as a task.gather event with reason
untrusted.instruction_like (once per task and source), named to the model in a fenced warning that says it is
data, and listed in the report’s “Suspicious text” section together with what the review flagged. It never blocks
a task: a pattern match catches common phrasings, misses reworded ones and can fire on harmless text (a security
doc, a test fixture), so blocking on it would be both unreliable and easy to trip. The protection against planted
instructions remains structural: the nonce fence (I9), edits limited to declared files (I2), and gates deciding
done (I7).
Discarded replies are rejections
Section titled “Discarded replies are rejections”A live run showed a gap: the planner’s first reply (thinking on) ran out of output room, the server returned 502,
and the client retried with thinking off. The run succeeded, but its events showed 0 rejections; only the request
records showed the lost reply. Now the client calls on_discard for every reply it throws away and retries, and
Run.observe turns that into a rejected event under the role’s stage with the reason code (for example
model.truncated), so the report counts it. Records also keep the HTTP status and the server’s error text
(redacted with the log’s token patterns, first 300 characters), and log lines carry the run id and the task id.
The app’s report view renders Markdown tables; before, they showed as raw | text.
Chat retries a reply that only thought
Section titled “Chat retries a reply that only thought”A chat reply can spend its whole output allowance on thinking and end with finish_reason=length and no text. The
app showed an empty message. Now, when a reply ends that way and the profile’s probe found a thinking control, the
chat sends a notice event (“asking again with thinking off”) and asks once more with thinking off. A message
saved after a retry is marked retried, and the app shows a short note on it. If the second reply is also empty,
the chat sends an error that names the output allowance and saves nothing, so an empty message never enters the
history.
Plan checks: files from dependencies, gates that cannot run
Section titled “Plan checks: files from dependencies, gates that cannot run”Two plan-check gaps showed up in a live Astro run:
- Files a dependency creates. A task that read or modified a file created by an earlier task was refused as
plan.missing_path, so a correct plan (“T1 creates global.css, T2 imports it”) could never pass. A task may now read or modify any file that a task in its transitivedependsOncreates. The planner prompt says so, and the refusal names the fix: add the creating task todependsOn. - Gates the baseline skipped. The planner was offered every configured gate, including one the baseline could
not run (a test runner that was not installed). It chose that gate, the gate was “unavailable” for the task, the
task blocked, and every dependent task was skipped. The planner prompt now lists only gates the baseline could
run.
plan.gate_unavailablerefuses a plan that names a skipped gate, or that addstestitems when every gate with scopetestswas skipped. The refusal names the reason and the gates that can run.
A crashed run is marked failed
Section titled “A crashed run is marked failed”The same live run hit a Ruby error in the fix loop: astro check reported one diagnostic with a code and three
without one, and sorting [file, code, message] keys compared nil with a string. The fix compares the fields as
strings. The crash also showed a second gap: the app’s build thread kept the error in memory, but run.json stayed
running forever, so the build card spun and nothing said why. Now a build thread that raises appends a
harness.crashed event with the error text and marks the run failed (exit code 1). harness resume can continue
it from the last committed task.
A gate missing at intake does not fail integration
Section titled “A gate missing at intake does not fail integration”Integration reruns the test gates. If a test runner is not installed, that gate is skipped at intake and again at
integration, and the run used to end failed even when every task was done. A gate that was skipped at both points
says nothing about the run’s changes, so integration now records it as skipped (“could not run at baseline
either”), not as a rejection, and it does not fail the run. The report still lists it in the gates table. A gate
that ran at intake and is skipped at integration still fails the run.
Implementers see the request; fixers see where a missing file is
Section titled “Implementers see the request; fixers see where a missing file is”The same run’s code compiled, but the theme was wrong: the planner shortened the stylesheet task to “the required CSS custom properties”, and the implementer, which saw only the task, invented every colour. Gates prove code compiles, not that it matches the request. Three changes, none tied to a model:
- The request goes to the implementer and the fixer. Fenced as data after the task, with “take exact values, names and text from it; build only this task’s part”. It is reserved in the context budget like the diff, and cut at 6,000 characters so a long request cannot crowd out the files.
- The planner keeps exact values. Its prompt says to copy colours, sizes, wording and names into the intent of the task that needs them, and that a task using names another task defines (CSS variables, a component, a type) depends on that task and reads its file.
- A module that cannot be found comes with the files of that name. T8 spent three fix rounds guessing
../components/Layout.astrofor a file atsrc/layouts/Layout.astro. When a diagnostic reads “Cannot find module ‘x’” (or a bundler’s “Could not resolve” / “Failed to resolve import”), the fixer is shown up to five repository files with the same base name.
A role that keeps running out of room stops thinking
Section titled “A role that keeps running out of room stops thinking”In the same runs, a verifier reply with thinking on often ran for about three minutes, used its whole output
allowance on thinking, and then answered in seconds when asked again with thinking off. Run 2 had 17 such replies.
The client now counts, per role, the replies that ran out of room (model.thinking_exhausted or
model.truncated) while thinking was on. After two, that role asks with thinking off from the start for the rest
of the run, and the log says so once. Other roles keep thinking. The count lives in the client, which is made per
run, so the next run starts with thinking on again.
A retry sees the file as the accepted edits left it
Section titled “A retry sees the file as the accepted edits left it”In the cookbook’s layout run, a task sent three edits to one file. The first changed the file, the other two searched for text the first had already replaced, and both were refused. The retry showed the model the file as it was before any edit, while its retried edits were applied after the accepted one, so they failed again and the task was blocked. Now the edit rules say that edits to one file apply in order and must not overlap, and that a small file can be rewritten whole instead. When a retry is needed, it includes each file with refused edits as the accepted edits left it (fenced, up to 12,000 characters in all), and says to copy the search text from that.
Built by V and Bruno LLC: Astro sites & AI tooling. Work with us →