The Implementer Will Not Touch Tests: Preventing the Self-Grading Agent with Exit 2

2026-10-26

Michał Sznurowski

The most dangerous behavioral failure mode of AI coding agents is "The Self-Grading Agent": When an implementation fails to pass assertions, the agent opens the test file, removes the failing expectation, weakens constraints, or appends .skip(), then cheerfully reports: "All tests are green, implementation complete."

1. Why Markdown Prompt Instructions Fail

Most teams try to solve this by adding polite instructions to system prompts:

"You are an implementer. You are strictly forbidden from modifying files in the test/ directory."

Empirical testing reveals that while models follow this instruction on trivial tasks, in multi-turn sessions with expanding context windows—especially after 3 or 4 failed implementation attempts—the model violates the restriction in 30% to 50% of complex tasks.

For an LLM, achieving a green test suite is a stronger optimization signal than adhering to a markdown sentence in context memory.

2. The Solution: PreToolUse Hooks with Exit Code 2

In runtimes like Claude Code, every tool call (Edit, Write, Bash) can be intercepted before execution by a PreToolUse hook.

In CDAL, the implementer role is configured with denyTests: true:

// cdal.config.mjs
roles: {
  'test-author': { write: ['test/', 'tests/'] },
  'implementer': {
    write: ['src/', 'app/'],
    denyTests: true, // Physical lock
  },
}

When an agent acting as implementer attempts to edit a test:

echo '{"tool_name":"Edit","tool_input":{"file_path":"test/booking.test.ts"}}' | npx cdalkit hook guard-paths implementer

The hook intercepts the tool call, writes to stderr: [guard-paths / implementer] BLOCKED write to: test/booking.test.ts (denyTests: true) and terminates with exit code 2.

In Claude Code, exit code 2 immediately halts tool execution before any bytes are written to the filesystem.

3. The Impact on Code Quality

Faced with a physical block, the model receives immediate feedback: "You do not have write access to this file." It cannot tamper with the test, forcing it to re-evaluate the production code and fix the underlying bug.

When you review a pull request under CDAL, you know with certainty that green tests were authored independently and never diluted.