Evaluate Claude Fable 5.2 on representative repository tasks with fixed acceptance criteria, identical environments, and a baseline you already use. Measure accepted changes, review effort, elapsed time, and total cost together. Repeat runs, preserve failures, and decide adoption by task category rather than a single benchmark score.
Anthropic announced Fable 5.2 on November 9, 2026. Its announcement reports improvements in coding and reasoning. Those are vendor claims about its evaluations and experiences. They don’t establish how the model performs against your dependencies, conventions, and release requirements.
Your evaluation should answer a narrower question: which work can you assign with an acceptable review burden? If several coding agents already share that work, Sharkly gives you a place to organize the tasks and their review context.
This guide proposes an experiment. It contains no measured comparison results.
Build a representative task set
A repository evaluation is a controlled comparison of coding workflows against requirements taken from your own development work. The unit of success is an accepted change, including its tests and review. A plausible patch, passing test command, or confident completion message alone doesn’t establish acceptance.
Start with twelve tasks drawn from recent maintenance work. That number is a suggested pilot size, not a statistical guarantee. Include bug fixes, small additions, refactors, and investigation tasks in proportions that resemble your backlog.
Choose work with a clear starting commit and a reviewer who understands the expected behavior. Avoid selecting only issues with obvious answers. Include at least one task where the right response requires asking a question or explaining a missing dependency.
For historical issues, remove the solution from the supplied context. Record whether the fix was already public: the model may have encountered it before. Keep a separate set of unpublished tasks for a later confirmation round.
Write the acceptance criteria before any run. For an API pagination fix, specify empty results, page boundaries, invalid cursors, and compatibility with existing clients. Include the permitted scope of changes and the checks required for acceptance.
| Category | Example task | Acceptance evidence |
|---|---|---|
| Bug fix | Correct cursor boundary handling | Regression test and existing suite pass |
| Small addition | Add optional request filtering | Contract tests and documented defaults |
| Refactor | Extract duplicated validation | Unchanged public behavior and review |
| Investigation | Diagnose intermittent timeout | Reproduction and supported causal explanation |
Use a small task when you need a clean comparison. Use a larger task when coordination is part of the question, and score that experiment separately.
Freeze the conditions before running agents
Record the repository commit, operating system, dependency lockfiles, tool versions, model identifier, and configuration. Pin the agent harness too: the harness is the software that supplies tools, context, and execution behavior around the model.
Compare Fable 5.2 with your current workflow. A model comparison needs the same harness where supported. A workflow comparison can use different tools, but its conclusion must name those complete configurations.
Keep permissions, network access, starting context, and time limits equivalent. Define whether dependency installation counts toward elapsed time. Don’t let one configuration inherit another run’s edited files, caches containing answers, or conversation history.
Use a fresh checkout or isolated worktree per attempt. Git’s worktree documentation explains how multiple working trees share a repository. Isolation of files doesn’t isolate databases, ports, or external services; give those separate test resources too.
Set an explicit intervention rule. For example, allow one clarification response containing only information available to every configuration. Record all intervention time. A reviewer quietly repairing an implementation changes what the trial measures.
Run paired attempts and retain failures
Run each configuration against the same tasks. Alternate execution order so a temporary service slowdown doesn’t consistently favor one candidate. Repeat attempts when resources permit, using a fixed repetition count chosen before viewing the results.
A practical pilot could use three attempts per configuration per task. Treat that as an engineering starting point. A small sample can’t support precise claims about rare failures or universal superiority.
Store the prompt, configuration, timestamps, transcript, patch, tool output, and final disposition for every attempt. Record timeouts and abandoned runs as outcomes. Excluding them makes the successful runs look cheaper and faster than the workflow you would deploy.
Keep test fixtures and credentials separate from production. Follow your repository’s existing testing process. For Python repositories, pytest’s documentation describes fixtures and test execution; the framework doesn’t decide whether your acceptance criteria cover the requirement.
Review correctness before speed
Have a reviewer inspect the patch without seeing the model label where practical. Use a fixed rubric covering requirement satisfaction, regression risk, maintainability, test quality, and unnecessary scope.
Passing tests is necessary when the task requires them, but tests can encode the wrong behavior. Check assertions against the original requirement. Watch for disabled checks, widened tolerances, swallowed errors, and changes that bypass the failing path.
Separate three dispositions: accepted unchanged, accepted after repair, and rejected. Record repair minutes and the nature of each repair. Decide in advance whether repaired changes count toward your primary acceptance rate.
For investigation tasks, score the evidence chain. A useful report identifies observations, distinguishes hypotheses, and gives a reproducible next step. Don’t reward a confident diagnosis without evidence.
Calculate cost per accepted change
Cost per accepted change divides the total evaluation cost by the number of changes accepted under your predefined rule. Include failed attempts in the numerator. If no changes are accepted, report the cost and zero acceptances; the ratio has no finite value.
Track these measures together:
- Acceptance rate by task category.
- Wall-clock time from task start to review-ready output.
- Human review and repair minutes.
- Model and tool charges for all attempts.
- Infrastructure costs where material.
- Regressions discovered during the observation window.
Record token usage and the dated price source alongside billed amounts. Cached tokens, retries, and execution tools can change costs. Don’t estimate the bill from output length alone.
If you assign a monetary rate to reviewer time, label it as an internal assumption. Show cash spend and labor minutes separately so readers can recompute the comparison. A cheaper generation can still require more engineering attention.
Keep evaluation evidence with the work
Sharkly manages the work around coding agents. Its documented workflow connects Tasks, Agents, Computers, execution context, and human review. Check the Sharkly documentation for the current product vocabulary and setup.
For this experiment, put the acceptance criteria in each Task and return the patch, test evidence, and outcome to the same work record. Keep the evaluation ledger in a spreadsheet or versioned data file if that’s easiest to analyze. Don’t assume the platform calculates your experimental metrics automatically.
Use a single Agent for a bounded trial. Introduce a Crew only when coordination itself belongs in the evaluation. Changing team structure midway introduces another variable.
Agents execute and report evidence. People define acceptance, inspect changes, and authorize release. That division remains useful even when the code passes every automated check.
Decide adoption by category
Write the adoption rule before opening the results. Require acceptable correctness first, then compare cost and elapsed time among configurations meeting that bar. Keep security-sensitive or poorly covered categories outside the initial rollout.
Report counts alongside percentages. A result of two accepted tasks out of three tells a different story from two hundred out of three hundred. Describe uncertainty and failure patterns instead of declaring a universal winner.
Repeat the evaluation after meaningful changes to prompts, harnesses, model versions, or repository structure. Preserve the old configuration so a later improvement has a reproducible baseline.
Start with one representative maintenance task in Sharkly, attach explicit acceptance criteria, and compare the returned evidence with your current workflow. Expand after you can explain both the successes and the failures.
FAQ
Can a public coding benchmark replace this evaluation?
It can help you select candidates. It doesn’t cover your repository’s conventions, dependencies, review expectations, or operating constraints.
Should every attempt use identical prompts?
Use equivalent task information and a predefined prompting policy. Record tool-specific wrappers so differences remain inspectable.
Do repaired patches count as successes?
Choose the rule beforehand. Report repaired and unchanged acceptances separately, including repair time in the workflow comparison.
How should timeouts be scored?
Retain them as failures under the chosen time budget. Include their spend and elapsed time in the results.
When should the team expand the rollout?
Expand when a task category meets your acceptance rule across repeated trials and reviewers understand its recurring failure modes.
Additional resources
Read the Git documentation for repository isolation, pytest fixtures guide for controlled test setup, and Sharkly for organizing agent work and human review.
