TL;DR
To test a new AI coding model, pick 3 to 5 real tasks from your backlog, run the new model and your current one on the same tasks under the same conditions, and have the same reviewer judge both. Score time to a reviewable change, review comments, and rework rounds. Then decide: adopt, route some task types, or wait.
A new AI coding model ships with a launch post and a lot of opinions. Neither tells you whether it handles your repository and your review bar. On November 11, 2026, Alibaba’s Qwen team announced Qwen4-Max in preview and closed the post with a line worth taking literally: “Put it to the test on a real problem.”
This guide turns that line into a one-week team playbook, with Qwen4-Max as the worked example. If your team already runs coding agents on a shared board, Sharkly lets you put both models on the same tasks and compare the runs side by side.
Why launch claims don’t answer your question
A model trial is a short, controlled comparison of two models on work your team would have done anyway. Its output is a decision about your workflow, not a leaderboard score.
Public benchmarks such as SWE-bench measure models on fixed open-source issues. They help you spot a capable model. They can’t tell you how it behaves with your test fixtures or the reviewer who rejects any patch without a migration note.
Qwen4-Max makes the gap obvious. As of November 11, 2026, the Token Plan page linked from the announcement lists qwen4-max-preview as a supported model, marked as a limited-time discount. That’s a model name inside a subscription plan, not a pay-per-token listing: the page’s API Pricing tab didn’t include Qwen4-Max that day, and the QwenCloud model marketplace had no model card for it. A third-party report from AlphaSignal notes a qwen4-max identifier in a Qwen Code client change, but it describes that as client configuration, not a confirmed server spec.
Step 1: Pick 3 to 5 real tasks from your backlog
Choose tasks your team would assign this week, and skip anything already solved in a public commit the model may have seen. A good trial set:
- One bug fix with a clear reproduction and an existing failing test.
- One small feature that touches two or three files.
- One refactor where public behavior must not change.
- One investigation task, where the right answer may be a question rather than a patch.
- Optionally, one task your current model handled badly last month.
Write the acceptance criteria before anyone runs anything. For a pagination fix, name the edge cases: empty results, the last page, an invalid cursor. A reviewer who decides what “done” means after reading the patch will grade on taste.
Five tasks won’t give you statistical certainty, but they show within a week whether a model deserves a bigger trial. For the heavier version with 12 tasks, repeated attempts, and blind review, see our guide to evaluating a model on your own codebase.
Step 2: Freeze everything except the model
The comparison only means something if the model is the one variable. Hold these constant across both runs:
- the starting commit and branch;
- the task description and acceptance criteria, word for word;
- the coding agent harness (the CLI that gives the model tools, files, and a shell);
- permissions, network access, and time limit;
- the reviewer.
Give each run its own working copy. Git’s worktree feature lets several checkouts share one repository, so the second run never starts from the first run’s edits or cache.
Set one intervention rule up front, such as one clarifying answer per run. If someone quietly fixes a patch before review, you’re no longer testing the model.
Run both models as Agents on the same tasks
This is where a shared board earns its keep. In Sharkly, an Agent is an AI teammate that runs on a Computer through a Runtime, such as Claude Code or Codex. Each Agent has a default model, picked from the catalogue its Runtime reports.
You have two clean ways to set up the comparison:
- Per-run model choice. When you assign a Task, hover the Agent in the Assignee picker and choose a model for that run. It overrides the Agent’s default for this task only, as described in the Agent task execution docs.
- A variant per model. An Agent variant inherits the original Agent’s instructions and Skills live, and can override the Runtime and model. Create one variant for the challenger model and keep everything else inherited, so the prompt stays identical.
Use per-run choice when you’re testing a handful of tasks once. Use a variant when the trial runs for a week and several people will assign work to it.
Point both at the same tasks with the temporary-directory working directory mode, which gives each run its own isolated worktree. Every run leaves its transcript and patch on the Task, so the reviewer compares two outcomes on one record.
Model usage stays on your own accounts: runs bill against the subscription or API key attached to the Runtime. Sharkly doesn’t sell tokens.
What if the new model can’t run in your harness yet?
Qwen4-Max sits in between. The Token Plan exposes qwen4-max-preview to tools that speak the OpenAI and Anthropic protocols, and its supported tools list names Claude Code and Codex. It’s still a preview, and we haven’t run it through a Sharkly Runtime, so we’re not claiming Sharkly runs it.
For any model without a usable ID, a fair trial still works outside the harness:
- Keep the same Task as the source of truth.
- Run the challenger in the vendor’s own app, using the exact task text.
- Paste its patch and a short note back onto the Task as a comment.
- Review it with the same rubric as the Agent run.
The harness differs, so label the result “workflow comparison,” not “model comparison.” When a stable model ID ships and your Runtime accepts it, you can set it in Custom arguments on the Agent or Runtime, for example specify model: model-name, and rerun the set properly.
Step 3: Score results your reviewer already sees
Keep the scorecard small. Three numbers per run carry most of the signal:
| Metric | How to record it | What it tells you |
|---|---|---|
| Time to reviewable change | Run start to the first patch a reviewer will read | Speed, including stalls and retries |
| Review comments | Count of blocking comments on the first patch | How close the first attempt lands |
| Rework rounds | Extra runs or edits before acceptance | The real cost of a “fast” model |
| Accepted (yes/no) | Did the change merge as written? | The only outcome that ships |
Rework matters most: a model that answers in four minutes and needs three rounds of fixes costs more reviewer time than one that lands on the first pass. We break that math down in when cheap models cost more.
Read the patches, not only the test results. Our checklist for catching coding agents gaming tests applies to any model, new or old. Google’s code review guide is a solid reference for what a reviewer should check on each patch.
Step 4: Decide with one of three outcomes
End the trial with a written decision, not a vibe. Three outcomes cover almost every case:
- Adopt. The new model matched or beat the current one on most tasks with fewer rework rounds. Make it the default model on the Agent.
- Route some tasks. It won one task type, say refactors, and lost the rest. Keep both, and send that task type to a variant on the new model. Our post on matching models to agent roles covers how to write that routing rule down.
- Wait. Results were close, or the model is still in preview with only a preview ID and no pay-per-token pricing. Note the date, keep the task set, and rerun it when the model reaches general availability.
For a preview model like Qwen4-Max, “wait” is often the honest answer. You still keep a reusable task set, so the next trial takes an afternoon.
FAQ
How many tasks do you need to test a new AI coding model?
Three to five real tasks give you a directional answer within a week. If the result is close or the stakes are high, expand to around 12 tasks with repeated attempts before you switch defaults.
Is Qwen4-Max available through an API?
As of November 11, 2026, Alibaba’s Token Plan lists qwen4-max-preview as a supported model, so subscribers can call it from tools on the OpenAI or Anthropic protocols. The pay-as-you-go API Pricing tab didn’t list Qwen4-Max or a per-token price that day. Check both pages for current details.
How long should a model trial take?
Plan for one working week: a day to pick tasks and write acceptance criteria, two or three days of runs, and a day for review and the decision.
What should I measure besides speed?
Measure review comments on the first patch, rework rounds, and whether the change merged as written. Speed without acceptance is a misleading number.
How does Sharkly help with model trials?
Sharkly puts both runs on the same Task. You can pick a model per run in the Assignee picker or create an Agent variant with a different model, and each run’s transcript and patch stay on the Task for review.
Your next step
Pick three tasks from this week’s backlog tonight and write their acceptance criteria. Tomorrow, assign each one twice: once to your current Agent, once to a variant on the challenger model. Sharkly is free to start, and you bring your own coding agents. Set up your first Agent at sharkly.ai and run the trial on your real work.
