Test health
Flaky tests stop blocking you. Broken ones never hide.
CoderHelm reads every end to end run your CI already produces, tells a flaky test from a broken one, and acts on it.
| Test | Status | Last 10 runs |
|---|---|---|
| checkout › pays with saved card | flaky | ✓✓✕✓✓✓✕✓✓✓ |
| account › updates email | broken | ✕✕✕✕✕✕✕✕✓✕ |
| search › filters by size | quarantined | ✓✕✓✓✓✓✕✓✓✓ |
| cart › removes last item | passing | ✓✓✓✓✓✓✓✓✓✓ |
Fix PR #327 for the flaky checkout test: 20 of 20 green runs, ready for review
Flaky, broken or a bad run
Told apart
A test that passed and failed on the same commit is flaky. One that fails almost every time is broken. A run where many tests fail together is a bad run, not many flaky tests.
"Is this failure mine?"
Each failing test on a pull request gets one line in the review: known flaky with its history, a bad run, or likely caused by the files this pull request changed.
Quarantine, never for broken tests
When your team turns it on, flaky tests keep running and being recorded but stop failing the build. A broken test is never quarantined. Owners get a ticket after two days by default; each team picks between 1 and 30.
Quarantined failures don't block
When the only failing tests in a run are quarantined ones, CoderHelm treats the run as passing: it does not start a CI fix, armed auto merge goes ahead, and a note on the pull request names the tests.
Fix pull requests with proof
CoderHelm reproduces the flake, fixes it, and marks the pull request ready only after twenty green runs in a row.
Tests you are missing
Uncovered changes and bug fixes without tests become suggested tests, each proposed only after it passes ten repeat runs.
Every run on one page
Runs, each test's history, failures grouped by cause, the slowest tests and the quarantine list.
Fits your CI
Quarantined tests still run in your CI. The Tests page shows the workflow step that keeps them from failing the build itself.
Reports it reads
Playwright (JSON or HTML), JUnit XML and Cucumber JSON. Results that never reach GitHub, such as a device farm, can be uploaded with a GitHub Actions token, so there is no secret to store.
Related
- Maintenance
CoderHelm stays on the code after the pull request: it answers review comments, fixes failing CI on its pull requests, resolves merge conflicts, checks deploys after merge and writes release notes.
- Code review
Add a label to any GitHub pull request and CoderHelm reviews it with the context of the whole repository, comments on exact lines with suggested fixes, rates risk and reviews again on every push.
- Tickets to PRs
Assign a GitHub issue or Jira ticket to CoderHelm and it plans the change, writes it on its own branch, runs your build and tests, reviews its own work and opens a pull request.
Put CoderHelm on your next ticket.
CoderHelm is in open beta. Sign up and bring your own Anthropic API key.