What Is Evalship? The Beginner's Guide to Finding the Flaws in Your LLM Evals

CodexFuse Team · 2026-11-06T01:46:40.002Z · 26 min read min read

What Is Evalship? The Beginner's Guide to Finding the Flaws in Your LLM Evals

Evalship is a paid service that checks whether your LLM evals are any good. If you searched "what is evalship", the short answer is this: it reviews your eval suites for flaws, watches every pull request for eval regressions, and tests prompt and model changes to find ones that work better or cost less. It does not write your evals, and it does not run them — it reads the results files your existing tools already produce and tells you where the measurement itself is broken.

The product launched publicly on 4 November 2026, when its founder posted it to Hacker News, and this guide was written two days later, on 6 November 2026. That makes Evalship brand new, and the honest starting point is that no independent review of it exists yet.

Everything about the product below comes from its own site, its own docs, and its founder's launch post. If you download any file from a vendor before you run it, you can confirm it is intact with a SHA-256 hash generator.

Teams that run evals in CI usually run them on rented machines, and VPS hosting is where many small teams host that work. This guide explains what Evalship is, how its three jobs work, what it costs, and where you should not trust it yet.

What is Evalship, exactly?

Evalship is a hosted service that calls itself "quality control for your LLM evals". An LLM eval is a repeatable test that scores an AI feature's answers against fixed cases and agreed criteria, run on every change so regressions show up before users see them.

People who search "evalship llm evals" or "eval flaws detector" are asking the same question this guide answers: how do I know whether my evals are measuring the right thing? Evalship's pitch is that the evals themselves are often the broken part: a test case that cannot fail, a judge that grades its own model's answers, a scoring rule that lets a leaking reply pass. It finds those flaws, then keeps watching your pull requests so a product change cannot quietly break the evals that were supposed to catch it.

The service does three things. First, it reviews your evals: your coding agent reads every scorer, judge and prompt and proposes upgrades.

Second, it checks every pull request: your evals run in GitHub Actions, upload a results file, and Evalship compares the run with your main branch's history, commenting when something looks off. Third, it optimizes your prompts and models: your agent tries different prompt and model combinations against your evals, and Evalship says which ones win beyond chance. A weekly email every Monday tells the team what needs attention and how the week went.

One distinction matters from the start, and it is worth stating once: Evalship does not run your evals. It reads the results files your tools already write — from pytest, Vitest, Jest, promptfoo, Langfuse, Braintrust or LangSmith — and compares every pull request with your default branch, test case by test case. If you have no evals at all, Evalship has nothing to check.

Why do LLM evals fail quietly?

LLM evals fail quietly because a passing number can hide a broken test. A judge model can be down and still return a pass.

A scoring formula can weight a critical rule so lightly that a bad answer still clears the bar. A test case can be so easy that no model could ever fail it, so the suite stays green while the product gets worse. The founder's own research article, "How noisy are LLM evals in CI?", published 8 October 2026, measured the scale of the problem across five public repositories.

The headline finding is that passing evals failed on the next run 1 time in 40. Across the five repos, 68 of 2,690 passing evals failed on the following run, which is 2.5 percent.

The authors note this is an upper bound on chance alone, because code changes landed between the runs, so the true noise rate is at most this figure, not exactly it. One caution belongs here and is worth stating once: the 2.5 percent figure is Evalship's own measurement on five public repositories, not an independent benchmark, and the authors themselves call it an upper bound.

The same article looked at a large suite, onyx-dot-app/agent-wiki, which runs about 2,268 evals every night. Over 56 nights, a median of 73 evals that passed one night failed the next, and the count never hit zero.

On the 22 nights when no code changed at all, the median was 73.5. Across the whole period, 541 of the 2,268 evals — about 23.9 percent — both passed and failed at least once. The article's rule of thumb follows from the same arithmetic: at 2.5 to 3 percent noise, a 50-eval suite loses 1 to 2 evals between any two runs, and a 300-eval suite loses 8 to 9.

Three case studies in the article show what noise looks like in practice. In HolmesGPT/holmesgpt pull request 1452, one eval that had passed all 230 previous runs failed after a refactor removed the runbook catalog it depended on.

In PrefectHQ/prefect-mcp-server pull request 200, 14 of 20 evals failed on the first push — all with the same 404 error, because the judge model claude-opus-4-1-20250805 had been retired. In launchdarkly/ai-tooling pull request 189, a supposed regression had also failed 7 of the 12 prior nights on main, and a rerun passed. Each case is a different failure: a hidden dependency, a dead judge, and plain flakiness.

What are the three jobs Evalship does?

Evalship's three jobs cover the full life of an eval suite: review it once, watch it on every pull request, and keep improving the prompts and models it tests. The table below is the whole product at a glance, and each job gets its own section after it.

The split between free and paid is deliberate. The review and the optimization run on your own coding-agent subscription, so Evalship charges nothing for them.

The PR checking runs on Evalship's side and is what the paid plans meter. Public repositories are free on every plan, and the review and Optimize features are free for every repository, public or private, because your agent does that work.

How does the eval review work?

The eval review is the first thing a new user does, and it is the part of Evalship that finds the flaws in your LLM evals. Setup is one prompt pasted into a coding agent: "Set up https://evalship.com/setup.md".

The agent reads every scorer, judge and prompt in the repository, writes a review on your machine, and pushes it to a branch called evalship/review as a single file, evalship-review.json. Evalship then checks every citation in that review against the repository and displays it as the repo's report. If you are learning how to drive a coding agent through a setup like this, our guide on how to use Claude Code covers the same kind of agent workflow.

The first review takes 10 to 45 minutes, and the review itself is free because it runs on your agent's own subscription. The review covers the suites inventory — name, framework, path, test-case count, deterministic versus LLM-judge assertion counts, code exercised, models called, run command, CI triggers, results format, estimated cost per run, and the line where pass or fail is decided — plus coverage, prompts, flaws, a plan of up to six upgrade steps, and CI. Flaws are ranked most valuable first, up to 12 of them, with lower-stakes ones marked as warnings.

The review checks for eleven flaw kinds: evals missing from CI, untested prompts, untested instructions, eval bugs, evals that are too easy, judge flaws, judge-only scoring, thin datasets, cost problems, duplicated setup, and nondeterminism. Each flaw carries a severity of critical, important or warning, plus the evidence — file, line, exact quote and a note — why it matters, and a concrete action. Evalship adds one flaw of its own, an eval gap: every prompt or AI call that no suite exercises, ranked between critical and important.

Evalship verifies the citations before it shows the review. After a review is claimed, it fetches every cited file and checks that the file exists, the line is in range, and the quoted text appears within three lines of the cited line.

A flaw is visible only if all of its evidence holds; the rest are hidden with a count. The reviewed badge stays green for 90 days after the latest review. Fixed flaws are tracked per pull request, so the dashboard can mark a flaw as being fixed in one PR and fixed in another.

Running your first review takes five steps:

1. Paste the setup prompt into your coding agent and let it read your evals.
2. Wait for the agent to write the review file and push it to the evalship/review branch.
3. Claim the review through the link Evalship gives you, signing in with GitHub and installing its GitHub App if needed.
4. Read the flaws, most valuable first, and dismiss any you disagree with as wrong, won't fix, fixed, or later.
5. Work through the plan's upgrade steps, and let each pull request mark flaws as fixed.

The claim link expires after seven days if it is not claimed, and an unconnected repository can also post its review to Evalship's audits API. Someone with push access can dismiss flaws, and scoring-line changes get a note so a team can see when a threshold moved. The review is a snapshot, not a permanent verdict: the badge goes stale after 90 days, which is the product's way of saying evals rot and need re-checking.

What flaws did Evalship find in its own demo?

The clearest way to see what an eval flaw looks like is Evalship's own demo, built around a fictional support bot called Eliza. One caution belongs here and is worth stating once: the demo repository, acme_demo/support-bot, is a sample — its GitHub address returns a 404, so the numbers below are a worked example, not real-world results. The demo review found three flaws across 44 test cases.

The first flaw is the one to remember. It is rated critical: replies that leak customer data still pass.

The scoring formula is score = 0.5 times helpfulness + 0.3 times tone + 0.2 times privacy, so privacy is only one fifth of the total. A reply that leaks a customer's address can score 0.80 and pass, because 0.5 plus 0.3 equals 0.80 even when privacy scores zero. The test case is not wrong about what it measures — it is wrong about how much the leak matters.

The second flaw is rated important: nothing checks that replies match the refund system. The refund instruction lives in the system prompt, but no test case exercises it, so a reply that invents a refund policy would pass.

The third flaw is rated a warning: evals pass when the judge is down. A judge failure returns 1.0, so a broken judge looks like a perfect product. Each flaw is the same lesson in a different shape: the eval measures something, but not the thing that would hurt a user.

The demo dashboard shows the same story in numbers: 45 of 46 evals passing on main, which is 97.8 percent, and one of three flaws fixed. In the demo pull request 128, which shortens the prompt, 42 of 46 pass — 91.3 percent, three fewer than main — with two regressions: one refund eval falling from 0.96 to 0.22, a drop of 0.74, and another from 0.94 to 0.18, a drop of 0.76. A single percentage would hide that damage; the per-eval comparison shows it.

How does Evalship check every pull request?

Evalship's PR check, sometimes called eval CI, works like code coverage for evals. Teams that search "eval ci github actions" want exactly this: their evals running in CI, compared with main, on every pull request.

An llm eval regression is a drop in an eval's score after a code change that should not have affected it, and the check exists to catch that drop before the change merges. Your eval workflow runs in GitHub Actions on pull requests and on pushes to the default branch, then uploads a results file as an artifact.

When the workflow run completes, GitHub sends Evalship's App a workflow_run event, and Evalship reads the result files from the run's artifacts. There is no Evalship GitHub Action to install and no token or secret to configure — your own workflow uploads the file with actions/upload-artifact, set to run even when earlier steps fail, under the recommended artifact name evalship-results. If your team is still getting comfortable with branches and pull requests, our guide on how to use Git covers the workflow this gate sits on top of.

Two requirements make the comparison work. The eval workflow must run on pull_request and on push to the default branch, because the default-branch runs are the baseline each pull request compares against.

The eval step typically gets continue-on-error set to true, so the job stays green and the evalship check decides pass or fail. Limits apply: an artifact may be at most 20 MB, a result file at most 10 MB, and a workflow run may hold at most 20,000 evals. Results are processed about 15 seconds after the latest artifact arrives, and multiple workflows or artifacts for the same commit merge into one run.

Each completed run is stored per commit. A pull request is compared with the baseline run for its merge-base commit, falling back to the latest run on the base branch or the default branch.

The evalship-app bot comments on the pull request with badges for evals passed and eval flaws, a status table showing each eval's main score against its PR score, failure-pattern notes with a likely cause, untested AI calls, and a details section. A check named evalship is posted on the commit, and it can block merging if branch protection requires it. Each new push gets a fresh report at the bottom of the pull request, and the old one collapses as outdated.

The comparison is noise-aware, which is the feature that separates it from a plain pass-or-fail gate. Noise is measured over the suite's last 30 default-branch runs, and a regression counts only when the suite got worse beyond chance — a level chance would reach in fewer than 1 in 20 pull requests.

The default score tolerance is 0.05 and can be changed in a .evalship.yml file. Flaky evals are ignored in the gate, with an opt-out for teams that want them counted, and results from a local machine can be sent in by posting to Evalship's results API with an upload token. A push to the default branch that breaks previously passing evals emails the account owners.

Which eval formats and frameworks does it plug into?

Evalship reads six result formats, auto-detected from content rather than filename. JUnit XML covers pytest, Vitest, Jest, RSpec, Go, Java, PHPUnit and Rust test runners, with each test case counting as one eval and scores and thresholds passed through properties.

Promptfoo JSON comes from running promptfoo eval with a JSON output, one eval per test times prompt times provider. The other formats are pytest-json-report JSON, Vitest or Jest JSON, Inspect AI log, and evalship JSON for hand-rolled setups. Coding agents reach outside tools through protocols such as the Model Context Protocol, and the same idea applies here: Evalship plugs into the tools you already use instead of replacing them.

The frameworks it plugs into natively are pytest, Vitest, Jest, promptfoo, DeepEval through pytest, and Inspect AI, plus the JUnit XML runners named above. Langfuse, Braintrust and LangSmith connect through a recipe of about 15 lines added to the user's experiment script: the script keeps reporting to the platform and also writes an Evalship JSON file that CI uploads, with each eval's URL pointing at its trace or experiment row in that platform.

A public example repository, evalship-hq/llm-evals-github-actions, shows the workflow wiring. The platforms stay where datasets and traces live; Evalship only reads the results.

How does Evalship compare with Braintrust, LangSmith, Langfuse and promptfoo?

Evalship is not a replacement for the eval platforms it plugs into — it is a checker that sits on top of them. Braintrust is an AI agent evaluation and observability platform for running evals and experiments, scoring with more than 20 scorers, and comparing experiments.

LangSmith is LangChain's agent engineering platform for observing, evaluating and deploying agents, with offline evals on datasets and online evals on live traffic. Langfuse is an open-source AI engineering platform for agent evals and observability, with traces, evaluations, datasets and prompt management, self-hosted or in the cloud, whose core is MIT-licensed and which became part of ClickHouse in January 2026. Promptfoo is an open-source MIT-licensed CLI and library for testing, evaluating and red-teaming LLM apps, acquired by OpenAI in March 2026 and still open source.

The comparison article on Evalship's own site, dated 2 November 2026, is the source for the PR behaviour in the table above, and it is a vendor comparison, so read it as the maker's view of its rivals rather than a neutral test. One related fact matters for teams choosing tools: OpenAI's hosted Evals service shuts down on 30 November 2026, and OpenAI recommends promptfoo instead. That leaves promptfoo as the natural eval runner for teams migrating off OpenAI Evals, with Evalship as an optional checker on top of it.

How does Optimize find cheaper prompts and models?

Optimize is Evalship's third job, and it answers a different question from the review: not "are my evals broken" but "could a cheaper prompt or model pass just as well". People searching "prompt optimization evals" are asking that question: whether a cheaper prompt or model still passes the same tests.

Your coding agent starts from why a set of evals fails, tries changes on your machine, and Evalship says which changes win beyond chance. The statistical bar is strict.

A gap counts only when chance would produce it less than 1 time in 40 for instructions, and less than 1 in 80 for models, and a change matters at 5 points of pass rate. A difference smaller than that is treated as noise, not a win.

The homepage shows two worked examples. A prompt variant passes 41 of 42 evals, which is 97.6 percent, against 34 of 42 for the current prompt, which is 81.0 percent — a gap of 7 evals, or 16.7 points, at 8 percent higher cost.

A model switch from gpt-4o-mini to gpt-4.1-nano is 33 percent cheaper and 43 percent faster, passing 658 of 680 evals, or 96.8 percent, against 669 of 680, or 98.4 percent, for the current model — 11 evals fewer, a gap of 1.6 points, which the homepage describes as passing about as often. The point of both examples is the same: a cheaper option is only a win if the evals still pass, and the evals are only trustworthy if the review has already checked them.

What does Evalship cost?

Evalship is a paid SaaS product, not open source — no open-source product code was found, and it runs as a hosted service. That is worth stating once, because the review and Optimize features being free can make it sound like a free tool.

They are free because they run on your coding-agent subscription; the PR checking is the paid part. Pricing below was verified on 6 November 2026, and the launch offer has no stated expiry date, so the 9-spot count will change.

The launch offer is 60 percent off forever for the next 9 paid accounts, with Pro at $19.60 a month instead of $49. That saves $29.40 a month, or $352.80 over a year.

The ladder after that is 50 percent off for the next 25 accounts, then 40 percent, stepping down to 10 percent, and list price after 134 accounts. Yearly billing costs 10 months' price, which is two months free, and paid plans have a 7-day trial. Billing goes through Polar as merchant of record.

Every plan includes the review, the monitor, Optimize, results from every run, the evalship check, badges, PR comments, eval flaws, gating, untested-change detection, failure patterns and the weekly report, with no limits on evals, runs or pull requests. What the plans meter is users, private repositories, PR analyses and history.

Public repositories are free on every plan. A PR analysis means a pull request in a month where a model writes part of the comment — one that changes prompts, AI calls or eval code, or has unexplained new failures — and later pushes to the same PR that month are free. Extra private repositories cost $10 a month each, extra users $2 a month, and extra PR analyses $0.10 each.

What does Evalship see, and what stays private?

Evalship's security page answers the first question a team should ask of any service that reads their evals: what leaves your machine. Your evals run in your own CI or on your own machine with your own keys.

Evalship sees results, short excerpts of pull requests, and diffs the coding agent checks before committing. Its own wording is that it never takes your secrets or a copy of your repository.

Eval inputs and outputs are kept up to 4 KB each, and the model that writes PR notes gets the first 600 characters of a newly failing eval's input and output, plus 400 characters of its failure message. Owners can switch off keeping eval inputs and outputs, and switch off sending anything to a model.

The model notes come from Claude — Haiku 5.5 today, or Opus when Haiku declines — through Anthropic's API, which does not train on them and deletes them within 30 days. Only the founder can reach user data, through a read-only admin view.

The GitHub App is read-only to code and writes only its PR comment and the evalship checks, with per-repo tokens that expire within an hour. Removing a repository or uninstalling deletes eval results, code excerpts and names 30 days later, with backups inside three months.

The company is Garm Tech B.V. in the Netherlands, under the GDPR, with hosting on Railway in the US.

Its security reviews are dated 19 and 29 October 2026. One incident is disclosed on the same page: on 21 October 2026, Railway reported that environment variables of hosted services may have been read, and Evalship rotated every key and secret the next day, found no misuse, and noted that no customer accounts existed yet. That last line is a useful date marker for how new the product is — the incident predates the public launch by two weeks.

How far should you trust Evalship?

Evalship is two days past its public launch, and the honest answer is: far enough to try, not far enough to believe without checking. The product's dated record runs from its first article on 8 October 2026 to the Show HN launch on 4 November 2026 — the launch came 27 days after the first article, and this guide was written 29 days after it.

There is no dated changelog on the homepage — the 4 November Show HN post is the launch evidence — and as of 6 November 2026 no independent review, news coverage or Product Hunt listing was found. All product claims in this guide come from the vendor and its founder.

The launch itself drew almost no attention, which is worth stating once: the Show HN post had 1 point and 1 comment when checked on 6 November 2026, and that single comment was the author's own launch text, because the story body was empty. The founder, Garm from Amsterdam, wrote that in testing on public repos Evalship consistently identifies flaws that nothing else is flagging yet.

That claim is his own, and no independent verification of it was found — so treat it as a pitch to test, not a result. The same caution applies to the site's other research articles: the November 2 piece found that every one of 10 open-source eval suites had a test case that could not fail, and that in 6 of 9 suites the LLM judge had a flaw, most often grading its own model's answers. Those are striking findings, and they are the vendor's own findings.

None of that makes the product wrong. The design is sound in the ways that matter: it checks the measurement before trusting the number, it compares against main with noise in mind instead of failing on every wobble, and it charges for the hosted checking while leaving the review free on your own agent.

The right way to use it is the way its own docs suggest — run the review on a real repository, read the flaws with their citations, fix one, and see whether the next pull request comment tells you something you did not already know. If it does, keep it.

If it only repeats what your evals already said, the review cost you nothing but 10 to 45 minutes. CodexFuse does not offer an Evalship tool, and this guide promises none.

Sources

Evalship homepage — https://evalship.com/ — fetched 2026-11-06. Evalship pricing docs — https://evalship.com/docs/pricing — fetched 2026-11-06.

Evalship eval review docs — https://evalship.com/docs/audit — fetched 2026-11-06. Evalship CI docs — https://evalship.com/docs/ci — fetched 2026-11-06.

Evalship result formats docs — https://evalship.com/docs/formats — fetched 2026-11-06. Evalship integrations docs — https://evalship.com/docs/integrations — fetched 2026-11-06.

Evalship PR comparison docs — https://evalship.com/docs/comparison — fetched 2026-11-06. Evalship security page — https://evalship.com/security — fetched 2026-11-06.

Evalship setup prompt — https://evalship.com/setup.md — fetched 2026-11-06. "How noisy are LLM evals in CI?" — https://evalship.com/articles/eval-noise — published 2026-10-08, fetched 2026-11-06.

Evalship articles index — https://evalship.com/articles — fetched 2026-11-06. Show HN launch item 50363399 — https://news.ycombinator.com/item?id=50363399 — posted 2026-11-04T10:03:08Z, fetched via the HN Algolia API 2026-11-06.

Braintrust — https://www.braintrust.dev/ — fetched 2026-11-06. LangSmith — https://www.langchain.com/langsmith-platform — fetched 2026-11-06.

Langfuse — https://langfuse.com/ and https://raw.githubusercontent.com/langfuse/langfuse/main/README.md — fetched 2026-11-06. Promptfoo — https://www.promptfoo.dev/ — fetched 2026-11-06.

Frequently Asked Questions

What is Evalship?

Evalship is a paid service that checks the quality of your LLM evals. It reviews your eval suites for flaws, watches every pull request for eval regressions against your main branch, and tests prompt and model changes to find ones that work better or cost less. It does not write your evals and it does not run them — it reads the results files your existing tools already produce.

Is Evalship free?

Partly. The eval review and the Optimize feature are free, because they run on your own coding-agent subscription.

The pull-request checking is paid: a Free plan covers 1 user, 1 private repository and 100 PR analyses a month, while Team, Pro and Business plans raise those limits. Public repositories are free on every plan, and paid plans have a 7-day trial.

Does Evalship work with my eval framework?

Probably, if your framework writes a standard results file. Evalship reads JUnit XML, promptfoo JSON, pytest-json-report, Vitest or Jest JSON, Inspect AI logs, and its own JSON format, which covers pytest, Vitest, Jest, promptfoo, DeepEval, Inspect AI and any JUnit XML runner. Langfuse, Braintrust and LangSmith connect through a short JSON bridge recipe in your experiment script.

How is Evalship different from Braintrust or LangSmith?

Braintrust and LangSmith are eval platforms: they run your evals, store your traces and experiments, and score results. Evalship is a checker that sits on top of them: it reads the results they produce, reviews the evals themselves for flaws, and compares every pull request with main using noise-aware statistics. The platforms stay where your datasets and traces live.

Should I trust Evalship's findings?

Check them the way Evalship checks its own reviews: read the citation. Every flaw comes with a file, a line and an exact quote, and a flaw is only shown when all of its evidence holds. The product launched publicly on 4 November 2026, and as of 6 November 2026 no independent review of it exists, so the safest approach is to run the free review on one real repository and judge it on what it finds there.