Verifying a fix with a one-sided z-test
Passing CI shows that a fix builds, not that it works. How xlens compares issue rates across builds to decide.
Placeholder post. Technical deep-dive outline below.
Before and after, per build
How the server picks the first build after a fix with at least verifyMinSessions sessions on the screen.
The test
// one-sided two-proportion z-test: did the affected rate drop?
const p = (x1 + x2) / (n1 + n2);
const z = (x1 / n1 - x2 / n2) / Math.sqrt(p * (1 - p) * (1 / n1 + 1 / n2));verified · not-improved · pending
What each outcome means and when it is logged.