I find the breaking point first. so production never has to.

✅ SLA gate — perf tests that can fail the build

Latency budgets live in slas.yml, per endpoint. A script parses Locust's CSV, prints a verdict table into the GitHub job summary, and exits non-zero on any breach. “A dashboard needs a human — an exit code scales.”

🧠 Memory-leak hunter

Samples container memory under constant load, discards warm-up, fits a least-squares trend — and fails only when both growth rate and total growth blow their budgets. Demonstrated against a simulated ~50 KB/request leak:

🛡 Gates hardened against a real pipeline

A green suite that lies is worse than a red one. A floating geventhttpclient stopped URL-encoding raw spaces, so ~29% of /search requests failed on CI and every nightly run went red — a tooling bug wearing a performance bug's costume. Search terms are now encoded, Locust is pinned to 2.32.4 to match the Compose image, and the gates learned to tell noise from signal:

• Runs with --exit-code-on-error 0 — the SLA gate is the judge, not Locust's any-failure default (the API injects ~1% payment failures by design)
• Survives Locust's N/A percentiles, and fails when an SLA'd endpoint is missing from the results instead of silently passing
• Skips low-sample endpoints rather than gating on quantization noise; flags 0→X regressions

⚡ Load shapes as ~15 lines of code

Stress, spike and soak profiles are LoadTestShape classes selected by one env var — one reviewed workload model serves every scenario.

class SpikeShape(LoadTestShape):
    """20-user baseline, sudden 10× spike."""
    def tick(self):
        t = self.get_run_time()
        if t < 120: return (20, 5)
        if t < 180: return (200, 50)  # the spike
        if t < 360: return (20, 50)   # recovery
        return None

📉 Regression detection, run over run

One run tells you where you are; comparing runs tells you where you're heading. Nightly CI diffs endpoint-by-endpoint against the previous nightly's artifact and fails on p95 growth, throughput drops, or worsening error rate:

Checkout is the interesting row. At ~0.3 rps it clears --min-requests nowhere near, so it is reported and deliberately not gated — a ±30% p95 swing computed from eight requests is quantization noise, not evidence.

🧪 The gates have their own tests

The scripts that decide pass or fail are themselves production code, so they are tested like it: 25 pytest cases across all four scripts, run in their own CI job before any load is generated. A broken gate can't quietly wave a regression through.

🐳 Distributed by default

make up WORKERS=8 — Locust master/worker split in plain Docker Compose, FastHttpUser for ~10× RPS per generator. The same images spread across extra hosts by pointing workers at the master.

📊 Live observability

Pre-provisioned Grafana dashboard: throughput and average response time per endpoint, response time plotted against the user ramp, failures/s, and the leak-hunting memory panel — problems visible the moment they start, not in a PDF afterwards.

🎯 Error budgets with a rationale

Checkout gets a 5% budget, and the config says why: with ~100 checkout samples in a 2-minute run and ~1% injected gateway timeouts, a 3% budget goes flaky on binomial noise alone. Thresholds you can defend in review.

🩹 Self-healing locator engine

Every element is an ElementDefinition with ordered fallback candidates. When the primary selector dies mid-day (booking.com A/B tests constantly), candidates are tried in order after a grace period, the winner is logged as a healing event, and non-volatile heals are persisted across runs so future runs try the working selector first — then quietly retest the primary and switch back once it recovers.

destinationInput: {
  key: 'stays.searchbox.destinationInput',
  candidates: [
    'input[name="ss"]',                            // primary
    '[data-testid="destination-container"] input', // fallback 1
    'input[placeholder*="Where are you going"]',   // fallback 2
  ],
}

📡 A real heal, from the committed store

Not a hypothetical. This is an actual entry from .healing/healing-store.json after a live run — booking.com dropped an aria-label input and the engine recovered on a role selector:

{
  "key": "cars.search.pickupInput",
  "failed": ["input[aria-label='Pick-up location']"],
  "healed": "role=combobox[name='Pick-up location']",
  "persistPreferred": true
}

🩹 healed → logged → persisted → suite stays green

🌍 Live-site quirks, engineered around

• Flights served as Kayak white-label in some regions — variant auto-detected, stable deep-link scheme used
• Sign-in popup at random moments — addLocatorHandler auto-dismisses whenever it blocks
• Bot challenges and upstream error pages are detected explicitly and skipped, never mistaken for product bugs
• Assertions read rendered outcomes — site totals, offer content, property identity — never a URL the framework built itself

🧱 Layered architecture

Plus a data layer with fluent builders — dates always generated relative to “now”, so the suite never rots from hardcoded test data.

🔦 “Green via fallback” is never invisible

Self-healing has an obvious failure mode: it hides rot. A suite that silently limps along on fallbacks looks identical to a healthy one. So after every run the custom HealingReporter prints the active preferred overrides — not just this run's events — and npm run healing:show dumps the store on demand. A heal keeps the build green and files a visible debt: a primary selector is broken and someone should look. CI never writes the store back — heals arrive as an uploaded artifact and a run summary, so adopting a healed selector stays a deliberate human commit.

♿ Beyond the happy path

axe-core WCAG 2.0 A/AA scan gating on critical and serious violations, toMatchAriaSnapshot structural assertions, and APIRequestContext canaries at the HTTP level — so a green UI run can't hide a broken contract underneath it.

⚙️ CI built to scale

Static checks fail fast → sharded test matrix → blob reports merged into one HTML report. PR annotations, a nightly full-regression cron, manual dispatch with tag filter. Scaling = extending the shard matrix.

🏷 15 tests · 9 suites · 3 browsers

Tag-driven: @smoke @regression @stays @flights @cars @attractions @a11y @api @healer compose with --grep regexes. Three browser projects are configured and run locally; CI keeps the live-site budget honest by sharding Chromium two ways, with the nightly cron carrying the full regression tag.

📸 Documentation it generates itself

The README's screenshots aren't hand-captured and going stale — npm run docs:screenshots drives a separate Playwright config to re-shoot them. The docs are an artifact of the suite.

🔍 Proven to catch real bugs

A screenshot test is only convincing if it catches something. The same suite runs against two deploys of the Toolshop app — a clean build and an intentionally broken one — backed by identical data, so any pixel difference is a planted defect the suite must flag. On the with-bugs build the affected pages light up red — a shifted layout and typos like Contakt and Massage all caught. The trick: there, a passing run would be the real bug.

🧬 Data-driven registry — add a row, get a test

Coverage lives in one array; a single generated spec turns each entry into a test, multiplied across every project. Adding a screenshot is one object, not a new file — the design that keeps the jump to 1000+ snapshots linear.

// pageRegistry.ts — one entry = one test
{
  category: 'home',
  name: 'home-grid',
  tags: ['@smoke', '@visual'],
  run: async ({ homePage }) => {
    await homePage.open();
  },
}

🤖 The CI proves the suite still bites

A 4-way sharded matrix runs in the pinned Playwright container. Each shard records baselines from the clean build, sanity-checks the clean build against them (no false positives), then runs the with-bugs build and asserts diffs were found — and that every failure is a visual diff, not a timeout or a crash wearing a regression's badge. An over-tolerant threshold can never let the suite go silently blind.

🧊 Flake frozen from one place

Every visual assertion inherits one tuned config: animations and caret killed, a shared stylesheet that freezes transitions and hides the polling chat widget, and a resolution-independent tolerance that absorbs anti-aliasing noise without hiding real change.

toHaveScreenshot: {
  animations: 'disabled',
  caret: 'hide',
  scale: 'css',
  stylePath: './tests/support/visual-stabilize.css',
  threshold: 0.2,
  maxDiffPixelRatio: 0.01,
  maxDiffPixels: 1000,
}

🧭 Ten weeks of red, and the right fix

The Toolshop's v5.0 rewrite started sending product search as an HTTP QUERY with a JSON body, while the with-bugs build still sent GET ?q=. A wait pinned to one HTTP verb timed out for ten weeks of CI. The fix wasn't to wait on the other verb — it was to stop coupling to the transport at all: the suite now waits on the state the app derives from the response, the results container it stamps search_completed plus the count that only renders once results arrive. “No copy of the API's shape to keep in sync — and an outage now fails loudly instead of baselining an empty grid.”

📊 A dashboard you can actually open

Allure runs as a secondary reporter feeding a hosted dashboard with history and trends, published to GitHub Pages. Every failed test carries its interactive image diff — Expected / Actual / Diff — and the exact reason it failed. It publishes only when every shard passed both its clean sanity check and its “all failures are visual diffs” assertion, so it never shows a run that broke for the wrong reason.

Open the live dashboard →

🖥 Cross-browser & responsive

Six scenarios run across five projects: Chromium, Firefox and WebKit at a 1366×768 desktop viewport, plus Galaxy Tab S4 and Pixel 7. Selectors target data-test via getByTestId — immune to CSS/text refactors.

🐳 Deterministic pixels in Docker

Fonts and anti-aliasing differ across macOS, Windows and Linux, so a baseline recorded on a Mac won't match one recorded in CI. The pinned playwright:v1.63.0-noble image gives local runs the same browser builds, fonts and OS libraries as CI — and a lockstep check fails the build if the image and the npm dependency ever drift apart.

🔒 Baselines under control

CI doesn't trust the committed PNGs — it re-records them from the clean build each run, so the comparison is always the two builds as they are right now, not against a baseline that quietly rotted months ago. Every step after that runs with updateSnapshots: 'none', so nothing can silently rewrite a snapshot to make itself pass. Locally, baseline:changed rewrites only what genuinely differs.