Enterprise Quality Observability & Analytics Platform

Head of Software Automation · 2021 - 2023

Quality & Automation2021 - 2023

Built an end-to-end test analytics and root-cause tracing system, reducing test suite flakiness to under 0.2% across 35,000 daily automated executions.

TypeScriptClickHouseOpenTelemetryPlaywrightRedisGrafana

Situation

With over 35,000 automated UI, API, and unit tests running daily across CI pipelines, transient network issues and unstable test data introduced silent flakiness. Engineers lost trust in build results, routinely clicking “re-run job” and ignoring failing checks.

Task

Eliminate test flakiness and create a single-pane-of-glass dashboard for engineering leadership to track test reliability, failure trends, and performance degradation.

Action

  • Telemetry Collector: Developed a custom test runner reporter injecting OpenTelemetry trace IDs into HTTP headers for every API/UI test request.
  • Storage & Analytics Engine: Ingested test execution data into ClickHouse via a high-throughput Redis buffer for real-time querying.
  • Auto-Quarantine & Root Cause AI: Engineered an algorithm that quarantines tests exhibiting >2% variance across 10 builds, automatically assigning tickets to owner squads with full trace logs and DOM snapshots.

Result

  • Dropped global test suite flakiness from 8.4% to 0.18%.
  • Saved an estimated 180 developer hours per month previously lost to manual debugging.