Situation
Performance testing was previously treated as an infrequent, manual pre-release activity. As a result, memory leaks, latency regressions, and fragile circuit-breaker configurations slipped into production, triggering cascading outages during peak load periods.
Task
Architect an automated performance and resilience testing platform that enforces p95/p99 latency budgets in continuous integration pipelines and continuously injects network and infrastructure faults to verify cluster resilience.
Action
- Automated Load Gates in CI: Built a lightweight k6 load testing execution runner integrated into GitHub Actions, evaluating custom latency SLOs ($p_{95} < 150\text{ms}$) against target microservices and failing pull requests that violated performance budgets.
- Chaos Mesh Kubernetes Fault Injection: Deployed Chaos Mesh custom resources to automatically simulate network partitions, packet loss (20–50%), and random pod evictions in staging environments while active synthetic load runs executed.
- Prometheus Alert-Driven Assertion: Wrote custom validation runners in Go that queried Prometheus metrics during chaos runs to confirm that Envoy circuit breakers tripped as expected and downstream retries gracefully shed load.
Result
- Shifted performance testing left, catching over 95% of latency regressions in pull requests before staging deployment.
- Verified microservice resilience across 100+ automated chaos experiments, completely eliminating cascading failures in production.