Convoy API & Database — Degraded performance and replica sync latency
- Started
- 14 Oct 2026, 17:00 UTC
- Duration
- 1h 15m
- Affected components
- Convoy APIMongoDB replica 1MongoDB replica 2Check-in Pipeline
Updates
Investigating
Monitoring systems detected elevated latency and intermittent query timeouts affecting the Convoy API and check-in processing pipeline.
Identified
Root cause identified: secondary database replication lag and SASL authentication retry storms on replica nodes caused connection contention during high read query periods.
Monitoring
Applied database configuration updates to route secondary queries via secondaryPreferred with bounded staleness, unified authentication credentials across application and admin databases, and added lag threshold hysteresis. Latency metrics are recovering.
Resolved
Resolved. Secondary replica replication lag has normalized, query load is evenly balanced, and CloudWatch agent metric dimensions have been aligned. API response times and check-in processing have returned to normal.