Degraded Performance
Some requests are slower or partially failing.
No incident has been opened yet, but automated checks are reporting a problem. We are investigating.
Components
90-day uptimeConvoy
Convoy API
UpRegistration, check-in, and admin API · ee34973 · deployed 11h ago
Convoy Web
UpStaff console and public registration · ee34973 · deployed 11h ago
TimescaleDB Metrics
UpOperational metrics store · ee34973 · deployed 11h ago
Redis / Valkey
UpCache and live broadcast · ee34973 · deployed 11h ago
Check-in Pipeline
UpDoor check-in processing · ee34973 · deployed 11h ago
Background Jobs
UpCalendar sync, Flock sync, snapshots · ee34973 · deployed 11h ago
Live WebSocket Feed
UpReal-time stats broadcast · ee34973 · deployed 11h ago
Databases
MongoDB
UpShared registration document store (quorum health) · ee34973 · deployed 11h ago
MongoDB replica 1
UpPRIMARY · ee34973 · deployed 11h ago
MongoDB replica 2
UpSECONDARY · lag 0s · ee34973 · deployed 11h ago
MongoDB replica 3
UpSECONDARY · lag 0s · ee34973 · deployed 11h ago
Identity Platform
Authentication & Sign-in
UpPassword, passkey and social sign-in · 1.3.1+8756f07 · deployed 11h ago
Accounts
Upmyaccount.tktchurch.com self-service · 1.1.7+7855d4f · deployed 12d ago
Developer Console
UpOAuth apps, API keys and admin tools · 1.1.7+7855d4f · deployed 12d ago
OAuth & Identity API
UpToken, authorization and userinfo endpoints · 1.3.1+8756f07 · deployed 11h ago
Email & SMS Delivery
UpVerification codes and notifications · 1.1.7+7855d4f · deployed 12d ago
Pathway
Pathway API
UpEvents API for the Connect app · latest+c60719d · deployed 17d ago
Pathway Connect
DegradedConnect app campaign replica · latest+c60719d · deployed 17d ago
Spark CRM
CRM API
UpMember, seeker and household records API
CRM Web
UpStaff console for congregation care
Past incidents
Convoy Web — Intermittent OAuth sign-in state mismatch
MinorResolvedResolved. Deployed a multi-transaction authorization state queue (retaining up to 5 concurrent states with 10-minute TTL), added a client-side single-flight navigation guard to eliminate redundant redirect cascades, and cleared shadowed parent-domain cookies. Fix is verified in production.
15 Oct 2026, 14:30 UTC · resolved in 1h 29m · Convoy Web, Authentication & Sign-in
Convoy API & Database — Degraded performance and replica sync latency
MinorResolvedResolved. Secondary replica replication lag has normalized, query load is evenly balanced, and CloudWatch agent metric dimensions have been aligned. API response times and check-in processing have returned to normal.
14 Oct 2026, 17:00 UTC · resolved in 1h 15m · Convoy API, MongoDB replica 1, MongoDB replica 2, Check-in Pipeline
Convoy API outage — expired edge TLS certificate
MajorResolvedResolved. Certificate renewed (valid to Jan 2027), API healthy, board green. Token rotated vault-side and fleet-wide except the CRM host (no access — tracked). TLS-expiry early warning (30d warn / 14d page) added to the status watcher so renewal failures page weeks before expiry.
8 Oct 2026, 16:13 UTC · resolved in 18m · Convoy API