Stabilising a VPN service with 50,000+ users

01Problem

50,000+ users, 147 nodes, around 20 TB of traffic a day — and a steady stream of complaints about drops. A rewrite was off the table: the job was to find the causes and steady what already ran, without taking production down.

02What was done

1. 01The audit came first: I looked at the infrastructure without touching it.
2. 02End-to-end connection quality checks across the key routes, with no new port opened to the outside.
3. 03Found the causes of the drops and backed them with measurements.
4. 04Fixes went out in stages, each with a result check and a way back.

03Outcome

• The causes behind the main complaints were found and confirmed by measurement within hours.
• The client ended up with a measurable picture of service quality.
• The work went unnoticed by users: not one switchover was spotted.

04Stack

• Linux
• Xray
• Prometheus
• Grafana
• nginx