Periodic 100% packet loss

[gerhardlazu]

This has been happening for many months now - https://forum.vodafone.co.uk/t5/Broadband-connection/gt-50-packet-loss-and-a-bad-experience-overall/... - and some days are better than the others.

Yesterday it got so bad that I had to capture the problem, and show just how bad things are: video calls keep stalling (UDP), web requests time out (TCP) and the overall experience is terrible. By the way, this is impossible to capture in speedtests or any other point-in-time troubleshooting. In other words, if I called Vodafone support, we wouldn't be able to reproduce the problem, so there must be no problem 🤨

No matter how many times I engage with Vodafone support, all fixes are temporary, and most of the time the problem only get worse. I sometimes wish I had these problems, instead of the current 100% packet loss one:

• https://forum.vodafone.co.uk/t5/Internet-speed/Paying-for-900Mbps-cannot-exceed-20Mbps-download-in-t...
• https://forum.vodafone.co.uk/t5/Other-broadband-queries/Milton-Keynes-routed-via-Manchester-high-lat...

So this is what happened so far today: I had 5 periods of 100% packet loss within 12:30pm - 12:50pm, as well as many other partial packet losses (20%).

This 100% packet loss is to the gateway (84.65.192.1), so from my router to Vodafone's ingress.

This is what the same period looks like inside my network (all wired, 0.2ms latency and no packet loss):

If the packet loss in the morning looks bad, it gets even worse in the evening, when I can't do much between 5:30pm and 6:30pm:

To me, this looks like load shedding, where people come back from work, and the network is dropping all the extra load which it cannot handle. It will take a few more days to start seeing a pattern, but for now, sharing this felt valuable enough.

[gerhardlazu]

Today it has been a very bad day, with 11 disconnects between 4:25pm and 5:35pm:

Same time interval, internal network has been fine:

 

This shows the number of pings lost in the last 24 hours, between my router, and the Vodafone gateway, which is the next hop:

 

Yesterday was not great either, but better than today:

In conclusion, today has been a bad day for packet loss on the Vodafone Gigafast network 😡

[gerhardlazu · Accepted solution]

Friday, Oct. 15, has been the worst day since I started monitoring this. Between 2pm - 6pm I couldn't have a video or audio conversation that wouldn't get interrupted (the nature of UDP traffic) and a lot of the TCP traffic would be super slow and sometimes fail due to retransmits or broken routes (the Vodafone gateway was just not available). The problem in one picture:

My internal network (Linux host CAT6a wired to the router) during the same period:

Today I had a single period with 20% packet loss, which confirms that it's the Vodafone network:

Another interesting finding is frequent PPP disconnects in my router logs (UDM-Pro) due to... packet loss:

Oct 15 06:07:39 UDMPro-Loft user.info ubios-udapi-server: ubios-udapi-server: wanFailover: ppp0 is down: {"alarm":true,"dns_alarm":0,"id":"ppp0","lat_avg":10620,"loss_pct":21.052600860595703,"std_dev":156,"type":"wanFailover"}
Oct 15 13:08:28 UDMPro-Loft user.info ubios-udapi-server: ubios-udapi-server: wanFailover: ppp0 is down: {"alarm":true,"dns_alarm":0,"id":"ppp0","lat_avg":10677,"loss_pct":21.052600860595703,"std_dev":311,"type":"wanFailover"}
Oct 15 13:36:53 UDMPro-Loft user.info ubios-udapi-server: ubios-udapi-server: wanFailover: ppp0 is down: {"alarm":true,"dns_alarm":0,"id":"ppp0","lat_avg":10703,"loss_pct":21.052600860595703,"std_dev":145,"type":"wanFailover"}
Oct 15 14:04:26 UDMPro-Loft user.info ubios-udapi-server: ubios-udapi-server: wanFailover: ppp0 is down: {"alarm":true,"dns_alarm":0,"id":"ppp0","lat_avg":10662,"loss_pct":21.052600860595703,"std_dev":205,"type":"wanFailover"}
Oct 15 14:36:08 UDMPro-Loft user.info ubios-udapi-server: ubios-udapi-server: wanFailover: ppp0 is down: {"alarm":true,"dns_alarm":0,"id":"ppp0","lat_avg":10667,"loss_pct":21.052600860595703,"std_dev":132,"type":"wanFailover"}
Oct 15 14:59:00 UDMPro-Loft user.info ubios-udapi-server: ubios-udapi-server: wanFailover: ppp0 is down: {"alarm":true,"dns_alarm":0,"id":"ppp0","lat_avg":10707,"loss_pct":21.052600860595703,"std_dev":207,"type":"wanFailover"}
Oct 15 15:04:03 UDMPro-Loft user.info ubios-udapi-server: ubios-udapi-server: wanFailover: ppp0 is down: {"alarm":true,"dns_alarm":0,"id":"ppp0","lat_avg":10701,"loss_pct":21.052600860595703,"std_dev":224,"type":"wanFailover"}
Oct 15 15:39:18 UDMPro-Loft user.info ubios-udapi-server: ubios-udapi-server: wanFailover: ppp0 is down: {"alarm":true,"dns_alarm":0,"id":"ppp0","lat_avg":10713,"loss_pct":21.052600860595703,"std_dev":203,"type":"wanFailover"}
Oct 15 15:43:12 UDMPro-Loft user.info ubios-udapi-server: ubios-udapi-server: wanFailover: ppp0 is down: {"alarm":true,"dns_alarm":0,"id":"ppp0","lat_avg":10692,"loss_pct":21.052600860595703,"std_dev":229,"type":"wanFailover"}
Oct 15 15:43:51 UDMPro-Loft user.info ubios-udapi-server: ubios-udapi-server: wanFailover: ppp0 is down: {"alarm":true,"dns_alarm":0,"id":"ppp0","lat_avg":10627,"loss_pct":21.052600860595703,"std_dev":133,"type":"wanFailover"}
Oct 15 15:46:03 UDMPro-Loft user.info ubios-udapi-server: ubios-udapi-server: wanFailover: ppp0 is down: {"alarm":true,"dns_alarm":0,"id":"ppp0","lat_avg":10664,"loss_pct":21.052600860595703,"std_dev":178,"type":"wanFailover"}
Oct 15 15:58:09 UDMPro-Loft user.info ubios-udapi-server: ubios-udapi-server: wanFailover: ppp0 is down: {"alarm":true,"dns_alarm":0,"id":"ppp0","lat_avg":10630,"loss_pct":21.052600860595703,"std_dev":186,"type":"wanFailover"}
Oct 15 16:14:00 UDMPro-Loft user.info ubios-udapi-server: ubios-udapi-server: wanFailover: ppp0 is down: {"alarm":true,"dns_alarm":0,"id":"ppp0","lat_avg":10743,"loss_pct":21.052600860595703,"std_dev":408,"type":"wanFailover"}
Oct 15 16:33:24 UDMPro-Loft user.info ubios-udapi-server: ubios-udapi-server: wanFailover: ppp0 is down: {"alarm":true,"dns_alarm":0,"id":"ppp0","lat_avg":10823,"loss_pct":21.052600860595703,"std_dev":613,"type":"wanFailover"}
Oct 15 16:46:30 UDMPro-Loft user.info ubios-udapi-server: ubios-udapi-server: wanFailover: ppp0 is down: {"alarm":true,"dns_alarm":0,"id":"ppp0","lat_avg":10706,"loss_pct":21.052600860595703,"std_dev":249,"type":"wanFailover"}
Oct 15 19:16:34 UDMPro-Loft user.info ubios-udapi-server: ubios-udapi-server: wanFailover: ppp0 is down: {"alarm":true,"dns_alarm":0,"id":"ppp0","lat_avg":10859,"loss_pct":21.052600860595703,"std_dev":446,"type":"wanFailover"}
Oct 15 19:28:40 UDMPro-Loft user.info ubios-udapi-server: ubios-udapi-server: wanFailover: ppp0 is down: {"alarm":true,"dns_alarm":0,"id":"ppp0","lat_avg":10894,"loss_pct":21.052600860595703,"std_dev":478,"type":"wanFailover"}
Oct 15 20:26:52 UDMPro-Loft user.info ubios-udapi-server: ubios-udapi-server: wanFailover: ppp0 is down: {"alarm":true,"dns_alarm":0,"id":"ppp0","lat_avg":10690,"loss_pct":21.052600860595703,"std_dev":181,"type":"wanFailover"}
Oct 15 21:35:58 UDMPro-Loft user.info ubios-udapi-server: ubios-udapi-server: wanFailover: ppp0 is down: {"alarm":true,"dns_alarm":0,"id":"ppp0","lat_avg":10632,"loss_pct":21.052600860595703,"std_dev":265,"type":"wanFailover"}

I have been on the phone with Vodafone support, and have concluded that they cannot help me. They kept asking me to reset the router, and I kept explaining why "stopping and starting it" won't help the situation. They kept telling me about Layer 7 fixes, and I kept explaining how this is a Layer 2/3 issue. It was a good & friendly conversation, and as a result I cancelled my contract. There is nothing wrong not knowing or not being able to help, and I would like to praise the person on the other end of that conversation for being honest about it. In case someone with access to support cases wants to find it, the reference number is MQFM4Y7A.

I will be around this forum for a few more weeks, in case anyone has any questions. I hope that sharing this experience will help others. Cheers!