Error Messages in the event Log and spikes in latency

[infinidim]

Folks,

I am getting a lot of messages in the event log of my Ultra Hub. See below -

Date Time Category Severity Log Details
18.08.2024 12:36:26 system Warning [ERROR pktrunner] runnerUcast_activate
18.08.2024 12:36:26 system Warning ERR: bdmf_attrelem_add_as_buf#4298: ucast: status:Entry doesn't exist. attribute:flow index:4294967295
18.08.2024 12:36:26 system Warning ERR: ucast_prepare_rdd_ip_flow_result#782: status:Entry doesn't exist ucast: US egress queue 0 is not configured
18.08.2024 12:36:22 firewall Warning DROP wan in: IN=pppoe-wan OUT= MAC= src=162.216.149.67 DST=84.67.50.152 LEN=44 TOS=0x00 PREC=0x00 TTL=251 ID=54321 PROTO=TCP SPT=54431 DPT=54500 WINDOW=65535 RES=0x00 SYN URGP=0 MARK=0x8000000
18.08.2024 12:36:17 system Error Failed to send to ff02::1%guest@br-guest (Cannot assign requested address)
18.08.2024 12:36:13 system Warning [ERROR pktrunner] runnerUcast_activate
18.08.2024 12:36:13 system Warning ERR: bdmf_attrelem_add_as_buf#4298: ucast: status:Entry doesn't exist. attribute:flow index:4294967295
18.08.2024 12:36:13 system Warning ERR: ucast_prepare_rdd_ip_flow_result#782: status:Entry doesn't exist ucast: US egress queue 0 is not configured

These messages seem to come in cycles and are the system error messages I see most in the hub.

Looking them up online it seems they relate to the configuration of the hub but there isn't anything in the hub to fix these errors.

The other thing I see is very variable latency on the connections to the hub. This does cause random slowness and sometimes packet loss. See the attached image of the latency issue, the yellow lines are increase in latency.

yellow lines are latency spikes

The minimum latency above (in green) I expect from the type of connection that we have from Vodafone (Openreach FTTC over copper with a 250 meter approx. from the FTTC cabinet) but the spikes (in yellow) are a totally different matter. On top of the image above are small red marks, these are packet drops. There are not very many but when the latency goes out to a high level then the packet drops start to occur.

We are due to get Cityfibre before the end of the year in our close as they have done the initial pull of fibre into the close, hopefully this will improve but I am seeing packet drops in the Vodafone core network though. Not that many now but they are there. See following image -

Vodafone Core network drops

Any ideas folks on what I can get done about this?

Thanks in advance

Infinidim

[infinidim]

The situation has got even worse on the latency front over the last couple of days. See the following two screen shots.

19082024

20082024

the first image is 19082024 and the second image is 20082024 up until 17:00. 

We are not getting any drops on the Openreach FTTC access network, the latency all seems to be coming from the Vodafone core network.

Hopefully the moderator might see this and raise it with the Vodafone core network team. When you talk to the front office support people they seem not to understand these sort of issues and won't pass them onto a higher level support.

Regards

Infinidim

[Cynric]

@infinidim Just as a test can you turn off IPv6 please? It may fix one part of the errors, though I'm reasonably sure it's not the cause for the latency drop at peak evening. 

You could just also be a victim of living 200 miles from where you are being routed into the VF infrastructure, they've done this before.

[infinidim]

Cynric,

I did turn off IP6 and it didn't effect anything

Interested in your 200 miles comment. I think that the Vodafone fibre backbone (used to me Mercury then Cable and Wireless) at it's nearest point is either Basingstoke or London. Can't be certain of that though.

Your thoughts.

Infinidim

[Ripshod]

We all know traffic causes latency spikes. Looks like a large something was doenloading between 18:30 and 00:15. Xbox/playstation update?

Woul be interesting to see a copy/paste of a traceroute to bbc.com, or a link to one you may have posted elsewhere.

[Ripshod]

I also noticed your minimum latency doesn't change - the green section is flat. If there was a connection problem one would expect to see a rise in minimum latency, like in my BQM, between 8pm and 10pm every  evening,

Don't worry about those spikes, I'm not. They are normal

[infinidim]

Ripshod,

Are you on fibre (Citryfibre or Openreach) or copper to the Openreach FTTC cabinet?

[Ripshod]

500Mb via Openreach. Download drops below upload during those "events" 

I wish they'd let us have signatures on here. 

[Cynric]

@Ripshod Something else I didn't spot earlier, the LAN goes 68.1 to 1.1, is this stacked routers?

[Ripshod]

I missed it 😔

Yep. Intentional dual NAT, or shared connection (a la HMO or apartments). So anyone else connected to the same service can affect the others with downloads etc.

Like I said, the minimum latency is flat, so likely a local problem rather than ISP