Error Messages in the event Log and spikes in latency

[infinidim]

Folks,

I am getting a lot of messages in the event log of my Ultra Hub. See below -

Date Time Category Severity Log Details
18.08.2024 12:36:26 system Warning [ERROR pktrunner] runnerUcast_activate
18.08.2024 12:36:26 system Warning ERR: bdmf_attrelem_add_as_buf#4298: ucast: status:Entry doesn't exist. attribute:flow index:4294967295
18.08.2024 12:36:26 system Warning ERR: ucast_prepare_rdd_ip_flow_result#782: status:Entry doesn't exist ucast: US egress queue 0 is not configured
18.08.2024 12:36:22 firewall Warning DROP wan in: IN=pppoe-wan OUT= MAC= src=162.216.149.67 DST=84.67.50.152 LEN=44 TOS=0x00 PREC=0x00 TTL=251 ID=54321 PROTO=TCP SPT=54431 DPT=54500 WINDOW=65535 RES=0x00 SYN URGP=0 MARK=0x8000000
18.08.2024 12:36:17 system Error Failed to send to ff02::1%guest@br-guest (Cannot assign requested address)
18.08.2024 12:36:13 system Warning [ERROR pktrunner] runnerUcast_activate
18.08.2024 12:36:13 system Warning ERR: bdmf_attrelem_add_as_buf#4298: ucast: status:Entry doesn't exist. attribute:flow index:4294967295
18.08.2024 12:36:13 system Warning ERR: ucast_prepare_rdd_ip_flow_result#782: status:Entry doesn't exist ucast: US egress queue 0 is not configured

These messages seem to come in cycles and are the system error messages I see most in the hub.

Looking them up online it seems they relate to the configuration of the hub but there isn't anything in the hub to fix these errors.

The other thing I see is very variable latency on the connections to the hub. This does cause random slowness and sometimes packet loss. See the attached image of the latency issue, the yellow lines are increase in latency.

yellow lines are latency spikes

The minimum latency above (in green) I expect from the type of connection that we have from Vodafone (Openreach FTTC over copper with a 250 meter approx. from the FTTC cabinet) but the spikes (in yellow) are a totally different matter. On top of the image above are small red marks, these are packet drops. There are not very many but when the latency goes out to a high level then the packet drops start to occur.

We are due to get Cityfibre before the end of the year in our close as they have done the initial pull of fibre into the close, hopefully this will improve but I am seeing packet drops in the Vodafone core network though. Not that many now but they are there. See following image -

Vodafone Core network drops

Any ideas folks on what I can get done about this?

Thanks in advance

Infinidim

[infinidim]

Ripshod

I am intentionally dual NAT. I already had a fibre based 6E mesh WiFi network before we moved to Vodafone and I am just using the Ultra Hub and it's mobile backup to get the internet into our property.

The local access to the property is Openreach FTTC about 250 meters from the cabinet with a mixture of copper and aluminium cables in the route. 

We already have Citryfibre in the Openreach boxes in our Close (8 properties) but it is not connected yet (due to be connected before the end of the year). Hopefully this issue will go away once the fibre is up and running because I am sure it relates to Openreach access delivery.

The image of the latency is actually driven from a remote location and is looking at the IP address of the  WAN interface of Vodafone Ultra Hub.

Hope this helps you understand what is going on here.

[Ripshod]

I'm tracking speed and latency on my router right now, as I agree that looking inward doesn't tell the whole story. I'll post the resulting graphs here later tonight. 

[Cynric]

@infinidim Have you tried something like WinMTR from inside your LAN?

[infinidim]

Cynric,

I actually have SAMKNOWS / OFCOM equipment on my site which monitors the link all the time. It is showing lot's of issues with latency and dropping of the WAN. Looks to me that there are some real issues with the Vodafone broadband in our area and over the Openreach access network to the Vodafone core. The errors come in waves which is really had to find problems in and then you have the issue that the Vodafone front line support only follow their scrips...

[infinidim]

Cynric,

Since I last posted on the forum I had a conversation with one of the Vodafone support people. He actually made some changes on Ultra Hub and updated the hub to a later version of the firmware than I had on it - before this update I checked for an update but there wasn't one. I also changed the events I could see in the Ultra Hub event log as well to just see WAN and LAN. It is much more readable now.

What I do see is -

8/26/2024 12:50:09 "HM_QS_WAN_RA_VC": 0 wan
08/26/2024 12:50:09 "HM_QS_WAN_RA_VA": 0 wan
08/26/2024 12:50:09 "HM_QS_WAN_RA_SL": 0 wan
08/26/2024 12:50:09 "HM_QS_WAN_RA_GPS": 0 wan
08/26/2024 12:50:09 "HM_QS_WAN_RA_GP": 0 wan
08/26/2024 12:50:09 "HM_QS_WAN_RA_GC": 0 wan
08/26/2024 12:35:28 Removing intercept reason: "wan_down" wan
08/26/2024 12:35:27 Device:pppoe-wan

But don't see the WAN port go down at all. There are also not as many errors across the Vodafone network as well.

What ever was changed seems to have improved the situation.

Latency is still quite high but I am sure we can live with this until we get Cityfibre access to the Vodafone network. Cityfibre are doing more work from tomorrow in our village - they are digging up the road that our close is off - so hopefully it won't be long before Vodafone say we can have it. I have talked directly with Cityfibre planners and they expect sometime before the end of the year.

[Cynric]

@infinidim Interesting. I suppose you may have hidden the connection failure depending on which elements are excluded from the log. But it does make me think of a signal issue, timeout or noise. But that's going to need some proper kit on the line. 

At least you have an idea when fibre is coming. In my town we still haven't got a fixed date.

[infinidim]

@Cynric Had another drop out last night. I have been also checking the ES, SES and UAS counts and there are some interesting figures appearing there. Here is a screenshot of the router errors after the failure last night -

It really makes interesting reading

The other thing that I have noticed is that the DSL link stays up yet the PPPOE level falls over. And example of this is -

DSL uptime over 2 days

PPP session uptime is only just over 9 hours as of now.

The conclusion I have made from this is the the Openreach FTTC copper link is stable (it was 2 days ago that The router was restarted) yet it is something to with the termination of the PPPOE path.

Hopefully someone will talk to me in Vodafone who actually knows what they are doing.

Keep safe 

[infinidim]

Vodafone still haven't found the issue.

Still seeing random high latency on the Openreach FTTC  provided access.

Still seeing 3/5 good days then 2/3 bad days for drops on access circuit but only for short periods.

Will keep this topic updated as and when I get it resolved.

Regards

Infinidim