pfsense behind Vodafone wifi-hub (ADSL) / double NAT / 50% plus packet loss all the time

[cmu2808]

This is a really strange one... so just moved into a new property and had to settle for a basic ADSL connection, Vodafone recently provided the connection to my property together with their standard wifi-hub... upon initial inspection simply running of the wifi-hub everything seemed ok, however now that I've put my pfsense (running version 2.6.0) behind the wifi-hub I'm seeing a lot of packet loss.  

So the setup is exactly the same as it was in my previous property (where i had no issues). I basically have the pfsense WAN interface connected to one of the Vodafone wifi-hub ethernet ports and wifi-hub sits in the same network range as the pfsense WAN interface IP - Vodafone wifi-hub is .1 and the pfsense WAN interface IP is .254.  I've placed the .254 address in the "DMZ"  and I disabled all functions on the Vodafone wifi-hub, turning it essentially into a ADSL modem.  But still have the annoying double NAT config!  

Doing a continues ping (towards 1.1.1.1) from my desktop, connecting to the internet via my AP (which connects into a switch, which connects into pfsense) - I'm getting 50% plus packet loss. 

The second hop below is the Vodafone wifi-hub, locally going from pfsense to wifi-hub it seems to be registering 65% loss!

Thinking this might be a wifi issue, I run a cable from my switch to my desktop - same result!

Thinking it might be a bad cable between the pfsense WAN interface and Vodafone router, I swapped cables multiple times - same result!  I've tried the different ethernet ports on the Vodafone router always the same result - packet loss!  Even played with the MTU and negotiation settings on the pfsense WAN interface - no luck!

Thinking this is pfsense related, I've done the same continues ping test connecting my desktop directly over the Vodafone wifi-hub, over both wifi and cabled connections registering 5-10% loss - which I guess is acceptable. 

Although at some point yesterday I was registering, 25% loss connecting directly over the Vodafone wifi-hub  - which led me to think there was something wrong with the line or the router - Vodafone engineer on the way to check line and router, who I doubt will find anything wrong.

Currently running quality monitor via thinkbroadband.com to make sure the issue is definitely not related with the connection itself.  

Now despite all of this packet loss I'm registering, when connecting to internet via my pfsense, don't seem to be having any connectivity issues.  No drops or buffering during streaming, no drops over VoIP, VPN connectivity seems solid,  So I'm really confused???

My next move I think is to order a simple ADSL modem (Draytek Vigor 130) and remove the Vodafone wifi-hub out of the equation, got the login details from Vodafone, but looking for alternative suggestions before I go ahead with that plan and spend the money.  

[cmu2808]

Well there is a problem between pfsense and Vodafone wifi-hub for sure.  Because loss goes up 50% plus (this apparently can happen in double NAT configs). 

No extension socket in flat - yes I'll check with engineer on Friday.  Thanks for the suggestions!  

[CrimsonLiar]

Starting from the top:

Using DMZ and turning off DCHP, DNS, and all of that does not make a router act as a modem - you still have NAT (Network Address Translation) in full flow, and hence your double NAT.

Constant high-speed pinging of many DNS servers will after a short period get your traffic downgraded or even ignored (ie packet loss).  On Vodafone, if you are testing your connection for a short period then use their commonly known default DNS servers at 90.255.255.90 or 90.255.255.255 and do it in a considerate manner!  A suggestion: if you could run MTR via SSH on the PFsense box you'd be able to see if that 65% packet loss and the associated 85ms maximum latency has to do with the local NAT setup (I suspect it does).  *Doesn't pfsense have some inbuilt connection monitoring ability?*  

---

Seriously@router:/tmp/home/root# mtr -rc 20 90.255.255.255
Start: 2022-03-24T18:44:55+0000
HOST: router                                 Loss% Snt Last  Avg Best Wrst StDev
1.|-- host-212-158-250-39.dslgb    0.0%     20   8.3   8.8   8.0  12.6      1.1
2.|-- 84.9.214.194                           0.0%     20   8.9   8.8  8.3     9.1      0.2
3.|-- 90.255.255.255                       0.0%     20 10.0 10.0  9.6  10.9       0.3

---

Also what happens if you bite the bullet and run path ping or MTR when directly connected to the VF router?

*Running an Asus RT-AC86U modem with a Draytek Vigor 130 modem!  I also have some limited experience of pfsense - most of it bad - and am likely to upgrade the router in the near future to a Mikrotik RB5009UG+S+IN.

[cmu2808]

Yes correct! Obviously there is a double NAT, with my initial comment "I disabled all functions on the Vodafone wifi-hub, turning it essentially into a ADSL modem." I was simply trying to suggest that I'm not using the Vodafone wifi-hub networking features and off-load all functionality to the pfsense.

Also, yes I'm aware of the downgrade behaviour of constant pinging towards DNS servers, but you have you to remember, I did not have this issue before, when my pfsense was behind the Virgin business router (same double NAT).  Zero packet loss!  And pfsense does have monitoring abilities, I have multiple monitoring G/Ws configured (towards well known DNS servers) - all reporting packet loss!  See below.  This wasn't happening before either (my G/Ws were always up).  Strange thing is, there is high packet loss between the pfsense and the Vodafone wifi-hub.  That's Layer 2 with no firewall in between, simply pfsense .254 pinging Vodafone wifi-hub .1 in the same network.  Another behaviour I didn't have prior to Vodafone.  

I've also just SSH-ed on to the pfsense and run a ping towards 1.1.1.1, packet loss!  I've also run a ping towards 90.255.255.90, less but still packet loss.

I've done multiple tests of running MTR over VF router before (some screenshots in earlier replies) - latest test towards 1.1.1.1 is producing 2% loss, I've seen this figure to be between 8% and 25% last few days.  But again that could be the downgrading effect.  

Pinging 90.255.255.90 seems to produce 0% loss.  Which is inline with what you were suggesting earlier in your reply.  

In conclusion, most likely the line is fine but issue is between pfsense and the VF router - hence why I opened this thread in the first place.  As I stated before, I didn't have this issue with the Virgin router, so provided the Openreach engineer finds nothing with the overall line health tmrw (I'll get the engineer to install a test socket and perhaps re-do the messy cabling within socket) - I will go and get myself a Draytek Vigor 130 and take the VF router out of the equation.  Maybe the double NAT causes more problems with VF router than it does for instance with the Virgin router - which is why I didn't have these problems before.

[Jayach]

---

@cmu2808 wrote:

Maybe the double NAT causes more problems with VF router than it does for instance with the Virgin router - which is why I didn't have these problems before.

---

Were you double NATing with the Virgin router though? The Virgin router has a true modem mode.

[cmu2808]

Yes I was 100% double NATting, I couldn't use the modem mode because of my static IP requirement.  I had to either drop the static IP or had to have 5 static IPs to be able use modem mode.  Can't remember the exact reason to be honest but was on the phone with support for a while.  NOTE - I was on Virgin Business not residential.  

[CrimsonLiar]

NAT operates at OSI layer 3, and the whole problem is that Double NAT tends to break the layer 3 packet headers.  Depending on how hardware and software are configured some devices handle those breakages better than others.  That the old Virgin equipment did a better job, doesn't mean that the VF equipment is failing to perform within spec!

*I trust you are also following this through on the pfsense support forums as they might have more answers for you.

**What are the specific strengths of pfsense that the VF platform fails to deliver (genuine question, I simply cannot live with the constraints of the VF router)?

[cmu2808]

I never said VF router is not delivering within spec.  It simply isn't working for my needs.  And yes I think I will open a thread in the Netgate forums now to see if I can resolve issues that come with the Double NAT.   

At a minimum VF router doesn't do VLANs and VPN, while it acts as a firewall, is not actually a proper firewall.  I simply prefer a segmented properly firewalled home network, a bit like what you would get in a corporate network and basic ISP routers usually cannot provide this functionality - hence the use of pfsense behind VF router.    

[cmu2808]

And the Open Reach engineer just told me there is an issue on the line, so helps to be persistent in these matters... Vodafone were absolutely sure the line was good!  I'll update status on pfsense end and packet drops once line issues are resolved.  

[Jayach]

@cmu2808 wrote:

Yes they are... i still see ping drops in cmd with the app not running.

@Jayach wrote:

In that case you have a problem with your line, nothing to do with the pFsense.

---

Thought you might.😀

To be fair, the Openreach engineers have better test equipment than Vodafone do.

[CrimsonLiar]

So far as I'm aware the VF routers offer the pretty much standard SPI (stateful packet inspection) type firewall on the router, which double NAT sometimes breaks at the router to router boundary.  But combine that with the basic filtering DNS, and it's a solid enough solution for most things (other than adblocking).  I believe the pfsense firewall though can be far more encompassing and pro-active!

*SPI is regularly misunderstood on these forums - any want a debate on this its probably better in a separate thread!