MC7354 inside GX450 keeps loosing service

Sierra Wireless AirPrime Modem Card Topics
Forum rules
Use the SEARCH function for related topics PRIOR to posting a new topic on the same subject.
Post Reply
axel1
Posts: 15
Joined: Mon Jun 21, 2021 2:40 pm
Has thanked: 1 time
Been thanked: 0

MC7354 inside GX450 keeps loosing service

Post by axel1 »

Hi all,

I've been testing out an AT&T 15GB hotspot plan on my Sierra GX450 (a vehicular router with an integrated MC7354).

I've got it up and running on the AT&T network, but I keep loosing service after an inconsistent amount of time. I'm running on a temporary antenna here on my desk, and signal levels are fine, but about five times in the past 6 hours I've lost connectivity with a "no service" status on the modem. I'm not moving the radio or otherwise changing signal, so I'm not sure what's going on. As far as I can tell, the only way to recover from this state is a hard reboot of the router.

Here's what my status page looks like when everything is good:
Image

And here's when things have died:
Image

As you can see there's almost no change in the signal levels, so something non-RF is going on here I think. MC7354 is on the latest AT&T firmware so I don't think that's the issue.

I'd love to know any good troubleshooting steps to figure out what's going on. I'm guessing it's likely a configuration issue somewhere with this new service. I was previously using this same router on US Mobile and never saw this issue. All I had to do for the switch to AT&T was update the APN to "Broadband" and things started working, so I can't think of what else could be going on.

Help is greatly appreciated!
axel1
Posts: 15
Joined: Mon Jun 21, 2021 2:40 pm
Has thanked: 1 time
Been thanked: 0

Re: MC7354 inside GX450 keeps loosing service

Post by axel1 »

Here's the output from the debug information tool in the router's web gui (with redactions) when the router gets stuck in the "no service" state:

Code: Select all

ATI
Manufacturer: Sierra Wireless, Incorporated
Model: MC7354
Revision: SWI9X15C_05.05.58.00 r27038 carmd-fwbuild1 2015/03/04 21:30:23
MEID: 35xxx
ESN: 128xxxxxxxx, 804xxxxx
IMEI: 359xxxxxxxxxxxxxxx
IMEI SV: 18
FSN: J86xxxxxxxxxxxx
+GCAP: +CGSM


OK
AT!GSTATUS?
!GSTATUS: 
Current Time:  1461Temperature: 37
Bootup Time:   0Mode:        ONLINE         
System mode:   LTE        PS state:    Attached     
LTE band:      B4     LTE bw:      10 MHz  
LTE Rx chan:   2000LTE Tx chan: 20000
EMM state:     Registered     Normal Service 
RRC state:     RRC Idle       
IMS reg state: No Srv  

RSSI (dBm):    -53Tx Power:    0
RSRP (dBm):    -81TAC:         486C (18540)
RSRQ (dB):     -10Cell ID:     0478E217 (75031063)
SINR (dB):     10.8


OK
Not sure if that helps with the troubleshooting at all. I'm guessing there's some AT commands I can run to get some more info as well.
User avatar
Didneywhorl
Posts: 3609
Joined: Fri Mar 23, 2018 5:37 pm
Location: USA
Has thanked: 1359 times
Been thanked: 754 times
Contact:

Re: MC7354 inside GX450 keeps loosing service

Post by Didneywhorl »

RRC State: Idle usually means the network is letting you login but is blocking your data, especially since you have a PS State: Attached.

Your SINR (Signal to noise ratio) isn't great. You'll want that to be closer to 20. https://thewirelesshaven.com/cellular-signal-guide/

Oddly though, all other stats are great. RSRP, RSRQ, RSSI.

If rebooting the modem fixes it it may be an ATT issue. Have you tried testing your connection over a simple VPN or proxy?

https://ngrok.com/

If you can use a proxy or VPN and it doesn't happen, then that is another sign it's ATT causing your issues, not your router.
axel1
Posts: 15
Joined: Mon Jun 21, 2021 2:40 pm
Has thanked: 1 time
Been thanked: 0

Re: MC7354 inside GX450 keeps loosing service

Post by axel1 »

I also noticed SINR being lower compared to the other signal levels. I figure there may be a bit of local interference in the area. I'm also not testing this on the world's greatest antenna, but it should be perfectly serviceable for a downtown location with sites all over the place. (Laird DBA series)

I've noticed now after letting the router sit overnight that the connection does eventually come back after ~ 15 minutes or so. But then it'll do the same thing a little later. Very strange.

Know of any extra logging or commands I can check out to better see what's going on? I think I can access the MC7354's AT terminal via the GX450's web GUI.
User avatar
Didneywhorl
Posts: 3609
Joined: Fri Mar 23, 2018 5:37 pm
Location: USA
Has thanked: 1359 times
Been thanked: 754 times
Contact:

Re: MC7354 inside GX450 keeps loosing service

Post by Didneywhorl »

AT!GSTATUS? is really your best command for overall info. That and any system log the router may have.

If you have a lot of cell towers with the same bands in your area, that will explain the noise. It's called cellular band cross-talk. Your modem is hooking up to one tower and the other towers are broadcasting in the same frequency range and causing noise at your radios receiver.
axel1
Posts: 15
Joined: Mon Jun 21, 2021 2:40 pm
Has thanked: 1 time
Been thanked: 0

Re: MC7354 inside GX450 keeps loosing service

Post by axel1 »

So here's that command again when I'm in the no service state. This time it shows as RRC Connected but IMS as No Srv.

Code: Select all

AT!GSTATUS?
!GSTATUS: 
Current Time:  3094Temperature: 39
Bootup Time:   0Mode:        ONLINE         
System mode:   LTE        PS state:    Attached     
LTE band:      B4     LTE bw:      10 MHz  
LTE Rx chan:   2000LTE Tx chan: 20000
EMM state:     Registered     Normal Service 
RRC state:     RRC Connected  
IMS reg state: No Srv  

RSSI (dBm):    -50Tx Power:    0
RSRP (dBm):    -77TAC:         486C (18540)
RSRQ (dB):     -11Cell ID:     0478E217 (75031063)
SINR (dB):      5.4


OK
I've enabled some extra logging on the GX450 so we'll see if anything stands out from those the next time it happens.
axel1
Posts: 15
Joined: Mon Jun 21, 2021 2:40 pm
Has thanked: 1 time
Been thanked: 0

Re: MC7354 inside GX450 keeps loosing service

Post by axel1 »

So here's the log from the most recent connection drop.

Here's the two entries prior to the connection failing. Just good signal reports and nothing abnormal.

Code: Select all

Jun 22 16:05:58 notice ALEOS_LINKMGMT_linkmon: Current: RSSI=-51, RSRQ=-13, RSRP=-75, SINR=5.8
Jun 22 16:20:58 notice ALEOS_LINKMGMT_linkmon: Current: RSSI=-50, RSRQ=-12, RSRP=-75, SINR=10.1
Then the connection drops at 16:23Z. The ping test warning is the first thing I see.

Code: Select all

Jun 22 16:23:04 warning ALEOS_LINKMGMT_servicemon:  Ping Test failed to reach 1.1.1.1, Cellular is not connected
Jun 22 16:23:04 notice ALEOS_LINKMGMT_linkmon: Radio Connection State - No Service
Jun 22 16:23:04 notice ALEOS_LINKMGMT_linkmon: New Link Status - No Service
Jun 22 16:23:10 warning ALEOS_EVENTS_dpRptGen: No GNSS Fix. Quality Indicator is 0 and 0 Satellites in Use. 0 seconds passed without fix.
Jun 22 16:23:10 warning ALEOS_EVENTS_dpRptGen: monitorFix : Lost fix at: Tue, 22 Jun 2021 16:23:10 UTC
Jun 22 16:23:11 alert ALEOS_CONNECTMGMT: IPV4 Disconnect Handled
Jun 22 16:23:21 warning ALEOS_EVENTS_dpRptGen: No GNSS Fix. Quality Indicator is 0 and 0 Satellites in Use. 0 seconds passed without fix.
Jun 22 16:23:21 warning ALEOS_EVENTS_dpRptGen: monitorFix : Lost fix at: Tue, 22 Jun 2021 16:23:21 UTC
Jun 22 16:28:03 notice ALEOS_LINKMGMT_linkmon: Network Watchdog - No reboot, Network Watchdog disabled by application [ 5 minutes ]
Jun 22 16:33:04 notice ALEOS_LINKMGMT_linkmon: Network Watchdog - No reboot, Network Watchdog disabled by application [ 10 minutes ]
Jun 22 16:34:44 warning ALEOS_LINKMGMT_servicemon:  Ping Test failed to reach 1.1.1.1, Cellular is not connected
Jun 22 16:35:58 notice ALEOS_LINKMGMT_linkmon: Current: RSSI=-50, RSRQ=-12, RSRP=-75, SINR=9.5
Jun 22 16:38:04 notice ALEOS_LINKMGMT_linkmon: Network Watchdog - No reboot, Network Watchdog disabled by application [ 15 minutes ]
Jun 22 16:43:04 notice ALEOS_LINKMGMT_linkmon: Network Watchdog - No reboot, Network Watchdog disabled by application [ 20 minutes ]
Jun 22 16:46:25 warning ALEOS_LINKMGMT_servicemon:  Ping Test failed to reach 1.1.1.1, Cellular is not connected
Jun 22 16:47:10 warning ALEOS_EVENTS_dpRptGen: No GNSS Fix. Quality Indicator is 0 and 0 Satellites in Use. 0 seconds passed without fix.
Jun 22 16:47:10 warning ALEOS_EVENTS_dpRptGen: monitorFix : Lost fix at: Tue, 22 Jun 2021 16:47:10 UTC
Jun 22 16:48:04 notice ALEOS_LINKMGMT_linkmon: Network Watchdog - No reboot, Network Watchdog disabled by application [ 25 minutes ]
The fact that I'm also getting a GNSS failure at the same time (I do have a GPS antenna connected and outside) makes me wonder if the module is faulting out somehow. Not sure how I would test that further though.

And here's the log sequence once things come back up:

Code: Select all

Jun 22 16:56:25 notice ALEOS_LINKMGMT_linkmon: Radio Connection State - Connected
Jun 22 16:56:25 notice ALEOS_LINKMGMT_linkmon: New Link Status - Network Link Down
Jun 22 16:56:33 notice ALEOS_CONNECTMGMT: *** Network Connection successful. WAN IP [Cellular] : 10.243.xx.xxx
Jun 22 16:56:33 notice ALEOS_CONNECTMGMT: *** IP UP 10.243.xx.xxx : wwan0
Jun 22 16:56:33 notice ALEOS_CONNECTMGMT: Setting WWAN MTU size to 1430
Jun 22 16:56:34 notice ALEOS_CONNECTMGMT: GDNS is restarting dnsmasq
Jun 22 16:56:45 notice ALEOS_LINKMGMT_linkmon: New Link Status - Network Ready
Jun 22 16:56:45 warning ALEOS_SERIAL_HMC: not in tcp mode current mode 0 
Jun 22 16:56:50 alert ALEOS_LAN: MTU Source Auto - Set DHCP option 26 MTU: 1430
Jun 22 16:56:50 alert ALEOS_LAN: MTU Source Auto - Ignore Opt 26 MTU size 1500
User avatar
Didneywhorl
Posts: 3609
Joined: Fri Mar 23, 2018 5:37 pm
Location: USA
Has thanked: 1359 times
Been thanked: 754 times
Contact:

Re: MC7354 inside GX450 keeps loosing service

Post by Didneywhorl »

Your SINR levels fluctuate a LOT. All the other stats seem to be pretty steady.

I'm not sure on the GPS stuff.
axel1
Posts: 15
Joined: Mon Jun 21, 2021 2:40 pm
Has thanked: 1 time
Been thanked: 0

Re: MC7354 inside GX450 keeps loosing service

Post by axel1 »

Part of the SINR fluctuation is likely band changes. I see it regularly switch between bands 2, 4, and 17, among others. However I don't think that's the issue with my connection drops.

I'm not sure what else to test at this point. AT&T support will be useless I'm sure. Perhaps the Sierra Wireless forums might be able to shed some light.
mtl26637
Posts: 321
Joined: Mon Jul 29, 2019 12:35 pm
Has thanked: 3 times
Been thanked: 102 times

Re: MC7354 inside GX450 keeps loosing service

Post by mtl26637 »

Dont know much about your specific modem, however, isn't it usually preferred to use 'generic' firmware rather than 'carrier' firmware? I'll let someone else chime in on that but wanted to throw it out there in case it was overlooked. You would think that the carrier firmware would be the way to go but that isn't always the case.
axel1
Posts: 15
Joined: Mon Jun 21, 2021 2:40 pm
Has thanked: 1 time
Been thanked: 0

Re: MC7354 inside GX450 keeps loosing service

Post by axel1 »

I'm giving the generic firmware a shot now. We'll see if it makes any difference.
axel1
Posts: 15
Joined: Mon Jun 21, 2021 2:40 pm
Has thanked: 1 time
Been thanked: 0

Re: MC7354 inside GX450 keeps loosing service

Post by axel1 »

No dice. Generic firmware installed and the same issue just happened again. Same exact log entries as before too.

GPS does seem to die at the same time as cell service. Maybe the entire modem module is experiencing some kind of fault? No clue how I'd even begin troubleshooting that.
mtl26637
Posts: 321
Joined: Mon Jul 29, 2019 12:35 pm
Has thanked: 3 times
Been thanked: 102 times

Re: MC7354 inside GX450 keeps loosing service

Post by mtl26637 »

Couldn't be that easy right. Unfortunately the logs don't help much in showing the problem. Possible lack of supply power causing it to randomly drop out? Hopefully someone else will jump in with some additional info to try too.
axel1
Posts: 15
Joined: Mon Jun 21, 2021 2:40 pm
Has thanked: 1 time
Been thanked: 0

Re: MC7354 inside GX450 keeps loosing service

Post by axel1 »

So I started plotting ping data last night, just to get a feel for how the modem was behaving. The following graph is ping times from the last ~12 hours, where the big triangles are loss of connection (average ping time slowly increasing as no pings make it through)

Image

Notice:
  • How the losses of connection are identical in duration and time.
  • How after 2:30 this morning they stopped
Very strange behavior. I'll be watching it again today just to see if things are magically fixed or if the issue still persists.
User avatar
Didneywhorl
Posts: 3609
Joined: Fri Mar 23, 2018 5:37 pm
Location: USA
Has thanked: 1359 times
Been thanked: 754 times
Contact:

Re: MC7354 inside GX450 keeps loosing service

Post by Didneywhorl »

If the device is officially registered to the network, then the official firmware is usually best, but not always. It may be worth trying, and also making sure the firmware is up to date. With the 7354, I believe it only holds one image at a time. If you need the latest, I have them on hard drive.
axel1
Posts: 15
Joined: Mon Jun 21, 2021 2:40 pm
Has thanked: 1 time
Been thanked: 0

Re: MC7354 inside GX450 keeps loosing service

Post by axel1 »

I've got the latest firmware I believe, both the ATT & the generic ones. It actually has enough space to store two images at a time which is handy. SWI9X15C_05.05.58.00 is what I've got, I downloaded it directly from Sierra's site.

Also more drops today, seems that maybe I just got lucky last night. Or it's a cell issue related to congestion/usage on the specific cell the radio is talking to.

Image
axel1
Posts: 15
Joined: Mon Jun 21, 2021 2:40 pm
Has thanked: 1 time
Been thanked: 0

Re: MC7354 inside GX450 keeps loosing service

Post by axel1 »

Well I now have a Sierra MP70 on the way thanks to a lowball eBay offer, so I'll have a second modem (also with an MC7354) to compare with once that arrives. Still, it's going to be a few weeks before it shows up so I'd like to try and figure out what the heck is going on here in the meantime.
Adm1jtg
Posts: 395
Joined: Sat Aug 08, 2020 7:19 pm
Has thanked: 8 times
Been thanked: 116 times

Re: MC7354 inside GX450 keeps loosing service

Post by Adm1jtg »

I havent read through all the posts here but have you tried locking to just one band? If it is a congestion issue some bands get more congested then others.
axel1
Posts: 15
Joined: Mon Jun 21, 2021 2:40 pm
Has thanked: 1 time
Been thanked: 0

Re: MC7354 inside GX450 keeps loosing service

Post by axel1 »

Unfortunately I can't find a way into the AT terminal for the modem itself to test that. The GX450 has band selections but they're more general for regions and modes as opposed to specific bands.

Not a bad theory. Seems like the radio is almost always on Band 4 so trying a different one could yield different results.
axel1
Posts: 15
Joined: Mon Jun 21, 2021 2:40 pm
Has thanked: 1 time
Been thanked: 0

Re: MC7354 inside GX450 keeps loosing service

Post by axel1 »

So again over the past day, lots of connection drops throughout the day and then absolutely zero connection drops between midnight and 9AM. Then they start up again. Same thing happened the day before.

Image

At this point I think it must be some kind of local issue with the cell sites near me. I can't think of any reason it would work fine all night and then start right back up with the hourly drops.
User avatar
Didneywhorl
Posts: 3609
Joined: Fri Mar 23, 2018 5:37 pm
Location: USA
Has thanked: 1359 times
Been thanked: 754 times
Contact:

Re: MC7354 inside GX450 keeps loosing service

Post by Didneywhorl »

I think your right. I wonder if the tower is freaking out under congestion load.
Post Reply

Return to “Sierra Wireless Modems”