Router restarting itself every 350 minutes, even after replacement

[sgb27]

Apologies in advance for the long post, but I have tried a lot of things already.

I have FTTP with a SR203 hub. I have a moderate number of devices connected to it (perhaps 30), many hard-wired through a couple of switches attached to the router. I also have an analog phone connected.

Anyway, about 3 weeks ago we noticed the internet would drop out (not WiFi) occasionally, then come back in 2-3 minutes. After a bit of checking, I found the router was rebooting itself, the logs confirmed that it had rebooted itself. Thinking the router or PSU was failing, I asked Sky to send me a new router. Yet the problem was exactly the same with thew new router and PSU installed! I hadn't added or changed anything on my network recently.

I then started investigating further, sending the logs to another device on the network to log (as the router seems to only show the log entries since restart, which is not very helpful for debugging). This revealed something very strange, the router was rebooting itself almost exactly every 350 minutes after the last restart. If I restarted it manually, it would go for 350 minutes, then restart itself.

So as any good trouble-shooter would do, I then disconnected half of the devices from the network to see what would happen, thinking it was something I had connected causing it to restart. It still restarted with half the devices, but less predictably, 526, 527, 971 and 1050 minutes between restarts. Note these numbers, as 526,527 are 1.5x 350 minutes, and 1050 is 3x 350 minutes, too much of a coincidence surely? Something is doing something at these specific time intervals after a reboot of the router.

I even ran wireshark and captured packets right before the restart, but nothing appeared unusal, just the last ping request from my computer that never got answered, then a bunch of ARP requests asking where the router had disappeared to!

Now I'm stumped, it's like the router is doing "something" every 350 minutes after startup, and the more devices I have connected, the more likely it is to restart itself at that time. 

Now onto the logs. Only once does anything interesting show up in the logs, right before it restarted I got this once. It might be a red herring though, a correlation rather than causation.

Nov 29 05:53:41 skyhub.ihr skytelemetry Device Doctor reboot. Lock duration: 608 secs, proc: voice, func: voicePerformFilterOperation, funcstep: 707 lockstatus: 1

And then I found out there is some more diagnostic data in the settings backup, but that always seems to be the same:

<X_SKY_REBOOT_CAUSE>
    <RebootTime>176xxxxxxx</RebootTime>
    <RebootReasonType>KLRT</RebootReasonType>
    <RebootReasonCode>1003</RebootReasonCode>
    <RebootInfo>CPE&#32;has&#32;been&#32;software&#32;resetted&#32;(possibly&#32;watchdog&#32;timeout,&#32;if&#32;no&#32;other&#32;indicators)</RebootInfo>
</X_SKY_REBOOT_CAUSE>

It's hard to find much online about any of this, but potentially KLRT means Kernal Restart or something? My guess is the router is getting stuck trying to do something at this specific time, and if it's under load/too many devices, it runs out of time and a watchdog fails and reboots itself for some reason? ChatGPT seems to agree with that, and suggested all the above was pointing towards an issue with the hub failing to renew its WAN session on Sky's side, and if that failed whilst it was under load it would crash. It suggested things like my line being misprovisioned and this could be due to DHCP/MER on Sky's side, or my line profile, but that might all be rubbish.

How on earth do I attempt to diagnose this further or get it fixed/looked at? Does anyone recognise the issue/symptoms? What are my next steps? Can Sky check for this type of issue from their end to rule out any issue, or is it just an inherant issue with the router firmware and I need to not connect so much to it?

[jamesn123]

Hi @sgb27 

Personally my next troubleshooting step would be to disconnect everything except 1 device and see if it still happens. I would make the one device a WiFi connected device as Sky's hubs have a history of not playing nicely with ethernet connected devices, so that could be your culprit. 

I understand this will be quite disruptive, perhaps do it overnight?

[JimM1]

@sgb27 I have setup my SR203 standalone, no wan connection and reset the unit at 6.30pm, will look at it tomorrow morning and use the Status uptime to see if it stay's up with no reset's as a runtime guide, was fresh FW updated last week and before that was not an issue on a FF500 FTTP connection but has been off for a while now.

[sgb27]

Hi @jamesn123 thanks for your reply. Sorry but troubleshooting is really slow, because I basically need to wait 6 hours each time to see if there is any improvement!

I've done similar to you suggest already, but also have done some other tests recently, let me explain.

First I disconnected almost everything (WiFi and wired), and then it didn't restart for over 12 hours, so I added back half my network. At that point it restarted after 526 and 527 minutes, which is almost exactly 1.5x the 350 minutes it restarts if I have everything connected. I then swapped over so that everything that was connected now was disconnected, and vice versa. It restarted after 971 and 1050 minutes, honestly I don't remember if the 971 was because I manually restarted it, but the 1050 is exactly 3x the 350 minutes it normally restarts.

So there doesn't seem to be one specific device causing the restart, more like a combination of devices, and the more that are connected, the more likely it is to restart at each 350 minute internal.

Yesterday I did another test. When it got near to the 350 minute interval, where I knew a restart was about to happen, I disconnected the WAN (from the router web UI) and pulled out the cable between the ONT and the hub. So the hub is just running isolated from the internet, but with everything my side still connected. It did restart, bang on 350 minutes again. So now my feeling is back to some issue on the LAN side, perhaps multiple devices are somehow overloading the hub, I have no idea where the 350 minutes is coming from though, it is like clockwork, 350 minutes after the hub is powered up.

My next test will be to replace the SR203 altogether with another non-Sky hub, and see how that goes. It's not a permanent solution as I need the voice port on the Sky hub, but at least it would help to diagnose, as the non-Sky hub probably has more options and diagnostic capability. The other thing to try will be to offload DHCP and DNS to a raspberry pi, see if that makes any difference, then a permanent solution could be to run a 2nd router after the Sky hub, the only complexity will be my VPN server that I'll have to try and figure out around the double-NAT issues that would introduce.

Anyway, I'm away for the next couple of weeks, so unlikely to be any further updates for a while now. Thanks for your suggestions though. I look forward to getting back to this when I return 🙄

[sgb27]

@JimM1 thanks for checking, what firmware version is it OOC? Mine is 7.04.0205.R

I can't see how to tell when the firmware was updated.

[JimM1]

@sgb27 Got the same version, going to say was updated couple weeks ago, that is something not a known factor unless someone is watching a sky hub and post's it up, just checked the hub at 16hrs running still there but it is all standalone so not on the wan, nothing strange on it at all, both bands up and no self hub reset.... Will look again after 6.30pm for the 24hr mark! Will also capture the stat's and attach the posted picture at that point for you, and also connect on the both bands to fill in the stat's fields! Can do Ethernet connection so will double the pic, no Lan and with Lan!

[JimM1]

@sgb27 SR203 Stat's below once cleared for viewing.

2.4Ghz band5Ghz band

[JimM1]

@sgb27 And the Ethernet connected to the SR203.

SR203 Ethernet capture!

[ugray]

I should add one detail about my setup. The only “change” I made in the past few weeks was adding more devices to the LAN and assigning them reserved IPs. Nothing structural, no new switches, no firmware tweaks, and no changes to WAN settings. Just more clients on the internal network.

I don’t think that alone should destabilise an SR203, but it’s the only variable that shifted before the reboots started. And even then, the behaviour doesn’t look like a simple load issue. Traffic levels are normal, CPU load isn’t exposed, and the reboot timing feels too consistent to be random or caused by a single device misbehaving.

So I’m in the same position as you:

• Full hub reboot

• Short outage

• No useful log entries right before the crash

Nothing added to the network that should realistically break a router

I still haven’t checked whether my reboot code matches yours, and I haven’t swapped hardware, so I can’t rule out a failing unit. But given the pattern and the fact that your replacement router behaved identically, it feels more systemic than a bad PSU or a dying hub.

If Sky can confirm whether MER sessions, DHCP renewals, or any scheduled processes on their end line up with these resets, that would be a useful next step. Right now it’s hard to tell whether this is a firmware bug triggered by a busy LAN or something upstream causing the hub to choke and restart.

If you get any updates from Sky, I’d appreciate if you could share them — your findings are the closest match to what I’m seeing.

[Chrisee]

@ugray the SR203 has been around for roughly 6 years and your's is the first such report I can remember seeing. I was sent one of the orginal batch to test and subsequentally was sent a second one as a Sky customer. I used both for quite a few years with no similar issues. Sky are estimated to have.  over 3 million of these units in use which are broadly reliable. Sky control the firmware each unit runs and it is quite common to have multiple releases in the fleet simultaenously as units are updated in batches. Given you have had two examples which have the same issue implies it was not caused by a faulty unit. 

The hub is designed for domestic use and is not the most powerful unit and there are well proven reports of the processor not being sble to cope with too many multiple simulteanous demands. The hub is pretty basic for example there is no QOS available unlike the newer SR213. The number of devices connected is far less important than the traffic they generate. IOT equipment cause minimal traffic so are not usually an issue but other devices vary. However it is known that for example more than 8 Sky Q nodes on a network can cause instability - the hub counts as a node, as do each Sky Q box and Q booster this is almost certainly down to processor.

I have no idea what is causing your issue as I dont know what you are running but I think simply you maybe overloading the processor causing a reboot. If you want the voice line I would suggest buying a better router and using that to manage your network connecting to the SR203 in a double NAT set up which causesxfar fewer issues in practice than most people think. Bybusing the SR203's DMZ you can minimise double firewall issues. 

If you simple want to keep the landline number investigate porting it to a VOIP provider which I believe can be done unlike the other way.

[JimM1]

@sgb27 As a little foot note, quick check 39Hrs up so far, will catch it at the 2 day mark after the 6.30pm, so sitting no device connected Hub sure does not care, wifi is all in line with the Asus Routers/Mesh that is connected, so apart from when i drop off and check the Hub it's not getting any problems as far as it is concerned!

Never had any issues with the SR, it sat double NAT for about 6 weeks in Mar/Apr when back to sky never threw a wobble in the slightest. and on the Lan Reservation list at that point had 33 ip's fixed and assigned, with a few extra that would pop in now and again, like son/daughter in-law/grand daughter/additional family members etc as they visited and got direct DHCP device assigned.

[sgb27 · Accepted solution]

I think I found the issue and have resolved it. I had a brainwave when thinking about the timings. What if I had 3 devices on my LAN that were somehow "filling up" some buffer on the router, and when full the router would reboot? This would perfectly explain when I reduced the number of devices to 2, the time between restarts would be 3/2, ie 1.5x longer, and when reduced to one, it would be 3x. This is exactly what I saw in my testing. 

So what do I have 3 of on my network? TP-Link AX53/55 access points. I checked them, all set as AP mode (rather than router mode), but the logs showed them communicating almost every second with the Sky router about DHCP and leases. I assume the Sky router just had since buffer overflow or something and crashed after 350 minutes of this from three access points. 

Anyway, I assigned the WAPs static IPs, and manually entered the IP in each of them, I also turned the DHCP server to "Off" rather than "Auto". After a reboot of everything, no more entries every second in the WAP logs, and the router is up for 18+ hours now. 

[JimM1]

@sgb27 You are always nearly better doing a static IP assignment on the LAN tab off the SR203, but should be no need to do it on the AP as well but what every it takes, AP mode should also work if you are not routing?  Will capture a 24hr on the hub, and keep it as is for a week, it's not causing any wireless wifi issues sitting were it is, also gives it a good run again as not been in use since end off April! Would say you have found the issue now!👍

The tp-link AX53/55 are really routers and may not be working true AP mode to a SR Hub!