Network outage same time every day

Talbs13 · 9 Jul 2015 at 15:03

Hey,

Got a funny one. Internal network goes down same time everyday. 12.02pm and 7.02am for the past 5 days. Doing a power cycle on the core switch fixes the issue. We replaced the core switch but it's still happening.

Setup is 4 3COM 4500 switches, 1 Core, 3 stacked, 2 VLAN's, Data and VOIP.

Installed packeteer and there is a massive spike in outbound traffic at those times. Unfortunately it doesn't show me where it's coming from / too.

No logs on switch, 'display logbuffer' doesn't show anything wrong.

It starting to feel like a virus?

Anyone had anything like this happen to them? Anything I can check. Next troubleshoot steps is set up SPAN and analyse the packets somehow.

Any help / suggestions appreciated.

Caged · 9 Jul 2015 at 23:58

In the long term you want to throw those switches away because they are garbage.

Can you turn on the logging feature and write logs out to a syslog server and see what's going on?

If you're totally losing network connectivity it sounds like the root bridge might be changing or some other need to change STP topology and the switch designated as the root bridge can't handle it.

Do you have BPDU guard (or the 3com equivalent) turned on for all the ports that don't have other switches connected? If the switches are full on crashing (can you get console access when they are down?) then I don't think you are going to be able to detect the problem happening at the IP level.

Caged · 10 Jul 2015 at 00:06

Also why have you designated one switch as a 'core' and then got yourself a stack of 3 when all switches are identical? Why not a stack of 4?

Talbs13 · 10 Jul 2015 at 11:30

Hey, thanks for the reply. I've turned syslog on and will check when it next goes down. I also haven't checked if I can console on when it goes down, will check today.

Thanks

Talbs13 · 10 Jul 2015 at 14:21

Little update. Syslog reported that Port 15 went DOWN for 3 seconds. Port 15 is a file server. Error log reports error with network bridge.

Packetshaper reported huge spike in outbound IPv6 at 12.01. We don't use IPv6 only 4.

Going to update drivers.

16-bit · 10 Jul 2015 at 15:48

If that doesn't sort it, do you have IPv6 still enabled? always good to check the scheduled tasks and event log for anything weird.

anything I don't mind · 13 Jul 2015 at 16:21

Got to be something on a schedule. WSUS or my first guess would be a rubbish antivirus software like mcafee. Ive seen mcafee bring a network to a halt at update time which by default is lunch time.

tris179 · 11 Aug 2015 at 21:36

You could try wireshark and capture the information at that time.. Could be a broadcast of some description.

nutcase · 12 Aug 2015 at 08:37

anything I don't mind said:
Got to be something on a schedule. WSUS or my first guess would be a rubbish antivirus software like mcafee. Ive seen mcafee bring a network to a halt at update time which by default is lunch time.

GFI used to do that to us.

peter212693 · 12 Aug 2015 at 22:32

Bit of a long shot but check for VSS / Shadow Copies on said Fileserver. 7am and 12pm are the default times windows is set to create a volume snap.

If its enabled on that server you could try moving the times from default and see if the issues moves to the new times. I have seen VSS causing servers to lockup and stop responding but not sure about the network traffic spike but as I say it probably a long shot.