Hi there
Hope some kind soul out there can help or point me in the right
direction.
For the last month or so our users have experienced a network failure
about once per week. Rebooting the main 48 port unmanaged switch
(netgear) resolves the problem.
I would like to inspect what's going on at the switch to try and get to
the bottom of the failure. I've downloaded and installed ethereal,
however, I have no idea what I should be looking for in the log files.
Can anyone help?
At present we have our W2K servers running on a copper gigabit switch,
this switch is then connected to the 48 port switch. Various other
switches are downstream of this switch.
I plan on hanging a hub between a downstream switch and its uplink to
the 48 port switch and then using ethereal to analyse what's happening.
However, I've no idea what I should be looking for! Can anyone help
please??
Thanks
BC
DComTalk.com
Discussion of VoIP, VPN, Video Conferencen, DSL and other data commucations.
Network Failure - No Idea How to Troubleshoot
20 posts
• Page 1 of 2 • 1, 2
Re: Network Failure - No Idea How to Troubleshoot
by Robert Redelmeier » Thu Nov 10, 2005 12:20 pm
BC <bcharlton@pchenderson.com> wrote:
This is a very tough problem to troubleshoot. First, what
has changed? Any new hardware, software or usage pattern?
There are some simple hardware things to check: have you
tried swapping out the switch? Is it's powerfeed [UPS] good?
Are any of the ports dead? Or likely to die, 'cuz they're
on long/outdoor runs?
Unfortunately, this will only sniff traffic on that branch
of the network. And may not catch malformed packets. This
is why people buy managed switches.
When do the hangs occur? During heavy usage, idle times?
-- Robert
For the last month or so our users have experienced a
network failure about once per week. Rebooting the main
48 port unmanaged switch (netgear) resolves the problem.
This is a very tough problem to troubleshoot. First, what
has changed? Any new hardware, software or usage pattern?
There are some simple hardware things to check: have you
tried swapping out the switch? Is it's powerfeed [UPS] good?
Are any of the ports dead? Or likely to die, 'cuz they're
on long/outdoor runs?
I plan on hanging a hub between a downstream switch and
its uplink to the 48 port switch and then using ethereal
to analyse what's happening.
Unfortunately, this will only sniff traffic on that branch
of the network. And may not catch malformed packets. This
is why people buy managed switches.
When do the hangs occur? During heavy usage, idle times?
-- Robert
- Robert Redelmeier
Re: Network Failure - No Idea How to Troubleshoot
by Alexey G. Khramkov » Thu Nov 10, 2005 12:20 pm
"BC" <bcharlton@pchenderson.com> writes:
First of all, recall all new h/w, especially different (non netgear)
verdors. Sometime new NICs can kill the switch.
Any nonstandard stuff. Our programmers added some IP and TCP
options which killed the switch. Usually that options are not
documented by firmware vendor.
Check the MAC capacity. If it overburdened the switch starts to work
like the hub. Thus collisions can kill all domain of collisions.
Bad Thing (TM). I have old 10Mb hub only. Thus it has bottleneck
without doubts. Are you lucky and have 100Mb hub at least?
HTH,
agkhram
--
status = rule("RFC1855"); /* avoid flame wars */
For the last month or so our users have experienced a network failure
about once per week. Rebooting the main 48 port unmanaged switch
(netgear) resolves the problem.
First of all, recall all new h/w, especially different (non netgear)
verdors. Sometime new NICs can kill the switch.
I would like to inspect what's going on at the switch to try and get to
the bottom of the failure. I've downloaded and installed ethereal,
however, I have no idea what I should be looking for in the log files.
Can anyone help?
Any nonstandard stuff. Our programmers added some IP and TCP
options which killed the switch. Usually that options are not
documented by firmware vendor.
At present we have our W2K servers running on a copper gigabit switch,
this switch is then connected to the 48 port switch. Various other
switches are downstream of this switch.
Check the MAC capacity. If it overburdened the switch starts to work
like the hub. Thus collisions can kill all domain of collisions.
I plan on hanging a hub between a downstream switch and its uplink to
the 48 port switch and then using ethereal to analyse what's happening.
Bad Thing (TM). I have old 10Mb hub only. Thus it has bottleneck
without doubts. Are you lucky and have 100Mb hub at least?
HTH,
agkhram
--
status = rule("RFC1855"); /* avoid flame wars */
- Alexey G. Khramkov
Re: Network Failure - No Idea How to Troubleshoot
by BC » Thu Nov 10, 2005 12:20 pm
Thanks for the response guys. To the best of my knowledge nothing has
changed on the network. No new hubs / switches, no new cabling, not
even new workstations. The hangs seem to occur about once per week,
during working hours, but not at periods of peak activity. Reboot of
the 48 port switch temporarily solves the problem. I'm going to try
swapping out the switch this evening and see how things go....
Thanks again.
BC
changed on the network. No new hubs / switches, no new cabling, not
even new workstations. The hangs seem to occur about once per week,
during working hours, but not at periods of peak activity. Reboot of
the 48 port switch temporarily solves the problem. I'm going to try
swapping out the switch this evening and see how things go....
Thanks again.
BC
- BC
Re: Network Failure - No Idea How to Troubleshoot
by James Knott » Fri Nov 11, 2005 2:37 am
BC wrote:
I assume you mean no traffic is getting through? With ethereal, you can
monitor on one computer, while pinging it from others. Does it get
through? When you ping, can you see lights flashing on the NICs and
switch? If nothing's getting through the switch and only rebooting clears
the problem, I'd say the switch is NFG.
As always with troubleshooting, take things one step at a time and verify
the simple stuff first.
I would like to inspect what's going on at the switch to try and get to
the bottom of the failure. I've downloaded and installed ethereal,
however, I have no idea what I should be looking for in the log files.
Can anyone help?
I assume you mean no traffic is getting through? With ethereal, you can
monitor on one computer, while pinging it from others. Does it get
through? When you ping, can you see lights flashing on the NICs and
switch? If nothing's getting through the switch and only rebooting clears
the problem, I'd say the switch is NFG.
As always with troubleshooting, take things one step at a time and verify
the simple stuff first.
- James Knott
Re: Network Failure - No Idea How to Troubleshoot
by bkbigpond » Tue Nov 15, 2005 11:35 am
"BC" <bcharlton@pchenderson.com> wrote in message
news:1131627636.314975.279840@g43g2000cwa.googlegroups.com...
Can you have the switch send syslog messages? Is there any spanning-tree
issues .. loops forming etc. I'm currently troubleshooting a Cisco
environment (6509, 6513 core switches). For no apparent reason (no changes
being made) the network has ground to a halt twice. Very hard to determine
what's going on if there's no logging available. External logging is ok but
can only happen while the network/switch is operational. Local debugging /
logging is better but if you have to shutdown a switch in a hurry (to
restore network services) you lose what logging was there.
BernieM
news:1131627636.314975.279840@g43g2000cwa.googlegroups.com...
Hi there
Hope some kind soul out there can help or point me in the right
direction.
For the last month or so our users have experienced a network failure
about once per week. Rebooting the main 48 port unmanaged switch
(netgear) resolves the problem.
I would like to inspect what's going on at the switch to try and get to
the bottom of the failure. I've downloaded and installed ethereal,
however, I have no idea what I should be looking for in the log files.
Can anyone help?
At present we have our W2K servers running on a copper gigabit switch,
this switch is then connected to the 48 port switch. Various other
switches are downstream of this switch.
I plan on hanging a hub between a downstream switch and its uplink to
the 48 port switch and then using ethereal to analyse what's happening.
However, I've no idea what I should be looking for! Can anyone help
please??
Thanks
BC
Can you have the switch send syslog messages? Is there any spanning-tree
issues .. loops forming etc. I'm currently troubleshooting a Cisco
environment (6509, 6513 core switches). For no apparent reason (no changes
being made) the network has ground to a halt twice. Very hard to determine
what's going on if there's no logging available. External logging is ok but
can only happen while the network/switch is operational. Local debugging /
logging is better but if you have to shutdown a switch in a hurry (to
restore network services) you lose what logging was there.
BernieM
- bkbigpond
Re: Network Failure - No Idea How to Troubleshoot
by Robert Redelmeier » Tue Dec 06, 2005 12:20 pm
BC <bcharlton@pchenderson.com> wrote:
As you say, difficult. One reset in 25 days isn't that horrible,
but may not be acceptable in a commercial environment.
There are two general causes of switches needing resets:
hardware and software. The hardware side would be things like
static electricity, lightening, poor grounding/interbuilding
These can also be permanent failures.
The software side of things is more likely to be cause by
unpredicted behaviour from high loads, buffer overflows,
evil packets. Make sure jumbo packets are turned off.
-- Robert
Hmmm.....replaced the suspect 48 port switch with a spare
24 port switch (turns out only 23 ports were in use on the
48 port). Everything fine for 25 days, then bang, network
failure again yesterday. Reset of the switch restores
connectivity. The only thing I haven't change on this level
is the server gigaswitch to which the 48porter uplinks.
I intend to test this out with a spare at the weekend.
If this fails I'm at a total lost. I'm pretty sure there are
no physical loops in the network. The fact that the
switches are unmanaged makes it difficult to troubleshoot.
As you say, difficult. One reset in 25 days isn't that horrible,
but may not be acceptable in a commercial environment.
There are two general causes of switches needing resets:
hardware and software. The hardware side would be things like
static electricity, lightening, poor grounding/interbuilding
These can also be permanent failures.
The software side of things is more likely to be cause by
unpredicted behaviour from high loads, buffer overflows,
evil packets. Make sure jumbo packets are turned off.
-- Robert
- Robert Redelmeier
Re: Network Failure - No Idea How to Troubleshoot
by BC » Tue Dec 06, 2005 12:20 pm
Fans seem to be spinning round nicely / clear of crap. Environmental
conditions are static. Thanks for the response tho!
conditions are static. Thanks for the response tho!
- BC
Re: Network Failure - No Idea How to Troubleshoot
by BC » Tue Dec 06, 2005 12:20 pm
Hmmm.....replaced the suspect 48 port switch with a spare 24 port
switch (turns out only 23 ports were in use on the 48 port).
Everything fine for 25 days, then bang, network failure again
yesterday. Reset of the switch restores connectivity. The only thing
I haven't change on this level is the server gigaswitch to which the
48porter uplinks. I intend to test this out with a spare at the
weekend. If this fails I'm at a total lost. I'm pretty sure there are
no physical loops in the network. The fact that the switches are
unmanaged makes it difficult to troubleshoot. Any further
troubleshooting advice would be appreciated!
switch (turns out only 23 ports were in use on the 48 port).
Everything fine for 25 days, then bang, network failure again
yesterday. Reset of the switch restores connectivity. The only thing
I haven't change on this level is the server gigaswitch to which the
48porter uplinks. I intend to test this out with a spare at the
weekend. If this fails I'm at a total lost. I'm pretty sure there are
no physical loops in the network. The fact that the switches are
unmanaged makes it difficult to troubleshoot. Any further
troubleshooting advice would be appreciated!
- BC
Re: Network Failure - No Idea How to Troubleshoot
by William P.N. Smith » Tue Dec 06, 2005 12:20 pm
"BC" <bcharlton@pchenderson.com> wrote:
Well, since I'm a hardware guy, is there any chance that the cooling
fan(s) are {clogged, stopped, blocked} or that the closet the switch
is in is warmer than before? Just a random thought...
troubleshooting advice would be appreciated!
Well, since I'm a hardware guy, is there any chance that the cooling
fan(s) are {clogged, stopped, blocked} or that the closet the switch
is in is warmer than before? Just a random thought...
- William P.N. Smith
Re: Network Failure - No Idea How to Troubleshoot
by BC » Wed Dec 07, 2005 11:17 am
Thanks for the reply Robert.
Switches are unmanaged therefore no way of connecting to the
switch in or out of band to view / manage, can't see if ARP tables
are overflowed etc. I'm suspecting either screwy internal ARP / MAC
tables or broadcast
storm. (or DoS attack?!)
I'm currently hanging ethereal off of a hub connected to the switch to
see if there's an issue with broadcast traffic.
As the problem is relatively infrequent I think a weekly switch reboot
procedure is probably the most pragmatic thing I can do. I'm hoping to
replace the backbone with a L2 managed switch early next year, which
might help me gain further insight into the problem.
Switches are unmanaged therefore no way of connecting to the
switch in or out of band to view / manage, can't see if ARP tables
are overflowed etc. I'm suspecting either screwy internal ARP / MAC
tables or broadcast
storm. (or DoS attack?!)
I'm currently hanging ethereal off of a hub connected to the switch to
see if there's an issue with broadcast traffic.
As the problem is relatively infrequent I think a weekly switch reboot
procedure is probably the most pragmatic thing I can do. I'm hoping to
replace the backbone with a L2 managed switch early next year, which
might help me gain further insight into the problem.
- BC
Re: Network Failure - No Idea How to Troubleshoot
by Paul Vacquier » Wed Dec 07, 2005 12:20 pm
Robert Redelmeier wrote:
There's not a chance that they are running a job locally that could
introduce a 'spike' is there?
Thinking of, say, payroll running up a 'burster' or the welding shop firing
up that old MIG welder
that only gets fired up once a month or so for the 'special jobs'?
Could you run a mains supply tester and look for brown? outs - or spikes
that could be picked up by the equipment.
May be that they are running a job that's chucking a massive load of data to
the network (just a straw - really wouldn't expect that)
--
regards
PPV.
--
Paul Vacquier | Tel: +44 1245 242229
BAE SYSTEMS Advanced Technology Centres | Fax: +44 1245 475244
Great Baddow, Chelmsford, CM2 8HN UK | paul.vacquier@b1a2e3systems.com
-----------------------------------------------------------------------
"This advice is offered in good faith but shall not be contractually
binding on and does not necessarily reflect the view of
CSC(Computer Sciences Corporation)."
BC <bcharlton@pchenderson.com> wrote:
Hmmm.....replaced the suspect 48 port switch with a spare
24 port switch (turns out only 23 ports were in use on the
48 port). Everything fine for 25 days, then bang, network
failure again yesterday. Reset of the switch restores
connectivity. The only thing I haven't change on this level
is the server gigaswitch to which the 48porter uplinks.
I intend to test this out with a spare at the weekend.
If this fails I'm at a total lost. I'm pretty sure there are
no physical loops in the network. The fact that the
switches are unmanaged makes it difficult to troubleshoot.
As you say, difficult. One reset in 25 days isn't that horrible,
but may not be acceptable in a commercial environment.
There are two general causes of switches needing resets:
hardware and software. The hardware side would be things like
static electricity, lightening, poor grounding/interbuilding
These can also be permanent failures.
The software side of things is more likely to be cause by
unpredicted behaviour from high loads, buffer overflows,
evil packets. Make sure jumbo packets are turned off.
-- Robert
There's not a chance that they are running a job locally that could
introduce a 'spike' is there?
Thinking of, say, payroll running up a 'burster' or the welding shop firing
up that old MIG welder
that only gets fired up once a month or so for the 'special jobs'?
Could you run a mains supply tester and look for brown? outs - or spikes
that could be picked up by the equipment.
May be that they are running a job that's chucking a massive load of data to
the network (just a straw - really wouldn't expect that)
--
regards
PPV.
--
Paul Vacquier | Tel: +44 1245 242229
BAE SYSTEMS Advanced Technology Centres | Fax: +44 1245 475244
Great Baddow, Chelmsford, CM2 8HN UK | paul.vacquier@b1a2e3systems.com
-----------------------------------------------------------------------
"This advice is offered in good faith but shall not be contractually
binding on and does not necessarily reflect the view of
CSC(Computer Sciences Corporation)."
- Paul Vacquier
Re: Network Failure - No Idea How to Troubleshoot
by BC » Wed Dec 07, 2005 12:20 pm
Thanks for the response Paul. Interesting idea about data "spikes".
I'm not aware of any high network utlising apps that are coinciding
with the network outages, however, it is a distinct possibility. We
have a document management app that IS used infrequently, and if used
incorrectly, can fire enormous amounts of data across the wire. I
think this is worth investigating.
I don't think theres a problem with the computer closet electrics.
I've recently had these tested in lieu of a new backup generator and
nothing has shown up. The actual hardware is protected by UPS with
surge protectors. Although again power spikes could be an issue. I'll
ask our sparkies to monitor over a longer period.
Thanks for your valid input!
I'm not aware of any high network utlising apps that are coinciding
with the network outages, however, it is a distinct possibility. We
have a document management app that IS used infrequently, and if used
incorrectly, can fire enormous amounts of data across the wire. I
think this is worth investigating.
I don't think theres a problem with the computer closet electrics.
I've recently had these tested in lieu of a new backup generator and
nothing has shown up. The actual hardware is protected by UPS with
surge protectors. Although again power spikes could be an issue. I'll
ask our sparkies to monitor over a longer period.
Thanks for your valid input!
- BC
Re: Network Failure - No Idea How to Troubleshoot
by BC » Wed Dec 07, 2005 12:20 pm
Thanks for the response Paul. Interesting idea about data "spikes".
I'm not aware of any high network utlising apps that are coinciding
with the network outages, however, it is a distinct possibility. We
have a document management app that IS used infrequently, and if used
incorrectly, can fire enormous amounts of data across the wire. I
think this is worth investigating.
I don't think theres a problem with the computer closet electrics.
I've recently had these tested in lieu of a new backup generator and
nothing has shown up. The actual hardware is protected by UPS with
surge protectors. Although again power spikes could be an issue. I'll
ask our sparkies to monitor over a longer period.
Thanks for your valued input!
I'm not aware of any high network utlising apps that are coinciding
with the network outages, however, it is a distinct possibility. We
have a document management app that IS used infrequently, and if used
incorrectly, can fire enormous amounts of data across the wire. I
think this is worth investigating.
I don't think theres a problem with the computer closet electrics.
I've recently had these tested in lieu of a new backup generator and
nothing has shown up. The actual hardware is protected by UPS with
surge protectors. Although again power spikes could be an issue. I'll
ask our sparkies to monitor over a longer period.
Thanks for your valued input!
- BC
Re: Network Failure - No Idea How to Troubleshoot
by Robert Redelmeier » Wed Dec 07, 2005 12:20 pm
BC <bcharlton@pchenderson.com> wrote:
I doubt something as common as ARP table overflow would
cause this. However, you could easly get a broadcast storm
if older computers running NetBEUI protocols get connected
and start trying to access network resources.
Broadcast traffic should be received on all ports.
It is pragmatic, but I'm not sure it will help much. Something
happens on your network to cause the hang. Up until it
happens, everything is likely fine and the reboots do nothing.
This isan't a case of slow deterioration.
That will help. Please remember that cheap unmanaged switches
are just that. They're not meant for much cascading. I would
also try to stick to the same manufacturer.
-- Robert
if ARP tables are overflowed etc. I'm suspecting either
screwy internal ARP / MAC tables or broadcast storm. (or
DoS attack?!)
I doubt something as common as ARP table overflow would
cause this. However, you could easly get a broadcast storm
if older computers running NetBEUI protocols get connected
and start trying to access network resources.
I'm currently hanging ethereal off of a hub connected to
the switch to see if there's an issue with broadcast traffic.
Broadcast traffic should be received on all ports.
As the problem is relatively infrequent I think a weekly
switch reboot procedure is probably the most pragmatic
thing I can do.
It is pragmatic, but I'm not sure it will help much. Something
happens on your network to cause the hang. Up until it
happens, everything is likely fine and the reboots do nothing.
This isan't a case of slow deterioration.
replace the backbone with a L2 managed switch early next
year, which might help me gain further insight into the
problem.
That will help. Please remember that cheap unmanaged switches
are just that. They're not meant for much cascading. I would
also try to stick to the same manufacturer.
-- Robert
- Robert Redelmeier
20 posts
• Page 1 of 2 • 1, 2
Who is online
Users browsing this forum: No registered users and 0 guests