| Author |
Message |
nntp chip
Guest
|
 Posted:
Tue Nov 30, 2004 2:19 pm Post subject:
Desktop switch kills routing |
|
|
Hello all,
I run a network where three different lan:s are used. Between the
buildings at every site the traffic flows through tagged ports in layer-2
switches. (ASCII-art and switchmodels below) When traffic need to go
somewhere outside that site a layer-3 switch routes it onto a carrier
network kept separated from the three other vlan:s.
Enabled spanning-tree on all switches to kill off nasty loops.
So far so good.
Then some student connected a simple desktop-switch and made a loop within
that little switch. Somehow the spanning tree did not work correctly in
that situation. The entire student-vlan stopped dead. While searching for
what was going on, the administration people started complaining too; They
could reach the local servers, but remote servers and internet was
unreachable.
Set up lab to study things a little closer.
Found out that when one of the vlan:s was looping, the other vlan:s worked
within that site, but routing soon stopped in the layer-3 switch. The very
second i disconnected the offending desktop-switch everything went back to
normal again.
Any ideas how to stop this from happening and keep the routing going? The
admin-network Must Always Be Reachable, so I dont like the idea that some
lousy desktop-switch can wreak such havoc...
TIA
------------
layer-2 switches are D-Link DES-3526
layer-3 switches are D-Link DES-3326S, DGS-3324SR, DGS-3312SR
vlan-10: link-net that connect all sites togehter.
vlan-110: students
vlan-120: administration
vlan-130: public hotspots etc.
(carrier network)
|
| vlan-10
|
__|_________________
| |
| switch-1 (layer 3) |
|____________________|
|
|
| tagged link with vlans-110,120,130
|
________________|___
| |
| switch-2 (layer 2) |
|____________________|
| | | |
| | | |
110 120 130 |
| tagged link with vlans-110,120,130
.
.
.
|
__________________|_
| |
| switch-n (layer 2) |
|____________________|
| | |
| | | |
|
| Back to top |
|
 |
glen herrmannsfeldt
Guest
|
| Posted:
Tue Nov 30, 2004 3:54 pm Post subject:
Re: Desktop switch kills routing |
|
|
nntp chip wrote:
| Quote: | I run a network where three different lan:s are used. Between the
buildings at every site the traffic flows through tagged ports in layer-2
switches. (ASCII-art and switchmodels below) When traffic need to go
somewhere outside that site a layer-3 switch routes it onto a carrier
network kept separated from the three other vlan:s.
Enabled spanning-tree on all switches to kill off nasty loops.
So far so good.
Then some student connected a simple desktop-switch and made a loop within
that little switch. Somehow the spanning tree did not work correctly in
that situation. The entire student-vlan stopped dead. While searching for
what was going on, the administration people started complaining too; They
could reach the local servers, but remote servers and internet was
unreachable.
|
If the loop is within a non-STP switch there isn't much
the others can do about it.
| Quote: | Set up lab to study things a little closer.
Found out that when one of the vlan:s was looping, the other vlan:s worked
within that site, but routing soon stopped in the layer-3 switch. The very
second i disconnected the offending desktop-switch everything went back to
normal again.
|
If vlan-110 is allowed out through the router, it could easily
saturate the router. A loop creates many copies of a packet, and
they can all get sent out through the router.
| Quote: | Any ideas how to stop this from happening and keep the routing going? The
admin-network Must Always Be Reachable, so I dont like the idea that some
lousy desktop-switch can wreak such havoc...
|
(snip of diagram, I believe it is explained well enough.)
If your equipment allow, limit the bandwidth allowed for
each vlan into the router. That at least gives you a chance,
as it might be that you are just saturating the uplink.
If you are saturating the router itself then it is harder.
Well, limiting on the vlans through the layer 2 switch would
help a lot. Not quite as much if you can only limit at the
router.
-- glen |
|
| Back to top |
|
|
Steve Horsley
Guest
|
| Posted:
Wed Dec 01, 2004 6:29 am Post subject:
Re: Desktop switch kills routing |
|
|
nntp chip wrote:
<snip long and thorough description>
My first guess is that the student VLAN was completely swamped
with a broadcast or multicast packet being repeated at full wire
speed by the rogue switch (effectively a traffic generator). And
I guess that this broadcast flood drove the router (switch 1)
into total CPU overload where it couldn't function properly.
It's difficult to know what to do about that, other than
perhaps using a separate router / L3 switch to route between
the non-admin LANS and the carrier network, just to separate
and protect the admin router. Perhaps with rate limiting.
Steve |
|
| Back to top |
|
|
stephen
Guest
|
| Posted:
Thu Dec 02, 2004 1:14 am Post subject:
Re: Desktop switch kills routing |
|
|
"Steve Horsley" <shoot@the.moon> wrote in message
news:coj6pc$4n3$1@news.freedom2surf.net...
| Quote: | nntp chip wrote:
snip long and thorough description
My first guess is that the student VLAN was completely swamped
with a broadcast or multicast packet being repeated at full wire
speed by the rogue switch (effectively a traffic generator). And
I guess that this broadcast flood drove the router (switch 1)
into total CPU overload where it couldn't function properly.
|
Agreed - i have seen this happen before as well.
it may be the broadcast level generally, or just IP broadcasts and
multicasts depending on your layer 3 kit and the traffic profile in the
network.
| Quote: |
It's difficult to know what to do about that, other than
perhaps using a separate router / L3 switch to route between
the non-admin LANS and the carrier network, just to separate
and protect the admin router. Perhaps with rate limiting.
|
Some layer 2 switches support broadcast rate limiting. The idea here is to
limit the number of broadcasts propagated to the rest of the network to
limit the spread of this kind of fault.
This does work, but probably isnt turned on by default if available on your
hardware, or the level is set so high that the limited stream still chokes
the layer 3 switch - experimenting may be the only way to choose a
compromise setting.
Alternatively, some layer 3 switches support control plane limiting,
intended to limit this kind of denial of service attack
These are the kind of "bells and whistles" features that isnt universal, so
may not be useful for your current kit.
Regards
Stephen Hope - return address needs fewer xxs |
|
| Back to top |
|
|
nntp chip
Guest
|
| Posted:
Thu Dec 02, 2004 10:07 pm Post subject:
Re: Desktop switch kills routing |
|
|
On Wed, 01 Dec 2004 20:14:31 +0000, stephen wrote:
| Quote: | "Steve Horsley" <shoot@the.moon> wrote in message
news:coj6pc$4n3$1@news.freedom2surf.net...
nntp chip wrote:
snip long and thorough description
My first guess is that the student VLAN was completely swamped with a
broadcast or multicast packet being repeated at full wire speed by the
rogue switch (effectively a traffic generator). And I guess that this
broadcast flood drove the router (switch 1) into total CPU overload
where it couldn't function properly.
Agreed - i have seen this happen before as well.
it may be the broadcast level generally, or just IP broadcasts and
multicasts depending on your layer 3 kit and the traffic profile in the
network.
It's difficult to know what to do about that, other than perhaps using
a separate router / L3 switch to route between the non-admin LANS and
the carrier network, just to separate and protect the admin router.
Perhaps with rate limiting.
Some layer 2 switches support broadcast rate limiting. The idea here is
to limit the number of broadcasts propagated to the rest of the network
to limit the spread of this kind of fault.
This does work, but probably isnt turned on by default if available on
your hardware, or the level is set so high that the limited stream still
chokes the layer 3 switch - experimenting may be the only way to choose
a compromise setting.
|
Broadcast rate limiting exists, but even when set to the lowest possible
limit it doesn't seem to work as intended. Looks like the excessive
traffic is not classified as broadcast and thus never triggers the
limiting behaviour.
I'll try experimenting with rate limiting for those locations where
bandwith is not an issue.
Still some places require huge bandwidh, so another router or layer
3-switch should do the trick there.
Thanks for the ideas.
| Quote: | Alternatively, some layer 3 switches support control plane limiting,
intended to limit this kind of denial of service attack
|
Do you know any brands/models that use control plane limiting? Might be
nice to read up on that.
| Quote: | These are the kind of "bells and whistles" features that isnt universal,
so may not be useful for your current kit.
Steve |
|
|
| Back to top |
|
|
stephen
Guest
|
| Posted:
Fri Dec 03, 2004 2:56 am Post subject:
Re: Desktop switch kills routing |
|
|
"nntp chip" <me@privacy.net> wrote in message
news:pan.2004.12.02.17.07.35.646781@privacy.net...
| Quote: | On Wed, 01 Dec 2004 20:14:31 +0000, stephen wrote:
"Steve Horsley" <shoot@the.moon> wrote in message
news:coj6pc$4n3$1@news.freedom2surf.net...
nntp chip wrote:
snip long and thorough description
My first guess is that the student VLAN was completely swamped with a
broadcast or multicast packet being repeated at full wire speed by the
rogue switch (effectively a traffic generator). And I guess that this
broadcast flood drove the router (switch 1) into total CPU overload
where it couldn't function properly.
Agreed - i have seen this happen before as well.
it may be the broadcast level generally, or just IP broadcasts and
multicasts depending on your layer 3 kit and the traffic profile in the
network.
It's difficult to know what to do about that, other than perhaps using
a separate router / L3 switch to route between the non-admin LANS and
the carrier network, just to separate and protect the admin router.
Perhaps with rate limiting.
Some layer 2 switches support broadcast rate limiting. The idea here is
to limit the number of broadcasts propagated to the rest of the network
to limit the spread of this kind of fault.
This does work, but probably isnt turned on by default if available on
your hardware, or the level is set so high that the limited stream still
chokes the layer 3 switch - experimenting may be the only way to choose
a compromise setting.
Broadcast rate limiting exists, but even when set to the lowest possible
limit it doesn't seem to work as intended. Looks like the excessive
traffic is not classified as broadcast and thus never triggers the
limiting behaviour.
I'll try experimenting with rate limiting for those locations where
bandwith is not an issue.
Still some places require huge bandwidh, so another router or layer
3-switch should do the trick there.
Thanks for the ideas.
Alternatively, some layer 3 switches support control plane limiting,
intended to limit this kind of denial of service attack
Do you know any brands/models that use control plane limiting? Might be
nice to read up on that.
|
cisco have put this on a lot of their recent stackables, as well as the
chassis switches - they also support broadcast and multicast storm control.
here is a link to the Catalyst 3550 with some info in the feature list
(layer 3 enet stackable)
http://www.cisco.com/en/US/products/hw/switches/ps646/products_data_sheet09186a00800913d0.html
| Quote: |
These are the kind of "bells and whistles" features that isnt universal,
so may not be useful for your current kit.
Steve
-- |
Regards
Stephen Hope - return address needs fewer xxs |
|
| Back to top |
|
|
|
|
|
|
FAQ
Memberlist
Register
Profile
Log in to check your private messages
Log in