Announcement

Collapse
No announcement yet.

Help in ending an IGMP Storm on my network.

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Help in ending an IGMP Storm on my network.

    Help in ending an IGMP Storm on my network, Just don't have enough knowledge.

    I've been trying to find the source of an IGMP storm on my network for weeks now. I start trying to find the source the hard way, which at this point is the only way I know. By disconnecting drops from the switches and waiting to see if the storm leaves but it never seems to be one specific drop that stops the storm.

    Eventually after getting complaints that servers can't be accessed or software is crashing (because it needs file shares that are getting disconnected) I plug everything back in to appease all the people trying to use network resources. By this time the storm has magically moved on and I can no longer see it on any switch which I plug my laptop into running wireshark.

    Then an hour later it returns, or a day later, or a minute later. When the IGMP Storm is happening, I can't access any of my switch's Web Management interfaces. They just will not load.

    Save for one older 10/100 switch. my 'Guest Network' switch. I was logged into it and loaded up the "Address Table" list, and in that list was the same MAC address that Wireshark said is the "Source: Cisco_Li_7e:38:ab (00:14:bf:7e:38:ab)".. and in the Guest Switch's address table was "0014BF-7E38AB, VLAN1, PORT4, Dynamic"..

    Port 4 of that switch is the patch to the "Core Network" switch. So I'm guessing that the storm is coming from the Core Network switch, I go to the Web Management on that switch which I only had access too temporarily when the storm was missing. I looked in the Address Table on the Core Network switch and it listed the same MAC address on VLAN1, PORT24, Dynamic.

    Port 24 On the Network Core, Is the Fiber Module which goes to another building (the one I'm in now where my office is). But again the storm is back and I'm unable to access the Web Management GUI on any of my switches. (All my switches are SMC of various models. Like and 8612T, or 8624T these are my GIGE switches, and I have some older 10/100 switches but they are all SMC)..

    I'm a very competent Network and Computer Technician, I recently got my Microsoft Systems Administrator cert and moved up to being a Network Administrator, I'm a noob at this position I'll admit it but I am competent. I just need some help, advice, something regarding this Broadcast Storm. I just can't figure it out. Am I on the right path? or just chasing geese here?

    I'm going to attach a screenshot of Wireshark with all the information from the IGMP packet (they are all the same from what I can tell). Perhaps it'll mean more to someone with experience with this.

    Hope there are some advanced users with time to help me out!, Thanks in advance.
    Attached Files

  • #2
    Re: Help in ending an IGMP Storm on my network.

    Do you run multicast on your network and what source is sending the multicast stream for group 224.0.0.252? What type of multicast do you run? Pim dense, sparse,sparse-dense , SSM, ?
    CCNA, CCNA-Security, CCNP
    CCIE Security (In Progress)

    Comment


    • #3
      Re: Help in ending an IGMP Storm on my network.

      Originally posted by auglan View Post
      Do you run multicast on your network and what source is sending the multicast stream for group 224.0.0.252? What type of multicast do you run? Pim dense, sparse,sparse-dense , SSM, ?
      I feel quite stupid after reading your reply. heh.

      I was handed this network, didn't build it, previous IT guys at this company were not even certified. Just computer technicians at best. I'm an MCSA/MCITP:SA.. but I have very limited experience with managing network hardware. So...

      Multicast is a broadcast to select destinations, or a group. 192.168.104.1 is trying to join group 224.0.0.252. I think I understand this, as it's what Wireshark is telling me.

      The Source IP 192.168.104.1 isn't even in my network's address range. I know it's bad practice but again it was done before I was handed this network. 172.21.36.0/22 (4 Class Cs. .36.0 to .39.255). Would then, this be the result of a piece of hardware connected to my network that has that 192. IP address defined manually (static) even though it can't effectively communicate with the rest of the network because, it's not the right subnet. And it's just thrashing away?

      Again regarding your question, embarrassingly I don't have the knowledge to answer that. I barely know what Multicast is other than (A multicast is a packet that is sent to an IP address that will be accepted by any device that is set to listen to packets destined for that address..), let alone why I would need it, or not need it on any specific devices on my network.

      Could you dumb down your inquiry a little more? (or try a different angle that I might better understand?) I really appreciate any help thank you!

      Comment


      • #4
        Re: Help in ending an IGMP Storm on my network.

        You must have multicasts running on your network as that host is sending IGMP joins for multicast group 224.0.0.252. Muticast is not broadcast. Yes that 192 host could be a static host on your network. I would check all your switches and look at the arp cache and see if you see that ip address/mac address in there. Also check your switches and routers layer 3 interfaces. Are these cisco devices? If so look for any config on the interfaces that says

        ip pim dense-mode
        ip pim sparse-dense-mode
        ip pim sparse-mode

        ip igmp join-group


        If your not running multicast there is no reason to have that on the interfaces. Also disabling it on the interfaces will also disable igmp on the interfaces. IGMP is messages between ends hosts and routers and PIM messages are between multicast routers
        CCNA, CCNA-Security, CCNP
        CCIE Security (In Progress)

        Comment


        • #5
          Re: Help in ending an IGMP Storm on my network.

          Multicast can be used for a variety of things. Streaming video, IPTV, imaging software, routing protocols use it in their updates. Im assuming you have your network segmented into vlans. Do you have any routes anywhere for that 192. network?
          CCNA, CCNA-Security, CCNP
          CCIE Security (In Progress)

          Comment


          • #6
            Re: Help in ending an IGMP Storm on my network.

            No the network has no VLANs setup as of yet, that is actually on my to-do list but I've not got there yet. I came to this company and knew after the first few days I would be rebuilding this network from scratch, previous "IT Guys" left this place a complete disaster. The network hardware setup is just part of the issue.

            Wireshark Says the SOURCE ADDRESS for the Multicast IGMP Flood is:
            Cisco-Li_7e:38:ab (00-14-BF-7E-38-AB)
            IP: 192.168.104.1

            Console connections to SMC TigerSwitches using command:
            CONSOLE# SHOW MAC-ADDRESS-TABLE
            Gives following data, regarding the above MAC Address.

            172.21.39.248 MAC-ADDRESS-TABLE says 0014BF-7E38AB is on Port 25 Dynamic, which is Patched to 172.21.39.250
            172.21.39.249 MAC-ADDRESS-TABLE says 0014BF-7E38AB is on Port 24 Dynamic, Which is Patched to 172.21.39.250
            172.21.39.251 MAC-ADDRESS-TABLE says 0014BF-7E38AB is on Port 23 Dynamic, Which is Patched to 172.21.39.250
            172.21.39.250 MAC-ADDRESS-TABLE says 0014BF-7E38AB is on Port 12 Dynamic, Which is Patched to 172.21.39.254
            <----- FIBER BETWEEN BUILDINGS ----->
            172.21.39.254 MAC-ADDRESS-TABLE says 0014BF-7E38AB is on Port 12 Dynamic, Which is Patched to 172.21.39.250
            172.21.39.253 MAC-ADDRESS-TABLE says 0014BF-7E38AB is on Port 25 Dynamic, Which is Patched to 172.21.39.254
            172.21.39.252 MAC-ADDRESS-TABLE says 0014BF-7E38AB is on Port 04 Dynamic, Which is Patched to 172.21.39.254


            Am I making any progress here?

            My two GIGE "Core" switches are both saying the MAC address is from the other one. The patch between them is the fiber run between buildings using the Module ports in the two switches. Both are SMC TigerSwitch 8612Ts.

            I haven't delved into routing just yet. but we currently have a Cisco PIX 501 as our boarder router which is patched to 172.21.39.253... And the PIX is connected to a Cisco 1841 which is owned by our provider. We're in a Rural(ish) area on a single T1.

            ...Sigh... Again thanks for any help/suggestions/etc!

            Comment


            • #7
              Re: Help in ending an IGMP Storm on my network.

              And I just disconnected the Fiber patch from Port 12 on 172.21.39.254 which is patched to Port 12 on 172.21.39.250, the fiber link between buildings.

              And the IGMP Multicast Flood is gone.... I'd have to drive over to the other building to see if its gone there as well because now it's completely cut off from the rest of the network and the internet gateway. But im assuming it is because, my logic/guess tells me the issue is between the two 8612Ts, the two Core GIGE switches.

              Suggestions?
              Last edited by Griff.J; 26th April 2012, 02:14.

              Comment


              • #8
                Re: Help in ending an IGMP Storm on my network.

                Well the vendor id from the wireshark capture says its a cisco device. Since the only cisco
                Device you control on the network is the pix, i would have a look at that.
                CCNA, CCNA-Security, CCNP
                CCIE Security (In Progress)

                Comment


                • #9
                  Re: Help in ending an IGMP Storm on my network.

                  I just happened to be doing an all-nighter, tore the 4 post rack apart and rebuilt it all rearranging the servers and using the cable management that came with them this time. Instead of the spaghetti rats nest that was left behind that rack previously. Anyway...

                  After my last post the business was more or less closed so I was able to disconnect the fiber which connects the two buildings. once I did that the IGMP flood was gone from both segments of the network.

                  Then I started poking around the MAC-ADDRESS-TABLEs again in the switches on the side of the network where I was working, and once again I found that MAC address 00-14-BF-7E-38-AB.... So no flood but it was still in the Table but this time I noticed it was going in a different direction out a different port on Core switch, instead of out the fiber port like before. So, I consoled on that next switch and it said it was on port X (whatever). Followed that port to a single drop on one of the patch panels.

                  At this point I'm not sure where the other end of that drop is but... I disconnected that drop and then reconnected the fiber between my two network segments (buildings) and nothing! No more IGMP floods. No more 192.168.104.1, I left WireShark running all night as I worked on the server rack and brought the rest of the network back online. Once I was done I used a filter in WireShark to see if there was even a single chirp from that 192 host. And there was nothing.

                  So, is it safe to say once I have time to find where that drops goes; the hardware on the other end of it is the culprit? Or would it more likely be a victim of another device on the network which is still the cause.

                  You say the source says it's a Cisco device, which I agree I see it in the WireShark frames.

                  Huuurrrmmmm. It says Cisco_Li.... I wonder if its. Cisco_Linksys in full. They have used residential broadband WiFi Linksys routers throughout the business setup as Access Points.

                  Huh.... I wonder if it's one of those.

                  Comment


                  • #10
                    Re: Help in ending an IGMP Storm on my network.

                    One positive about this broken-ass-network good place for a new Network Admin to start. I have to learn so many new things every day as I run this network, having to repair all it's issues. If everything was working perfectly I'd have nothing to do and would be SOL if the crap hit the fan. LOL..

                    Comment


                    • #11
                      Re: Help in ending an IGMP Storm on my network.

                      Yeah I missed the cisco_li on the capture. Cisco/linksys does make sense though.
                      CCNA, CCNA-Security, CCNP
                      CCIE Security (In Progress)

                      Comment


                      • #12
                        Re: Help in ending an IGMP Storm on my network.

                        ...

                        Cisco Linksys WAP54GP screwed to the bottom side of a random desk, I guess chosen because of was more or less in the middle of that area. Found it with Ekahau HeatMapper, WiFi signal surveying software.

                        And because all hot spots have the same SSID/PW so that people can roam around from hot spot to hot spot. I guess no one really noticed that one wasn't working properly.

                        Next task in this place for me involves a Label Maker, a Tone Gen and Probe. The lack of documentation about this network is frustrating, Now I'm noticing even the drops that are labeled, aren't correct..

                        Thanks for the help!

                        Comment


                        • #13
                          Re: Help in ending an IGMP Storm on my network.

                          Yeah rogue devices around the network can be a nightmare especially if no one has documented it. Another great thing to do is get Visio (or something similar) and diagram the network. Makes life much easier.
                          CCNA, CCNA-Security, CCNP
                          CCIE Security (In Progress)

                          Comment

                          Working...
                          X