Announcement

Collapse
No announcement yet.

Multicasting Problem - OUT OF SEQUENCE PACKETS‏

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Multicasting Problem - OUT OF SEQUENCE PACKETS‏

    Hi,

    First off I'd like to thank all those who will be helping me through my dilema and provide my current setup:

    video server / smart bit (used for testing - outputs multicast streams)
    |
    cisco 2900 (switch) --> multicast flooded to encapsulator
    |
    encapsulator --> converted into dvb packets
    |
    modulator
    |
    sat dish (upconverting)
    |
    Satellite (specific transponder)
    |
    sat dish (downconverting)
    |
    Satellite Receivers (4 of them - outputs multicast back on another LAN)
    ||||
    cisco 4948 (Layer 3 Switch)
    |
    client's requesting multicast (Smartbits used again for testing - smartbits sending igmp join request to all multicast addresses)

    Description of what i'm trying to accomplish:

    I need to send a video/audio feed from one province to another via satellite where client's will be using igmp to request the feeds.

    Broken down steps
    -smartbit outputs multicast test stream (230.0.0.1, 230.0.0.2, 230.0.0.3, 230.0.0.4)
    -stream is flooded onto 2900 switch which the encapsulator is listening
    -encapsulator converts ip packets into dvb packets which are then modulated into a specific frequency to my upconverter which simulates my satellite connection.
    -sat receivers receive the packets via sat0 interface and outputs the traffic to the cisco 4948.
    -client's send igmp join requests to get the video stream but for now i use the smartbit controller to do this.

    I have not implemented ip pim nor cgmp for controlling multicast. (when i try i get the same results)

    Description of the problem:
    I can get the multicast traffic through the satellite connection but my smart bits is detecting a lot of out of sequence packets. I need to find a way to reduce the amount of out of sequence packets but I cannot figure out what's the cause. When I only output one multicast stream through the smart bits, the receiving end has no problems. (1 or 2 out of sequence packets which is normal from what i've been told). The minute I add more streams, I have a serious issue with out of sequence packets.

    My co-worker hates cisco and blames the switch. To prove to me that he was right, he took a linksys 24 port switch (igmp passthrough enable) and replaced the 4948 with it. Low and behold, 4 streams with hardly any out of sequence packets. So the issue has been confirmed, it's 4948 switch.

    Anyone know what needs to be enable on the 4948 to solve my problem? I'm CCNP certified with a huge background in networking. I've read tones of documentation on multicasting with cisco but nothing regarding the problem i'm facing. Plz I need to get this setup working without using a linksys 24 port switch ASAP.

    Hope to hear from someone,

    Evan

    PS. IF you need more info ask for it or you can e-mail me directly at [email protected]

  • #2
    Re: Multicasting Problem - OUT OF SEQUENCE PACKETS‏

    Massively way past my knowledge, so laugh away, but I assume you set the interfaces rather than leaving auto? It's not just a duplex mismatch? I see it supports 10/100/1000BASE-T. From what you describe, if you canjust drop a switch in and it works then it is something basic like that.
    cheers
    Andy

    Please read this before you post:


    Quis custodiet ipsos custodes?

    Comment


    • #3
      Re: Multicasting Problem - OUT OF SEQUENCE PACKETS‏

      Hi Andy,

      Thanks for the quick response. I think you are on to something. I hard coded the duplex and speed to 100 full on the switch ports and the 2 receivers I hooked up.

      Here's some stats that I took:
      *Original results from smartbitts plugged into Cisco L3 4948 with 1 Sat receivers sending multicast.

      port 1/1: Sat Receiver 1
      receiver 10.233 - 230.1.1.1
      frm rcvd: 94102
      in sequence: 94102
      Outofsequence:0

      *sh interface g1/1 controller when multicast coming in

      GigabitEthernet1/1 is up, line protocol is up (connected)
      Hardware is Gigabit Ethernet Port, address is 001e.f71a.5940 (bia 001e.f71.5940)
      MTU 1500 bytes, BW 100000 Kbit, DLY 10 usec,
      reliability 255/255, txload 3/255, rxload 6/255
      Encapsulation ARPA, loopback not set
      Keepalive set (10 sec)
      Full-duplex, 100Mb/s, link type is auto, media type is 10/100/1000-TX
      input flow-control is on, output flow-control is off
      ARP type: ARPA, ARP Timeout 04:00:00
      Last input never, output never, output hang never
      Last clearing of "show interface" counters 00:00:16
      Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
      Queueing strategy: fifo
      Output queue: 0/40 (size/max)
      5 minute input rate 2680000 bits/sec, 429 packets/sec
      5 minute output rate 1519000 bits/sec, 231 packets/sec
      29801 packets input, 22469954 bytes, 0 no buffer
      Received 29801 broadcasts (29801 multicast)
      0 runts, 0 giants, 0 throttles
      0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
      0 input packets with dribble condition detected
      8 packets output, 544 bytes, 0 underruns
      0 output errors, 0 collisions, 0 interface resets
      0 babbles, 0 late collision, 0 deferred
      0 lost carrier, 0 no carrier
      0 output buffer failures, 0 output buffers swapped out
      wireshark dump of singe multicast stream

      No. Time Source Destination Protocol Info
      1 0.000000000 Cisco_1a:59:4e Spanning-tree-(for-bridges)_00 STP Conf. Root = 32769/00:1e:f7:1a:59:40 Cost = 0 Port = 0x800f
      2 0.735340000 192.168.10.233 230.1.1.1 IP Bogus IP header length (0, must be at least 20)
      3 0.736127000 192.168.10.233 230.1.1.1 IP Bogus IP header length (0, must be at least 20)
      4 0.736153000 192.168.10.233 230.1.1.1 IP Bogus IP header length (0, must be at least 20)

      *Original results from smartbitts plugged into Cisco L3 4948 with 2 Sat receivers sending multicast.

      port 1/1: Sat Receiver 1
      multicast 230.1.1.1
      frm rcvd: 184409
      in sequence: 83033
      Outofsequence: 101376

      port 1/5: Sat Receiver 2
      Multicast 230.1.1.2
      frm rcvd: 143599
      in sequence: 22516
      Outofsequence: 121083

      Wireshark dump

      No. Time Source Destination Protocol Info
      1 0.000000 192.168.10.233 230.1.1.2 IP Bogus IP header length (0, must be at least 20)
      2 0.000011 192.168.10.233 230.1.1.1 IP Bogus IP header length (0, must be at least 20)
      3 0.000138 192.168.10.233 230.1.1.2 IP Bogus IP header length (0, must be at least 20)
      4 0.000149 192.168.10.233 230.1.1.1 IP Bogus IP header length (0, must be at least 20)
      5 0.000258 192.168.10.233 230.1.1.1 IP Bogus IP header length (0, must be at least 20)
      6 0.000268 192.168.10.233 230.1.1.2 IP Bogus IP header length (0, must be at least 20)
      7 0.000337 192.168.10.233 230.1.1.1 IP Bogus IP header length (0, must be at least 20)
      8 0.000347 192.168.10.233 230.1.1.1 IP Bogus IP header length (0, must be at least 20)
      9 0.000360 192.168.10.233 230.1.1.1 IP Bogus IP header length (0, must be at least 20)
      10 0.000370 192.168.10.233 230.1.1.1 IP Bogus IP header length (0, must be at least 20)
      11 0.000501 192.168.10.233 230.1.1.2 IP Bogus IP header length (0, must be at least 20)
      12 0.000512 192.168.10.233 230.1.1.1 IP Bogus IP header length (0, must be at least 20)
      13 0.000639 192.168.10.233 230.1.1.2 IP Bogus IP header length (0, must be at least 20)
      14 0.000650 192.168.10.233 230.1.1.1 IP Bogus IP header length (0, must be at least 20)
      15 0.000749 192.168.10.233 230.1.1.2 IP Bogus IP header length (0, must be at least 20)

      GigabitEthernet1/1 is up, line protocol is up (connected)
      Hardware is Gigabit Ethernet Port, address is 001e.f71a.5940 (bia 001e.f71a.5940)
      MTU 1500 bytes, BW 100000 Kbit, DLY 10 usec,
      reliability 255/255, txload 27/255, rxload 28/255
      Encapsulation ARPA, loopback not set
      Keepalive set (10 sec)
      Full-duplex, 100Mb/s, link type is auto, media type is 10/100/1000-TX
      input flow-control is on, output flow-control is off
      ARP type: ARPA, ARP Timeout 04:00:00
      Last input never, output never, output hang never
      Last clearing of "show interface" counters 00:00:26
      Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
      Queueing strategy: fifo
      Output queue: 0/40 (size/max)
      5 minute input rate 13622000 bits/sec, 2248 packets/sec
      5 minute output rate 11863000 bits/sec, 1953 packets/sec
      255671 packets input, 192774426 bytes, 0 no buffer
      Received 255671 broadcasts (255671 multicast)
      0 runts, 0 giants, 0 throttles
      0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
      0 input packets with dribble condition detected
      227810 packets output, 171754192 bytes, 0 underruns
      0 output errors, 0 collisions, 0 interface resets
      0 babbles, 0 late collision, 0 deferred
      0 lost carrier, 0 no carrier
      0 output buffer failures, 0 output buffers swapped out

      GigabitEthernet1/5 is up, line protocol is up (connected)
      Hardware is Gigabit Ethernet Port, address is 001e.f71a.5944 (bia 001e.f71a.5944)
      MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
      reliability 255/255, txload 5/255, rxload 5/255
      Encapsulation ARPA, loopback not set
      Keepalive set (10 sec)
      Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000-TX
      input flow-control is on, output flow-control is off
      ARP type: ARPA, ARP Timeout 04:00:00
      Last input never, output never, output hang never
      Last clearing of "show interface" counters 00:01:28
      Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
      Queueing strategy: fifo
      Output queue: 0/40 (size/max)
      5 minute input rate 20702000 bits/sec, 3425 packets/sec
      5 minute output rate 21608000 bits/sec, 3575 packets/sec
      858241 packets input, 647114192 bytes, 0 no buffer
      Received 858239 broadcasts (858237 multicast)
      0 runts, 0 giants, 0 throttles
      0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
      0 input packets with dribble condition detected
      934960 packets output, 704915401 bytes, 0 underruns
      0 output errors, 0 collisions, 0 interface resets
      0 babbles, 0 late collision, 0 deferred
      0 lost carrier, 0 no carrier
      0 output buffer failures, 0 output buffers swapped out

      Comment


      • #4
        Re: Multicasting Problem - OUT OF SEQUENCE PACKETS‏

        Too big for one post... here's the rest:

        *after hardcoding speed/duplex

        GigabitEthernet1/1 is up, line protocol is up (connected)
        Hardware is Gigabit Ethernet Port, address is 001e.f71a.5940 (bia 001e.f71a.5940)
        MTU 1500 bytes, BW 100000 Kbit, DLY 10 usec,
        reliability 255/255, txload 31/255, rxload 33/255
        Encapsulation ARPA, loopback not set
        Keepalive set (10 sec)
        Full-duplex, 100Mb/s, link type is auto, media type is 10/100/1000-TX
        input flow-control is off, output flow-control is off
        ARP type: ARPA, ARP Timeout 04:00:00
        Last input never, output never, output hang never
        Last clearing of "show interface" counters 00:33:11
        Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
        Queueing strategy: fifo
        Output queue: 0/40 (size/max)
        5 minute input rate 13275000 bits/sec, 2189 packets/sec
        5 minute output rate 12386000 bits/sec, 2044 packets/sec
        2693881 packets input, 2031208307 bytes, 0 no buffer
        Received 2693860 broadcasts (2693845 multicast)
        15 runts, 0 giants, 0 throttles
        145 input errors, 130 CRC, 0 frame, 0 overrun, 0 ignored
        0 input packets with dribble condition detected
        2515113 packets output, 1895510729 bytes, 0 underruns
        0 output errors, 0 collisions, 0 interface resets
        0 babbles, 0 late collision, 0 deferred
        0 lost carrier, 0 no carrier
        0 output buffer failures, 0 output buffers swapped out

        GigabitEthernet1/5 is up, line protocol is up (connected)
        Hardware is Gigabit Ethernet Port, address is 001e.f71a.5944 (bia 001e.f71a.5944)
        MTU 1500 bytes, BW 100000 Kbit, DLY 10 usec,
        reliability 255/255, txload 25/255, rxload 27/255
        Encapsulation ARPA, loopback not set
        Keepalive set (10 sec)
        Full-duplex, 100Mb/s, link type is auto, media type is 10/100/1000-TX
        input flow-control is off, output flow-control is off
        ARP type: ARPA, ARP Timeout 04:00:00
        Last input never, output never, output hang never
        Last clearing of "show interface" counters 00:34:08
        Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
        Queueing strategy: fifo
        Output queue: 0/40 (size/max)
        5 minute input rate 10865000 bits/sec, 1788 packets/sec
        5 minute output rate 10163000 bits/sec, 1670 packets/sec
        2589247 packets input, 1952539041 bytes, 0 no buffer
        Received 2589237 broadcasts (2589221 multicast)
        82 runts, 0 giants, 0 throttles
        1199 input errors, 1117 CRC, 0 frame, 0 overrun, 0 ignored
        0 input packets with dribble condition detected
        2644325 packets output, 1992900719 bytes, 0 underruns
        0 output errors, 0 collisions, 0 interface resets
        0 babbles, 0 late collision, 0 deferred
        0 lost carrier, 0 no carrier
        0 output buffer failures, 0 output buffers swapped out

        port 1/1 receiver 1
        multicast 230.1.1.1
        frm rcvd: 122911
        in sequence: 122862
        Outofsequence: 49

        port 1/1 receiver 2
        multicast 230.1.1.2
        frm rcvd: 122270
        in sequence: 121580
        Outofsequence:690

        This is a big drop but the out of sequence packets are still too high. Technically, there shouldn't be any out of sequence packets except maybe 1 or 2.

        Andy... anything else that you can think of that I could try? Or is there someone that you know of or could direct me to that could help me out? The cisco 4948 was recomended by a junior tech and is what is being implemented at the site. I know that the linksys switch will not be able to handle the amount of multicast traffic generated by all the video/audio feeds.

        My current config is:

        4948
        ip multicasting

        int vlan 10
        ip address 192.168.10.222 255.255.255.0
        ip pim sparse-dense (adding once operational)
        ip cgmp (adding once operational)
        no shut

        int g1/1
        speed 100
        duplex full
        switchport access vlan 10

        int g1/5
        speed 100
        duplex full
        switchport access vlan 10

        Andy or whoever that solves this, i will ship you a bottle of premium vodka or a case of beer.

        Hope to hear from you soon,

        Evan

        Comment


        • #5
          Re: Multicasting Problem - OUT OF SEQUENCE PACKETS‏

          Could be way out in left field here but try disabling PIM Dense mode it is probably enabled by default.

          PIM Dense Mode Fallback Prevention in a Network Following RP Information Loss

          The PIM Dense Mode Fallback Prevention in a Network Following RP Information Loss feature enables you to prevent Protocol Independent Multicast (PIM) dense mode (DM) fallback when all rendezvous points (RPs) fail. Preventing the use of PIM-DM is very important to multicast networks whose reliability is critical. The feature provides a mechanism to keep the multicast groups in sparse mode. The feature also allows you to block multicast traffic for groups not specifically configured.

          http://www.cisco.com/en/US/docs/ios/.../gtautorp.html
          Stacey Smith
          Sr. Systems Engineer

          The rule is perfect: in all matters of opinion our adversaries are insane --Samuel Clemens

          Comment


          • #6
            Re: Multicasting Problem - OUT OF SEQUENCE PACKETS‏

            Thanks Stacey,

            I'm confused right now and very stressed out. I really appreciate the help but unfortunetly my IOS is version 12.2 and the command is not available.

            Here's my current config for the 4948.

            Switch#sh run
            Building configuration...

            Current configuration : 2644 bytes
            !
            version 12.2
            no service pad
            service timestamps debug uptime
            service timestamps log uptime
            no service password-encryption
            service compress-config
            !
            hostname Switch
            !
            boot-start-marker
            boot-end-marker
            !
            !
            no aaa new-model
            vtp mode transparent
            ip subnet-zero
            !
            ip multicast-routing
            !
            no file verify auto
            spanning-tree mode pvst
            spanning-tree extend system-id
            power redundancy-mode redundant
            !
            !
            !
            vlan internal allocation policy ascending
            !
            vlan 10
            name primary
            !
            vlan 20
            name secondary
            !
            interface GigabitEthernet1/1 (connected to sat receiver 1)
            switchport access vlan 10
            speed 100
            duplex full
            !
            interface GigabitEthernet1/2
            switchport access vlan 10
            !
            interface GigabitEthernet1/3
            switchport access vlan 10
            !
            interface GigabitEthernet1/4
            switchport access vlan 10
            !
            interface GigabitEthernet1/5 (connected to sat receiver 2)
            switchport access vlan 10
            speed 100
            duplex full
            !
            interface GigabitEthernet1/6
            switchport access vlan 10
            !
            interface GigabitEthernet1/7
            switchport access vlan 10
            !
            interface GigabitEthernet1/8
            switchport access vlan 10
            !
            interface GigabitEthernet1/9
            switchport access vlan 10
            !
            interface GigabitEthernet1/10
            switchport access vlan 10
            !
            interface GigabitEthernet1/11 (smartbit)
            switchport access vlan 10
            !

            interface Vlan1
            no ip address
            !
            interface Vlan10
            ip address 192.168.10.115 255.255.255.0
            ip pim sparse-mode
            ip cgmp
            !
            no ip http server
            !
            ip pim rp-address 192.168.10.115
            ip pim send-rp-announce Vlan10 scope 16 group-list 1
            access-list permit 192.168.10.115
            !
            !
            !
            line con 0
            stopbits 1
            line vty 0 4
            !
            !
            end

            It's a basic one Layer 3 Switch setup on the receiving end.

            smartbit generating multicast traffic for testing (headend)
            |
            Cisco 2900 (headend)
            |
            Satellite
            |
            2 Sat Receivers (port g1/1 and g1/5) -> receiving end
            ||
            Cisco 4948 -> receiving end
            |
            smartbits receiving its own gererated multicast from over the satellite (port 1/11)

            Is my designed flawed?? i cannot allow multicast traffic from flooding the cisco 4948 so i've implemented PIM and CGMP

            With this current setup, my sniffer does not see the flooding of multicast traffic to all ports which is good.. but my smartbits is not receiving any of the traffic even though I have it send up to send IGMP joing requests.

            *sh interface command that displays the multicast traffic coming in

            GigabitEthernet1/1 is up, line protocol is up (connected)
            Hardware is Gigabit Ethernet Port, address is 001e.f71a.5940 (bia 001e.f71a.59
            40)
            MTU 1500 bytes, BW 100000 Kbit, DLY 10 usec,
            reliability 255/255, txload 1/255, rxload 10/255
            Encapsulation ARPA, loopback not set
            Keepalive set (10 sec)
            Full-duplex, 100Mb/s, link type is auto, media type is 10/100/1000-TX
            input flow-control is off, output flow-control is off
            ARP type: ARPA, ARP Timeout 04:00:00
            Last input never, output never, output hang never
            Last clearing of "show interface" counters 00:35:50
            Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
            Queueing strategy: fifo
            Output queue: 0/40 (size/max)
            5 minute input rate 4302000 bits/sec, 706 packets/sec
            5 minute output rate 0 bits/sec, 0 packets/sec
            1244080 packets input, 950518882 bytes, 0 no buffer
            Received 1244071 broadcasts (1244051 multicast)
            6561 runts, 0 giants, 0 throttles
            31176 input errors, 24615 CRC, 0 frame, 0 overrun, 0 ignored
            0 input packets with dribble condition detected
            874246 packets output, 658148088 bytes, 0 underruns
            0 output errors, 0 collisions, 0 interface resets
            0 babbles, 0 late collision, 0 deferred
            0 lost carrier, 0 no carrier
            0 output buffer failures, 0 output buffers swapped out

            GigabitEthernet1/5 is up, line protocol is up (connected)
            Hardware is Gigabit Ethernet Port, address is 001e.f71a.5944 (bia 001e.f71a.59
            44)
            MTU 1500 bytes, BW 100000 Kbit, DLY 10 usec,
            reliability 255/255, txload 1/255, rxload 12/255
            Encapsulation ARPA, loopback not set
            Keepalive set (10 sec)
            Full-duplex, 100Mb/s, link type is auto, media type is 10/100/1000-TX
            input flow-control is off, output flow-control is off
            ARP type: ARPA, ARP Timeout 04:00:00
            Last input never, output never, output hang never
            Last clearing of "show interface" counters 00:36:22
            Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
            Queueing strategy: fifo
            Output queue: 0/40 (size/max)
            5 minute input rate 5093000 bits/sec, 839 packets/sec
            5 minute output rate 0 bits/sec, 0 packets/sec
            1662763 packets input, 1254087837 bytes, 0 no buffer
            Received 1662742 broadcasts (1662719 multicast)
            339 runts, 0 giants, 0 throttles
            2679 input errors, 2340 CRC, 0 frame, 0 overrun, 0 ignored
            0 input packets with dribble condition detected
            522691 packets output, 393062643 bytes, 0 underruns
            0 output errors, 0 collisions, 0 interface resets
            0 babbles, 0 late collision, 0 deferred
            0 lost carrier, 0 no carrier
            0 output buffer failures, 0 output buffers swapped out

            Do i need to redo my network???

            Thanks again,

            Evan

            Comment


            • #7
              Re: Multicasting Problem - OUT OF SEQUENCE PACKETS‏

              Hmm. fragments maybe? MTU to be made smaller?
              I'm only guessing
              cheers
              Andy

              Please read this before you post:


              Quis custodiet ipsos custodes?

              Comment


              • #8
                Re: Multicasting Problem - OUT OF SEQUENCE PACKETS‏

                Am I trying to lower the mtu on the L3 switch? If so, do I use the global config command ip tcp mss 1200?
                Can I run PIM and CGMP on a single l3 switch the way i have it setup?
                Last edited by mamoser; 17th April 2008, 16:44.

                Comment


                • #9
                  Re: Multicasting Problem - OUT OF SEQUENCE PACKETS‏

                  Hmm, I assume you have set the Sat boxes to 100 full as well. There are a few crc errors but not massive compared to the transfered. Is it possible to clear the stats and watch them when you enable 1 then 2 etc?

                  MTU bits not sure, I reemmber an issue we had with a Cisco PIX where, due to the devices either end, we had to reduce the MTU to get smooth running. Was a VPN I believe. Do the satellite boxes have recommendations?
                  cheers
                  Andy

                  Please read this before you post:


                  Quis custodiet ipsos custodes?

                  Comment


                  • #10
                    Re: Multicasting Problem - OUT OF SEQUENCE PACKETS‏

                    Thanks Andy for the reply. I'll get you that information as soon as possible. I'm currently working on another high priority project and had to take down my lab setup. If all goes as planned, I will rebuild this setup on Monday and provide you the interface information with 1 receiver plugged in and then the results when i plug in the second receiver.

                    I am still trying to comprehend why a crapy 3000$ linksys works and a 11000$ layer 3 switch does not.

                    Have a great weekend...

                    Evan

                    Comment


                    • #11
                      Re: Multicasting Problem - OUT OF SEQUENCE PACKETS‏

                      My modulator has been taken by a coworker... I will keep this thread going once I have it back.

                      Sorry,

                      Evan

                      Comment


                      • #12
                        Re: Multicasting Problem - OUT OF SEQUENCE PACKETS‏

                        We had somone who kept borrowing things. He came in one day to find his chair wheels replaced by small bricks..
                        cheers
                        Andy

                        Please read this before you post:


                        Quis custodiet ipsos custodes?

                        Comment

                        Working...
                        X