No announcement yet.

Packets being dropped

  • Filter
  • Time
  • Show
Clear All
new posts

  • Packets being dropped

    I have two applications need need to see mirrored traffic. One application sees the traffic fine while the other does not. The data it does see doesn't make sense and is inconsistent. The traffic is coming from a 1 Gb Dell 6248 switch, to a 10/100 Cisco 3550 and from there to the two application servers. We had to put the Cisco switch in the middle because the Dell switch was not capable of having to destination ports for mirroring.

    Now we eliminated the possibility of it being the application by performing a tcpdump on the application server running the latest RHEL. After importing into WireShark and a little filtering, I see a lot of TCP ACKed lost segment data coming across. It seems that packets are just being cut off. This means that the data is lost prior to being processed by the application.

    In terms of troubleshooting, I have bypassed the Cisco switch and connected the monitor port directly to the application and ended up with the same issue. I have also tried hard coding the speeds on the NIC. The cable has been replaced with a new one. Traffic to the monitored port is around 10Mbps, with spikes to 25Mbps.

    Any ideas as to why this would be occurring? I don't know Linux too well and we do not have a network engineer on staff. I welcome any ideas as for I have devoted tons of hours into trying to resolve this issue. Thank you in advance.

  • #2
    Re: Packets being dropped

    Hi SudoNim,

    Can you please provide a basic diagram? Is there any routing happening in between or any load balancing going on? You may be experiencing a symptom of asynchronous routing.



    • #3
      Re: Packets being dropped

      Thank you for your reply. I'll try to describe the layout the best way I can. -

      The Dell switch is plugged into a load balancer appliance. The port that the load balancer is connected to is being mirrored. The destination port is connected to the Cisco Switch, which is connected to the two application servers. So its:

      Load Balancer
      -->Dell Switch
      ---->Cisco Switch
      ------>Applications Servers

      It's a flat network. There is no special configuration for switch other than the SPAN port sessions setup. I hope this help and again, thanks for the response.


      • #4
        Re: Packets being dropped

        Based on your description and your diagram I would be almost certain it is your load balancer. I'm not say it is mis-configured, but I bet it as to do with your load balancing algorithm (i.e. Round Robin, First Available, etc). What I would suggest is run tcpdump on the loadbalancer to see how it is forwarding your packets; or place a sniffer before traffic hits the load balancer and place one behind and you should be able to find the trouble spot.



        • #5
          Re: Packets being dropped

          I'm not familiar with load balancers, however I've seen some network protection features in some switches "choking" some of the packets. I'[m sure this can happen in any network equipment. Worth a shot.

          MCSE 2003, MCITP EA, VCP4.


          • #6
            Re: Packets being dropped

            Thanks for the responses. I haven't had the chance to complete the suggestions regarding the load balancer but did find some more information.

            As I stated below, the traffic is coming from the load balancer into the Dell Switch on g14. It's monitor session destination port is g44.

            From there, it connects to the Cisco Switch, f0/3, which has 2 monitor sessions, 1 going to f0/5 and the other f0/7.

            Most importantly, I want to share with you some screenshots regarding FCS errors. This evidently is my problem or at least a good starting point. I'm not familiar with this type of issue and am currently trying to understand it. Also, the final screenshot is the bits/sec on the ports for the Cisco switch. They show traffic over 90Mbps. We do not have this much traffic flowing to our entire environment, so I do not understand how this is possible.

            Hmmm...I am unable to post the screenshots due to being new (need 5 or more posts). Will try to explain via text.

            The first screenshot shows the Dell switch managment Web GUI and displays "Frame Check Sequence(FCS) Errors" as a bar chart. The only port to display any errors is port g14, and it shows 1093.

            The seconds screenshot is probably less important but shows the "Received Rate (MFrame Bits/sec)" which displays a lot of traffic on g14 and very little traffic on g44.

            Finally, the Cisco switch, which displays 90+Mbps across all ports consistently.

            Is the root of my problem this error. I also get the same amount when selecting to display "CRC & Align Errors". Thank you for your attention.


            • #7
              Re: Packets being dropped

              Adding screenshots:

              Last edited by SudoNim; 9th November 2010, 18:54. Reason: adding screenshots


              • #8
                Re: Packets being dropped

                Since you are recieving errors on the dell switch a port g14 have you tried using another port to connect your loadbalancer? FCS errors are typically a hardware type issue. Every time a frame traverses a link (switch interface, router port, etc) the FCS get checked and recalculated through each hop. You may have a bad port on the dell switch or the load balancer. If the FCS is bad then the packet will get dropped. I suggest you put a sniffer between your load balance and your switch (the dell switch). I'm not sure what type of load balancer you use but some of the high-end ones (such as F5) will allow you to run tcpdump directly off the appliance. This should help you determine where the packets are getting mangled. Also check your cables with a tester to ensure there isn't any type of line noise because that can cause the FCS errors.

                As far as the traffic across the backplane of the switch (all ports at 90Mbps) that could be anything. Mis-configured STP (causing sub optimal switch path to overload the switch itself. I have actually experienced that in a past life.) or you could be the victim of a broadcast storm, especially since you are on a flat network.

                Just my $0.2

                Last edited by ryansmitty; 10th November 2010, 04:54.