Announcement

Collapse
No announcement yet.

Server fails Ping tests when teamed NIC is disconnected!?

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Server fails Ping tests when teamed NIC is disconnected!?

    I have a server, CLA-TADCM, which has two NICs [Broadcom BCM5708C Nextreme II GigE] teamed using Broadcom Advanced Control Suite 2 version 8.3.11. The network guys use a server called CLA-MON to monitor all servers using Ping tests. CLA-MON runs ServersAlive V6.2.2324.

    Recently, one of our servers has been failing the Ping tests intermittently. One of the NICs was disconnected as the switch was needed in another part of the data centre then the Ping tests started failing.

    The network guys investigated the issue and here's what they said:
    I have investigating this issue a bit more this after noon.

    First I added a static ARP (an IP->MAC mapping) on CLA-MON for CLA-TADCM and this seemed to resolve the issue for a few hours.
    I then removed the static ARP entry from CLA-MON and again and the alerts started.
    I then did the same on CLA-TADCM adding an entry for CLA-MON and the false alerts continued.

    This pointed to an issue with CLA-TADCM not responding correctly to ARP requests, if the server was not responding correctly to ARP requests this would have caused these false positives, this obviously would not have affected the already established connections.

    This indicated that maybe the ARP requests were being sent down the inactive NIC, I then took the inactive NIC out of the teaming, leaving only the active NIC in teaming. I have since noticed that no more false alerts are being generated for this host.

    The underlaying cause was obviously a teaming/driver issue with the server.
    I don't understand how a server can suddenly stop responding, intermittently, to PING tests when the issue is related to the NIC drivers. I have not updated the drivers yet as I want to know what is happening first.

    Is there any way to test if it is actually an ARP problem?
    |
    +-- JDMils
    |
    +-- Regional Systems Engineer, DotNet programmer & Jack of all trades
    |

  • #2
    Re: Server fails Ping tests when teamed NIC is disconnected!?

    Did you check on the monitoring server to see what ARP address was cached there? It likely had the removed teamed nic in the cache and failed when it attempted to resolve the physical address for the ping further down the chain.
    Andrew

    ** Remember to give credit where credit is due and leave reputation points sigpic where appropriate **

    Comment


    • #3
      Re: Server fails Ping tests when teamed NIC is disconnected!?

      ARP cache addresses last for approx 20mins max. When a system performs a new broadcast, it asks the destination IP for for its MAC address. The destination IP did not exist if the IP for your system was the one that was disconnected. Therefore, a connection was not made and the ping failed.

      As the person says, once the redundant NIC was removed, that left just the other IP address registered with the server. Because it was still active, ARP requests for the IP's MAC address succeeded.

      I've never used teaming, but assume that the ping test used does not cycle through all registered IP's for a server. I assume that's why it failed with a 'false alert'.

      I'm still learning this too...
      Last edited by Blood; 11th March 2010, 14:38. Reason: clarity
      A recent poll suggests that 6 out of 7 dwarfs are not happy

      Comment


      • #4
        Re: Server fails Ping tests when teamed NIC is disconnected!?

        Originally posted by Blood View Post
        ARP cache addresses last for approx 20mins max. When a system performs a new broadcast, it asks the destination IP for for its MAC address. The destination IP did not exist if the IP for your system was the one that was disconnected. Therefore, a connection was not made and the ping failed.

        As the person says, once the redundant NIC was removed, that left just the other IP address registered with the server. Because it was still active, ARP requests for the IP's MAC address succeeded.

        I've never used teaming, but assume that the ping test used does not cycle through all registered IP's for a server. I assume that's why it failed with a 'false alert'.

        I'm still learning this too...
        True, time is definately at play here. Still leaves the whole thing unanswered though. Strange behavior.
        Andrew

        ** Remember to give credit where credit is due and leave reputation points sigpic where appropriate **

        Comment


        • #5
          Re: Server fails Ping tests when teamed NIC is disconnected!?

          Only problem with the discussions so far as that in Teaming, although the server has two NICs, the teaming application creates a new virtual NIC with relevant details. For example:

          NIC1:
          MAC: 00-18-8B-76-C5-DB
          IP Address: N/A
          Connection Status: Connected
          NIC2:
          MAC: 00-18-8B-76-C5-DD
          IP Address: N/A
          Connection Status: Disconnected
          Teaming (Virtual) NIC:
          Windows IP Configuration

          Host Name . . . . . . . . . . . . : cla-tadcm
          Primary Dns Suffix . . . . . . . : xxx.aaaa.com.au
          Node Type . . . . . . . . . . . . : Unknown
          IP Routing Enabled. . . . . . . . : No
          WINS Proxy Enabled. . . . . . . . : No
          DNS Suffix Search List. . . . . . : xxx.aaaa.com.au
          yyy.aaaa.com.au
          dmz.aaaa.com.au
          zzzz.bbbb.com
          cccc.com.au

          Ethernet adapter Team #1:

          Connection-specific DNS Suffix . :
          Description . . . . . . . . . . . : BASP Virtual Adapter
          Physical Address. . . . . . . . . : 00-18-8B-76-C5-DB
          DHCP Enabled. . . . . . . . . . . : No
          IP Address. . . . . . . . . . . . : 10.3.1.172
          Subnet Mask . . . . . . . . . . . : 255.255.252.0
          Default Gateway . . . . . . . . . : 10.3.0.1
          DNS Servers . . . . . . . . . . . : 10.3.0.50
          10.16.1.10
          NetBIOS over Tcpip. . . . . . . . : Disabled
          The teaming app has thus taken the NIC1 MAC and the IP set aside for the server. Thus a ping to the server will always be to the same IP and to the same MAC address.

          So the question remains, why did the Ping test fail when the second NIC was disconnected?

          I have re-added the 2nd NIC to the team, although it is still disconnected, and have not had a Ping alert from the monitoring system yet.
          Last edited by JDMils; 11th March 2010, 22:46.
          |
          +-- JDMils
          |
          +-- Regional Systems Engineer, DotNet programmer & Jack of all trades
          |

          Comment


          • #6
            Re: Server fails Ping tests when teamed NIC is disconnected!?

            Ah then the problem isn't with the hardware or the abscence of it but rather with the teaming driver/application imo.
            Andrew

            ** Remember to give credit where credit is due and leave reputation points sigpic where appropriate **

            Comment


            • #7
              Re: Server fails Ping tests when teamed NIC is disconnected!?

              Thanks for that. I will thus upgrade the drivers.
              |
              +-- JDMils
              |
              +-- Regional Systems Engineer, DotNet programmer & Jack of all trades
              |

              Comment

              Working...
              X