Announcement

Collapse
No announcement yet.

Desktops/Server drop from network until power cycle?

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Desktops/Server drop from network until power cycle?

    Hi All,

    I've had a long standing issue here which is totally confusing me now and I need to resolve it as its affected a server today.

    Basically, every so often a desktop will drop of the network and shows a disconnected. Nothing you do in the OS (windows or linux) brings it back online, even a reboot doesn't help. Its only when you power down and pull the power cable out for 10 seconds or so, to let it discharge, and then connect and power up does the card work again. It seems random what it affects, one day it can happen and then not again, or happen every hour for a week, then not again. Had one desk where it happened with 3 PCs, but not on the 4th, but the PCs worked fine elsewhere? Today though its happened on a server which has now got me more worried.

    Affected systems are spread between 8 floors of a building, all of different hubs (mix of Cisco and DLink), cabling swapped and checked by network techs as fine. Network cards are different makes.

    Any ideas? Could there be a rouge packet or something like that which is knocking the cards out, and only when discharged they come back up? How would I check for something like this?

    Thanks,

    Andy

  • #2
    Re: Desktops/Server drop from network until power cycle?

    A rogue packet? Discharging the network card? I'm thinking no to both of those.

    My hunch is that you have a problem with one or more of your switches; you either have a switching loop or a bad ARP table somewhere.

    Comment


    • #3
      Re: Desktops/Server drop from network until power cycle?

      Originally posted by joeqwerty View Post
      A rogue packet? Discharging the network card? I'm thinking no to both of those.

      My hunch is that you have a problem with one or more of your switches; you either have a switching loop or a bad ARP table somewhere.
      Great thanks. I've been thinking a problematic switch too but as I've had the problem affect systems on multiple floors (each floor has its own switch), the only consistant switch would be the main one which all switches go into in our server room. I'll start looking there.

      How would I check for a bad ARP table on so many different machines and/or a switching loop?

      Comment


      • #4
        Re: Desktops/Server drop from network until power cycle?

        I would look for a loop first, if there is a loop then that could cause messed up ARP tables in the switches, so it naturally follows that if there's a loop, and you fix it, that you'll fix the ARP tables at the same time.

        As for looking for a loop, what you can do (at least the way I do it) is to draw all of your switches on a piece of paper and draw all of the switch to switch links connecting all of the switches together, then log in to each switch and look at the STP settings for the ports connecting the switches together, depending on how things are connected you should find at least one link in the blocking or discarding state, X out that link or links on the drawing, then draw a host connected to one of the switches and trace all of the possible paths to that host, you should have only one, if you have two or more then you have a switch loop.

        It sounds rudimentary but it's the only way I can visualize the switch connections and analyze the path to my host.

        Comment


        • #5
          Re: Desktops/Server drop from network until power cycle?

          If his network is spread over more then one floor, a paper floor plan might not be able to show you what you want (as it will be drawn by what he believes should be in the walls (unless he is going to check the cables running between floors and such)).

          If you network gear is Cisco, you should be able to get some network topology software to map the network for you.

          I use nMap, but its all manule and on a larger network might take some time to trace to all the work stations.

          Have you checked your licensing? No idea what license could affect unix and windows together, but if you can think of anything on your network, check it.

          Next time it happens on a work station, leave it down while you do some tests like check the Arp table on the work station and on the server (switch, hub, modem, what ever device the work station talks to first), see if the entrys match.

          If you plug the disconnected computer into a different network device (something with a different arp table), does it reconnect?

          Does your Cisco Diagnostics show anything (the switchs should have there own logs, you should be able to see signs of a loop in the logs).

          And a oldie but a goldy, If your work stations are white boxes, or techs that services them are not the best, check to make sure that the NIC is earthed. HIGHLY unlikly, but I have seen Unearthed NIC drop connections till the charge is discharged. If its a PCI card, then its earth if screawed in, if its on board make sure you have the faceplates metal tab on top of the NIC data port.

          Wofen

          PS: Sorry for how un-ordered that last post is, it was typed over 45mins in bits and pieces. And the red bull does not help.
          Last edited by Wofen; 13th October 2010, 03:27. Reason: Appologising for drink a can of redbull.
          Good to be back....

          Comment


          • #6
            Re: Desktops/Server drop from network until power cycle?

            Thanks for all that, I really appreciate it.

            I've taken a glance through our cabling setups on each of the floors/switches and everything looks to be OK. 1 cable from the main switch in the server room, to the first switch on each floor, and any floors which have more than one switch are connected port 48-24 from that so looks OK to me.

            2 of the floors have unmanaged switches <groan, want them replaced>, but the others don't show any errors on ports, EXCEPT for one port on the main switch in the server room. It was kicking up thousands of RXerrors on one port which I traced to an old DC. I've installed 2 new DCs recently, but this old one remains as it holds the print queues which I haven't got round to migrating yet. I replaced every option of cable switch-patch-socket-server and direct but still get errors so it looks like the card/driver so I'll try a driver update first.

            Also, don't know if this could be it, but it surprised me. I tried that nmap software and when I did a scan on my local machine it returned the wrong hostname? I checked in the reverse DNS zone on my DNS server and found hundreds of duplicate entries for system where it looks like the previous IT bod who I have recently replaced, never cleaned up. My system alone had 5 entries, 1 correct but 4 which hadn't been used in several years. Forward look ups was the same so I've gone through and pulled out all the ones which haven't updated in the last couple of weeks.

            Think this could be the culprit?

            Will keep working on it for now. Thanks for the help so far guys.

            PS. Wofen - post was fine, and Red Bull always helps

            Comment


            • #7
              Re: Desktops/Server drop from network until power cycle?

              Umm, I can not say for sure if the rDNS mess could be causing this, but at the very lest there is no way that it is helping.

              I would attempt to sort that out first, and see if it helps.

              If you do need help to resolve this, please let us know, but I am over rDNS atm (they are braking all over the place in the last month), so I will only type out my advice if you need it. There are currently 3 rDNS problems on the forums atm.

              Ok, one thing then, are you able to just delete all the rDNS entries (please test by removing the entries for a single machine first)? and If so, does it recreate the correct entry, or all of them, inculding the wronge.

              How does Nmap show the topolgy of your network? It should look like a wheel and spoke with no side channels (a line that connects 2 of the spokes).

              If in doubt, take a screen shot of the topolgy and we can have a look, just please remove any publice IP info, as well as your SBS info on the image.

              Wofen

              PS: Thank you for taking the time to up my Ref and give feed back.
              Last edited by Wofen; 15th October 2010, 07:15. Reason: Giving Thanks
              Good to be back....

              Comment


              • #8
                Re: Desktops/Server drop from network until power cycle?

                It's doubtful that the rDNS problem is causing this network connectivity problem. Windows computers find other Windows computers via hostname, which is resolved to the ip address, via the forward lookup zones, not the reverse lookup zones.

                In fact, as a test, delete yout rDNS zones. They're not required and not having them will do no harm. If the problem goes away then you'll know that was it, if not then you'll need to keep digging.

                I'm wondering about the old DC and what it's sending out on to the network. It could be flooding the switches.
                Last edited by joeqwerty; 15th October 2010, 12:25.

                Comment


                • #9
                  Re: Desktops/Server drop from network until power cycle?

                  Thanks for the help.

                  I deleted all the rDNS entries which I knew were wrong or had just not been updated in the last 2 months. Also did the same for any forward DNS entries too, and since doing that last week none of the wrong entries had re-appeared and everything seems to be working fine.

                  nMap has come up with this when scanning my local IP range


                  Hopefully that looks right?

                  So far though, no further network drops since last week when I tidying up DNS. I'm trying to migrate all the users print queues at the moment of the old DC and onto the new one so once this is done I can pretty much decommission the old DC then (just have to remove it as my secondary DNS) as I don't like that kicking out errors like that.

                  Comment


                  • #10
                    Re: Desktops/Server drop from network until power cycle?

                    First off, I have no idea how you even made nmap generate that image. It looks like a fibonacci image.

                    A topology image should be a map of your network, with any IP device being a point, and each line a network cable (or connection type). Can you please tell me the steps you took, and what version you used to create that image. It could mean there is a loop in your network (as there should be a start and end to the topolgy map), or it could mean you have a different version/methiod that produced something I am not fimilar with.

                    That image is pretty....

                    While I do not believe the problem is caused by the rDNS entrys, I do believe they could be a syptom of the problem as well (each time the computer has to be power cycled, the DNS can not work out how to clear it).

                    Other then this, I would say anytime a computer goes offline, go though it and see if there is any errors in the event logs, test all types of network access, test anything and everything you can think of on the work station, and then on the next. Hopfully we can find a pattern or something, as right now this is a needle in a hay stack, and we need to shink the haystack.

                    Wofen

                    PS: You said every 2 hours... is it Exactuly every 2 hours (give or take 5 seconds), and is it on the hour (or half). IE, is there something that happens every 2 hours that creates this problem?
                    Last edited by Wofen; 18th October 2010, 03:16. Reason: One more quesiton
                    Good to be back....

                    Comment


                    • #11
                      Re: Desktops/Server drop from network until power cycle?

                      Originally posted by Wofen View Post
                      First off, I have no idea how you even made nmap generate that image. It looks like a fibonacci image.

                      A topology image should be a map of your network, with any IP device being a point, and each line a network cable (or connection type). Can you please tell me the steps you took, and what version you used to create that image. It could mean there is a loop in your network (as there should be a start and end to the topolgy map), or it could mean you have a different version/methiod that produced something I am not fimilar with.

                      That image is pretty....
                      I used zenmap 5.21 and to get that I just put in my network range (172.25.1.0/24) and picked ping scan. Let it do its thing for a few seconds and then selected topology and thats the image I get.

                      Originally posted by Wofen View Post
                      While I do not believe the problem is caused by the rDNS entrys, I do believe they could be a syptom of the problem as well (each time the computer has to be power cycled, the DNS can not work out how to clear it).

                      Other then this, I would say anytime a computer goes offline, go though it and see if there is any errors in the event logs, test all types of network access, test anything and everything you can think of on the work station, and then on the next. Hopfully we can find a pattern or something, as right now this is a needle in a hay stack, and we need to shink the haystack.
                      OK, since I deleted all the old rDNS entries and false forward entries none of them have come back and everything seemed OK. I thought it had solved the problem but then this morning 3 machines dropped off, all at different times starting from 9pm last night to 7am this morning.

                      As I've seen on all the machines before, nothing in the logs, you just suddenly get the time service saying it cant update and thats the first error. Nothing else related that I can spot. Also, when I check on the machine it just shows the NIC as disconnected (with no lights on the NIC card either). If I look at the ARP cache etc it all looks fine, cant spot anything particular.

                      Really bizarre. Gradually getting users of the old DC print queues so hopefully I can get rid of that soon too.

                      Originally posted by Wofen View Post
                      PS: You said every 2 hours... is it Exactuly every 2 hours (give or take 5 seconds), and is it on the hour (or half). IE, is there something that happens every 2 hours that creates this problem?
                      Errrr 2 months I think, well thats the last time and of the rouge DNS entries which I deleted showed up.

                      Comment


                      • #12
                        Re: Desktops/Server drop from network until power cycle?

                        solarwinds engineer's toolset is quite a good tool for ur diagrams and monitoring, will probibaly have some tools to help narrow down the problem more too.
                        Last edited by uk_network; 19th October 2010, 00:55.
                        Please remember to award reputation points if you have received good advice.
                        I do tend to think 'outside the box' so others may not always share the same views.

                        MCITP -W7,
                        MCSA+Messaging, CCENT, ICND2 slowly getting around to.

                        Comment


                        • #13
                          Re: Desktops/Server drop from network until power cycle?

                          I hate to say it, but I am coming to a loss. I would recommend creating a log to log all the times and dates it happens ot what machine, and see if you could isolute it.

                          I will keep thinking about this, and if I come up with any ideas, I will let you know.

                          All I can recommend is log everything, no matter how small. I would also get some packet sniffers onto the network, see if any packets are behaving strange.

                          One thing to test, When a computer does this, if you pull the network cable out of it, and plug a 2nd connected cable in, does the computer reconnect (Ie, is the problem in the computer/network, or in the device suppling connection).

                          I wish I could of been more help.

                          Wofen
                          Good to be back....

                          Comment


                          • #14
                            Re: Desktops/Server drop from network until power cycle?

                            Just realized I never updated this topic.

                            I fixed it in the end with a driver update to the Realtec Network Cards in the PCs. Turned out all the affected machines were using the same network card (onboard even though different mobos) and as soon as I updated the drivers on the affected PCs the problem went away, and touch wood, hasn't re-occured in nearly 6 months.

                            PHEW!

                            Thanks guys.

                            Comment


                            • #15
                              Re: Desktops/Server drop from network until power cycle?

                              This is an increasingly common problem these days - although usually on Broadcom chips in my opinion, I've never had a problem with Realtek drivers before. Out of interest, what was the chipset they use?

                              In any case well done for fixing it and thanks for letting us know the outcome. rep++;
                              Gareth Howells

                              BSc (Hons), MBCS, MCP, MCDST, ICCE

                              Any advice is given in good faith and without warranty.

                              Please give reputation points if somebody has helped you.

                              "For by now I could have stretched out my hand and struck you and your people with a plague that would have wiped you off the Earth." (Exodus 9:15) - I could kill you with my thumb.

                              "Everything that lives and moves will be food for you." (Genesis 9:3) - For every animal you don't eat, I'm going to eat three.

                              Comment

                              Working...
                              X