Announcement

Collapse
No announcement yet.

Server 2008 unexpected shutdowns

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • crowntech
    started a topic Server 2008 unexpected shutdowns

    Server 2008 unexpected shutdowns

    Hi everyone,

    I have noticed that every 2 weeks our server will unexpectedly restart. The event logs show event id 6008 saying "The previous system shutdown at xxxx on xx/xx/xxxx was unexpected." The only other event prior to event id 6008 is event id 5 stating "The virtual storage filter driver is disabled through the registry. It is inactive for all disk drives." It appears event id 5 occurs during system startup and may not have anything to do with the restarts but I thought it might be worth mentioning. I have checked the task scheduler for scheduled reboots and there is nothing enabled that would allow anything like that to occur. I checked Windows update and that hasn't updated for a few months. This problem began at the end of February and I figured it might have to do with a system update but it began on Feb 23rd and our last update (Windows Powershell & WinRM) was Feb 24th.

    I started to think it might be hardware issue so we purchased a redundant power supply and have both PSUs connected to a backup UPS. For the time being, the restarts are not occurring during business hours which makes it difficult to troubleshoot but fortunately does not half our operations. I'm at a complete loss here, if anyone is willing to point me in the right direction I would greatly appreciate it.

  • crowntech
    replied
    Re: Server 2008 unexpected shutdowns

    Sorry for the delayed response but I tracked it down to the battery backup not being sufficient enough to power the server while on battery. Of course my boss knew the battery wasn't meant for the server but waited to tell me after I spoke with CDW techs.

    Leave a comment:


  • crowntech
    replied
    Re: Server 2008 unexpected shutdowns

    Looks like I've got quite a bit of testing to do this weekend. I'll make sure to run a memory test, disk check, and swap the PSUs. I just checked the power connections and it looks like both of the PSUs are connected to the same battery. We currently have two battery backups, they are APC Back-UPS NS 1250, not entirely sure if this is enough to power everything on them.

    On the first battery we have a total of 5 switches connected (one 8 port, one 16 port, and three 24 ports) and a CCTV PSU connected. On the second battery, we have two 24-port switches, VoIP system (Allworx 6x), and a DVR connected. I can post the voltage specs if anyone thinks that would be helpful.

    On the first battery, I can see the "Building wiring fault" light is on but not on the second battery. Time to dig up the manual!

    Leave a comment:


  • Ossian
    replied
    Re: Server 2008 unexpected shutdowns

    Don't remember if it has been asked, but is the beast under warranty -- if so, get HP onto it

    Leave a comment:


  • RobW
    replied
    Re: Server 2008 unexpected shutdowns

    I would go back to my originial suggestion of a bad power supply failing the BIOS update. If the power is abruptly cut, there is no opportunity to write an event to the log. The HP has redundant power supplies, but only one is working at a time. If one PSU is failing on a temporary basis, maybe the alternate PSU is not working.

    Do you have a good UPS with power conditioning like an APC SC-1500 for example? Are you using a 'Y' power cable to the UPS instead of individual power cords to a power bar?

    Maybe you could see if the server will run on just one PSU. If it does, shut down and switch the PSU and see if it runs on the other one.

    As far as I know, you can't silence the alarm on an HP if you remove a PSU so you will have to live through that. You just want to ensure both PSU are working.

    Leave a comment:


  • wullieb1
    replied
    Re: Server 2008 unexpected shutdowns

    Is there anything in the event logs???

    Have you scanned for viruses and malware???

    Have you tested the hardware in the server???

    I would reckon if the OS was to blame then you would at least see something in the event logs.

    Leave a comment:


  • crowntech
    replied
    Re: Server 2008 unexpected shutdowns

    The option is selected but nothing is generated.

    Leave a comment:


  • tehcamel
    replied
    Re: Server 2008 unexpected shutdowns

    are you generating memory dumps ?

    Leave a comment:


  • crowntech
    replied
    Re: Server 2008 unexpected shutdowns

    Just an update, we had ANOTHER unexpected restart a couple of hours ago. It had been about 9 days since I went and did the BIOS update. We have about 30 OS updates but I wanted to refrain from doing them until I was sure they were necessary. I'm guessing the next steps would be to run a memory and hard drive test? I'm open to other suggestions.

    Leave a comment:


  • crowntech
    replied
    Re: Server 2008 unexpected shutdowns

    Its been about a couple weeks since my last post and I finally got the go ahead to do the BIOS update. I did it this past Sunday and everything seems to be working out just fine. I did have a scare at one point when the OS wouldn't boot but that was because the machine was trying to boot from our external drives attached to it. I guess another couple weeks need to go by until I can really say it's been fixed.

    As a side note, I did notice our server did the unexpected restart 6 days after the previous one (usually its 14). That was pretty much the winning argument since it was starting to occur more frequently.

    Leave a comment:


  • crowntech
    replied
    Re: Server 2008 unexpected shutdowns

    Believe me, I'm all for updating the firmware and while in the middle of composing a convincing email my boss walked in. Of course I took the opportunity to explain my reasoning and gave him a description of the fix I plan on applying. Here is an excerpt directly from the HP website regarding a critical Systems ROMPaq firmware upgrade for the Proliant DL180 G5 series

    Important Notes:
    NoneFirmware Dependencies:
    NoneProblems Fixed:
    Resolved a processor issue where an unlikely and complex sequence of conditions operating in 64-bit mode could result in an application-level error, blue screen, kernel panic, or other unpredictable system behavior. This issue affects platforms utilizing Intel Xeon 5400-series processors. This issue is not unique to HP ProLiant servers and could impact any system using affected processors. This revision of the System ROM contains an updated version of Intel microcode for affected processors that addresses this issue. Due to the potential severity of the issue addressed in this revision of the System ROM, this System ROM upgrade is considered a critical fix. HP strongly recommends immediate update to firmware revisions with required critical fixes.Known Issues:
    None

    I have been given the go-ahead but will have to come in on a day that nobody is here, basically a Sunday which will have to be on the 24th since I won't be in town this weekend. Until then I'll just keep on truckin'
    Attached Files

    Leave a comment:


  • RobW
    replied
    Re: Server 2008 unexpected shutdowns

    I don't understand why someone should provide you a "convincing argument'? It's not like this is a law debating class

    If you have a system whose BIOS is out of date, would you not want to bring everything current before you start to diagnose it? Maybe there was as issue that caused random reboots that was addressed in the update? You won't know until you try. It's akin to trying to fix Windows problems prior to doing any patches, service packs or updates.

    If you and your boss are that anal about trying to fix this, contract the work out to someone else that can assume the responsibility. In my case on our Dell PowerEdge, we have been OK for almost a month now on randon reboots which used to occur weekly. All we did was firmware and BIOS updates but I;m sure you read that already.

    Leave a comment:


  • crowntech
    replied
    Re: Server 2008 unexpected shutdowns

    So it pretty much happened again like clockwork except at a slightly different hour. The interesting thing to note is that I shutdown the VMs on Monday for some system maintenance and did a reboot on the host machine after applying a few updates. This "should" have reset the time period between reboots but it still went ahead and did it last night.

    As suggested, I removed the option to automatically reboot on error but when my boss came in this morning everything was already up and running. Still no memory dumps and the event logs are the same as before.

    I am pretty much convinced that a firmware update will work but my boss isn't. Given that this is our only server capable of housing our 3 VMs, the fear of something going wrong is hindering me from doing what needs to be done. Would someone provide a convincing argument to proceed with the firmware update? I was given the go ahead to do OS updates but I had to cancel in the middle of them on Monday.

    Leave a comment:


  • tehcamel
    replied
    Re: Server 2008 unexpected shutdowns

    Ok, they've told you not to install those updates.
    They've also told you to delay the installation of other patches that could assist the situation.

    So, just wait for it to fall over again. then say "told you i needed to do this"

    Leave a comment:


  • crowntech
    replied
    Re: Server 2008 unexpected shutdowns

    We are running an HP ProLiant DL180 G5. Thank you for your feedback, I will keep you posted on the situation.

    Update: So the time approached to take care of the situation but my boss wants me to save it for next week due to the high work volume. One thing I did notice is the server is due for quite a few updates (30+), I asked to install them but the fear of breaking something was too high so I was told to leave it alone.
    Last edited by crowntech; 1st July 2011, 23:49.

    Leave a comment:

Working...
X