Announcement

Collapse
No announcement yet.

Hard drive spike above 65 ms everyday in the morning

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Hard drive spike above 65 ms everyday in the morning

    Hi All,

    I'd like to know what could be the problem or indication if I got this warning every morning between 6 AM - 7 AM ?

    Code:
    Status: Warning (Yellow)
    Alarm: Host disk write latency
    Time: 6/11/2010 6:26:56 AM
    Level of Disk write latency is above 65 Millisecond
    FYI here's the hardware specs:

    ESXi 4.0 with local VMFS datastore of RAID-1
    2x 1 TB WDC10EARS 64 MB buffer 7200 rpm SATA-II

    it happens just in the morning everyday when there is no significant workload in the server (it is still outside of business hours).

    any kind of suggestion would be greatly appreciated.

    Thanks.

  • #2
    Re: Hard drive spike above 65 ms everyday in the morning

    Do you have a backup running on the storae at that time?

    AV client scan, patch managemanet agent scan?

    How many VM's?

    All the same os?
    Maish
    ----------------------------------------------------------
    Technodrone|@maishsk|Author of VMware vSphere Design
    VMware vExpert 2013-2010,VCAP5-DCA/DCD,VCP
    MSCA 2000/2003, MCSE 2000/2003
    A proud husband and father of 3 girls
    ----------------------------------------------------------
    If you find the information useful please don't forget to give reputation points sigpic.

    Have a good one!!

    Comment


    • #3
      Re: Hard drive spike above 65 ms everyday in the morning

      Do your clients all login at the same time???

      Comment


      • #4
        Re: Hard drive spike above 65 ms everyday in the morning

        thanks for the reply man, i don't think so.
        i was thinking that this is hard drive problem.

        Comment


        • #5
          Re: Hard drive spike above 65 ms everyday in the morning

          Originally posted by Albertwt View Post
          thanks for the reply man, i don't think so.
          i was thinking that this is hard drive problem.
          Yeah i know that but if your clients are all logging into the server at the same time it creates load on the disks.

          So what is happening at 630 in the morning on your servers???

          Are your ESX boxes updating??

          Are your servers updating AV engines or DATS or are they running a scan???

          Are your SQL servers running a big report or loads of transactions???

          We need loads more info to help you determine the root cause of the problem.

          Comment


          • #6
            Re: Hard drive spike above 65 ms everyday in the morning

            Originally posted by wullieb1 View Post
            Yeah i know that but if your clients are all logging into the server at the same time it creates load on the disks.

            So what is happening at 630 in the morning on your servers???

            Are your ESX boxes updating??

            Are your servers updating AV engines or DATS or are they running a scan???

            Are your SQL servers running a big report or loads of transactions???

            We need loads more info to help you determine the root cause of the problem.
            thanks Wullie,
            it is bit weird, since the alarm is now keep on coming every 30 minutes, so far i only keep one VM running which is a web server with almost nill users.

            there is no AV at all.

            Comment


            • #7
              Re: Hard drive spike above 65 ms everyday in the morning

              Have you checked your disks for errors??

              Personally, if you can get the downtime, i'd take the ESX box offline for a bit and check the disks using some sort of diagnostics check. Most drive manufacturers have them.

              What type of server is it that you are using and what are the specs???

              Comment


              • #8
                Re: Hard drive spike above 65 ms everyday in the morning

                Originally posted by wullieb1 View Post
                Have you checked your disks for errors??

                Personally, if you can get the downtime, i'd take the ESX box offline for a bit and check the disks using some sort of diagnostics check. Most drive manufacturers have them.

                What type of server is it that you are using and what are the specs???
                thanks for your response Wullie, it is Dell R200 1U rack server to host ESXi onUSB drive.

                Most of the VM is just DMZ web server.

                Comment


                • #9
                  Re: Hard drive spike above 65 ms everyday in the morning

                  I'd start by checking if there are any orange lights on the drives.

                  Next shutdown and run the Dell Diagnostics.

                  If that doesn't show anything then i'd start looking at running the drive manufacturers utility.

                  You also haven't answered if there is anything disk intensive running on the machine at the time.

                  Comment


                  • #10
                    Re: Hard drive spike above 65 ms everyday in the morning

                    Originally posted by wullieb1 View Post
                    I'd start by checking if there are any orange lights on the drives.

                    Next shutdown and run the Dell Diagnostics.

                    If that doesn't show anything then i'd start looking at running the drive manufacturers utility.

                    You also haven't answered if there is anything disk intensive running on the machine at the time.
                    so far nothing that i know of running on that disk, i guess it is time to do the diagnostic as you suggest.

                    Thanks mate.

                    Comment


                    • #11
                      Re: Hard drive spike above 65 ms everyday in the morning

                      Originally posted by wullieb1 View Post
                      I'd start by checking if there are any orange lights on the drives.

                      Next shutdown and run the Dell Diagnostics.

                      If that doesn't show anything then i'd start looking at running the drive manufacturers utility.

                      You also haven't answered if there is anything disk intensive running on the machine at the time.

                      ok, I forgot to include the real screenshot from my ESXi, here it is, it averages above 40 ms for both disk command latency and the disk commands issued. Is this normal or not OK for a low load web server ?

                      I know that the performance is very slow but i couldn't found anything peculiar in CPU or memory contention.
                      Attached Files
                      Last edited by Albertwt; 24th November 2010, 07:58.

                      Comment


                      • #12
                        Re: Hard drive spike above 65 ms everyday in the morning

                        Looks like your disks go to sleep mode and then when there are waking up by disk activity you see a spike in latency...
                        Check and turn off any sleep mode or energy saver thingy on the disks.
                        Rgds,
                        Didier

                        Freelance, Business Owner, Virtualisation & Cloud Computing Fan Boy, VCP4/5/Cloud, VCAP4/5-DCA/DCD, vEXPERT '10-'14, EMCCA-Specialist, VMware Alumni.

                        Comment

                        Working...
                        X