Announcement

Collapse
No announcement yet.

slow drive performance...

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • slow drive performance...

    HP DL380 G4 server with (3) 146GB 15k SCSI RAID drives. RAID controller has 64mb memory and is set for 100%read/0%write.

    No problems whatsoever on this server for the last year/year and a half. Today however is a different story.

    I was asked by the CEO to change permissions on a bunch of shared folders. Previously, permissions were set at the NTFS level using AD global groups. That wasn't adequate enough for the security she wanted, so she asked me to specify individual user accounts for a slew of directories and subdirectories.

    Now, maybe it's just coincidence (or maybe I didn't notice until now), but as soon as I started making the changes (or shortly thereafter), I noticed I was getting an hourglass (cursor) when trying to make changes to permissions, access folders, etc.

    It seems a bit dramatic that my disk drives would be causing the slowness just because I was requested to change permissions on a bunch of folders, but I ran W2003 performance monitor and sure enough, whenever I get an hourglass cursor, "Disk Time % - Write" goes up to 100% and Disk Queues also shoot up. "Disk Time % - Read" also goes up sometimes, but not nearly as often...

    As soon as the % Disk Time drops down from 100%, and the queue length drops down, the hourglass turns back into the cursor and server performance seems normal again.

    Prior to these changes earlier today, I did not (nor was it reported to me) notice any slowdowns or problems with the speed of the server...

    So... is changing NTFS permissions from global groups to specific users this detrimental? Sure seems to be - I'm quite shocked if that's the case.

    I'm wondering how much improved it would be from upgrading the RAID controller to something that could cache writes, and not just reads, or something with a little more cache than 64mb. It would be difficult to convince her to go back to using global groups for NTFS permissions on some of these directories...

    Anyone experience something like this? I'm going to try and track down what specifically the drives are doing when the Disk Time % - Write goes to 100%, using programs like processmonitor.exe from SysInternals, but not having much luck yet...

    JR

  • #2
    Re: slow drive performance...

    How many users are we talking here?

    Always better to abstract permissions through a functional / logical permissions group then assign users to the group.

    How did you get such a hands-on CEO? A bad security audit or ???
    Cheers,

    Rick

    ** Remember to give credit where credit is due and leave reputation points sigpic where appropriate **

    2006-2099 R Valstar. This post is offered "as is" for discussion purposes only with no express or implied warranty of any kind including, but not limited to, correctness or fitness for use. Nothing herein shall be construed as advice. Attempting any activity based on information in this post is done at your own risk.

    Comment


    • #3
      Re: slow drive performance...

      The folder (and subfolders) she wanted me to set up specifically is a folder that, in all honesty, she should be the one to determine who has access to the files... It's not like she decides permissions on everything, just this folder and subfolder.

      I have about 1000 users hitting the server in general. Home drives, profiles, public shared drives, etc. Again, didn't notice any performance problems until yesterday. It might have been coincidence... not sure yet.

      I've ordered a memory upgrade for the RAID controller card, should be here Monday. I'll give that a shot. I'm just trying to specifically determine what is doing this...

      FWIW, I'm not sure I understood your statement about abstracting permissions, heh...

      Comment


      • #4
        Re: slow drive performance...

        Originally posted by jrink View Post
        The folder (and subfolders) she wanted me to set up specifically is a folder that, in all honesty, she should be the one to determine who has access to the files... It's not like she decides permissions on everything, just this folder and subfolder.

        I have about 1000 users hitting the server in general. Home drives, profiles, public shared drives, etc. Again, didn't notice any performance problems until yesterday. It might have been coincidence... not sure yet.

        I've ordered a memory upgrade for the RAID controller card, should be here Monday. I'll give that a shot. I'm just trying to specifically determine what is doing this...

        FWIW, I'm not sure I understood your statement about abstracting permissions, heh...
        So you went from what you had (a few groups per object) to 100's up to 1000 individual users permitted to the same objects. That will kill the performance.

        By abstracting I mean making logical groups that mean something to your business: OrderEntry, CustomerService, AccountsPayable, Finance, ManufacturingSupervisors, whatever and permitting these domain groups to directories and files. Then add the appropriate users in those roles to the groups. Not only does this reduce the number / size of ACL's but it makes it much easier to turnover personnel through promotion or
        Cheers,

        Rick

        ** Remember to give credit where credit is due and leave reputation points sigpic where appropriate **

        2006-2099 R Valstar. This post is offered "as is" for discussion purposes only with no express or implied warranty of any kind including, but not limited to, correctness or fitness for use. Nothing herein shall be construed as advice. Attempting any activity based on information in this post is done at your own risk.

        Comment


        • #5
          Re: slow drive performance...

          Nono.

          I have a nested subfolder off a shared drive. That subfolder's (lets call it Folder 1)permissions used to be based on global groups. The amount of people belonging to these global groups was <30. I've since removed the global groups, and assigned individual user permissions to the Folder1 folder. These individual users number less than 10 people.

          This is why it seems so strange to me. It's not like I added 1000, or even 100, individual users to the ntfs permissions of Folder1.

          Hope that makes sense.

          As for what you said about abstracting groups, yes that is what I was doing. I had a specific group of people as member of a certain global groups. The Folder1 folder had these global groups as either read or read/write permissions. But, based on her recommendation, we've axed the global groups from the Folder1 permission structure and added each individual user. Again, we are talking around 10 users only that have access to Folder1.



          Originally posted by rvalstar View Post
          So you went from what you had (a few groups per object) to 100's up to 1000 individual users permitted to the same objects. That will kill the performance.

          By abstracting I mean making logical groups that mean something to your business: OrderEntry, CustomerService, AccountsPayable, Finance, ManufacturingSupervisors, whatever and permitting these domain groups to directories and files. Then add the appropriate users in those roles to the groups. Not only does this reduce the number / size of ACL's but it makes it much easier to turnover personnel through promotion or

          Comment


          • #6
            Re: slow drive performance...

            OK then the obvious, easy stuff is not it.

            1) You've checked your RAID monitoring tools, the event log, etc. and the RAID array appears fine?

            2) You've done a chkdsk and a defrag?

            3) When's you next available outage window to cycle the server? I'm a programmer by training so I like to rotate the tires to prove we have a flat

            4) Is it possible someone else is writing a bunch?

            5) Your 100% Read / 0% Write on that RAID controller means ???

            6) When you go to the 100% Disk Time % - Write, what process(es) in Task Manager is(are) the biggest CPU consumers?

            I'm hoping 1-3 fix it. Otherwise, we'll see again.
            Cheers,

            Rick

            ** Remember to give credit where credit is due and leave reputation points sigpic where appropriate **

            2006-2099 R Valstar. This post is offered "as is" for discussion purposes only with no express or implied warranty of any kind including, but not limited to, correctness or fitness for use. Nothing herein shall be construed as advice. Attempting any activity based on information in this post is done at your own risk.

            Comment


            • #7
              Re: slow drive performance...

              Hey Rick, thanks for working with me on this...

              FWIW, here's what I'm noticing when running processmonitor.exe on the server. It is showing a very HIGH amount of events by this process "dfssvc.exe" using this path: "c:\windows\system32\activeds.tls". This process easily dominates by far any other process which doesn't even come close to the amount of times/activity of this one in particular.

              For kicks, I ran processmonitor.exe on a few other servers; another domain controller (which is only access by about 80 PCs), and an application server which is accessed by a few hundred people. Neither of those servers are showing that dfssvc.exe and activeds.tls processes/paths in processmonitor.exe. I'm wondering if this is part of the issue. I'm going to try and access another domain controller offsite I have access to that has 2000+ users on it and see if it is also showing high amounts of dfssvc.exe and activeds.tls or not...

              As for your questions, I don't have answers yet... but.
              I did check RAID controller and RAID drives in the HP RAID controller software. No problems reported I also checked disk defrag and that seems fine too. I will reboot the server tomorrow and see if that helps at all... I don't have answers for the rest of the questions yet.

              HTH?

              Comment


              • #8
                Re: slow drive performance...

                So what exactly are you doing w/ DFS? Google on:

                http://www.google.com/search?hl=en&q=dfssvc+100

                Curious.
                Cheers,

                Rick

                ** Remember to give credit where credit is due and leave reputation points sigpic where appropriate **

                2006-2099 R Valstar. This post is offered "as is" for discussion purposes only with no express or implied warranty of any kind including, but not limited to, correctness or fitness for use. Nothing herein shall be construed as advice. Attempting any activity based on information in this post is done at your own risk.

                Comment


                • #9
                  Re: slow drive performance...

                  Therein lies the mystery. I'm not using DFS at all. I don't even have a DFSRoot setup on the server or my domain.

                  Comment


                  • #10
                    Re: slow drive performance...

                    I found this http://support.microsoft.com/kb/285115 but I have a 2003 DC, not 2000....

                    Still interesting.

                    Comment


                    • #11
                      Re: slow drive performance...

                      I take it your W2K3 is patched to current levels. When, again, can you reboot the server (the programmer in me

                      Here's a bedtime read on DFS:

                      http://technet2.microsoft.com/Window....mspx?mfr=true

                      Seen mention of SYSVOL in some other threads.

                      How many DC's do you have?
                      Cheers,

                      Rick

                      ** Remember to give credit where credit is due and leave reputation points sigpic where appropriate **

                      2006-2099 R Valstar. This post is offered "as is" for discussion purposes only with no express or implied warranty of any kind including, but not limited to, correctness or fitness for use. Nothing herein shall be construed as advice. Attempting any activity based on information in this post is done at your own risk.

                      Comment


                      • #12
                        Re: slow drive performance...

                        Hi, jrink.
                        The 100% read / 0% write balance on your RAID card bothers me a little. This means that the readings from the RAID are cached, but the info written is written immediately. While this pattern does have its advantages (if no data is cached, then no damage will result when the server fails, on a power failure, for instance. If the data was cached, then it was lost on such a failure), it can be a real problem when doing a major change that involves disk activity, like the one you describe... Hmmmm... It makes me scratch my beard...
                        If this is the case, then that extra memory on the 6i card is useless and a waste of money.
                        I am trying to find you a paper that describes how to change the balance, but finding something in HP's site is not that easy. I'll find you something and post it.
                        Meanwhile, think of it.
                        Good luck.

                        Sorin Solomon


                        In order to succeed, your desire for success should be greater than your fear of failure.
                        -

                        Comment


                        • #13
                          Re: slow drive performance...

                          I agree with the posibility of not having write cache causing me problems especially since the reporting shows delays when trying to write to the drives...

                          the extra memory I bought for the raid controller 6i card is going to allow me 128mb of write back battery backed up cache... along with the 64mb read cache. i'm hoping that improves things. it'll be here on monday so i'll report back on tuesday.

                          JR

                          Comment


                          • #14
                            Re: slow drive performance...

                            Originally posted by jrink View Post
                            128mb of write back battery backed up cache... along with the 64mb read cache.
                            Sorry, I misunderstood you. I thought you have the extra battery already installed. In this case, it will really make the things easier.
                            I am still trying to get that HP stuff... I am certain you have to do it with SmartStart, but want to be sure... Don't want you to destroy your server and come looking after me :grin

                            Sorin Solomon


                            In order to succeed, your desire for success should be greater than your fear of failure.
                            -

                            Comment


                            • #15
                              Re: slow drive performance...

                              I believe you configure the read/write cache during boot... it can be done with smartstart or before i think by configuring the raid controller before the OS boots or smart start boots. I'll have to find out for sure... but I believe that's what I remember.

                              Comment

                              Working...
                              X