Announcement

Collapse
No announcement yet.

Migrating disks from MSA SAN to new NVX SAN- disk locking issues

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Migrating disks from MSA SAN to new NVX SAN- disk locking issues

    We have a number of Windows 2008 R2 SP1 servers which have data disks presented to them from an HP MSA1000 SAN and I am in the process of migrating these disks to our new EMC VNX SAN. The servers are part of an Active/Passive Windows & SQL cluster.

    We use PowerPath to manage the SAN connections for the VNX SAN.

    I am using EMC Open Migrator to migrate the data from the source disks (MSA SAN) to the destination disks (VNX SAN) which works really well. When I get my outage for the clusters, what I do is click on Migrate on each disk in the cluster (in OM), and Open Migrator (OM) automatically "swaps" the disks so that the servers maintain the same disk number in Disk Manager and drive letters in the OS.

    This has proven to work with every clustered Windows/SQL migration I have done so far (around 15) but I had one issue on the weekend which not only brought the migration of cluster X to a sudden stop, but killed my confidence in the migration process which has shattered me.

    When I clicked on "Migrate" in OM, I got the following warning:

    "Following processes are accessing the source volume:
    System, sqlserver.exe, fdlauncher.exe, SQLAGENT.EXE"

    I was able to stop the processes "fdlauncher.exe, SQLAGENT.EXE" simply by killing the processes in Task Manager, but problem #1 was "sqlserver.exe". If I killed the process, the cluster would automatically migrate the disks to the other node which would kill the OM copy process. You see, OM has to have the disks on the node it is installed on 100% of the time so it can keep an eye on the data copy (it's a block-to-block copy). As soon as OM loses the disk(s) to the other node of the cluster, I have to restart the copy process which takes around 1 day.

    So how do I stop SQL from writing to the source disk without causing the cluster to failover?

    The other problem was that "System" was accessing the disk. What exactly is "System"? I think it may be a generic name for "a Windows process" but I was unable to find the process which was accessing the disk. Is there a fool-proof way of getting this info?

    I have a few more clusters to do in the coming weeks, all Active-Active clusters where each node has their own preferred disks from the cluster, so I need to find out why this last cluster migration failed before I can attempt the others.

    Can anyone help with suggestions?
    Last edited by JDMils; 27th August 2013, 02:25. Reason: Resolved: Page File setting was still using the SAN disks even though there was no page file there!
    |
    +-- JDMils
    |
    +-- Regional Systems Engineer, DotNet programmer & Jack of all trades
    |

  • #2
    Re: Migrating disks from MSA SAN to new NVX SAN- disk locking issues

    You'll probably need to bring the resource offline to ensure it doesn't failover.

    Check out these steps: http://msmvps.com/blogs/clusterhelp/...l-22-2005.aspx
    Regards,
    Jeremy

    Network Consultant/Engineer
    Baltimore - Washington area and beyond
    www.gma-cpa.com

    Comment


    • #3
      Re: Migrating disks from MSA SAN to new NVX SAN- disk locking issues

      Taking SQL offline in the cluster is a valid point. This would take care of "sqlserver.exe", but what about the "System" process which seems to be using the disk? How do I decipher which process this is?
      |
      +-- JDMils
      |
      +-- Regional Systems Engineer, DotNet programmer & Jack of all trades
      |

      Comment


      • #4
        Re: Migrating disks from MSA SAN to new NVX SAN- disk locking issues

        Download Process Explorer from www.sysinternals.com and then you can search in it for the lock. It will give you a lot of information about the process.
        Regards,
        Jeremy

        Network Consultant/Engineer
        Baltimore - Washington area and beyond
        www.gma-cpa.com

        Comment


        • #5
          Re: Migrating disks from MSA SAN to new NVX SAN- disk locking issues

          Thanks Jeremy. I retried the migration tonight and again it failed with the same processes locking the source drive.

          I tried taking the SQL resources offline but it also takes the disk offline- the disk is NOT visible at all to the OS and Disk Manager shows the disk but no actions are possible. Plus this process killed the disk sync which Open Migrator was taking care of since it lost "sight" of the source drive!

          Last Saturday, I was able to kill the SQL services but shutting down the second node, so the disk resources had no where to go and they stayed on the node with Open Migrator installed and thus kept the disk sync alive.

          However the System process was still locking the drive, same as tonight, and I can't figger out what it is!
          |
          +-- JDMils
          |
          +-- Regional Systems Engineer, DotNet programmer & Jack of all trades
          |

          Comment


          • #6
            Re: Migrating disks from MSA SAN to new NVX SAN- disk locking issues

            I'm looking at the server right now and I noticed that the page file was set to automatic:

            "Automatically manage paging file size for all drives" is checked

            Listed disks:

            C:
            D: [SQL_DATA]
            M: [DTC]
            Q: [QUORUM]

            The shared SAN cluster disks are D, M & Q so does the above mean that the System will use the clustered disks for paging as well?

            I've looked at the D, M & Q drives and I cannot see the "pagefile.sys" file on them.

            Could this be my problem?
            |
            +-- JDMils
            |
            +-- Regional Systems Engineer, DotNet programmer & Jack of all trades
            |

            Comment


            • #7
              Re: Migrating disks from MSA SAN to new NVX SAN- disk locking issues

              Originally posted by JDMils View Post
              I'm looking at the server right now and I noticed that the page file was set to automatic:

              "Automatically manage paging file size for all drives" is checked

              Listed disks:

              C:
              D: [SQL_DATA]
              M: [DTC]
              Q: [QUORUM]

              The shared SAN cluster disks are D, M & Q so does the above mean that the System will use the clustered disks for paging as well?

              I've looked at the D, M & Q drives and I cannot see the "pagefile.sys" file on them.

              Could this be my problem?
              No. Typically the page file is stored on the boot drive so in your case it automatically goes there.

              http://support.microsoft.com/kb/314482

              Comment


              • #8
                Re: Migrating disks from MSA SAN to new NVX SAN- disk locking issues

                Originally posted by JDMils View Post
                I tried taking the SQL resources offline but it also takes the disk offline- the disk is NOT visible at all to the OS and Disk Manager shows the disk but no actions are possible. Plus this process killed the disk sync which Open Migrator was taking care of since it lost "sight" of the source drive!
                That stinks. Can you open a case with EMC? The might/should have documentation on how to perform a cluster disk migration.
                Regards,
                Jeremy

                Network Consultant/Engineer
                Baltimore - Washington area and beyond
                www.gma-cpa.com

                Comment


                • #9
                  Re: Migrating disks from MSA SAN to new NVX SAN- disk locking issues

                  I performed the same type of migration on an Active/Active SQL cluster and everything went really smoothly with NO issues! And I've done this around 15 times already on Windows 2003 & Windows 2008 all without issues.

                  I found that I could still take the SQL cluster resources offline without the cluster failing over. Note that I'm talking about the cluster resource NOT the cluster group. So, in Cluster Manager, viewing the SQLServer group, I can see:

                  Summary of SQL Server (MSSQLSERVER):

                  * Server Name => SQL Cluster Name & IP.
                  * Disk Drives => SQL Disk Volume: (D.
                  * File Share Resources => MSSQLSERVER.
                  * Other Resources => SQL Server, SQL Server Agent.

                  I thus right-click the SQL Server Agent and/or SQL Server resources and I can "Take Offline". This does not cause the cluster to failover, but stops SQL from locking the disk.

                  The only thing I can find is the Page File setting which could be the culprit. I've scheduled a server reboot for tonight where I will fix the Page File to the C drive and on Saturday we'll see what happens!

                  Thanks for your time so far guys!
                  |
                  +-- JDMils
                  |
                  +-- Regional Systems Engineer, DotNet programmer & Jack of all trades
                  |

                  Comment


                  • #10
                    Re: Migrating disks from MSA SAN to new NVX SAN- disk locking issues

                    UPDATE: After resetting the Page File settings for the server so that it did not try to use the data (SAN) disks for Page File, I set the Page File as System Managed but only for the system drive which required a reboot.

                    We re-tried the Open Migrator migration on the weekend and it worked! Not an issue! Wow, what a relief.

                    Thanks everyone for your help.
                    |
                    +-- JDMils
                    |
                    +-- Regional Systems Engineer, DotNet programmer & Jack of all trades
                    |

                    Comment


                    • #11
                      Re: Migrating disks from MSA SAN to new NVX SAN- disk locking issues

                      Excellent! Thanks for posting back.
                      Regards,
                      Jeremy

                      Network Consultant/Engineer
                      Baltimore - Washington area and beyond
                      www.gma-cpa.com

                      Comment


                      • #12
                        Re: Migrating disks from MSA SAN to new NVX SAN- disk locking issues

                        Thanks guys for your help- which is always plentiful!

                        FYI, I use Open Migrator to migrate from non-EMC SANs to EMC SANs (although you could probably use it to migrate TO non-EMC SANs as well). It installs a device driver which, once configured, monitors the source disk(s) and automatically block-copies new data to the destination disk- while the source disk is still the disk which the server & SQL is using for storage.

                        When the source & destination disks are in sync, it keeps them in sync as data is written to the source disks. You click Migrate when you are ready and OM automatically swaps the source & destination disks by swapping the drive letters on both drives so the destination disk is now the same drive letter as what the source disk was and this keeps apps happy. It also reconfigures the Windows cluster to suit.

                        When you are done, just release the OM device driver from the disks, uninstall OM, reboot and manually reconfigure your DTC & Quorum (2 minute job). Your clustered disks are now migrated and running on the new EMC SAN.

                        And I have done this with SQL still running! It's documented to do this well. And of course, I use PowerPath to handle the multipathing.

                        The other option is to actually use PowerPath to do the migration, and this is a much easier and more graceful option, but only works if PowerPath can see the source & destination disks- which is when you are using EMC SANs for source & destination. It's called PowerPath migrator and is an extra option you select when installing PowerPath- note it requires a separate serial number and thus license, but it is WELL worth it!

                        Using PP Migrator, you select the source & destination disks and start the sync via the command line. When done, you tell it to switch the disks and the migration is over! Really simple.

                        In both case you can control the rate of copy from source to destination disks but PP Migrator is just that more intuitive and comfortable to use and you don't need to install & uninstall drivers and software.

                        If anyone needs more info then just let me know.
                        |
                        +-- JDMils
                        |
                        +-- Regional Systems Engineer, DotNet programmer & Jack of all trades
                        |

                        Comment


                        • #13
                          Re: Migrating disks from MSA SAN to new NVX SAN- disk locking issues

                          Again, thanks for the info.
                          Regards,
                          Jeremy

                          Network Consultant/Engineer
                          Baltimore - Washington area and beyond
                          www.gma-cpa.com

                          Comment

                          Working...
                          X