Announcement

Collapse
No announcement yet.

Disaster...how to recover?

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Disaster...how to recover?

    I'm in a major mess! I have a domain with 3 DC's (3x2008 ). 2 of those DC's went down this week as a total loss. I may (slim chance) be able to get system state restores off a tape, but it's a slim chance. We have backup issues as well of course...

    So I currently have 1 working DC, I've added another to the domain (2003 machine) and need to work on replacing my 2 downed machines. What's the best course of action?
    Last edited by biggles77; 16th February 2009, 00:58. Reason: Fix 2008) smilie issue.

  • #2
    Re: Disaster...how to recover?

    Have you made sure all FSMO roles are active on the current domain and hold the GC? It's important to do that first of all.

    As you have a DC up you shouldn't need to use any restores on the other DCs unless you are missing AD objects.

    Your are probably best to cleanup the metadata of the failed DCs. Reimage them, bit first find out why they failed, and then rejoin them to the domain.

    Comment


    • #3
      Re: Disaster...how to recover?

      As it is the weekend you have a little time which is good.
      Ifyou have already added another DC then, as Virtual state, make sure you have sorted your FSMO roles and GC status.
      If you use DHCP then check it is on the live server, if not set it up and test.
      Make sure any servers are using a correct IP address for DNS.
      Check remote logins, IAS etc are working.
      cheers
      Andy

      Please read this before you post:


      Quis custodiet ipsos custodes?

      Comment


      • #4
        Re: Disaster...how to recover?

        I would start by trying to find out why the DC's failed in the first place.

        Sieze all the FSMO roles onto the remaining server, they can be moved when the other servers are up and running.

        http://www.petri.com/seizing_fsmo_roles.htm

        Then i would move onto clearing the metadata from AD on the downed servers.

        http://www.petri.com/delete_failed_dcs_from_ad.htm

        Move all services that were on the failed servers to the new one temporarily.

        Me personally i would then sort the issue as to why the DC's failed then rebuild from scratch doing one server at a time.

        Seems strange that both your server 2008 machines have fallen over. Did you make any chages to them prior to failure??

        Comment


        • #5
          Re: Disaster...how to recover?

          Thanks for the tips guys, I'm going to start today on sorting this all out. The DC's failed due to a SAN catastrophe that took the virtual servers with it. Of course these DCs were VS'...

          From this point I see my GC missing, all roles are showing ERROR and can't be transferred via GUI.

          Comment


          • #6
            Re: Disaster...how to recover?

            You will need to seize the roles via ntdsutil. Wullieb1 has some links to some related articles. Once all FSMO roles are on a machine, you can then make it a GC.

            Comment


            • #7
              Re: Disaster...how to recover?

              Ok thanks to the guides I'm getting there! I have seized the roles over to a working DC. DCPromo'ed 2 replacement DCs into place etc. Things seem to look good overall.

              Now, any tips on how to get rid of my old DFS that was hosted (root) on the downed machines? I've looked at a few guides and can't figure out how to form my command. They all reference machine and share names, but I want the root removed all together.

              Dfsutil /UnmapFtRoot /RootFS root /Server:RootTargetServer /ShareFS share name

              I'm stuck on 'Server:RootTargetServer /ShareFS share name'

              Comment


              • #8
                Re: Disaster...how to recover?

                Is it a GC as well?

                Not sure about the DFS side of it.

                Comment


                • #9
                  Re: Disaster...how to recover?

                  Originally posted by Virtual View Post
                  Is it a GC as well?

                  Not sure about the DFS side of it.
                  When I added the 2 new DCs (2008 ) they went in as GCs. Should I avoid having more than 1 GC in my environment? I have 2 sites, with 2 DCs at one, 3 DCs at the other. The site that went down is the 3 DC site (was 2 before disaster), which is 2 Virtual and 1 physical.

                  Comment


                  • #10
                    Re: Disaster...how to recover?

                    If it's just 1 domain then it doesn't really matter.

                    I have posted the below today on another thread.

                    Depending on the number of users at your annex site, you could use Universal Group Membship Caching on Server 2 that will help minimise replication traffic across the VPN tunnel. It details it Here

                    Some of the criteria for placement of GC roles is dependent on whether there are multiple domains, you appear to have one domain. The GC should then not be placed on the DC holding the Infrastructure Master role in a multi domain environment, unless all other Infrastructure Master role holders are also GCs.

                    Making them both GCs will give redundancy but monitor the replication traffic just in case this effects bandwidth.

                    GCs will often be placed at sites where aplpications need a GC. e.g. Exchange.

                    When a user logs on, it will always query a GC, so your logon traffic will go down your VPN.

                    Universal Group membership is listed on a GC, so it also depends on your Universal Groups in use but either way, logon traffic will go there for checking membership.
                    Last edited by Virtual; 16th February 2009, 19:17.

                    Comment


                    • #11
                      Re: Disaster...how to recover?

                      Thanks so much for the help guys. Things are looking good over hear now. Worked the DFS out too.

                      Man without this sites guides and community.

                      Comment


                      • #12
                        Re: Disaster...how to recover?

                        Glad we could help.

                        How did you resolve the DFS issue? It would be good to know in case others have the same.

                        Comment


                        • #13
                          Re: Disaster...how to recover?

                          Originally posted by Virtual View Post
                          Glad we could help.

                          How did you resolve the DFS issue? It would be good to know in case others have the same.
                          Actually it was with a thread from here: http://forums.petri.com/showthread.php?t=1360

                          Just followed the steps on a linked MS document and I was more or less good to go. For whatever reason I couldn't get a new DFS root up on my 2008 machines, but the 2003 machine worked without issue. I then added the 2008 machines as replicas and it's looking good so far.

                          Comment


                          • #14
                            Re: Disaster...how to recover?

                            I've got another minor issue that needs working out. I have a few servers that seems to be struggling to find the domain again. If RDP'd too they report that the RPC service is unavailable, a quick search on this error seems to be issues contacting the domain. As well when logging in on the console of these machines the process is VERY long, I'm guessing due to again not contacting the domain correctly.

                            Any advice on how to fix these up? I was thinking of removing them from the domain then re-joining but there's 1 server in particular that I don't want to do this with.

                            Comment

                            Working...
                            X