Announcement

Collapse
No announcement yet.

Hyper V and HRL files

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Hyper V and HRL files

    I have been running 2012R2 Hyper-v for several months. I have been using the Replica functionality from day 1. I have 18 VMs running on 3 different servers, 4 VMs on one, 5 VMs on another and 9 VMs on another. All host servers and VMs are running 2012R2. I have NIC teams on each server. Each server has 64Gb of RAM. This is a small shop, so I am the only IT person here. I use Symantec BUE 2014 with the Hyper-v option, Shavlik Patch Management and Protect and Condusive V-locity. Everything has been running fine. Replications are fast and take just seconds.

    I decided to use Extended Replication over a WAN for a Disaster Recovery solution. I have a 10Mb pipe to a local server farm. All of the servers replicate fine. Butů

    A couple of times a day, each VM replicates a large HRL file, anywhere from 1.2Gb to 2.3Gb in size. When the replication was local, this was not a problem and pretty much unnoticed by me. When the Extended Replica tries to replicate this large HRL file over the WAN, as you can imagine, everything goes to a crawl.

    I have looked through Task Scheduler, Event logs, files, etc. trying to find what is causing this large file to develop. I have excluded files from virus scanning, removed patch scanning, removed disk defrag, basically about everything I can think of. Can anyone think of what to do next? Any place to look? Anything? Any ideas are welcome.

  • #2
    Re: Hyper V and HRL files

    I haven't worked much with Hyper-V replica and from what I gather, it is working by Design and nothing has changed. It's just that locally you don't have the same bandwidth restrictions. Have you considered throttling the traffic and then reviewing if this still meets your requirements?

    http://www.thomasmaurer.ch/2013/12/t...plica-traffic/

    As ever, we tend to not replicate all VMs. Where possible, we use the application's failover capability. Recently, we have also been using Zerto for DR VM replication. Not cheap but meets the requirements of failover to be fairly rapid including for legacy apps and VMs.

    Comment


    • #3
      Re: Hyper V and HRL files

      Thanks for the reply.
      Throttling will not really help because it is going as fast as it can. I could buy more bandwidth but I want to know where the large HRL files are coming from.

      Comment


      • #4
        Re: Hyper V and HRL files

        Have a hunt for and read the Best Practices for Replication. The size of your HRL files is almost certainly down to the amount of changes recorded in the paging file inside each VM's VHD file. Best Practice is to create a new VHD(x) file, one for each VM, and move the paging files to those volumes. when you set up replication, UNtick the box next to that volume.

        Also, if any VM hosts SQL, move the TempDB files to a separate VHD file as well, and don't replicate that one.

        Every time you start a server or SQL instance, the paging files and Temp DBs are empty, so nothing is lost if you fail over to the replica and carry on. But, the replica destination has to have the same additional VHD files in the same folder name, so the replicated VMs can point to the same resources if they need to start. It's a bit tedious to set up, but once done, you're fine.
        *RicklesP*
        MSCA (2003/XP), Security+, CCNA

        ** Remember: credit where credit is due, and reputation points as appropriate **

        Comment


        • #5
          Re: Hyper V and HRL files

          I have separated the pagefiles and SQL is not running on 2 of the 4 VMs. The SQL on the VMs is SQL Express with very small databases.

          Comment


          • #6
            Re: Hyper V and HRL files

            The database size isn't the issue, it's the constant activity of the SQL-system TempDB. That DB is SQL's version of the Windows swap file. The more changes a server has in any of it's volumes, the larger the HRL files get. The more active a server is, the faster that file grows. So to keep from running into exactly the situation you're in, you really should think about removing the swap file activity from the VHD(x) files that are being replicated.

            I also run a replication setup for 22 VMs, across a 10G fiber link, and I had no end of replication errors to clear on a daily basis. Once I made the adjustments for the TempDB and paging files, all my repl issues went away. I do a quick check of repl status once a week now using powershell, and I can't remember the last time I had to clear any warnings or force a repl to resume.

            FYI: the powershell command I use is;
            Get-VMReplication -Computername (host1,host2,host3,etc) | Format-Table

            'Host1, Host2,Host3...' are the names of each hypervisor, both at source and replication ends. I get each VM named twice, once for each end, but I get it all in one command so it's easy to see when there's an issue. If you want to read the details of the command I gave you, here's the Technet link:
            'http://technet.microsoft.com/en-us/library/hh848570.aspx'

            Good luck.
            *RicklesP*
            MSCA (2003/XP), Security+, CCNA

            ** Remember: credit where credit is due, and reputation points as appropriate **

            Comment


            • #7
              Re: Hyper V and HRL files

              Okay, that answers the question for 2 of the VMs.
              But what about the other 2 VMs that are not running SQL and have the pagefile on a separate drive? One of the VMs is a clean install with the applications I stated in the first post. Is the virus scanner making a large HRL file?

              Comment


              • #8
                Re: Hyper V and HRL files

                Your first post talked about what apps you're running like BUE 2014. I use BUE as well, but don't know the other 2. If BUE is running inside one of the 2 suspect VMs, there are a lot of catalog files written each time a backup job runs, so may explain that. As I said, I can't speak for the other 2 apps as I've never heard of them.

                Are they all (other than the Windows OS itself) that are running on either or both of those last 2 VMs, or are there more apps? Any 3rd-party app that runs it's own internal (non-SQL) database could be generating the volume you're seeing, without knowing a lot more I couldn't make a guess. If one hosts an AV server, any updates that come down could easily build up a lot of changes.

                I've just re-read your original post, and I must apologize for mis-reading it the first time. I thought you'd written that you had a 10Gig connection, not 10Meg. With a path to your replica server that small, I think you'll struggle with the # of VMs you have. One alternative without spending anything is to limit the window of time the replicas can copy, which means your copies will be so many hours out of sync. Can your operation cope with that kind of failure? If not, increased b/w is the only real answer.
                *RicklesP*
                MSCA (2003/XP), Security+, CCNA

                ** Remember: credit where credit is due, and reputation points as appropriate **

                Comment


                • #9
                  Re: Hyper V and HRL files

                  Thanks for the reply. I believe the problem is the virus definition file. I am working with the virus company, but so far, no luck. Thanks

                  Comment


                  • #10
                    I'm not at all sure it is a virus.. We have the exact same on a Server 2012 that replicates back to our offices.. I have just built a second VM, which has nothing on it.., no shadow copy, no AV, nothing..., and it too sends between 1.5GB and 2.4GB files back to us just about everyday...
                    Defiantly no viruses on it..., only just been built...#
                    We also notice that if I do any kind of Windows Update on it, even a 2mb patch, that kicks off a 2.3gb replication back to us...
                    Any one got any ideas..?

                    Comment


                    • #11
                      Pls go back and read my posts, esp. the issues of Windows paging files for every VM being replicated, and any SQL database TempDB files. That will help with part of it. as for the rest, remember that when Windows is running, esp with an Update cycle, temp files are being written to either C:\Temp or C:\Windows\Temp (depending on your environment), and in most cases those temp files are not deleted when no longer needed. That's over/above the swap files. Also, when you do Updates, the content of your %windir%\SoftwareDistribution grows by the size of whatever update installation files just downloaded, and then those installs are run which grows Windows more. The only way I know to prevent HRL file growth of a VM server with replication is to not use that server.
                      *RicklesP*
                      MSCA (2003/XP), Security+, CCNA

                      ** Remember: credit where credit is due, and reputation points as appropriate **

                      Comment


                      • #12
                        We tried an idea at the weekend of totally removing the swapfile and using memory as we have shed loads on this Hypervisor, and we still got this 2.3GB go down the line.. We have one established server and one that I have just built and they both have the same issue.. The new server has nothing installed on it, no SQL, nothing, not even AV..

                        We look after many customers replicating back to our Server 2012 R2 Datacenter Hypervisor and none of them have this problem..., no issues whatsoever with a massive HRL file being generated when any change is make to the server...

                        The only difference with this customer is that it is a Server 2012 Std Hypervisor with 2 x Server 2012 VM's...

                        Is this an issue unique to Server 2012?
                        Last edited by ColinA; 29th April 2017, 23:51.

                        Comment


                        • #13
                          If you remove the page file from Windows, it will automatically create a new one. it has to have a pagefile.

                          For replicas, you should put the page file on a VM hard disk drive dedicated to the page file. For gen 1 VMs it needs to be IDE, for gen 2 it can be SCSI (only option ).

                          You can safely add this to a VM that already has replication configured since you don't want to replicate this drive.
                          Regards,
                          Jeremy

                          Network Consultant/Engineer
                          Baltimore - Washington area and beyond
                          www.gma-cpa.com

                          Comment


                          • #14
                            Thanks for your help....

                            My latest test is that I have created a separate vhdx file which I have moved the swapfile over to, and that in turn is excluded from the replication....

                            Still doing 2.3GB every time something changes..

                            As I say, we don't get this with Server 2012 R2...

                            Any other ideas here are very much appreciated..

                            Comment

                            Working...
                            X