Announcement

Collapse
No announcement yet.

Current Updates to ESX 3

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Current Updates to ESX 3

    Just a warning to others...

    there was a buttload of updates to the ESX server that i found day before yesterday... it was somewhere in the neighborhood of 20 tar ballz that i downloaded.

    un tarred the files and placed them on each ESX server in a directory created on the root on the ESX drive /updates/

    as each server was upgraded, the virtuals that reside on that server were moved to a different server and it was placed in maintance mode while the patches were applied.

    there was a problem (well, 2 known issues) that occured with us.

    one of the servers hosted an SQL server with RDM to a LUN on the SAN. due to the rmd, i couldnt migrate the server. the best idea would have been to define fail over servers on the blade chassis so that this virtual with RDM could be migrated. we had not to do so nor even considered the implication... here is the problem it created...

    when the vserver with the RDM was "powered on" it didnt. i got an error saying that there was no disk 3 (which should have been scsi0:4:02) even thouhgt it could be seen by the ESX host. it actually said Disk 3 was "innaccessable" i could SSH and see the LUN... the virtual's setting's dialog revealed that the scsi ID had been removed from the RDM on the virtual. it knew it had a disk, but it didnt have the location... it says "scsi0:0:0" which is BS. even though the scsi ID was wrong, the second box still held the location for the store "/vmfs/vm-sc-vd01/vsc-images"

    the same happened with an image store that was about 2 terabytes in size.

    for some reason the upgrade updated the UUIDs of the LUNs and the drive was no longer visable.

    the fix was somewhat simple, but i didnt like the feeling none the less.

    the drive was leftwhere it was on the virtual settings page. a second drive was added, and it was mapped to the location that appeared in the settings window. then the first (defuct) mapping was removed. the virtual fired right up...

    my thought was to find the location of the raw disk mappings and to change the UUID value, but ill be damned if i couldnt find where ESX kept this info... i check the etc/fstab, the etc\scsid file... the best i could find was a pointer to the UUID, but this wasnt where the UUID originiated for the ESX host. that file is somewhere else...

    in addition, the upgrade also updated the NIC drives for the blades, and this caused then to loose the static IP they had. this was an easy one to figure out, and had to be reset on the virtual host.

    just wanted to give a heads up and make sure that anyone with RDM take special precaution before patching your ESX servers.
    its easier to beg forgiveness than ask permission.
    Give karma where karma is due...

  • #2
    Re: Current Updates to ESX 3

    and since i already have this thread, ill append to it and save the space...

    i also upgraded the instance of Virtual Center Server and the VIclient the other day...

    totally sucked. after the upgrade i had no servers in my client. WTF! so i had to go back thru and recreate my DRS and HA groups. really cool.

    whats weird is the servers seemed to still act as if in a DRS or HA cluster. they would migrate while i was trying to shut down the hosts... a pain in the arse.

    and i got failures on all the hosts that had virtuals running. i had to migrate one at a time, and then wait till everyone left to migrate the dB servers.

    once the virtuals were moved, the server was rebooted and the client install was upgraded.

    i really like ESX and VMWare, but these updates are killing me. seems like every day is a new patch, and there is always something that doesnt quite go as planned. just wanna give a heads up and vent. thanks.
    its easier to beg forgiveness than ask permission.
    Give karma where karma is due...

    Comment


    • #3
      Re: Current Updates to ESX 3

      Originally posted by James Haynes View Post
      Just a warning to others...

      there was a buttload of updates to the ESX server that i found day before yesterday... it was somewhere in the neighborhood of 20 tar ballz that i downloaded.

      un tarred the files and placed them on each ESX server in a directory created on the root on the ESX drive /updates/

      as each server was upgraded, the virtuals that reside on that server were moved to a different server and it was placed in maintance mode while the patches were applied.

      there was a problem (well, 2 known issues) that occured with us.

      one of the servers hosted an SQL server with RDM to a LUN on the SAN. due to the rmd, i couldnt migrate the server. the best idea would have been to define fail over servers on the blade chassis so that this virtual with RDM could be migrated. we had not to do so nor even considered the implication... here is the problem it created...

      when the vserver with the RDM was "powered on" it didnt. i got an error saying that there was no disk 3 (which should have been scsi0:4:02) even thouhgt it could be seen by the ESX host. it actually said Disk 3 was "innaccessable" i could SSH and see the LUN... the virtual's setting's dialog revealed that the scsi ID had been removed from the RDM on the virtual. it knew it had a disk, but it didnt have the location... it says "scsi0:0:0" which is BS. even though the scsi ID was wrong, the second box still held the location for the store "/vmfs/vm-sc-vd01/vsc-images"

      the same happened with an image store that was about 2 terabytes in size.

      for some reason the upgrade updated the UUIDs of the LUNs and the drive was no longer visable.

      the fix was somewhat simple, but i didnt like the feeling none the less.

      the drive was leftwhere it was on the virtual settings page. a second drive was added, and it was mapped to the location that appeared in the settings window. then the first (defuct) mapping was removed. the virtual fired right up...

      my thought was to find the location of the raw disk mappings and to change the UUID value, but ill be damned if i couldnt find where ESX kept this info... i check the etc/fstab, the etc\scsid file... the best i could find was a pointer to the UUID, but this wasnt where the UUID originiated for the ESX host. that file is somewhere else...

      in addition, the upgrade also updated the NIC drives for the blades, and this caused then to loose the static IP they had. this was an easy one to figure out, and had to be reset on the virtual host.

      just wanted to give a heads up and make sure that anyone with RDM take special precaution before patching your ESX servers.
      The RDM metadata file is located on whichever VMFS volume you put it on when you created the Raw Device Mapping. Look in VirtualCenter at the VM properties for it's physical location, or open up the .VMX file for the VM and look for the location of the RDM file in there.

      As far as all your problems, the advice I can offer is this:
      1) Get all your VMs migrated off your host before you start patching it
      2) Put your host in maintenance mode before you start patching it
      3) Patches must be rolled out by order of the month they were released!!!! Never ever install January patches after you've already installed March patches!
      4) The need for patch management in ESX is quickly growing. I suggested for now using VMware's HTTP or FTP repository method to quickly and uniformly roll out patches to each of your ESX servers. Here, I'll give you my automated patch script which you can butcher to fit your own needs:

      #Install ESX patches:
      #Patches are extracted to a web server directory and placed in their own directory by date released.
      #Patches are deployed from a web server in order to save time and automate the patching process.

      #Patches should be installed in order by month released (ie. First January, then February, March, etc.)
      #There should be no running VMs on the host prior to patching or patching may fail.
      #Place ESX host into maintenance mode using VirtualCenter.

      #Paste the following commands into PuTTY (logged in as root):

      esxcfg-firewall -allowOutgoing

      esxupdate --noreboot -r http://yourwebserver/esxupdate/11-30-06/ESX-1006511 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/11-30-06/ESX-1410076 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/11-30-06/ESX-2158032 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/12-28-06/ESX-2066306 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/12-28-06/ESX-6921838 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/12-28-06/ESX-8173580 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/12-28-06/ESX-9986131 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/01-31-07/ESX-1271657 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/01-31-07/ESX-1917602 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/01-31-07/ESX-2031037 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/01-31-07/ESX-2092658 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/01-31-07/ESX-3996003 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/01-31-07/ESX-5497987 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/01-31-07/ESX-6075798 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/03-05-07/ESX-3199476 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/03-05-07/ESX-5031800 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/03-05-07/ESX-5885387 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/03-05-07/ESX-6050503 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/03-05-07/ESX-6856573 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/03-05-07/ESX-9865995 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/03-29-07/ESX-1541239 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/03-29-07/ESX-2257739 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/03-29-07/ESX-2559638 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/03-29-07/ESX-6431040 update
      esxupdate --noreboot -r http://yourwebserver/esxupdate/03-29-07/ESX-9916286 update

      esxcfg-firewall -blockOutgoing

      reboot






      Also see my post at http://www.vmware.com/community/mess...=595706#595706 for configuring your IIS server to allow a few additional MIME types so that you can use IIS as a patch repository server.

      Jas
      Last edited by jasonboche; 11th April 2007, 02:40.
      VCDX3 #34, VCDX4, VCDX5, VCAP4-DCA #14, VCAP4-DCD #35, VCAP5-DCD, VCPx4, vEXPERTx4, MCSEx3, MCSAx2, MCP, CCAx2, A+
      boche.net - VMware Virtualization Evangelist
      My advice has no warranties. Follow at your own risk.

      Comment


      • #4
        Re: Current Updates to ESX 3

        Originally posted by James Haynes View Post
        and since i already have this thread, ill append to it and save the space...

        i also upgraded the instance of Virtual Center Server and the VIclient the other day...

        totally sucked. after the upgrade i had no servers in my client. WTF! so i had to go back thru and recreate my DRS and HA groups. really cool.

        whats weird is the servers seemed to still act as if in a DRS or HA cluster. they would migrate while i was trying to shut down the hosts... a pain in the arse.

        and i got failures on all the hosts that had virtuals running. i had to migrate one at a time, and then wait till everyone left to migrate the dB servers.

        once the virtuals were moved, the server was rebooted and the client install was upgraded.

        i really like ESX and VMWare, but these updates are killing me. seems like every day is a new patch, and there is always something that doesnt quite go as planned. just wanna give a heads up and vent. thanks.
        You shot yourself in the foot my friend and I know exactly how you did it.

        When upgrading VirtualCenter, if the VirtualCenter Server database is already populated with data from previous usage, the following dialog box will be displayed warning you that you can choose YES to wipe out the database and start from scratch, or choose NO to leave the existing database in tact. In almost all cases, you will want to choose NO here. Choosing YES is going to wipe out the VirtualCenter Server database… be careful! The only reason you’d choose YES here is if you wanted to start over from scratch and build out a new VI3 infrastructure.

        VCDX3 #34, VCDX4, VCDX5, VCAP4-DCA #14, VCAP4-DCD #35, VCAP5-DCD, VCPx4, vEXPERTx4, MCSEx3, MCSAx2, MCP, CCAx2, A+
        boche.net - VMware Virtualization Evangelist
        My advice has no warranties. Follow at your own risk.

        Comment


        • #5
          Re: Current Updates to ESX 3

          you rock. i wish i knew all that before hand...

          im still a fledgling in the ESX stuff.. but eveything is working together fine. getting ready to test the failover and vmotion next weekish.

          thanks for the great write up. i can only hope its stickied so others see that. i have saved the page for myself.

          thanks man!
          its easier to beg forgiveness than ask permission.
          Give karma where karma is due...

          Comment

          Working...
          X