Copy Files In Azure Using Free Tool AzCopy

cloud-computing-hands-hero
In this post I’ll show you how to copy files up to an Azure storage account, down from a storage account, and even between storage accounts (without upload/download) using a free tool called AzCopy.

AzCopy

If you use Azure, then you’ve probably faced similar scenarios that I’ve experienced. Let’s say that you want to move files up to Azure and down from Azure. There are many options, including PowerShell and third-party tools that you can use. But what if you want to use a solution that offers the best performance? A free command-line tool, shared on GitHub, called AzCopy is intended to give you the ability to move files to, from, and inside of Azure, across Blobs, files, and tables with the best possible performance.

Getting AzCopy

As with all tools that you use with Azure, you should use the latest version of AzCopy, which you can download from here. After installation, you can find AzCopy.exe in “%ProgramFiles(x86)%\Microsoft SDKs\Azure\AzCopy” on 64-bit computers and “%ProgramFiles%\Microsoft SDKs\Azure\AzCopy” on 32-bit computers.
By default, the AzCopy folder is not added to the system path on your computer, so you’ll have to navigate on command-line or specify the full path to AzCopy.exe to run the tool. You can add the installation folder to your path.

Uploading a File to Azure

I am working with a container called demo in a storage account called pazcopy. I have copied the primary access key of the storage account; it will be required to use AzCopy. The basic syntax of AzCopy is as follows:

AzCopy /Source:<source> /Dest:<destination> [Options]

I can upload a single file, called Upload1.TXT, from C:\Temp to the demo container in the storage account (blob service) using the following command:

AzCopy /Source:C:\Temp /Dest:https://pazcopy.blob.core.windows.net/demo /DestKey:QB/asdasHJGHJGHJGHJ+QOoPu1fC/Asdjkfh48975845Mh/KlMc/Ur7Dm3485745348gbsUWGz/v0e== /Pattern:"Upload1.TXT"

Some notes:

  • Note how the URL of the destination storage account specifies the blob service.
  • I have specified the storage account access key with the /DestKey option. This key is unique to the storage account (source or destination).
  • The /Pattern option specifies the file being copied.
Copying a file to an Azure storage account with AzCopy (Image Credit: Aidan Finn)
Copying a file to an Azure storage account with AzCopy (Image Credit: Aidan Finn)

Upload All Contents in a Folder to Azure

I can modify the previous upload command to use a /S flag (recursive) instead of /Pattern to upload all files within a folder. This will search the source folder for sub-folders and files and upload all results to the Azure storage account.

AzCopy /Source:C:\Temp /Dest:https://pazcopy.blob.core.windows.net/demo /DestKey:QB/asdasHJGHJGHJGHJ+QOoPu1fC/Asdjkfh48975845Mh/KlMc/Ur7Dm3485745348gbsUWGz/v0e== /S

Upload Files Matching a Pattern

You might have a scenario where you want to upload a sub-set of files that have some pattern to their naming. You can use the /Pattern option to specify a query that will select those files. In my example, I can upload files called Upload1.txt, Upload2.txt, and Upload3.txt with a single command:

AzCopy /Source:C:\Temp /Dest:https://pazcopy.blob.core.windows.net/demo /DestKey:QB/asdasHJGHJGHJGHJ+QOoPu1fC/Asdjkfh48975845Mh/KlMc/Ur7Dm3485745348gbsUWGz/v0e== /Pattern:"Upload*.TXT"

Downloading Blobs or Files from Azure

You can use AzCopy to download files from Azure, using a similar structure to the upload commands. Downloads will require you to use a /SourceKey option instead of /DestKey.

AzCopy /Source:https://pazcopy.blob.core.windows.net/demo /Dest:C:\Temp /SourceKey: QB/asdasHJGHJGHJGHJ+QOoPu1fC/Asdjkfh48975845Mh/KlMc/Ur7Dm3485745348gbsUWGz/v0e== /Pattern:"Upload1.txt"

There is a gotcha here: Note that the pattern that specifies the name of the blob that you want to download is case sensitive; Upload1.TXT and Upload1.txt are two different blobs.
The combinations of flags we used with uploads can also be used with downloads. The following example will download all blobs, including virtual directories, from a container in a storage account:

AzCopy /Source:https://pazcopy.blob.core.windows.net/demo /Dest:C:\Temp /SourceKey: QB/asdasHJGHJGHJGHJ+QOoPu1fC/Asdjkfh48975845Mh/KlMc/Ur7Dm3485745348gbsUWGz/v0e== /S

Copying Between Storage Accounts

You might have a situation where you need to copy blobs from one storage account to another. For example, maybe you need to move a virtual machine’s hard disk (VHD blob) to another storage location. Without AzCopy, you will have to download the file and upload it again. Even a simple 127 GB C drive will take quite a while to move with that technique. Thanks to the usage of the Copy Blob API, AzCopy can move the file directly between two storage accounts without a tedious download/upload process.
In the following example, I am copying a blob (this could be a VHD) from a storage account in one Azure subscription to a different storage account in a separate subscription. Note that we will specify the relevant access keys for source and destination storage accounts.
The syntax is:

AzCopy /Source:<source> /Dest:<destination> /SourceKey<SourceKey> /DestKey:<DestinationKey> /Pattern:<BlobName>

This example moved a file between two storage accounts in completely different Azure subscriptions:

AzCopy /Source:https://pazcopy.blob.core.windows.net/demo /Dest:https://pazdest.blob.core.windows.net/demo /SourceKey: QB/asdasHJGHJGHJGHJ+QOoPu1fC/Asdjkfh48975845Mh/KlMc/Ur7Dm3485745348gbsUWGz/v0e== /DestKey:D67f/asdasHJGHJGHJGHJ+QOoPu1fC/Asdjkfh48975845Mh/K


AzCopy is a very useful tool. I have used it to move the blobs of Azure virtual machine OS drives between storage accounts and it takes seconds instead of hours/days to do a wasteful download or upload.