VMware Disk Mount: Access Your VM Disks Directly for Recovery
How to Mount Your ESXi Virtual Disks and Access Their Files
It’s going to happen sooner or later: you’re going to need to recover some files from a VM disk after a server has crashed, been decommissioned, or otherwise. Either your backups weren’t current enough, or you though you had everything, or the crash was sudden. For whatever reason, you’re wishing you could just pull the hard drive, hook it up to a USB – SATA adapter, and pull files off. Wouldn’t it be nice if there was a VMware disk mount utility? Good news, if you have a Linux workstation, there is a free one you can use to recover your files. As with all my Linux Tutorials, this is an Ubuntu based tutorial. I am running Ubuntu 10.04 LTS. If you are running a different version of Linux, your mileage may vary.
The Recovery Procedure
Step 1: Recover the Hard drives, and connect to your computer.
Pull the SATA hard drives from your crashed or defunct server, and take them to your Linux workstation. Connect them up with a USB to SATA adapter. Once you do, Linux will automatically recognize some of the paritions on the disk.
As you can see from the picture above, Linux immediately recognizes a few of the paritions from my VMWare ESXi 4.1 server. The problem is, it only contains things like a boot image, and some other data that is critical to the VMWare System, but doesn’t help us recover any files. But, this is a first and positive step in the vmware disk mount process: it indicates the drive is usable and in good shape. (Nothing can help you but prayer and a good data recovery service it the drive won’t even spin up).
Loading the VMWare Disk Mount Utility: vmfs-tools
In order to be able to read, mount, and write to VMWare Disks, we need to install the VMware File System tools package. Using the Linux package manager, apt, install the vmfs-tools package:
sudo apt-get install vmfs-tools
Discovering the Right Parition to Mount
Once this downloads the vmfs-tools package, you’ll be able to mount the VMware Disk Partitions to the Linux file system to directly access them.First, we need to discover which partition is the partition that holds our treasured files. We can now do this with fdisk:
sudo fdisk -l
This will list all the available disks and partitions. In my example case, we see that disk /dev/sdj contains multiple partitions, and many are related the VMWare:
We are interested in sdj3 for two reasons: first, it is huge (it has 483271704 blocks, which is massive and therefore indicative of a main system drive), and secondly, it is a VMware VMFS volume. This is the partition we want to mount.
Mounting the Partition on Your Computer to Gain Access to the Disk Files.
First, let’s prepare a directory on which we can mount the partition. Let’s create /mnt/thisvm as our mount point:
sudo mkdir /mnt/thisvm
To do this, we’ll use vmfs-fuse, which is a vmware disk mount utility that comes with vmfs-tools:
Â vmfs-fuse /dev/sdj3 /mnt/thisvm/
This mounts the drive to /mnt/thisvm. You can now change directory to /mnt/thisvm and see the files:
You can see that we have access to the entire datastore at this point. We can navigate through the directory structure and get to the vmdk files with ease. You can copy them off at this point for recovery or backup, or you can copy files into the store for restoring them. But… suppose you want to get a specific file off of one of the specific VM’s?
Mounting the VMWare Disk Image to Access Files within the Virtual Disk
This requires a little more advanced knowledge of Linux, so if you get lost, keep reading, and keep trying. You can always ask questions below. First, you need to find a free loopback device. These are located in /dev/loopN where N is a number from 0 to 9. A loop device allows a file system to be mounted and read as a block device. Common usages of a loop device are mounting ISO files, CDROMs, etc…
To find a free loop device, run this command and take note of the loop device number it produces:
sudo losetup -find
In mycase, it comes back as /dev/loop1, but you may have loop0 or loop 7. It just depends on what you’ve been doing and what your system has mounted. We want to access a backup server to recover the httpd.conf file (Apache’s config file). This file is inside the primary vmdk of the Backup01 Virtual machine. So, let’s change directory to Backup01:
If we change directory into this directory, we see the following files are available:
We are only interested in the flat file. Double check to make sure it is the right size. (In this case, 40GB):
So, the first step is to mount the flat file vmdk to the first available free loop device:
losetup /dev/loop1 /mnt/thisvm/Backup01/BackupServer_esxi-flat.vmdk
Ninja Shortcut: You can use the single quote notation syntax to find the first available loop device by putting losetup –find in single quotes (the ones that share the tilde to the left of the “1” key), and mount it like this example:
losetup -v `losetup –find` /mnt/thisvm/ReVoip.highpoweredhelp.com/ReVoip.myserver.com-flat.vmdk
*Make sure you put the -v switch in there! Otherwise you won’t really know which loop now has your vmdk!
Determining the File System
If you have a regularly formatted file system (ext4 for example) you are in for an easy road. But if the system uses LVM as the file system, we have more work to do before we can actually get the volume mounted. So, let’s use fdisk to find the system volume information:
fdisk -l /dev/loop1
This shows us that the big device that is available is a Linux LVM filesystem:
In other cases, you’ll end up with a regular old Linux file system:
You’ll use different steps to gain access to different files systems. Since the LVM is both the default file system for Ubuntu installs and it is more complicated to gain access to, I am going to discuss that one first (below). If you have a regular Linux file system, click here to skip to your next steps.
Working with an LVM File System
Because LVM requires that you find the volume and group (as well as other fun stuff), we’ll need to start up the LVM services before we can get access to it. To prevent damage to the vmdk, we have to grab the disk and throw it to an image. This requires an adequate amount of space (in our case, 40GB), so you’ll need to find an appropriate place to store the image. In my case, I have a raid array on this workstation with 3TB of space, so that will do nicely. I created a directory called Backup01-New on the raid array as a place where I can store the image file. Then, I used dd to create an image from the loop device:
dd if=/dev/loop1 of=/mnt/raid/Backup01-New/backup01.img bs=1024
Once this completes (and it may take a while depending on the size of the volume. My 40G drive took around 30 minutes (45 seconds / GB). Your mileage may vary.
Once this completes, let’s remove the association between the flat file and the image:
losetup -d /dev/loop1
Now, let’s associate the image we just created with /dev/loop1:
losetup /dev/loop1 /mnt/raid/Backup01-New/backup01.img
This allows us to deal with the new image (a copy of the original) without damaging or changing the original vmdk file.
Creating the device maps form the Partition Table
In order for the LVM system to be able to see the partitions that are on the loop device, and detect the LVM, we need to tell Linux that there are partitions on this device it should care about. To do this, we’ll use kpartx:
If you need to install kpartx:
sudo apt-get install kpartx
Once it is installed, add the partition mappings using this command:
kpartx -av /dev/loop1
This should give you an output similar to this:
Installing LVM Tools
Now that the files are prepared, we’ll need the LVM tools to detect and manage the logical volumes. If you don’t already have the LVM tools installed,Â install the tools for dealing with logical volumes with this command:
sudo apt-get install lvm2
Mounting the Logical Volume to Gain Access
Now, the system has been prepared properly, vgscan will be able to see the partition. Execute this command:
Now, we’ll use lvm pvscan to scan and find the physical volume for the LVM:
sudo lvm pvscan
Next, we need to activate the volume group. Execute:
sudo lvm vgchange -ay
Now, it’s time to admire our handy work: let’s list the available logical volumes to make sure that our LVM is active and usable:
sudo lvm lvs
Now, one last step before we mount the LVM: we need to make sure it shows up in the device mapper. Execute:
You should get an output similar to this one. Notice that our volume group backup01 is promptly displayed:
Now, let’s mount the LVM. We already know that the volume group backup01 is showing in the device mapper. We also know that there are two volumes in it: root and swap_1. So, we want to mount the root volume from the backup01 group onto /mnt/tmp:
mount /dev/backup01/root /mnt/tmp -o ro,user
Now, when you change directory to /mnt/tmp, you see the contents of your drive!
Mounting Regular Partitions (ext3,ext4)
If fdisk shows you that you have a regular Linux file system, you will be able to gain access to it very quickly. Confirm you have a Linux file system by creating a directory for a mount point, then mounting the file system.
Now, let’s create a mount point for the file system:
Now that that is taken care of, let’s go examine our disk. We’ll want to:
- Mount the VMDK
- Identify the partition we want to access,
- Identify that starting sector of that partition
- Calculate the offset we need to successfully isolate to be mounted, and
- Mount the partition.
To mount the vmdk, we’ll use the same command as we did above:
losetup -v `losetup –find` /mnt/thisvm/ReVoip.highpoweredhelp.com/ReVoip.myserver.com-flat.vmdk
Make sure you have the -v in there so you can see which loop device was associated with your file, In this case, it was /dev/loop2.
We’ll use fdisk to look at the disk as it is mounted on /dev/loop2, run the command:
fdisk -lu /dev/loop2
Which should give you an output like the one below.
There are three vital pieces of information here:
- We have identified that the device on /dev/loop2p1 is a Linux file system. This is the partition we want to mount.
- This partition starts at sector 2048.
- Each sector contains 512 bytes.
Therefore, the offset we need is calculated as the number of sectors times the size of the sector. In this case, 2048 * 512 = 1048576 (bytes).
Armed with this information, we can put that partition (by itself) on the next available loop device:
losetup -v `losetup –find` /dev/loop2 -o1048576
Don’t forget the -v switch. Otherwise, you won’t know which loop device has your partition! In this case, we got /dev/loop3.
Now that we have this partition isolated and ready, the last thing we need to do is mount it to gain access to the files:
Now, let’s mount the file system to gain access to its contents:
mount /dev/loop3 /mnt/linux-fs
You now have full access to the files to read / write and recover.
Mounting your VMWare vmdk files on Linux is easy when you know how to do it, and when you follow the procedure. Whether or not you have a LVM or Linux file system determines how easy it is get gain access to the files. In my opinion, using a regular file system and NOT an LVM makes recovery more simple; however, your choice to use LVM versus a standardized single file system should be based on the merits of each and not on the ease of recovery because (hopefully) recovery won’t be necessary. But if (and when) it is, follow this guide and you should be able to recover everything you need.
I will warn you, however, this guide and the knowledge it contains is NO SUBSTITUTE for proper backups.