Shrinking VMDK files

With the steadily approaching release of Gallery 2.2, currently available as Release Candidate 1 we discovered the need to set up some sort of test environment. to run the unit tests that tests the various bits and pieces of the Gallery 2 application. The test environment would have to be flexible enough to run MySQL, Postgres and DB2 to cover some of the RDBMS that it supports. Since we already offer the Gallery Appliance, setting up a VMware Server/Player appliance was a natural thought for us, and this time around we opted to go with a basic Debian install, based on the minimal images from www.thoughtpolice.co.uk Without tweaking Debian packages all that much, we ended up with a compressed VM of about 700MB (715 242 245 bytes), which is somewhat large to distribute between the different team members. So, I decided to try out the technique outlined by a random comment by "Guest" on a previous post. Basically what was suggested was to zero-fill the remaining disk space in the VMDK files, and them use vmware-vdiskmanager.exe to run an offline shrink operation on the virtual disks. The steps I did was:
  • 1. Run cat /dev/zero > zero.fill;sync;sleep 1;sync;rm -f zero.fill inside the VM
  • 2. Run vmware-vdiskmanager.exe -k g2-developer-vm.vmk
The results were as follows: Before:
  • 2 093 547 520 g2-developer-vm-s001.vmdk
  • 2 084 110 336 g2-developer-vm-s002.vmdk
  • 786 432 g2-developer-vm-s003.vmdk
After:
  • 1 888 944 128 g2-developer-vm-s001.vmdk
  • 1 930 625 024 g2-developer-vm-s002.vmdk
  • 786 432 g2-developer-vm-s003.vmdk
Not that huge a difference, but a total VM size of 4 178 540 714 bytes vs 3 820 609 962 bytes does add up to roughly 340MB space saved, and when compressed the results are 682MB vs 512 MB - 170MB less to download by each of the testers. I'm sure we can get the base Debian installation tweaked even more, lose packages we don't need etc., but this still proves that the procedure of zero filling and then shrinking the VMs before distribution does have it's merits and should be done before distributing any VM.

Posted by Christian Mohn aka h0bbel

Post metadata


Published January 2, 2007 01:17
19 comments

Tagged with , , , and


19 Responses to Shrinking VMDK files:

  • coComment -
    0000-00-00 00:00:00

    view blog [IMG]

  • ConnectBlogs
    0000-00-00 00:00:00

    for that. If you've just got one big partition, then you can't use the file transfer option and still have exactly the same machine. Fortunately there's a relatively painless way of reducing the size of the disk to just what's needed (thanks to Christian Mohn for the technique). The first step is to zero out all the free space on each partition of the drive you want to shrink. This, in effect, marks the free space. You can do that easily with this command:

  • Planet Identity
    0000-00-00 00:00:00

    for that. If you've just got one big partition, then you can't use the file transfer option and still have exactly the same machine. Fortunately there's a relatively painless way of reducing the size of the disk to just what's needed (thanks to Christian Mohn for the technique). The first step is to zero out all the free space on each partition of the drive you want to shrink. This, in effect, marks the free space. You can do that easily with this command:

  • Marc Liyanage - Home
    0000-00-00 00:00:00

    ./diskTool -k 0 /path/to/f7.vmdk Shrink : 100% (9060864/9060864) done. This took a while and I noticed no change in size. I read somewhere that the utility needs a huge zero-filled file to mark the unused space. Back in the guest OS, I created a new partition /dev/sda3 with all the remaining space, formatted it as ext3 and put the zero-filled file in there:

  • VMTN Discussion Forums: Shrinking disks ...
    2007-03-27 05:03:17

    [...] this an acceptable way to shrink the disk if the defrag and shrink does nto appear to do anything? http://h0bbel.p0ggel.org/2007/01/02/shrinking-vmdk-files/i tried it but got a "cat: write error: No space left on [...]

  • Drool
    2007-03-27 19:26:58

    The error is normal. It fills up the remainder of the disk (it will dump zeros to the huge file until there's no space left), sync (dump whatever's left in the cache) and finally, delete the file.

  • VirtuaMag.net
    2007-04-04 03:32:32

    [...] [...]

  • Eric Grejda
    2007-04-06 18:17:47

    Thank you very much for the article - it came in handy this afternoon while trying to back up a virtual server.

    However, we ran into a gotcha that everyone should be aware of: There cannot be any snapshots extant for the virtual server in question. When attempting to shrink the disk image, we got the error "The selected operation can only be executed on a disk with no snapshots." This was fixed by using the VMware server console to delete the snapshots. It should be noted that this can take a while to run.

    Also, make sure that there are no lock files for a virtual machine. I found that shutting down the instance and disconnecting all consoles from it remedied this.

    When performing the disk space consolidation procedure it is also advisable to drop the virtual server into single user mode, if applicable, when creating the disk file full of zeroes. This prevents the operating system from panicking when the disk space is used up.

    I hope that this helps other people who encounter this.

  • h0bbel
    2007-04-06 20:31:53

    Nice pointers, Eric. All of this is absolutely true, in my case I did this on a single user machine that we were packaging as an appliance. Doing this for backup purposes for a live running VM is something else, of course and your steps highlight that.

  • -X-
    2007-04-06 22:48:50

    Hey h0bbel,

    I worked with Eric on shrinking our Solaris 10-x86 VM and wanted to post some numbers:

    Before Shrink:
    6,527,320,064

    After Shrink:
    6,393,692,160

    However, after compressing the .vmdk we were able to fit it (4,651,149,126) onto a 4.7 GB DVD, which was our goal.

    BTW, step two in your procedure has the name of the file as .vmkd, FYI.

    ;)

  • h0bbel
    2007-04-10 20:59:27

    @-X-: Where? I see no .vmdk :-)

    Thanks, I've updated the post.

  • Scott Hanselman's Computer Zen - VM Performance Checklist - Before you Complain that your Virtual Machine is Slow
    2007-05-29 21:13:58

    [...] with the Virtual PC Pre-Compactor that comes with Virtual PC when hosting Windows, and there are Linux options for shrinking VM hard drives as [...]

  • jars.de » How-to: VMware Ubuntu Image verkleinern - Java und Technologie Blog von Markus Junginger
    2007-06-24 09:20:27

    [...] Virtual PC Guy, h0bbel) Inwieweit das in Kombination mit Shrink und Defragmentieren nötig ist, beziehungsweise etwas [...]

  • kanunt about shrink
    2007-07-03 08:48:34

    [...] Shrink Splitting guide JavaScript::Squish - Compact JavaScript code to minim.. Shrinking VMDK files - h0bbel ProGuard Automated Food Product Packaging Machines - Shrink Fi.. Step by Step: Ripping DVD Movies [...]

  • Phil Windley's Technometria
    2007-08-20 18:57:23

    for that. If you've just got one big partition, then you can't use the file transfer option and still have exactly the same machine. Fortunately there's a relatively painless way of reducing the size of the disk to just what's needed (thanks toChristian Mohnfor the technique). The first step is to zero out all the free space on each partition of the drive you want to shrink. This, in effect, marks the free space. You can do that easily with this command:

  • VMware experience sharing - LINUX
    2007-08-21 17:06:50

    [...] system.  Is it installed in Linux guest?   By the way, have you read this shrinking-vmdk-files   He suggest to do a   cat /dev/zero > zero.fill;sync;sleep 1;sync;rm -f [...]

  • Marc Lorenz
    2008-03-12 05:48:54

    Finally, I found a process that might work. Go h0bbel.p0ggel. :) I've got a VM that has grown from 12GB to 28+ due to the fact that MySql tran logging was on and generating 1GB logs.

    I'm going to play with this on my dev boxes this week. Thanks!

  • RevRagnarok
    2008-03-15 15:40:40

    I've been doing something similar for years.

    dd if=/dev/zero of=delme bs=102400 || rm -rf delme

    You need to do it on each partition, but not hard to do.

    Also, zero out your swap partition. Paraphrased:

    swapoff -a
    dd if=/dev/zero of=/dev/hdc1
    mkswap /dev/hdc1

    (then shutdown without a "swapon")

  • qwerty
    2008-06-01 10:19:05

    Since it is a Debian system, don't forget to delete the contents of /var/cache/apt before zero-filling the disk! This can save quite a lot of space!

11 Pingbacks to Shrinking VMDK files:

  • coComment -
    0000-00-00 00:00:00

    view blog [IMG]

  • ConnectBlogs
    0000-00-00 00:00:00

    for that. If you've just got one big partition, then you can't use the file transfer option and still have exactly the same machine. Fortunately there's a relatively painless way of reducing the size of the disk to just what's needed (thanks to Christian Mohn for the technique). The first step is to zero out all the free space on each partition of the drive you want to shrink. This, in effect, marks the free space. You can do that easily with this command:

  • Planet Identity
    0000-00-00 00:00:00

    for that. If you've just got one big partition, then you can't use the file transfer option and still have exactly the same machine. Fortunately there's a relatively painless way of reducing the size of the disk to just what's needed (thanks to Christian Mohn for the technique). The first step is to zero out all the free space on each partition of the drive you want to shrink. This, in effect, marks the free space. You can do that easily with this command:

  • Marc Liyanage - Home
    0000-00-00 00:00:00

    ./diskTool -k 0 /path/to/f7.vmdk Shrink : 100% (9060864/9060864) done. This took a while and I noticed no change in size. I read somewhere that the utility needs a huge zero-filled file to mark the unused space. Back in the guest OS, I created a new partition /dev/sda3 with all the remaining space, formatted it as ext3 and put the zero-filled file in there:

  • VMTN Discussion Forums: Shrinking disks ...
    2007-03-27 05:03:17

    [...] this an acceptable way to shrink the disk if the defrag and shrink does nto appear to do anything? http://h0bbel.p0ggel.org/2007/01/02/shrinking-vmdk-files/i tried it but got a "cat: write error: No space left on [...]

  • VirtuaMag.net
    2007-04-04 03:32:32

    [...] [...]

  • Scott Hanselman's Computer Zen - VM Performance Checklist - Before you Complain that your Virtual Machine is Slow
    2007-05-29 21:13:58

    [...] with the Virtual PC Pre-Compactor that comes with Virtual PC when hosting Windows, and there are Linux options for shrinking VM hard drives as [...]

  • jars.de » How-to: VMware Ubuntu Image verkleinern - Java und Technologie Blog von Markus Junginger
    2007-06-24 09:20:27

    [...] Virtual PC Guy, h0bbel) Inwieweit das in Kombination mit Shrink und Defragmentieren nötig ist, beziehungsweise etwas [...]

  • kanunt about shrink
    2007-07-03 08:48:34

    [...] Shrink Splitting guide JavaScript::Squish - Compact JavaScript code to minim.. Shrinking VMDK files - h0bbel ProGuard Automated Food Product Packaging Machines - Shrink Fi.. Step by Step: Ripping DVD Movies [...]

  • Phil Windley's Technometria
    2007-08-20 18:57:23

    for that. If you've just got one big partition, then you can't use the file transfer option and still have exactly the same machine. Fortunately there's a relatively painless way of reducing the size of the disk to just what's needed (thanks toChristian Mohnfor the technique). The first step is to zero out all the free space on each partition of the drive you want to shrink. This, in effect, marks the free space. You can do that easily with this command:

  • VMware experience sharing - LINUX
    2007-08-21 17:06:50

    [...] system.  Is it installed in Linux guest?   By the way, have you read this shrinking-vmdk-files   He suggest to do a   cat /dev/zero > zero.fill;sync;sleep 1;sync;rm -f [...]

Leave a Reply


Contact me

Lets talk!
Get in touch