With the steadily approaching release of Gallery 2.2, currently available as Release Candidate 1 we discovered the need to set up some sort of test environment. to run the unit tests that tests the various bits and pieces of the Gallery 2 application. The test environment would have to be flexible enough to run MySQL, Postgres and DB2 to cover some of the RDBMS that it supports.
Since we already offer the Gallery Appliance, setting up a VMware Server/Player appliance was a natural thought for us, and this time around we opted to go with a basic Debian install, based on the minimal images from www.thoughtpolice.co.uk
Without tweaking Debian packages all that much, we ended up with a compressed VM of about 700MB (715 242 245 bytes), which is somewhat large to distribute between the different team members. So, I decided to try out the technique outlined by a random comment by "Guest" on a previous post.
Basically what was suggested was to zero-fill the remaining disk space in the VMDK files, and them use vmware-vdiskmanager.exe to run an offline shrink operation on the virtual disks.
The steps I did was:
- 1. Run cat /dev/zero > zero.fill;sync;sleep 1;sync;rm -f zero.fill inside the VM
- 2. Run vmware-vdiskmanager.exe -k g2-developer-vm.vmk
The results were as follows:
Before:
- 2 093 547 520 g2-developer-vm-s001.vmdk
- 2 084 110 336 g2-developer-vm-s002.vmdk
- 786 432 g2-developer-vm-s003.vmdk
After:
- 1 888 944 128 g2-developer-vm-s001.vmdk
- 1 930 625 024 g2-developer-vm-s002.vmdk
- 786 432 g2-developer-vm-s003.vmdk
Not that huge a difference, but a total VM size of 4 178 540 714 bytes vs 3 820 609 962 bytes does add up to roughly 340MB space saved, and when compressed the results are 682MB vs 512 MB - 170MB less to download by each of the testers.
I'm sure we can get the base Debian installation tweaked even more, lose packages we don't need etc., but this still proves that the procedure of zero filling and then shrinking the VMs before distribution does have it's merits and should be done before distributing any VM.
21 Comments so far
coComment - , on January 1, 1970 at 1:00am, said:
view blog [IMG]
Edit Comment
ConnectBlogs, on January 1, 1970 at 1:00am, said:
for that. If you've just got one big partition, then you can't use the file transfer option and still have exactly the same machine. Fortunately there's a relatively painless way of reducing the size of the disk to just what's needed (thanks to Christian Mohn for the technique). The first step is to zero out all the free space on each partition of the drive you want to shrink. This, in effect, marks the free space. You can do that easily with this command:
Edit Comment
Planet Identity, on January 1, 1970 at 1:00am, said:
for that. If you've just got one big partition, then you can't use the file transfer option and still have exactly the same machine. Fortunately there's a relatively painless way of reducing the size of the disk to just what's needed (thanks to Christian Mohn for the technique). The first step is to zero out all the free space on each partition of the drive you want to shrink. This, in effect, marks the free space. You can do that easily with this command:
Edit Comment
Marc Liyanage - Home, on January 1, 1970 at 1:00am, said:
./diskTool -k 0 /path/to/f7.vmdk Shrink : 100% (9060864/9060864) done. This took a while and I noticed no change in size. I read somewhere that the utility needs a huge zero-filled file to mark the unused space. Back in the guest OS, I created a new partition /dev/sda3 with all the remaining space, formatted it as ext3 and put the zero-filled file in there:
Edit Comment
VMTN Discussion Forums: Shrinking disks ..., on March 27, 2007 at 5:03am, said:
[...] this an acceptable way to shrink the disk if the defrag and shrink does nto appear to do anything? http://h0bbel.p0ggel.org/2007/01/02/shrinking-vmdk-files/i tried it but got a "cat: write error: No space left on [...]
Edit Comment
Drool, on March 27, 2007 at 7:26pm, said:
The error is normal. It fills up the remainder of the disk (it will dump zeros to the huge file until there's no space left), sync (dump whatever's left in the cache) and finally, delete the file.
Edit Comment
VirtuaMag.net, on April 4, 2007 at 3:32am, said:
[...] [...]
Edit Comment
Eric Grejda, on April 6, 2007 at 6:17pm, said:
Thank you very much for the article - it came in handy this afternoon while trying to back up a virtual server.
However, we ran into a gotcha that everyone should be aware of: There cannot be any snapshots extant for the virtual server in question. When attempting to shrink the disk image, we got the error "The selected operation can only be executed on a disk with no snapshots." This was fixed by using the VMware server console to delete the snapshots. It should be noted that this can take a while to run.
Also, make sure that there are no lock files for a virtual machine. I found that shutting down the instance and disconnecting all consoles from it remedied this.
When performing the disk space consolidation procedure it is also advisable to drop the virtual server into single user mode, if applicable, when creating the disk file full of zeroes. This prevents the operating system from panicking when the disk space is used up.
I hope that this helps other people who encounter this.
Edit Comment
h0bbel, on April 6, 2007 at 8:31pm, said:
Nice pointers, Eric. All of this is absolutely true, in my case I did this on a single user machine that we were packaging as an appliance. Doing this for backup purposes for a live running VM is something else, of course and your steps highlight that.
Edit Comment
-X-, on April 6, 2007 at 10:48pm, said:
Hey h0bbel,
I worked with Eric on shrinking our Solaris 10-x86 VM and wanted to post some numbers:
Before Shrink:
6,527,320,064
After Shrink:
6,393,692,160
However, after compressing the .vmdk we were able to fit it (4,651,149,126) onto a 4.7 GB DVD, which was our goal.
BTW, step two in your procedure has the name of the file as .vmkd, FYI.
;)
Edit Comment
h0bbel, on April 10, 2007 at 8:59pm, said:
@-X-: Where? I see no .vmdk :-)
Thanks, I've updated the post.
Edit Comment
Scott Hanselman's Computer Zen - VM Performance Checklist - Before you Complain that your Virtual Machine is Slow, on May 29, 2007 at 9:13pm, said:
[...] with the Virtual PC Pre-Compactor that comes with Virtual PC when hosting Windows, and there are Linux options for shrinking VM hard drives as [...]
Edit Comment
jars.de » How-to: VMware Ubuntu Image verkleinern - Java und Technologie Blog von Markus Junginger, on June 24, 2007 at 9:20am, said:
[...] Virtual PC Guy, h0bbel) Inwieweit das in Kombination mit Shrink und Defragmentieren nötig ist, beziehungsweise etwas [...]
Edit Comment
kanunt about shrink, on July 3, 2007 at 8:48am, said:
[...] Shrink Splitting guide JavaScript::Squish - Compact JavaScript code to minim.. Shrinking VMDK files - h0bbel ProGuard Automated Food Product Packaging Machines - Shrink Fi.. Step by Step: Ripping DVD Movies [...]
Edit Comment
Phil Windley's Technometria, on August 20, 2007 at 6:57pm, said:
for that. If you've just got one big partition, then you can't use the file transfer option and still have exactly the same machine. Fortunately there's a relatively painless way of reducing the size of the disk to just what's needed (thanks toChristian Mohnfor the technique). The first step is to zero out all the free space on each partition of the drive you want to shrink. This, in effect, marks the free space. You can do that easily with this command:
Edit Comment
VMware experience sharing - LINUX , on August 21, 2007 at 5:06pm, said:
[...] system. Is it installed in Linux guest? By the way, have you read this shrinking-vmdk-files He suggest to do a cat /dev/zero > zero.fill;sync;sleep 1;sync;rm -f [...]
Edit Comment
Marc Lorenz, on March 12, 2008 at 5:48am, said:
Finally, I found a process that might work. Go h0bbel.p0ggel. :) I've got a VM that has grown from 12GB to 28+ due to the fact that MySql tran logging was on and generating 1GB logs.
I'm going to play with this on my dev boxes this week. Thanks!
Edit Comment
RevRagnarok, on March 15, 2008 at 3:40pm, said:
I've been doing something similar for years.
dd if=/dev/zero of=delme bs=102400 || rm -rf delme
You need to do it on each partition, but not hard to do.
Also, zero out your swap partition. Paraphrased:
swapoff -a
dd if=/dev/zero of=/dev/hdc1
mkswap /dev/hdc1
(then shutdown without a "swapon")
Edit Comment
qwerty, on June 1, 2008 at 10:19am, said:
Since it is a Debian system, don't forget to delete the contents of /var/cache/apt before zero-filling the disk! This can save quite a lot of space!
Edit Comment
Lucius, on February 12, 2009 at 11:06am, said:
This works very well.
I used it to recover 5 GB (after a big deletion on a linux guest).
So, I can confirm that this method works, I'm using VMWare Server 2 on a Widows 2003 Server host.
Thank you !
Edit Comment
tips for virtualbox, on May 17, 2010 at 6:56am, said:
...RIVERS=="?*", GOTO="persistent_net_generator_do" I also used to shrink my disk (http://h0bbel.p0ggel.org/shrinking-vmdk-files) 1: # fill the useless disk with zero be well compressed 2: >cat /dev/zero > zero.fill;sync;s...
Edit Comment