h0bbel.p0ggel.org

I'm setting up a new web server for a client of mine, and was supposed to start installing it this evening. After unpacking it I realized that this new HP ProLiant DL 120 came without a cd-rom and thus the Debian ISO netinstall image I had downloaded was useless. Considering I'm doing this install at home, I don't exactly have a PXE server available either

My quest for installation via an USB stick started, but most of the information I found on how to create one was geared towards users that already have Debian or any other Linux distribution already installed. Again, this is being done at my home, which currently is a Linux free zone. I do have a Linux server available, the one that hosts this site amongst other things, but that is located elsewhere.

After fiddling with WinImage a bit, I was still unable to properly create a bootable stick. Everything seemed to go ok, until I actually tried booting it. All I got was a unpleasant "Boot Error" on the server, and no installation love at all. Trying to run RawWrite for Windows on Vista x64 didn't yield any better results, as that doesn't even see the USB stick as a device it can write to at all.

I came very close to giving up creating this on Windows and was ready to do a virtualized Debian install and then try to create the boot device from there, but then I remembered seeing a Windows version of dd somewhere. It turns out that Chrysocome, that also makes RawWrite for Windows, also maintain a port of dd for Windows

As a last ditch attempt before trying a virtualized Debian instance I decided to give it a try, and it turned out to work beautifully!. So, here is my step-by-step guide for creating a bootable USB stick for Debian on Windows

1. Download the required Debian files.
   I ended up with boot.img.gz and debian-LennyBeta2-amd64-businesscard.iso. I used the businesscard version since the USB stick i was using was only a 128MB one.
2. Download dd for Windows and extract it somewhere
3. Extract the boot.img file from the downloaded boot.img.gz file.
   I placed the boot.img file in the same directory where I extracted the dd binary to avoid path problems
4. Use dd to write the boot.img to your USB stick
   Open a command prompt and run dd. I used the following command: dd if=boot.img of=\\.\g: where g: is the drive letter Windows assigned to my USB stick.
5. After dd finishes, copy the debian-LennyBeta2-amd64-businesscard.iso file to the USB stick
6. Boot target computer with the USB stick connected
7. Install Debian

Of course, your target computer or server must be able to boot from a USB device and it's BIOS must be configured to actually try booting from USB. The HP ProLiant DL120 did this out of the box, so I didn't have to configure anything at all.

I'm sure this is pretty common knowledge for a lot of people, but I wasn't able to quickly find a concise description on how to use Windows to create the bootable USB stick so I thought it would be best to record my steps in case it helps someone else. Besides, I can always look at this post again the next time I need to do something like this.

Using iSCSI as a cheap storage solution is something that has been on my mind for a while, and today I finally got around to actually playing with it. I have a grand master plan for this, but for now I'll just highlight what the results of todays experiments were. My iplan was as follows:

Phase 1:
Codename Intense Hope

1. Install Debian in VMware Server

Originally I had planned to use Openfiler, since they have a preinstalled Virtual Appliance for this, but their website was down earlier today when I got started. Another reason for using Debian, and setting everything up manually, was that I wanted to really try and understand how everything is connected. I might very well opt to use Openfiler when I set up my final "product" though.

While it may seem like a useless task to set up an iSCSI target inside a Virtual Machine running on the same computer as the initiator, it serves the testing purposes perfectly.

I set up a basic netinstall of Debian Sarge, upgraded it to Etch and installed the 2.6.18 kernel. The next steps were to add a new HDD to the Virtual Machine, and set it up with LVM and mount it inside the VM. This volume is what the iSCSI instance will offer my iSCSI initiators.

2. Configure iSCSI Target

To get a Debian package for iSCSI Enterprise Target I added http://debian.hug.cx/debian/ to my Debian Sources, and the Sarge based install there installed without any problems as long as all the requirements are present.

The iSCSI configuration ended up like this:

/etc/ietd.conf

:
Target iqn.2007-02.int.mystorage:storage.lvm Lun 0 Path=/dev/volgroup/logicalvol,Type=fileio Alias LVMVOL

Which basically just offers the LVM volume via iSCSI to any client that connects.

Note that there is no security in this whatsoever, so everyone who connects will have full read and write permissions. You can add security on it, but for the sake of simplicity I opted to not add any in my test environment.

3. Test Windows XP and Windows Vista initiators

Windows XP doesn't have a "native" preinstalled iSCSI initiator, but you can download it directly from Microsoft.

After the initiator was installed, all that was required was to add the IP of the target to the initiator configuration and logon.

My shared volume then became available in the normal Windows XP Disk Management tool, and Windows treats it just like a normal directly attached HDD.

The setup in Vista was exactly the same, but without the need to install the iSCSI Initiator as it comes pre-installed and it was just a matter of configuring it.

4. Conclusion

Setting up an iSCSI target on Debian and connecting to it from Windows XP and Windows Vista is really easy. It didn't take very long to configure, and so far it seems to work really well. I now have two computers connecting to it, without any problems at all and once again VMware Server makes configuring a test environment like this a much less painful (and expensive) venture than actually having the physical hardware to play around with.

I don't know when I'll post Playing with iSCSI - Part II (Codename Bright Spoon), it depends on when I can get the actual physical hardware my final solution will run on. I might do something with Openfiler over the next few days though, as their website seems to be up again and I'm currently downloading their pre-configured Virtual Appliance.

Anyone care to guess what my final goal is?

With the steadily approaching release of Gallery 2.2, currently available as Release Candidate 1 we discovered the need to set up some sort of test environment. to run the unit tests that tests the various bits and pieces of the Gallery 2 application. The test environment would have to be flexible enough to run MySQL, Postgres and DB2 to cover some of the RDBMS that it supports.

Since we already offer the Gallery Appliance, setting up a VMware Server/Player appliance was a natural thought for us, and this time around we opted to go with a basic Debian install, based on the minimal images from www.thoughtpolice.co.uk

Without tweaking Debian packages all that much, we ended up with a compressed VM of about 700MB (715 242 245 bytes), which is somewhat large to distribute between the different team members. So, I decided to try out the technique outlined by a random comment by "Guest" on a previous post.

Basically what was suggested was to zero-fill the remaining disk space in the VMDK files, and them use vmware-vdiskmanager.exe to run an offline shrink operation on the virtual disks.

The steps I did was:

• 1. Run cat /dev/zero > zero.fill;sync;sleep 1;sync;rm -f zero.fill inside the VM
• 2. Run vmware-vdiskmanager.exe -k g2-developer-vm.vmk

The results were as follows:

Before:

• 2 093 547 520 g2-developer-vm-s001.vmdk
• 2 084 110 336 g2-developer-vm-s002.vmdk
• 786 432 g2-developer-vm-s003.vmdk

After:

• 1 888 944 128 g2-developer-vm-s001.vmdk
• 1 930 625 024 g2-developer-vm-s002.vmdk
• 786 432 g2-developer-vm-s003.vmdk

Not that huge a difference, but a total VM size of 4 178 540 714 bytes vs 3 820 609 962 bytes does add up to roughly 340MB space saved, and when compressed the results are 682MB vs 512 MB - 170MB less to download by each of the testers.

I'm sure we can get the base Debian installation tweaked even more, lose packages we don't need etc., but this still proves that the procedure of zero filling and then shrinking the VMs before distribution does have it's merits and should be done before distributing any VM.

Word of warning, don't! I did an upgrade of my debian box, the very box you're connected to right now. Since I use eAccelerator to speed up my Gallery 2 install I had added the dotdeb sources to my /etc/apt/sources.list. So, it only took a few seconds of apt-get update && apt-get upgrade to leave my box in a world of hurt. I got the new PHP 4.4.1 version, which should have been a good thing considering the recent security issues found in >4.4.1. Well, the result was that both Wordpress and Gallery 2 stopped working entirely. Both applications were working fine before the upgrade, and as far as I can tell the problem must be with the dotdeb version of PHP 4.4.1 and not the actual applications I run. Other people I've talked with, that also run Wordpress and Gallery 2, did not have any problems with upgrading to 4.4.1 at all. The errors I saw was pretty strange though, Wordpress was unable to include any of the files it's supposed to include. This was the case, even if the include_path was set correctly in php.ini and the files it tried to include was also placed where they always has been. Gallery 2 showed a different error alltogether, it compained about not being able to redeclare a previously declared function. The strange thing is that the function is only declared once, so the error made no sense. In fact, none of the errors made any sense at all. I was forced to switch back to the official Debian Sarge sources, and install PHP from there instead. Phew, at least the site is back up again now.

 Added 4th. Nov: Seems like dotdeb isn't to blame after all. Numerous problems has been seen with PHP 4.4.1 and it looks like the actual culprit is a buggy version of PHP.