A couple of days ago I finally got a chance to play around with a IBM BladeCenter S chassis with a HL21 blade server. The BladeCenter S is positioned as an entry level blade solution for the SMB market and looked like the perfect solution for a small Lotus Domino based local consultant company that I was hired to perform the installation for. The usage scenario was a powerful, extensible and manageable solution that they could utilize in their development and testing work as well as allow for further growth. Initially it will be used to run VMware Server and run a few development VMs, but the plan is to migrate to a full VMware ESX based infrastructure as the company and their computing needs continue to grow.
But, this is supposed to be about the BladeCenter "experience", so I'll get right back on track. Last week we got the equipment and we spent a some time unpacking it and assembling all the components, fitting it in the rack on the serverroom etc. Then, the first disappointment hit us. The power cables that were delivered with the power supplies, were for use with a PDU only. In other words, they simply didn't fit into the power sockets in the server room. Remember, this is a pretty small firm that hasn't got a full datacenter setup with all the bells and whistles that bigger clients have. So, we had to order some new cables just to be able to supply some power to the thing in the first place. Of course the IBM site that explains what features the BladeCenter S has, only claims
BladeCenter S uses standard office power supplies, enabling you to build your IT outside of the data center
so we thought we got what we needed. Sadly that wasn't the case, but I admit we probably should have checked that part a bit better with our hardware vendor.
After this little setback, we had to wait a couple of days before we had the time to get the setup working and configured. So this afternoon we started fresh, armed with proper power cables. Initial setup of the chassis was a breeze, so was also the case for the Nortel Layer 2/3 10 Gigabit Uplink Ethernet Switch Module. That little gem of a switch really impressed me with easy setup and dynamic routing between different networks on different ports. Since we're doing virtualization here we had a need to connect it to both the DMZ and the LAN as there will be VMs running in both zones. The Nortel made that very easy to accomplish, as well as a very nice way to rewire the existing servers into the same segments. Good one!
Feeling very upbeat about the whole BladeCenter, we continued to setup the first (and so far only) HL21 Blade and get the basic Windows 2003 Server Enterprise x64 setup running before diving into VMware Server.
Thats when the problems really started. The SAS controller module inside the HL21 Blade could not see any of the SAS disks that were present in the SAS Connectivity Module.
That lead us to really digging into the documentation, without really finding any clues at all. The only thing we discovered by doing that was that the SAS Expansion card doesn't support Raid5. Come again? No Raid5? I seriously never anticipated that to become a problem. I still have a very hard time wrapping my head around the fact that a "state-of-the-art" Blade Server/Chassis, manufactured by IBM and introduced in 2007 (!!) doesn't support Raid5 out of the box.
2.4.4 RAID options
The SAS Expansion Card supports three RAID levels:
RAID-1 (called Integrated Mirroring)
RAID-1E (called Integrated Mirroring Enhanced)
RAID-0 (called Integrated Striping)
This is taken right out of the IBM RedBook "Implementing the IBM BladeCenter S Chassis", also known as redp4357. Can you believe it? According to a forum post somewhere, that I'm unable to find again right now, Raid5 support will be available in Q2 2008. It better just be a firmware upgrade when it's released, and not a replacement part.
In pursuit of the lost SAS drives we also ran into a blog post from IBM about the release of a new Storage and Configuration Management software (SCM) and in out of pure desperation we thought that might be the solution to our problems. A very slow 250MB download later I had the SCM installed on my laptop, but that did us no good at all. In fact, it gave us nothing at all. Simply put, it just didn't work. After installing it my browser tries to open http://localhost/Console/faces/protected/MainConsole.jsp and that doesn't exist. Going back through the documentation it seems like it should be using port 32100, so I simply tried adding that to the URL. Still nothing. A big blank nothing once again appeared in my browser, and that was that. I tried uninstalling, reinstalling, rebooting but I still couldn't get the thing to work. I checked firewall settings, ran netstat -a to view open ports and nothing was to be found. The SCM documentation mentions the IBM SCM Service, but that service is nowhere to be found on my machine.
After scratching our heads until we nearly started bleeding, it turns out that the problem with the missing SAS drives was twofold.
Firstly, the Options Rom on the blade doesn't have enough storage space for two SAS controllers and PXE boot on the network cards. As unbelievable as it might sound, it's actually true. A small hint IBM; Expand the size of the options rom when you add new features?
The second part of the problem was equally stupid. It turns out that the firmware revison of the SAS Connectivity Module we got wasn't compatible with the BladeCenter S. After all, we only ordered the BladeCenter as a complete pre-configured setup from IBM it's not like the components we get should work together. Upgrading the firmware worked though, and we got the blade software installation running pretty quickly after all of the problems were sorted out.
Now, all that remains is configuring Windows 2003 Server, VMWare Server, creating some template VMs and setup the VMware virtual switches and whatnot. I'll leave that to another day though, I've had my share of BladeCenter experiences today.
12 Comments so far
Brad R. Cooper, on January 31, 2008 at 5:01am, said:
Sounds like you need to take a second look at your hardware vendor. I'm not referring to IBM - but to the business partner you worked with. They should be responsible for clearing up these issues before anything is even shipped... Good article though..
Edit Comment
David Tareen, on January 31, 2008 at 9:24pm, said:
Would like to learn more about your experience. Please email. Thank you.
Edit Comment
Christian Mohn, on January 31, 2008 at 10:08pm, said:
@Brad: While I agree that the vendor should have checked with us, especially with regards to the power cables, I don't think the vendor is to blame for the firmware revisions and lack of Options ROM space. I'm just surprised that the BladeCenter S was shipped with "incompatible parts" out of the box. As for the Options ROM issue, that should really be adressed by IBM.
@David: Feel free to contact me (h0bbel [at] p0ggel.org), but I would really like to answer any questions you might have (if I can) in public.
Edit Comment
Chris Jensen, on February 13, 2008 at 6:09pm, said:
Like Brad said... compatibility runs through the vendor. They are supposed to know what your needs are before you order the equipment. I’ve been an IBM hardware vendor for over 10 years now and get a lot of request for BladeCenter quotes. Either they didn't ask you the right questions.. i.e. are you connecting to a PDU or wall outlet... If they did, they would have told you that you needed different cables. Also, they should have asked you what type of RAID you planned on doing... b/c you can do RAID 5 in the BladeCenter S chassis, just not with the configuration you ordered. In order to do RAID 5 you needed to order the Storage and I/O expansion blade in addition to the Servraid 8k adapter for each blade you planned on doing RAID on. Then you can do RAID 5. A competent Hardware vendor would have asked the right questions and got you the right equipment for your needs. Again, if what was ordered was what you needed, you would have had everything initially. That’s not the manufactures fault. That’s the person doing the orderings fault. I hope this helps.
Edit Comment
SteveJHU, on June 5, 2008 at 3:36pm, said:
Hi,
We just got one of these. Did you ever figure out what the two slots on the right side of the DVD drive are for? They have icons of what appear to be batteries next to them, but all of the documentation seems to leave out what their purpose is. We found written somewhere that they are for "future use," whatever that means. Thanks.
Edit Comment
Christian Mohn, on June 9, 2008 at 9:26am, said:
@SteveJHU: No, I haven't investigated that at all so I have no idea.
Edit Comment
SteveJHU, on August 12, 2008 at 7:08pm, said:
I found out from an IBM engineer that the two slots are for battery-backed write cache. It said "future use" because they did not distribute a BBWC for it at that time.
Edit Comment
CobraIT, on January 22, 2009 at 11:07pm, said:
First, we sell BC-S occasionally, and they come with 6 different power cords. 2 for the house 110V style 5-15, 2 for the PDU style, and 2 for the 220V style 6-15. They send more cables than you can use, so someone must have stolen yours.
Also, any competent IT person knows to UPGRADE all the firmware, BIOS,etc. before you try to configure any server, SAN, network,etc.
All the documentation stated the BC-S SAS Connectivity module only supported Raid 0,1. The clown that didn't bother to research this customer's need before he ordered the equipment should be scolded, not fired.
The new SAS Raid module is a new part, supports Raid 5 and comes with battery BU. Like all new IT equipment,IBM, HP, Dell, it has some bugs, so it takes a little work.(which it sounds like some people are averse to)
Edit Comment
Christian Mohn (h0bbel), on January 23, 2009 at 12:04am, said:
@CobraIT: remember, this post was done about a year ago. The Bladecenter we got did not come with more than the PDU cables. They might have changed that now, or something went wrong with this particular shipment. I don't know.
As for the firmware revisions, I agree. You should always check for upgrades etc. before starting to configure anything, perhaps especially in this case as the BladeCenter S was a very new product at the time.
I was genuinely surprised to see that Raid-5 support wasn't supported by default, but as I wasn't the one who ordered it (I only installed it) I'll be sure to make the "Clown" know that some anonymous person who commented on this post is such a great judge of character.
Edit Comment
Dafortune, on November 16, 2009 at 2:29pm, said:
Hi Christian,
Well I ran into trouble when i was setting up Vmware VSphere 4.0 on the bladecenter S. Here are a few questions and I would be very grateful if you can help me out with this.
Current Config:
- Two HS21 Blades with common storage and running ESX 4.0
- Two " Server connectivity Modules" on the bladecenter
- Each blade is equipped with a 2 port Ethernet Expansion Card
1) How do I configure Service Console, Virtual machine network and VMkernal using just 2 ethernet ports available
2) If I have a 2 port Ethernet Expansion card in each blade... how do i configure then
3) I have limited or no knowledge about the " Server connectivity module "
Thanks in advance
Edit Comment
David Watts, on November 17, 2009 at 3:42pm, said:
What BC-S bays are the Server Connectivity Modules installed in? If you really have a 2-port Ethernet Expansion Card installed in the blades, then that means you have 4 Ethernet ports per server - the two onboard Ethernet ports plus the two from the Ethernet expansion cards.
The onboard ports are both routed to the I/O module installed in Bay 1 of the BC-S chassis. The two ports from the card go to the I/O modules in bays 3 and 4 - one port to each module.
So that means, you really need three I/O modules - Server Connectivity Modules or other Ethernet modules - to use the 4 ports that the servers have: in bays 1, 3, and 4 of the chassis.
Check the BladeCenter Products and Technology redbook for more details.
-- David.
Edit Comment
Tim, on January 9, 2010 at 4:45pm, said:
Dafortune,
The Server Connectivity Modules would have to configured for VLANs if you wanted to separate the Service Console, VM Network and VMKernel ports. If VLANing is not an options, you can simply create a single virtual switch with both vmnics attached and create your SC and VMKernel ports on that vswitch all on the flat network. This is not best practices for security, but it will work.
Not sure if you still need help, but maybe this can answer someone's question in the future.
--Tim
Edit Comment