Our house has a wonderful, unfinished basement. There is a plethora of reasons why I simply love the fact that it is unfinished:
- It doesn't matter if it gets dirty (well, Elise will eventually complain and make me clean it up
- If you poke a hole in the wall, it doesn't really matter (unless it's an outside wall, of course)
- Concrete walls. They're just plain awesome.
- ...
The list goes on and on. Of course, when *I* look at an unfinished basement, that's just the icing on the cake. When no drywall is hung, one can run as many cables as one likes, wherever and (just about) however one pleases. Thus, my Frankenstein is conceived...
Over the time that we've had this house, I've slowly gone about making certain technological ...upgrades. It began with bringing in a standard DSL internet connection with a static IP address, built to adding a *second* internet connection with static IP address and BGP routing, offering transparent redundancy and increasing connection speeds, and has continued from there with over 500 ft. of new CAT5 cable run throughout the house. A bit of blatant foreshadowing: a full *five* separate CAT5 cables were run into the basement...
The project reached a fever pitch over the past two weeks with the acquisition of a 42U server rack, one 4U file server chassis (24 bay, SATA-II, hotswap, awesome...), one 2U chassis for a virtual server host (via vmware, my xen is a bit rusty), and a 24 port managed gigabit switch (netgear, sadly).
I will note, at this juncture, that while I am jumping up in down with glee at this, I recognize that the vast majority of those reading this will, sadly, simply not appreciate this post. You have my condolences, of course. For those of you in that demographic, there are some pictures at the bottom so you can see at a glance what I will, undoubtably, spend several more paragraphs describing in the most painfully dull detail. For the rest of you, read on.
The "Project", as I've labeled it, has involved some pretty complicated planning and implementation has been anything but simple. Of course, if it were just throwing up a rack with a few rack-mount servers, it wouldn't have been a challenge. No, that would have been easy. Honestly, though, when have I ever chosen the *easy* route?
The Plan
A project of this magnitude, with the grand work for which it is destined (yet more blatant foreshadowing...), simply cannot be started without a detailed plan. After several days, several pages of notes, and a to-do list larger than any I've seen, the Plan started to come together. I used a very cool tool called "OmniFocus" to build it up. Boy am I grateful for good software. The Omni Group also publishes a few other fantastic tools.
I managed to throw together an ordered to-do list with associated dates (otherwise known as a Gantt chart...), develop a purchase list, research pricing, and lock down the intended result. A short list of the basic requirements:
- 42U rack, floor mount
- Independent power circuit run from main
- Power redundancy by both UPS (for 15 minutes) and standby generator
- Switchable power control- meaning several 15 amp circuits, controllable by computer or network
- At least one high capacity storage server (24+ bays, redundant power supplies, hot-swap SATA-II controller, etc.)
- At least one 2U chassis to replace a desktop chassis for the primary domain name and secondary name server
- Managed, 24+ port, gigabit switch
- Two independent internet providers with separate lines and methods of communication (I chose a basic DSL line and a radio link for diversity)
- BGP routing for both networks
- There's more, but I'm sure you're bored by now
You can guess at the work (and budget) involved in a project like this. Speaking of budget...
The Budget
Recently, as you can read about in other posts, I put a fair bit of money into a very nice bike. There will be another post very soon regarding my intentions with that very fine ride, stay tuned! The only drawback (thus far) with ownership has been the smoking hole in the proverbial family pocket from the purchase. What does this mean? It means I had basically no money to accomplish The Plan. Not to worry! Not for naught did my father and I enjoy MacGyver! Throughout the years, making something out of nothing has become a talent.
There were several pieces that would undoubtably cost money. The real question was how much. I planned about $150 for the 42U rack (a steal, for a four post), the same for the 2U chassis, and $350 for the file server. As for the switch, I had originally hoped to spend <$200, but found that if I wanted a *decent* one, I'd have to up it to about $300. Total so far: $800.
Fortunately, Mozy, the company I work for, is in the data protection business. We were just replacing a few of our high capacity server chassis, so I made an offer on one. I got it for a steal- all the power supplies, the SATA-II backplane, the drive bays, everything for my budgeted $350. Thanks Mozy!
NewEgg was kind enough to supply the rest for me. I split this across three months- purchasing the server from Mozy first, and just recently the rack, 2U chassis, and switch. I can't tell you how exciting it was to see the boxes in the living room when I got home from work...
Now there are still some pieces missing here- some unfortunately pricy pieces. The most key of them is the switchable power distribution unit (PDU). The idea behind this little piece of magic is to allow an administrator to remotely power cycle a machine or piece of hardware. There is a good reason why this is expensive- it's essentially an embedded computing device inside a power switch. Generally, they run about $600 and up. Needless to say, I'm not spending that much on a power strip. Period. So...
The Construction
We build it. There is an astonishingly simple electronic component called a "Relay". A relay is essentially a magnetically controlled switch. Run some DC voltage through it and the switch is turned on (or off, depending on the model). Cut the voltage out and the switch toggles again. This is the building block for a remotely controlled power distribution unit. Once you have a few of these, you have some options:
The Dumb PDU
There are two pins in a serial port that carry a (supposedly) consistent voltage: the DTR and RTS pins. They are generally used to indicate the state of the device for communication- DTR indicating that transmission is pending (more or less), and RTS indicating a ready to receive state. The simplest PDU in the world disregards these finely tuned purposes and employs these two pins as simple drivers for the relays. What I mean is this: turn the DTR/RTS pins up and your relay (if connected properly), becomes either active or inactive (your choice by design). Voila! You have a very stupid remotely switchable PDU!
The Smart PDU
There is a distinct drawback to the above approach- you can only control two circuits from a single RS-232 serial port. Most computers (including mine) only provide a single serial port, if any at all. I had a flaky DSL modem, a flaky wireless router, two servers and a switch, at a minimum, to control. If my math is right (generally questionable at best), that's five separate circuits. Problem? Not really. We just need to use the *rest* of that fine serial port.
There are more than a few ways that this can be done- you can buy the ICs, etch some boards, and produce a nice serial relay controller. Cost: somewhere around $50, from RadioShack estimates. The much faster, simpler, and reliable alternative is this: Serial Port Relay Board from Carl's Electronics
Let me tell you, it was a life saver. a $65 price tag gives control of 8 independent circuits, 15 amps each. I purchase and had this shipped to the house. Once received, the only piece left was a bit of code to control it. It uses an extraordinarily simply protocol (thank heavens) that was equally simple to implement. Ruby being my language of choice, I grabbed a copy of the Ruby SerialPort Library and started hacking. About 20 minutes later, I had a simple class put together implementing and nicely abstracting the serial controller protocol. You can find the code here. There are a few main methods- power-on, power-off, power-cycle and status. Returns true/false for power events, a hash structure of all relays and their status with the status command. Extremely useful.
To power cycle any of the hardware, I simply need to call "power-cycle" on the relay in question from the controlling computer. Next steps: grab a mini-itx or smaller m/b, cpu, and the rest to build a little embedded controller. I can then slap the thing into its own rackable box!
The Result
As things stand right now, there is a single rack sitting in the basement of my house with two entirely independent (albeit slow) internet connections, entirely controllable via ssh. I've run 4 cat5 cables down to that basement (one for each internet provider, one to route back to the rest of the house, and one for future scalability). The entire house is wired with a gigabit backbone (useful for the Apple TV, but I don't think the Tivo. Overkill? Of course. Can it be done any other way?
The Ulterior Motive
I must be honest about my intentions. As a simple hobby or experiment, this is insane. Fortunately, I plan to recoup the investment in more than one way. One of the biggest driving reasons behind the construction of The Project is the poor infrastructure in the development where we live. Harvest Hills is a beautiful place, but frequent electricity outages, brown-outs, by-law stipulations against Comcast and other more capable server providers, and finally but most critically a back-yard, barely legal (if at all) shared dish/direct-tv connection called the communities cable, and perhaps more critically, another back-yard DIY project sharing 3 T1 (a T1 is 1.5Mbs up/down) for 450 homes. They call it high-speed internet, but you can do the math on what it really is.
So one of the ulterior motives is to significantly improve both bandwidth and uptime of the service involved. As it stands, each power outage results in a prolonged outage of both cable and internet. There isn't even a single UPS batter backup on either! Insanity. Total insanity.
Stay tuned- I'll post a few updates here as progress is made.
