[Tech] Your very own server
16 years ago
This journal was originally rather lengthy. In hopes of more people not Ctrl+W-ing the tab, I've cut through the fluff [;_;] to get to the meaty center.
At least once a month, somebody I'm watching posts a journal about a hard drive death taking a whole swath of their digital life down with it. Invariably, these people don't have their files backed up. All they can do is rage while they hear their hard drive clicking and laying waste to gigabytes of carefully organized music, years of personal logs, artwork which isn't available anywhere else, and whatever else they might have. The common solution offered by folks like
dragoneer is to get an external hard drive and back up your files regularly. And you know, that works fine if you've got a single computer. But if you're under the age of 18 or above the age of 22, and possibly even if you're in that age range, you've got multiple computers on your home network. What do you do? One external drive for each computer? One external drive for a computer that's always on? And what if you have files you want to share between those computers, too? Like a music collection, or video files, or personal documents?
Windows Home Server is a handy ~$100 OS which you can put on old desktops or new systems built specifically for the role. It allows you to pack a computer case to its proverbial gills with hard drives--be they old 80 gig ones you have sitting around, or massive 1500 gig ones purchased to maximize storage--and integrate that storage into your home network. You can make virtual folders which span multiple drives, letting you store terabytes of files in a single 'directory' even if you've only got 320 GB drives. These virtual folders can be accessible by any computer on your home network, or restricted so that only certain users can change the contents or even just read the contents. The server will perform regular (in my case, nightly) backups of all your systems to make sure you never lose more than a day's work, and it does it intelligently so duplicate files only get stored once--no need to keep around gigabytes of duplicate Windows files or MP3s that you've got copies of on each computer. Although while you're at it, why not dump all your MP3s into a single directory on the server, so you can access your music more easily? With a planned update for Home Server, it'll even integrate your files automatically into the Windows 7 libraries, so you don't have to care about where they are.
And if the stuff you're storing on your server is really important to you, personal files or carefully organized music you'd rather not lose, you can enable data redundancy for the virtual folders. The server keeps an extra copy of the files in those folders on more than one drive, so even if one of your hard drives does crash, any files you really care about which were on it exist somewhere else in the system and will be promptly copied over to other drives to again keep them protected. It's not RAID, where you decide up front to commit a certain amount of a certain number of drives to always duplicate data. It's a folder-level redundancy which lets you use two 500 GB drives for 1 TB of non-backed-up data (what you'd typically do), 500 GB of backed-up data (something else you could easily do), or any flexible combination of redundant and non-redundant storage. Starting out with 700 gigs of anime that you don't care too much about losing? No problem. Add in 3 GB of personal documents and music which you want to be backed up and protected from drive failure? All you have to do is click a checkbox. It's a degree of flexibility that you aren't going to get with an external hard drive, NAS system, or Apple's Time Capsule.
So whether you just want your files to automatically be backed up, or you want a comprehensive system with indexed searching across redundant multi-terabyte network shares, Windows Home Server has your networked storage covered. But it doesn't stop there. Indeed, some of the important benefits you'll get will be side-effects of just having the system set up. You don't have to wonder about where you should put your documents or media, put it on the server. If you're having a LAN party and everyone's going to install a particular game or patch, don't put it on DVDs or flash discs--dump it on the server in a public directory. The server becomes your general-purpose file-dump. And because you're keeping most of your bulky files off of your personal computer, you'll probably be fine with a single hard drive, even a 750 gig drive will store a hundred modern games. Backups are smaller and faster. Everything feels more connected and organized. You feel secure in knowing that everything is backed up and an inevitable drive failure will just mean you have a few hundred fewer gigs of storage space until the manufacturer sends a replacement. Knowing this, you might even do riskier things with your computer--putting a couple drives in RAID 0 for example, for dramatically faster hard drive speeds. Who cares if it's more likely to experience a 'cripling' drive crash? You just pop in a new one and restore from your previous night's backup. Trust me. The security you feel from having all your stuff backed up, and the convenience you enjoy by having thousands of gigabytes of networked storage... they're well worth the cost of the OS.
I wrote this up (again, if you'll believe it, this is the shortened version) for several reasons, for those who are curious. First, two years back I had some journals discussing Windows Home Server. Now I have experience with it, and I'd like to share that experience with you. Second, a lot of people still don't have a backup solution, and this is a rather sexy one. Finally, as of late both for myself and some people, I've been seeking out components for a mature, modern home network. In a recent journal I detailed how to turn Blu-ray discs into gorgeous 1080p rips in admittedly large 10-30 gig files. Even a pretty large 1 TB drive will only hold a few dozen, a hundred at most, before it's full up. So where do you store these files in a cost-effective manner? Your WHS box. Grab the most cost-effective or largest drives you can get (1500 GB are good right now) and toss them in for terabytes of space. Sepf and I have over 9000 GB right now, approximately half of the 18000 we could have if we upgraded to exclusively 1500 GB drives.
If you want to make a cheap but good WHS box you will want (assuming you don't have other parts laying around) :
Windows Home Server, $95
A cheap 35-watt CPU, $40
A cheap 6-SATA, 4-PCI-Express port motherboard, $80
2x512 MB DDR2 modules, $28
A generic case with 6 HDD spots, expandable to 12, with 500W power supply, $70
If you've got access to a PCI-Express graphics card and a DVD-ROM drive for about an hour, you can put them in the server for installation, and your grand total is about $300 for the hardware and OS. If not, grab the cheapest graphics card ($25) and the cheapest DVD drive ($20) you can find, and still remove them after installation (they will draw power, after all).
That hardware up there, with your hard drive of choice, will give you a functional system that can house up to 6 hard drives. After that point, you'll need a 3x5.25"-to-4x3.5" drive bay ($23) and a 4 SATA port controller card ($39) for each additional 4 drives you want.
The first case will max out at 12 hard drives, but there's nothing stopping you from getting a second case and power supply, and filling that one up with hard drives as well. Connect the drives to the power supply in the second case, and use long SATA or eSATA cables to connect the drives to the server, simple as that. The practical limit of drives you can physically connect to the motherboard is 6 onboard + 3x4 (PCI-based) and 3x2 (PCI-Express-based) = 24, which is conveniently as many as you can fit in two full towers. Really, if you need any more than that, I think this guide is beyond your scope =P
At least once a month, somebody I'm watching posts a journal about a hard drive death taking a whole swath of their digital life down with it. Invariably, these people don't have their files backed up. All they can do is rage while they hear their hard drive clicking and laying waste to gigabytes of carefully organized music, years of personal logs, artwork which isn't available anywhere else, and whatever else they might have. The common solution offered by folks like
dragoneer is to get an external hard drive and back up your files regularly. And you know, that works fine if you've got a single computer. But if you're under the age of 18 or above the age of 22, and possibly even if you're in that age range, you've got multiple computers on your home network. What do you do? One external drive for each computer? One external drive for a computer that's always on? And what if you have files you want to share between those computers, too? Like a music collection, or video files, or personal documents?Windows Home Server is a handy ~$100 OS which you can put on old desktops or new systems built specifically for the role. It allows you to pack a computer case to its proverbial gills with hard drives--be they old 80 gig ones you have sitting around, or massive 1500 gig ones purchased to maximize storage--and integrate that storage into your home network. You can make virtual folders which span multiple drives, letting you store terabytes of files in a single 'directory' even if you've only got 320 GB drives. These virtual folders can be accessible by any computer on your home network, or restricted so that only certain users can change the contents or even just read the contents. The server will perform regular (in my case, nightly) backups of all your systems to make sure you never lose more than a day's work, and it does it intelligently so duplicate files only get stored once--no need to keep around gigabytes of duplicate Windows files or MP3s that you've got copies of on each computer. Although while you're at it, why not dump all your MP3s into a single directory on the server, so you can access your music more easily? With a planned update for Home Server, it'll even integrate your files automatically into the Windows 7 libraries, so you don't have to care about where they are.
And if the stuff you're storing on your server is really important to you, personal files or carefully organized music you'd rather not lose, you can enable data redundancy for the virtual folders. The server keeps an extra copy of the files in those folders on more than one drive, so even if one of your hard drives does crash, any files you really care about which were on it exist somewhere else in the system and will be promptly copied over to other drives to again keep them protected. It's not RAID, where you decide up front to commit a certain amount of a certain number of drives to always duplicate data. It's a folder-level redundancy which lets you use two 500 GB drives for 1 TB of non-backed-up data (what you'd typically do), 500 GB of backed-up data (something else you could easily do), or any flexible combination of redundant and non-redundant storage. Starting out with 700 gigs of anime that you don't care too much about losing? No problem. Add in 3 GB of personal documents and music which you want to be backed up and protected from drive failure? All you have to do is click a checkbox. It's a degree of flexibility that you aren't going to get with an external hard drive, NAS system, or Apple's Time Capsule.
So whether you just want your files to automatically be backed up, or you want a comprehensive system with indexed searching across redundant multi-terabyte network shares, Windows Home Server has your networked storage covered. But it doesn't stop there. Indeed, some of the important benefits you'll get will be side-effects of just having the system set up. You don't have to wonder about where you should put your documents or media, put it on the server. If you're having a LAN party and everyone's going to install a particular game or patch, don't put it on DVDs or flash discs--dump it on the server in a public directory. The server becomes your general-purpose file-dump. And because you're keeping most of your bulky files off of your personal computer, you'll probably be fine with a single hard drive, even a 750 gig drive will store a hundred modern games. Backups are smaller and faster. Everything feels more connected and organized. You feel secure in knowing that everything is backed up and an inevitable drive failure will just mean you have a few hundred fewer gigs of storage space until the manufacturer sends a replacement. Knowing this, you might even do riskier things with your computer--putting a couple drives in RAID 0 for example, for dramatically faster hard drive speeds. Who cares if it's more likely to experience a 'cripling' drive crash? You just pop in a new one and restore from your previous night's backup. Trust me. The security you feel from having all your stuff backed up, and the convenience you enjoy by having thousands of gigabytes of networked storage... they're well worth the cost of the OS.
I wrote this up (again, if you'll believe it, this is the shortened version) for several reasons, for those who are curious. First, two years back I had some journals discussing Windows Home Server. Now I have experience with it, and I'd like to share that experience with you. Second, a lot of people still don't have a backup solution, and this is a rather sexy one. Finally, as of late both for myself and some people, I've been seeking out components for a mature, modern home network. In a recent journal I detailed how to turn Blu-ray discs into gorgeous 1080p rips in admittedly large 10-30 gig files. Even a pretty large 1 TB drive will only hold a few dozen, a hundred at most, before it's full up. So where do you store these files in a cost-effective manner? Your WHS box. Grab the most cost-effective or largest drives you can get (1500 GB are good right now) and toss them in for terabytes of space. Sepf and I have over 9000 GB right now, approximately half of the 18000 we could have if we upgraded to exclusively 1500 GB drives.
If you want to make a cheap but good WHS box you will want (assuming you don't have other parts laying around) :
Windows Home Server, $95
A cheap 35-watt CPU, $40
A cheap 6-SATA, 4-PCI-Express port motherboard, $80
2x512 MB DDR2 modules, $28
A generic case with 6 HDD spots, expandable to 12, with 500W power supply, $70
If you've got access to a PCI-Express graphics card and a DVD-ROM drive for about an hour, you can put them in the server for installation, and your grand total is about $300 for the hardware and OS. If not, grab the cheapest graphics card ($25) and the cheapest DVD drive ($20) you can find, and still remove them after installation (they will draw power, after all).
That hardware up there, with your hard drive of choice, will give you a functional system that can house up to 6 hard drives. After that point, you'll need a 3x5.25"-to-4x3.5" drive bay ($23) and a 4 SATA port controller card ($39) for each additional 4 drives you want.
The first case will max out at 12 hard drives, but there's nothing stopping you from getting a second case and power supply, and filling that one up with hard drives as well. Connect the drives to the power supply in the second case, and use long SATA or eSATA cables to connect the drives to the server, simple as that. The practical limit of drives you can physically connect to the motherboard is 6 onboard + 3x4 (PCI-based) and 3x2 (PCI-Express-based) = 24, which is conveniently as many as you can fit in two full towers. Really, if you need any more than that, I think this guide is beyond your scope =P
FA+

Also, didn't you have a Win7 journal up? I could of swore I remember glancing over someone who had some neat little tips and tricks for it.
Some of us know just enough to be dangerous, but not enough to be sane.
15 TB of local hard drive space...also, you can't sneeze at 8 quad core CPUs and 128 GB of ram.
Doesn't stop me from wanting to put a gigabit ethernet card in it with a gigabit internet connection and be able to play TF2 at 9000 FPS with a ping of 2 to Japan.
http://www.ubuntu.com/products/what...../serveredition
its for FREE AND A MILLION TIMES BETTHER THEN WINDOWS
* Support indexed search which integrates with Windows
* Host game servers for popular multiplayer titles like Team Fortress 2, Left 4 Dead, Counterstrike Source and Call of Duty
* Easily integrate to do regular system backups and restore files from those backups
you need to do it all in comand lines like DOS
you can install the Simpel confing or you Built your own OS by choosing for what the server is
well the best is you start using the Desktop Ubuntu 9.4 its the easyest linux for beginners
Its Free
No Viruses
No Windows Bugs
Faster Boot and Shut Down
ALOT MORE SAVE ( keycode) and it use 3different parttitions on one kernenl
and yes it is like a KIT you can configute a lot of things
sadly my english is to bad to explain this to you
yes it can include windows Parts ( ntfs, fat32) but it will be bether you use the linux system (SWAP..)
yes you can setup a few backups like on external Harddrives or Store Boxes ( usb or lan)
or you install a RAID and config it ...
also you can include your full network stuff to linux ( rooting, HOMELAN and Internet )
(i would use a 3 way networkcard for that)
I use a multi raid and a external HDD BOX
you also could setup a older ZIP TAPE....or what you have left
game server... well i dont know ive that supports that
i guees you can EMULATE Windows using wine on Linux that can work maybe
but i think you use a older computer install it AND play arround then you will see what the Os is like
LOL im sorry again for my english
We haven't had bugs, and we never shut the thing off either. We picked parts which would be cheap when it comes to energy use, so we don't have to worry about that. It's a convenience thing.
Although we might be able to save $100 on the OS, having it integrate nicely and be easy to work with are important to us, and probably most people who would read this. I assume that most folks who already know the benefits of servers don't need me to explain this stuff to them; this is for 'lesser nerds' who know enough to put together a computer for games but not much more.
It's from some yaoi manga or something. There was a small meme going around where the face was photoshopped onto other faces. http://www.youtube.com/watch?v=tQynkChvg0A
The operating system doesn't have to say 'server' in the tittle to make the computer a server... The features the computer offers to another computer (or computers) makes it a server. Technically, all windows machines are both clients and servers (if file sharing is on).
I use an older computer that runs XP for my server OS. I stuffed three 1TB hard drives in a RAID level 5 array. I don't worry about data loss currently...
LINK
I prefer WHS's software RAID-type thing because unlike RAID 5 it doesn't have to read the contents of every single disk to restore any one disk that fails, which means there's a lower chance of subsequent drive failure. From the benchmarks I've seen, RAID 5 is the worst performance-wise and only useful if you can't afford RAID 1. In an environment where I'm going to be reading and writing to and from many files on a daily basis, spread across a dozen hard drives, I'd never consider RAID 5 to be a good option.
Restoring a dead drive in a RAID 1 array would technically be just as fast as restoring a RAID 5 drive. It would only be limited by the performance of the RAID controller and the drive that is being written to.
In theory RAID 5 should outperform RAID 1 because it can read parts of the data needed sequentially from ALL individual drives that compose the array, it all depends on the performance of the card being used.
I also have high doubts with the performance of RAID 5 using a chip-set controller. I'd get an add-in card regardless.
I WILL be using RAID 5 on my next desktop computer just to see how well the performance really is.
RAID 0: Obviously you're fucked >_>;
RAID 5: Bad. You have to read all the contents of every disk.
RAID 1: Acceptable. You have to read all the contents off of a single disk.
WHS RAID-like file duplication: Best. Assuming that every drive is full up and you have n drives, you read 1/(n-1)th of the files from each drive. If you've got 5 drives and one dies, you get a quarter of the file backups from each drive. It's the same amount of work as RAID 1, but distributed over multiple drives which further decreases the chance of any one drive failing and irreparably taking files with it.
Regardless, the WHS solution is nice because it doesn't require any expensive cards--anything that gives you more SATA ports, really. Any single 1 TB drive should be able to push a sizable portion of of the 1 Gbps that the network connection maxes out at, anyway (I have a very sexy screencap of a sustained >100 MB/sec file transfer from my desktop to the server). Really, the only downside is that you pay double for storage, but once you realize that storage is, at any given moment, at the lowest price ever, it's really not so bad. Two months of incremental backups of three computers in this house take up 75 GB. Doubled, that's 150 GB, or twelve dollars worth of storage. That's nothing.
RAID 5 does have crappy performance when writing to it, but again that is all up to the RAID controller's performance. I would rather pay money for a good card that makes handling the array invisible to the host system's CPU and software.
Copying a file from my server on onto the laptop uses a pretty steady >100Mb/s of the Gigabit port.
Another benefit of this approach over traditional RAID stuff is that you've got the -option- to make things redundant or not. With typical RAID you make an up-front commitment. With this, you can take a pair of 1 TB drives and decide to have 2 TB of files, or 1 TB of files with redundancy, or 750 GB with redundancy and 500 GB without, or any mixture in between. It's handy right now when I've got constantly changing amounts of files of varying importance to me.
CPU/system performance is a non-issue in a setup like this, a friggin' 1.6 GHz single core Celeron is able to handle processing needs :P
Me, I prefer to save money on not having hardware RAID 5 support and instead investing it in more drives.
Take that with a grain of salt if you want, but yeah, stuff like that makes me paranoid about using RAID 5. The fact that there's a group called BAARF (Battle Against Any Raid Five) makes me suspicious by itself >_>;
http://milek.blogspot.com/2006/08/h.....d-part-ii.html
http://milek.blogspot.com/2007/04/h.....-part-iii.html
http://blogs.sun.com/bonwick/entry/raid_z
I suppose it depends on *which* software RAID you mean.
(Note: the aforementioned software RAID technology has advanced considerably since 2007. Then, ZFS software RAID was comparable in speed to hardware RAID, and slightly faster than hardware RAID in some cases. That was two years ago; ZFS has only become faster and more reliable since then.)
You get an intensive application running along side your raid application in the background... performance will?
Software RAID would work quite dandy on a server that its only purpose is storage and sharing of data. But on a workstation or a server that does more than storage its not a great idea unless you have CPU power to burn.
I would rather stick with hardware RAID controllers and leave my CPU open for video rendering...
I'm just saying, look at the numbers. Speculate all you like but, at the end of the day, ZFS' RAID capabilities aren't like older software RAID technology. It is a vastly superior design and, even with the expense of CPU overhead, it manages to beat many, if not most, hardware RAID configurations on current hardware (or, as the links I provided indicate, on hardware from 2007). Look at the numbers. Look at what places like, oh, Los Alamos National Laboratories are doing with these systems, for that matter. It's no slouch.
I use it with a machine powered by four gigabytes of RAM and four 296MHz UltraSPARC-II processors. It handles a raidz2 array of a dozen disks without a problem, and I'm currently upgrading it to hold 20 disks. Two will be the boot disk and its mirror, and the rest will be one raidz2 array. The computer you're using to post comments here is probably more powerful than the one I describe by a large margin. You may dismiss this as an anecdote, but a whole lot of people are trying this out and finding the same to be true.
Also...as for video rendering or what have you, isn't this thread about a server dedicated to storage and backups? Besides, what server-class machine these days has only one core to go around? Even on my old beater from 1998, I can log in and compile something without issue while a big file transfer is going on.
More tests:
http://storagemojo.com/2006/08/15/z.....hardware-raid/
Windows Home Server, $95: here.
This is me just nitpicking at cents or dollars here and there XD in the end im probably going to go with plan b unless I hit some cash.
On a serious note this sounds a lot more like using an Abrams to kill a fly for the vast majority of people's needs. Hey, as much of a computer geek that I am my backups come in the form of .torrents for, uh, music, movies, games and pictures released under public domain, GPL or creative commons and using my web server or USB thumbstick as a remote backup for anything more personal. Just incase I skip a protection payment and the mafia burns my house down.
But if you're like me, and you positively hoard terabytes of files, and you've got several systems in the network which you share music and movies and documents between, then spending an extra $200-250 to upgrade your external drive to a home server is a pretty sexy idea. Let me tell you, it beats the hell out of having 4 125-disc DVD binders and several spindles full of discs, and files scattered across several computers.
Sent from my not-iPhone.
...
And viruses ._.'
Meanwhile, unlike Windows, you can leave this stuff up for years unless something goes catastrophically wrong on an electrical level. (I've had Solaris systems survive RAM and CPU failures, often without even going down.) The price is damn well right.
Oh, and the aforementioned filesystem is freakishly fast and self-repairing to such extent that fsck is unnecessary.
Copy files to the machine via rsync if you like--though, if you use ZFS locally, you can also just take filesystem snapshots and back those up instead. (This reminds me that I should rsync my ~/fux directory again after today's peruse of paws.ru.)
My current fileserver is an Enterprise 450 system that came from an auction. I stuffed it full of cheap SCSI disks from eBay, put Solaris 10 10/08 on it, and had smooth sailing thereafter.
Bonus: A budget lighter by $95, and hardware requirements gentle enough that a suitable server could be found in a dumpster (the thing I'm using is ten years old and uses four 296MHz processors).
So the current issues we're facing are A) how to get Server 2k8 and whatever we use in the third system to do a sort of JBOD storage pool with 12 to 20 drives on multiple SATA controllers, and how to get the two systems to do regular, incremental backups of each other (otherwise, backing up a mere 10 TB server will take a full 24 hours going at the theoretical max of 1 Gbit/sec (although we're considering giving each server two links to the system that backs it up to double that).
(Warning: Ignore everything after this point unless you are so bored that you are eating parts of your house.)
In the case of the setup you describe, I honestly think it would be easiest to eschew the system in the middle, as long as the first and the third have some kind of reliable storage; two machines should be fine, and adding a third in a setup which already involves two different operating systems would probably complicate and encumber the backup process--and especially the automation thereof.
As for the non-Windows backup system, my obnoxious and unsolicited advice for the simplest, fastest, most reliable, and cheapest solution I can think of with a heap of disks and some SATA controllers is definitely ZFS (via OpenSolaris or FreeBSD) and rsync on the not-Windows side and cwrsync for the Windows server being backed up. The non-Windows system could, in turn, be backed up to the Windows server by similar means (or by the backing up of regular ZFS snapshots). Your mileage may vary; even though I am entirely certain that that the aforementioned would work well and be fairly quick to get running, I am probably overstating how easy it is to set up, since I am familiar with a Unix environment.
ZFS is good at making storage pools out of piles of ordinary disks and, if specified, designating them as one of several possible RAID setups, without suffering from most of the traditional performance failures of conventional RAID topologies. Despite being free, it's solid, reliable, enterprise-quality stuff whose performance and capabilities are on par with (if not better than) that of software for which EMC or Symantec will charge many thousands of dollars. If it weren't for how awesome it is, I might feel awkward about talking at such length about how awesome it is.
An aside: Apparently, there is a cute little trick for getting at ZFS snapshot backups from Windows, involving the Shadow Copies for Shared Folders client:
http://sun.systemnews.com/articles/...../Windows/22112
(I just found out about this; I have no occasion to use it, but somebody who needs to back up a Windows system might.)
I want a computer that requires its own 200amp service...