<p>I haven't ever done exactly what you're trying to do, but I think I know how I could do it.  Hopefully this answer is useful, or at least starts some discussion from people that do have more concrete experience.</p>
<p>One of the important factors in your choice is going to be buying something that someone knows how to admin.  I think you'd be fine with a commodity fileserver appliance, where you admin it through a web interface or something, and just plug in hard drives + ethernet + power, and set it up to mail you when a drive dies.</p>
<p>If you use a normal PC, running Linux (with Linux's very excellent software RAID), and samba and/or NFS to share the files, you can easily stuff 6 hard drives in.  With any cheap PCIe SATA card, you can add another however many.  Your server will have lots of CPU time to spare, too, so you can have it do video encoding or something if you want.  Or just go really cheap on the CPU.  (Don't get slow RAM, though.  Copying around and checksumming data will potentially bottleneck on memory bandwidth.)</p>
<p>RAID6 is more CPU intensive, since the parity info has to use fancy modulo arithmetic, instead of just XOR, but again, with gigE as your bottleneck, and where you don't need the machine to have a lot of CPU time left over for anything else, it's fine.  You don't need an expensive RAID card, since cost per TB is more important for you than maybe 10% speed.  RAID cards often come with software to monitor them and send email when a drive needs replacing.  smartd from smartmontools, and/or mdadm can send email, too, if you set them up to do so.</p>
<p>gigabit ethernet is going to be by far the biggest bottleneck (100MB/s).  If you get a server with multiple gige ports, you could bond them (multiple cables all connected to the same switch), so multiple clients at once could read and write 100MB/s.</p>
<p>Since you're mostly going to be using this over gigE, and for video (big files read and written sequentially), something like WD Green hard drives would be most appropriate.  Lower idle and max power means you can more easily pack a lot of HDs into a case without overheating.  You'll have to shop for cases that can hold a lot of HDs, preferably with them accessible from the front for hotswapping.  Maybe a rackmount server case, esp. if you already have a rack.  I haven't ever looked into cases for boatloads of HDs, so there's probably stuff I'm missing here.</p>
<p>I think there are external chassis (with their own PS) for holding extra HDs, and they may even have SATA port multipliers.  If so, then you might be able to hook it all up to your mobo onboard SATA ports, since you'd have 4 or more hard drives on each of the 6 SATA connectors on your mobo.  I'm not sure this is how things work, so better read up.  (even sequential reads on current drives don't use the full bandwidth of 6Gb/s SATA, so the interface would be a bottleneck with 3 or more drives on a single SATA port.  But gigE is a bigger bottleneck.  And anyway, 6 mobo sata ports all running flat out would be helluva fast.)</p>
<p><strong>edit: apparently not all SATA controllers support port multipliers</strong>.  e.g. a <a href="http://www.newegg.ca/Product/Product.aspx?Item=N82E16817182280" rel="nofollow noreferrer">http://www.newegg.ca/Product/Product.aspx?Item=N82E16817182280</a> I was looking at for my own use has a manufacturer reply that mobo SATA ports often don't support port multipliers.</p>
<p><strong>edit</strong>: The <a href="https://forums.freenas.org/index.php?threads/hardware-recommendations-read-this-first.23069/" rel="nofollow noreferrer">FreeNAS HW recommendations</a> advise caution with SATA port multipliers.  Apparently they can eat your data.  That FreeNAS guide is exactly what you should look at for getting specific about hardware.  (But if you don't use ZFS, then tons of RAM is not very important.  Linux instead of FreeBSD will also probably have somewhat better hardware compatibility.  Remote management IPMI will be very valuable if anything goes wrong, other than disk HW failure.  Or even then, for determining which disk is failed.)</p>
<p>As for details of the RAID setup, yes, I'd suggest RAID5 is probably a good bet.  If you can afford to have at least a 2nd copy of your data somewhere, as a backup, then you don't need to be too paranoid about the reliability of either the main archive or the backup.  If not, well then you should probably go RAID6.</p>
<p>Even RAID6 still doesn't help you if someone accidentally deletes or overwrites the wrong directory on the shared server, or if a CPU or RAM failure leads to filesystem corruption.  RAID only helps you read back the data that was written.  If other errors lead to the wrong data being written, you are in trouble.</p>
<p>ZFS might be a good choice, for extra checksumming to detect corruption.  You can set it up to give RAID5 levels of redundancy (RAID-Z), or RAID6 (z2).  Linux BTRFS does basically the same stuff as ZFS, but its built-in RAID5/6 isn't mature yet.</p>
<p>If you end up using more than maybe 16 drives, you might want to think about breaking them into 2 smaller RAID5 or RAID6 arrays.</p>
<p>Off-site backups, updated at least weekly from your main storage server, are a really good idea.  For this special case of backing up video, maybe use the CPU time on your NAS to transcode your videos to lower bitrates for off-site backup, so you don't need so much space off-site.  Then if you lose everything, you have whatever you can pull off people's workstations that they had copied there, plus the lower-quality videos from your offsite backup.  Hopefully you never have to use it.</p>
<p>Oh, another idea.  The costs of having that much storage available online all the time is a bit of a factor (having to buy a server).  Possibly useful to use external hard drives as a 2nd copy.  Or even &quot;internal&quot; hard drives that you stick into a docking station.  (looks like a toaster, but takes SATA HDs instead of bread.)  Insert HD, sync some data to the drive, take it home for safe keeping.  Then if your studio burns down, someone has the data at home.  This might be easier than finding somewhere off-site to put a backup server for your archive.</p>
<p>Or just pay for cloud backups of your archive.  That might actually make the most sense, if your Internet connection can handle it.  (you could use the lower-bitrate off-site backup idea with cloud backups, too.)</p>
<p>This is all pretty theoretical, since my job as a sysadmin for some small Linux beowulf clusters was several years ago.</p>
<p>Anyway, remember that someone will have to admin this thing.  Being the admin probably won't mean any hours a week babysitting it, but you need someone that understands the setup and knows what to do if there's a problem.  Swapping a drive usually requires poking at the admin UI, not just physically changing hardware.</p>
<p>Have a spare HD ready to pop in to replace a failed one, so your data isn't going unprotected for long.  Probably you'd want a failed-drive email to go to multiple people, so whoever's around can replace a failed drive with the spare.  (If you have the drive in the NAS PC or appliance, then it's a hot spare and can get used automatically.)</p>

CC BY-SA 3.0
2015-01-26T14:11:34.407Z
Thanks a lot for a very detailed and thorough reply! Now I have a lot more info to digest to make a more informed decision.
westis
CC BY-SA 3.0
2015-01-26T20:53:06.240Z
Cheers.  I'd be interested to hear what you end up doing.  Maybe post a summary as an answer to your own question :P
Peter Cordes
CC BY-SA 3.0
2015-01-29T12:17:32.360Z
Yup, will post something when we've decided on exactly what to do.
westis
CC BY-SA 3.0
2015-01-23T23:36:12.057Z
2015-04-26T19:25:26.213Z
2020-06-17T09:53:03.607Z
Peter Cordes
<p>We are a community TV station, looking to improve our video archive. Currently we archive our master files on external drives. But to make the archive easier to access, both for saving and re-using, we are looking into a network-based solution.</p>

<p>We have a gigabit network and currently use about 18.5 TB (our exported files for broadcast already reside on another NAS). But we likely will need at least 30TB for now (we have about 34TB worth of disks available).</p>

<p>My question is, how to we best set up this archive, to make it available over LAN? RAID5? Do we need a RAID controller or can we get away with some kind of software/motherboard/PCIe RAID, as it's only for archiving? Is it better to buy a dedicated NAS solution, or to set up our own server with a good amount of swap drive slots? Or would we even need a SAN?</p>

<p>About six editing computers will on a more or less regular basis archive/backup to this archive server.</p>

<p>Our budget is pretty tight, so we're looking for a cheap, yet efficient solution.</p>

<p>Although we currently use Google Spreadsheets to track what's on each disk, a media asset management solution would probably be a nice addition to this as well :)</p>

<p>Thanks!</p>

CC BY-SA 3.0
2015-01-23T23:38:40.773Z
As I said, buried in my long answer, RAID controller cards are most useful for taking the load off the server CPU.  Also, battery backed cache to allow write caching.  But neither of those are important for you.  I'd prob. use a cheap PCIe SATA card just to hook up more HDs, and run software RAID.
Peter Cordes
CC BY-SA 3.0
2015-01-23T23:40:28.650Z
A SAN (storage area network) would be, in your case, a separate gigE network for carrying fileserver traffic only?  So every computer would need 2 gigE ports?  Doesn't make sense, IMO, esp. if cost-efficiency is more important than performance.
Peter Cordes
<video><hardware>
Video archive solution
westis
# Video archive solution

(Asked by: username_0 on 2015-01-22)

We are a community TV station, looking to improve our video archive. Currently we archive our master files on external drives. But to make the archive easier to access, both for saving and re-using, we are looking into a network-based solution.

We have a gigabit network and currently use about 18.5 TB (our exported files for broadcast already reside on another NAS). But we likely will need at least 30TB for now (we have about 34TB worth of disks available).

My question is, how to we best set up this archive, to make it available over LAN? RAID5? Do we need a RAID controller or can we get away with some kind of software/motherboard/PCIe RAID, as it's only for archiving? Is it better to buy a dedicated NAS solution, or to set up our own server with a good amount of swap drive slots? Or would we even need a SAN?

About six editing computers will on a more or less regular basis archive/backup to this archive server.

Our budget is pretty tight, so we're looking for a cheap, yet efficient solution.

Although we currently use Google Spreadsheets to track what's on each disk, a media asset management solution would probably be a nice addition to this as well :)

Thanks!

**Tags:** video, hardware

> **Comments:**
> **username_1**: As I said, buried in my long answer, RAID controller cards are most useful for taking the load off the server CPU.  Also, battery backed cache to allow write caching.  But neither of those are important for you.  I'd prob. use a cheap PCIe SATA card just to hook up more HDs, and run software RAID.  
> **username_1**: A SAN (storage area network) would be, in your case, a separate gigE network for carrying fileserver traffic only?  So every computer would need 2 gigE ports?  Doesn't make sense, IMO, esp. if cost-efficiency is more important than performance.  

---

## 1 answers

### Answer by username_1 on 2015-01-23. Score: 3

I haven't ever done exactly what you're trying to do, but I think I know how I could do it. Hopefully this answer is useful, or at least starts some discussion from people that do have more concrete experience.

One of the important factors in your choice is going to be buying something that someone knows how to admin. I think you'd be fine with a commodity fileserver appliance, where you admin it through a web interface or something, and just plug in hard drives + ethernet + power, and set it up to mail you when a drive dies.

If you use a normal PC, running Linux (with Linux's very excellent software RAID), and samba and/or NFS to share the files, you can easily stuff 6 hard drives in. With any cheap PCIe SATA card, you can add another however many. Your server will have lots of CPU time to spare, too, so you can have it do video encoding or something if you want. Or just go really cheap on the CPU. (Don't get slow RAM, though. Copying around and checksumming data will potentially bottleneck on memory bandwidth.)

RAID6 is more CPU intensive, since the parity info has to use fancy modulo arithmetic, instead of just XOR, but again, with gigE as your bottleneck, and where you don't need the machine to have a lot of CPU time left over for anything else, it's fine. You don't need an expensive RAID card, since cost per TB is more important for you than maybe 10% speed. RAID cards often come with software to monitor them and send email when a drive needs replacing. smartd from smartmontools, and/or mdadm can send email, too, if you set them up to do so.

gigabit ethernet is going to be by far the biggest bottleneck (100MB/s). If you get a server with multiple gige ports, you could bond them (multiple cables all connected to the same switch), so multiple clients at once could read and write 100MB/s.

Since you're mostly going to be using this over gigE, and for video (big files read and written sequentially), something like WD Green hard drives would be most appropriate. Lower idle and max power means you can more easily pack a lot of HDs into a case without overheating. You'll have to shop for cases that can hold a lot of HDs, preferably with them accessible from the front for hotswapping. Maybe a rackmount server case, esp. if you already have a rack. I haven't ever looked into cases for boatloads of HDs, so there's probably stuff I'm missing here.

I think there are external chassis (with their own PS) for holding extra HDs, and they may even have SATA port multipliers. If so, then you might be able to hook it all up to your mobo onboard SATA ports, since you'd have 4 or more hard drives on each of the 6 SATA connectors on your mobo. I'm not sure this is how things work, so better read up. (even sequential reads on current drives don't use the full bandwidth of 6Gb/s SATA, so the interface would be a bottleneck with 3 or more drives on a single SATA port. But gigE is a bigger bottleneck. And anyway, 6 mobo sata ports all running flat out would be helluva fast.)

**edit: apparently not all SATA controllers support port multipliers**. e.g. a <http://www.newegg.ca/Product/Product.aspx?Item=N82E16817182280> I was looking at for my own use has a manufacturer reply that mobo SATA ports often don't support port multipliers.

**edit**: The [FreeNAS HW recommendations](https://forums.freenas.org/index.php?threads/hardware-recommendations-read-this-first.23069/) advise caution with SATA port multipliers. Apparently they can eat your data. That FreeNAS guide is exactly what you should look at for getting specific about hardware. (But if you don't use ZFS, then tons of RAM is not very important. Linux instead of FreeBSD will also probably have somewhat better hardware compatibility. Remote management IPMI will be very valuable if anything goes wrong, other than disk HW failure. Or even then, for determining which disk is failed.)

As for details of the RAID setup, yes, I'd suggest RAID5 is probably a good bet. If you can afford to have at least a 2nd copy of your data somewhere, as a backup, then you don't need to be too paranoid about the reliability of either the main archive or the backup. If not, well then you should probably go RAID6.

Even RAID6 still doesn't help you if someone accidentally deletes or overwrites the wrong directory on the shared server, or if a CPU or RAM failure leads to filesystem corruption. RAID only helps you read back the data that was written. If other errors lead to the wrong data being written, you are in trouble.

ZFS might be a good choice, for extra checksumming to detect corruption. You can set it up to give RAID5 levels of redundancy (RAID-Z), or RAID6 (z2). Linux BTRFS does basically the same stuff as ZFS, but its built-in RAID5/6 isn't mature yet.

If you end up using more than maybe 16 drives, you might want to think about breaking them into 2 smaller RAID5 or RAID6 arrays.

Off-site backups, updated at least weekly from your main storage server, are a really good idea. For this special case of backing up video, maybe use the CPU time on your NAS to transcode your videos to lower bitrates for off-site backup, so you don't need so much space off-site. Then if you lose everything, you have whatever you can pull off people's workstations that they had copied there, plus the lower-quality videos from your offsite backup. Hopefully you never have to use it.

Oh, another idea. The costs of having that much storage available online all the time is a bit of a factor (having to buy a server). Possibly useful to use external hard drives as a 2nd copy. Or even "internal" hard drives that you stick into a docking station. (looks like a toaster, but takes SATA HDs instead of bread.) Insert HD, sync some data to the drive, take it home for safe keeping. Then if your studio burns down, someone has the data at home. This might be easier than finding somewhere off-site to put a backup server for your archive.

Or just pay for cloud backups of your archive. That might actually make the most sense, if your Internet connection can handle it. (you could use the lower-bitrate off-site backup idea with cloud backups, too.)

This is all pretty theoretical, since my job as a sysadmin for some small Linux beowulf clusters was several years ago.

Anyway, remember that someone will have to admin this thing. Being the admin probably won't mean any hours a week babysitting it, but you need someone that understands the setup and knows what to do if there's a problem. Swapping a drive usually requires poking at the admin UI, not just physically changing hardware.

Have a spare HD ready to pop in to replace a failed one, so your data isn't going unprotected for long. Probably you'd want a failed-drive email to go to multiple people, so whoever's around can replace a failed drive with the spare. (If you have the drive in the NAS PC or appliance, then it's a hot spare and can get used automatically.)

> **Comments:**
> **username_0**: Thanks a lot for a very detailed and thorough reply! Now I have a lot more info to digest to make a more informed decision.  
> **username_1**: Cheers.  I'd be interested to hear what you end up doing.  Maybe post a summary as an answer to your own question :P  
> **username_0**: Yup, will post something when we've decided on exactly what to do.  

---

