Building a highly write resiliant CCTV solution using SSD's

[dragmore]

Hi. At work we are redesigning a CCTV solution and we are evaluating the move from rotating drives to SSD's.

My main issue with SSD's in our scenario is the following:

1. The systems need to be able to receive from 36-72 4mbit streams of CCTV data concurrently = 18-36MB/S in 2MB random chuncks.
2. Since its recieving and has very little need for reading from the data, id guess we could say 95% writes and 5% reads.
3. We are gonna push the non-hot data over to a big NAS with Rotating SATA drives in sequencial streams so we mitigate the randomness of the writes.
4. Since these CCTV systems are offshore we need em to endure pressure changes, high temperature etc = hence the SSD's are perfect for this but one thing... Write ressiliancy..
5. Example 18MB/S = 1,5TB a day written.. A normal consumer SSD will die within 6-9months.. and these systems have 5 year life.. so we need to look at e-mlc or slc based solutions... Like the Samsung SM-825

So my question is, is this a good way of aproaching this? Since we need alot of SAS drives to be able to sustain a stable random IO performance of 18MB/s
Any input is most welcome!

PS! Systems will be a dell R510 with the H700 controller (lsi9265) running on w2008r2x64

Best regards TE

[canthearu]

Any input is most welcome!

PS! Systems will be a dell R510 with the H700 controller (lsi9265) running on w2008r2x64

Use 8 x Intel 520 - 240 gig drives in RAID5. Each drive at 60gig, can take at least 250tib of incompressible writes, as shown in my endurance test of the 520. Each 240gig drive therefore would tolerate approx 1PB of incompressable writes.

Total expected write endurance would be at least 7PB, or more then 6 years at 3TB per day.

This is assuming you would design with consumer drives. Switching to the Intel 710 drives, with 30k write cycles, would reduce the requirement to 1 or 2 drives.

[CrazyNutz]

Your in Luck! My company manufactures a Digital CCTV system called VistaPlex. I am part owner, and I'm the engineer who developed the system. It has been in use many Universities, Airports,
Casinos, Finacial, etc.. for about 12 years now. There are a lot of things we learn during this time.

1). CCTV is very demanding on the hardware, especially on the HDD/Storage subsystem.
2). Another massive problem is the File System. The most reliable FS we have found is XFS, all others failed badly overtime.
3). You absolutely must maintain 20% or more free space on your array so the File System can keep from having massive fragmentation. I know some don't think fragmentation can be an issue if you're on SSD, however it is an issue for CCTV. Most File systems leave resources allocated on the storage even after deletion these resources are not infinite (such as inodes) the more fragmentation you have the more these resources have to be used to reference all the fragments of file. We ran into this problem when we started using large arrays after 2 years, when these resources ran out nothing could be written to the array.
4). Use only motherboards with ALL SOLID CAPACITORS, trust me on this one. Also never overlook exceptional cooling.
5). Always use a UPS.
6). Do everything you can to reduce the amount of data actually written to the array. Utilise Motion Detection, or other analytics to determine when to record. Adjust the cameras to the minimal useful resolution, bitrate, and framerate.

About SSD's for CCTV. We have considered them, and concluded it too risky. However if I were to do it I would not even consider MLC. Only thing I would even think of considering is Intel's SLC.

Based on our empirical data, we find that HDD's last between 3 and 5 years in CCTV, and the 3 year mark is when we recommend customers replace them.

If you have any other questions or concerns I'll be happy to assist

[dragmore]

Thx for the response guys!!

Some comments/thoughts:

1.Since im forced to use windows 2003/2008 server this limits my options regards to better filesystems..
2.From my calculations i need a TBW of at least 4000 for my disk/stripeset
3.Im having trouble finding out where i can buy the SLC or e-MLC drives.. Both intel 720 and the samsung e-mlc drive is M.I.A
4.Anyone considered any kind of SSD caching solution that works with WRITES ? All the caching solutions ive seen is focused on read.. The main point is to mitigate the randomness of the dataflow with so many cameras sending to the server concurrently. So a SLC drive or 2 in front and a SATA drive would prolly work good..
5.Anyone know if the Dell H700 controllers will support non-dell branded drives?


br TE

[CrazyNutz]

Looks like RAID manufactures are building cards with SSD or NAND as Cache.
http://www.theregister.co.uk/2009/06...tec_capacitor/

As for mitigating the data flow, this is built in to the OS, File System, and Raid hardware. For example when an app calls write() (as in write this data to disk) what happens is the data is written to a File IO buffer (i.e. Memory). The kernel, and FS decides when it's a good time to flush this data from the memory location to disk. If there is a RAID controller in the mix it has it's own memory (cache) it will hold the data until it decides its a good time to commit it to the array.

So what you need to be concerned with is: Have alot of System Memory for the IO Buffers, Have a fast RAID controller with lots of cache, and Have an array of fast disks.

[dbguru]

You're seriously overestimating randomness of your IO workload (if it's described correctly below) - 18-36MB/sec in 2MB chunks is only 9-18 IOPS (i.e. VERY low - under 10% of limit for good 10K RPM drives).
Recently announced 1TB 10K RPM Velociraptors (about $300) looks like a good choice to me if used in RAID6 config (to prevent data loss).
UPS power and BBU for H700 controller are mandatory.

Just my $0.02.

Hi. At work we are redesigning a CCTV solution and we are evaluating the move from rotating drives to SSD's.

My main issue with SSD's in our scenario is the following:

1. The systems need to be able to receive from 36-72 4mbit streams of CCTV data concurrently = 18-36MB/S in 2MB random chuncks.
2. Since its recieving and has very little need for reading from the data, id guess we could say 95% writes and 5% reads.
3. We are gonna push the non-hot data over to a big NAS with Rotating SATA drives in sequencial streams so we mitigate the randomness of the writes.
4. Since these CCTV systems are offshore we need em to endure pressure changes, high temperature etc = hence the SSD's are perfect for this but one thing... Write ressiliancy..
5. Example 18MB/S = 1,5TB a day written.. A normal consumer SSD will die within 6-9months.. and these systems have 5 year life.. so we need to look at e-mlc or slc based solutions... Like the Samsung SM-825

So my question is, is this a good way of aproaching this? Since we need alot of SAS drives to be able to sustain a stable random IO performance of 18MB/s
Any input is most welcome!

PS! Systems will be a dell R510 with the H700 controller (lsi9265) running on w2008r2x64

Best regards TE

[dragmore]

You're seriously overestimating randomness of your IO workload (if it's described correctly below) - 18-36MB/sec in 2MB chunks is only 9-18 IOPS (i.e. VERY low - under 10% of limit for good 10K RPM drives).
Recently announced 1TB 10K RPM Velociraptors (about $300) looks like a good choice to me if used in RAID6 config (to prevent data loss).
UPS power and BBU for H700 controller are mandatory.

Just my $0.02.

Hi. Im probably am overestemating the IO-randomness. But the main problem is that i dont know the underlaying write code of the app and how it handles the writing of the chunks. In addition to my previous data i need to add that they need to handle up to twice the amount of cameras + 20-30 streams READ at the same time from time to time + archive mode which means that the app takes all the data written in the last time period and flushes it over to another drive..

So in the end we went for a solution like this:

2x200GB Samsung 825 SSD's in RAID1 for OS and to recive data (3,5PetaByteWritten endurance)
10x900GB SAS 10k drives in RAID6 for Archive
Dell H700 w/512mb NV Flash ram (LSI logic 9265)

Price was better and the performance should cover our needs now, future needs and scalability

best regards TE

[mobilenvidia]

Dell H700 is based on the LSI9260-8i not the LSI9265
It also has only 8 ports.
The new Dell Perc 710 = LSI9265 with dual core ROC but still only 8 ports

Possibly look at a LSI9260-16i if you decide to go with more than 8 drives in single array

[dragmore]

hei. Ah i didnt know that.. But the R510 Dell server has 12 slots, so its gotta be the 12 port version if it exists or the 16 or a small expander card.. Havnt gotten the server yet so im missing some info.

[mobilenvidia]

Looking at manual for R510 with 12x HDD bays, its got an expander.
One of the Perc 700 sas 4x ports will plug into the expander and then then controls the 12x HDD's

You can rest easy that all 10x drives will happily work with RAID6
Although saying that, you may not be able to have the RAID1 array on the same controller
I can't see RAID1 + RAID6 on the RAID configuration table from Dell, best is RAID1 + RAID5
You may need to experiment with this, but be warned.

[Gogeta]

Looking at manual for R510 with 12x HDD bays, its got an expander.
One of the Perc 700 sas 4x ports will plug into the expander and then then controls the 12x HDD's

You can rest easy that all 10x drives will happily work with RAID6
Although saying that, you may not be able to have the RAID1 array on the same controller
I can't see RAID1 + RAID6 on the RAID configuration table from Dell, best is RAID1 + RAID5
You may need to experiment with this, but be warned.

With 10 900GB drives, RAID5 with 1 spare is all you need. The expected rebuild time doesn't warrant the extra fault tolerance and loss of capacity for this application in my opinion. If capacity isn't a concern, there is no reason to not use RAID10 given the speculation about the write randomness.