GullLars' SSD discussion/rant corner

Like the title says, i felt we needed a thread for discussing, brainstorming, and ranting about various stuff concerning SSDs.
I will try to make interresting discussions here and post links to vairous interresting SSD material. I like to get into technical discussions or benchmark data, but i think we should have a dedicated thread for SSD benchmarking.

We could possibly make an SSD FAQ here, or start collecting Q's and A's for a dedicated thread. Feel free to rant about or ask about anything and everything regarding SSDs here, or post links to dedicated threads for help on subjects if you don't get answers or it takes too long ;)

I'll try to consolidate and post a lot of material here the comming days/weeks. For now, i have a 2000 word discussion/rant on ONFI 2, SSD scaling, some general architecture, and oppinion on upcomming generation SSDs. The main points of the post is performance density scaling and channel saturation/oversaturation.

I include a couple of pictures to illustrate and to break the wall of text :P
I have to upload some images, so it will take a couple of minutes before the second post is up.

ONFI 2 rant.

Topics:
Performance and scaling:
*MBps/GB
*R:W bandwidth ratio, raw and SSD aggregate, SLC vs MLC
*dies pr channel, # of channels, and channel saturation/oversaturation
complexity, and price

Target markets:
capacity points/ranges
IOPS points/ranges

Features and interface:
compression, NCQ, and the clean block pool writing method
fat/regular/slim SSDs
3Gbps vs 6Gbps
SATA vs SAS vs PCIe

misc:
SSD only RAID controllers?
DIMM-like SSD modules?
mixed SLC/MLC?
Multiple port SSDs?


Referances:
REF#1: http://onfi.org/wp-content/uploads/2...bottleneck.pdf
REF#2: http://www.haifa.ibm.com/conferences...pers/2_2_2.pdf



MBps/GB, a new meteric.
I want to start off by introducing you to a new meteric i made up a few weeks back, as this is the basis of much of my frustration with SSD companies, and is the core point of this post. Broken down, this is a simple ratio showing performance (MBps, sequential throughput) pr capacity (GB). This shows how much performance you get out of the flash you pay for, and can also be seen as performance density.
MBps/GB is a very relevant meteric for RAID arrays, where you often focus on aggregate performance, and it's easy to get more capacity than you need for high performance, making it wastefull. It's striking meteric for defining what sort of usage the SSD is intended for.

This is also a great meteric for showing how well performance scales at different capacity points of the same SSD line.
According to the paper i've given as REF#1, "ONFI 2 breaks IO bottleneck", Synchronous SLC and MLC have the following

MBps/GB given a dual plane SLC die holds 4GB, and a dual plane MLC die holds 8GB:
SLC; (120/4=) 30MBps/GB read, (28/4=) 7MBps/GB write.
MLC; (88/8=) 11MBps/GB read, (8/8=) 1MBps/GB write.

Before i go on to post tables of MBps/GB for a selection of SSDs, i'll also touch on a second central point to this post wich is nicely tied up to MBps/GB.

R:W bandwidth ratio.
This is another simple metric (with no unit), again a basic ratio showing performance characteristics of the storage. This ratio is very important to what areas an SSD can be expected to be used in. I'll give you the raw NAND ratios for sync dual plance die SLC and MLC in REF#1.
SLC; (120:28=) 4,29:1. (rounded up 4,3:1)
MLC; (88:8=) 11:1.
I will include the bandwidth ratio in the tables of MBps/GB.

Attachment 104745
Things worth noting:
*no SLC SSD has higher read performance density than Async SLC NAND
*many low-capacity MLC SSDs have higher performance density than Async NAND.
*The only SSDs with a >2:1 R:W ratio are: ioXtreme, Intel x25-V/M, JFM612 30GB, and C300 128GB, yet Async MLC and sync both SLC and MLC have R:W > 4:1.
*performance density scales very poorly with SSD capacity, especially for reads.


This brings me to my next topic/segment.


Dies pr channel, # of channels, and channel saturation/oversaturation.
In order to illustrate this more propperly i refer to page 22 and 23 of the ONFI 2 pdf, i've taken screenshots of the images to easier illustrate here (the first was barely over 200kb, so you have to click it to see the full pic...
http://bildr.no/image/654893.jpeg
Attachment 104746
In order to fully appreciate these 2 images, you should read the entire ONFI 2 pdf, it's pretty quick to skim through.
The first thing you will notice is that src sync (source synchrounous, the ONFI 2 flash buss spec) has a huge impact on read performance for both MLC and SLC, a slight impact on write performance for SLC, and almost no impact on MLC write performance.
You will also notice the differences make themselves more visible for more dies pr channel, especially 4 or more dies pr channel.

Here goes a quick round of some numbers and math:
The ONFI 2.1 buss spec (flash channel) allows 200MB/s transfer rate. Sync SLC can do 120MB/s read, 28MB/s write. Sync MLC can do 88MB/s read, 8MB/s write.
A single SLC die will give 120/200= 60% saturation of the channel for reads, and 28/200 = 14% saturation for writes.
Two SLC dies on a single channel will fully saturate the channel for reads with 20% oversaturation (40MB/s wasted read), and 28% saturation for writes.
To fully saturate a single channel for writes with SLC, you need 7 dies (196MB/s = 98% saturation), at wich point you oversaturate the channel for reads by 320%, wasting 640MB/s potential write speed.
A single MLC die will give 44% channel saturation for reads, and 4% for writes.
Two MLC dies will give 88% channel saturation for reads, and 8% for writes.
Three MLC dies will fully saturate the channel, with 32% oversaturation (64MB/s wasted read), and 12% saturation for writes.
To fully saturate a single channel for write with MLC, you need 25 dies (200MB/s), at wich point you oversaturate the channel by 1000% for reads, wasting 2000MB/s potential read speed.

From these numbers, i would say 1-2 SLC dies pr channel, and 2 MLC dies pr channel are fairly desirable to avoid wasting bandwidth, and securing a high performance density. Most consumer SSDs today have multiple dies pr channel, and likely slower channels than 200MB/s. This leads to oversaturation of the channels (or limiting controller throughput) and wasted bandwidth, resulting in low performance density for higher capacity drives.

Now on to the next part of this topic/segment, # of channels.
Many current SSDs use 4 channels, some use 8 or 10. As long as SSDs use SATA 3Gbps, it can be (over)saturated with 2 ONFI 2.1 channels (200MB/s) for reads. SATA 6Gbps can be saturated with 3 channels for reads.
Because of the correlation between # of channels and read IOPS, 2-3 channels may not be desirable, since it would limit read IOPS to 15-30K.
You could increase IOPS whitout oversaturating the SATA buss by only using 1 die pr channel.

On SATA 3Gbps, this would allow 3 channels of single die SLC (20% oversaturation for sequential read), up to 300MB/s read, 30K IOPS read, 84MB/s write, and 20K IOPS write (12GB capacity). With 3 channels of single die MLC, you'd get 264MB/s read, <22K IOPS read, 24MB/s write, and 6K IOPS write (24GB capacity).
This all result in low capacities and low write speeds to keep from oversaturating the SATA 3Gbps buss, so a bump up to 6Gbps will push the problem a bit back.

With 6Gbps and SLC, you could have 5 channels with single die SLC before saturating the buss for reads, at wich point you have 600MB/s read, 140MB/s write, <50K IOPS read, <35K IOPS write, 20GB capacity. Good IOPS, but low capacity.
With 6Gbps and MLC, you could have 7 channels with single die MLC and barely saturate the buss for reads, with 600MB/s 616) read, 56MB/s write, <50K IOPS read, <14K IOPS write, 56GB capacity. Again, good IOPS, a bit better but still low capacity, and very low write.

As a conclusion, it seems inevitable to oversaturate SATA 6Gbps for reads to reach usefull capacity points and/or write speeds above entry-level/RAID drives.
There's also the factor that controller complexity and cost increases with # of flash channels. From a cost vs performance density perspective, if you can get by with fairly low capacity and write speeds, 3 channels and 2 dies pr channel seems to be a sweet-spot for 6Gbps drives, regardless of SLC vs MLC. At that point, you also get 20-30K read IOPS, wich could qualify as "good enough" for a lot of uses.
3 channels 2 dies pr channel SLC: 24GB, up to 600MB/s read, 170MB/s write, 30K IOPS read, 40K IOPS write.
3 channels 2 dies pr channel MLC: 42GB, up to 530MB/s read, 42MB/s write, 20K IOPS read, 10K IOPS write.
Slightly /drool specs for RAID or OS+apps drives. Counting in controller complexity and ammount of flash, the MLC could cost below $200, and the SLC below $300.


So, with so difficult ballancing between capacity, complexity, R:W ratio, and random/seq performance, what targets are there for next generation SSDs? I'll try to draw some outlines for what i think.
*OS+application drive: 32-128GB, SATA 6Gbps, max seq read and read IOPS (<100K), >10K IOPS write, write 1 MBps/GB.
*RAID member drive: 32-128GB, SATA/SAS 6Gbps
*Laptop as only drive: 32-512GB, SATA 3/6Gbps
*Database drive 32GB-1TB: SATA/SAS 6Gbps or PCIe 2.0
*Workstation scratch-disk / active data: 64GB-nTB, sATA/SAS 6Gbps or PCIe 2.0



Features and interface.
IMHO, NCQ should be standard on all SSD using more than 1 channel and costing over $100. It's neccessary to fully utilize the parallell nature of SSDs, but in the lowest end of low-cost, the 5000-7500 read IOPS possible to get whitout NCQ could be "good enough".

The "clean block pool writing method", wich is just like it sounds, a pool of pre-erased blocks where random LBAs can be steamed sequentially to, is something i think will become common practice, and already is implemented by Intel, Micron, and SandForce (i strongly suspect, not confirmed). It's described in REF#2, here's a picture i "loaned" from Anandtech's SSD anthology to illustrate:
Attachment 104747
This basically allows the SSD to write random IOs at the same speed as sequential data, with a minor overhead penalty.

To avoid sustaining massive ammounts of random IOs and shortening drive life or high power draw, there could be implimented a sollution where you have full speed for random writes up to 100K IO's within 10 seconds, and after that 10K sustained untill the average is under 500-600K/min.

I also think more manufactureres should consider compression. It's a good performance booster for os+apps and many database usage patterns, and the space saved by compression acts as dynamic spare area to increase sustainable performance and drive longevity. It's a welcome write boost for lower capacity MLC boot drives, and could give many times write speed bonus for very compressible files (empty files could literary get a 10x write speed boost on some configurations).

Skinny/regular/fat SSDs is a term coined by Zslot kerekes, editor of Storagesearch.com. It reffers to ratio of RAM to flash. Link to article for details: http://www.storagesearch.com/ram-in-flash-ssd.html
I think it would be interresting if some manufacturer made a fat SSD with the RAM as a combination of burst-write buffer and read-cache. Fat SSDs are defined as 1% of total capacity or more in RAM size. I think this would be interresting for higher end OS+App drives.

Something i haven't discussed yet is the benefit of SAS over SATA. SAS is full duplex, and can sustain 6Gbps in both directions, wich is usefull if the SSD is able to oversaturate the interface for reads, since you could then write some to the drive whitout loosing much, if any, read speed. Because of the full duplex, it's also superior for mixed random read/write IOPS latancy. The downside of SAS is ofcause that it's not compatible with SATA HBAs and most motherboards.

There's also a strenghtened case for PCIe interface for upcomming SSDs, since it's possible to saturate 6Gbps interfaces with 3 channels (or 5-7 dies). With PCIe 2.0 x4 or higher, you have 2000MB/s or higher bandwidth availible. There's also the lower latency benefit, and avoiding HBAs as bottlenecks. By using compression togheter with a PCIe 2.0 interface, some files could get a significant speed boost.

In order to saturate a PCIe 2.0 x16 link whitout compression for reads, you need 40 saturated flash channels. With 2 MLC dies pr channel, you'd need 45 channels / 90 dies. With 8GB pr die, that's 720GB.If set up in with double-stacked packages with 2 dies pr package, it could be a 6 long 4 wide grid arrangement with one spot missing, wich should fit a card smaller than a full size graphics card.



Over to hypothetical thought experiments.
Speaking of compression and PCIe busses, there's a case to be made for a RAID controller with on-the-fly compression, like SandForce. It would neccesitate an additional level of abstraction, but could boost the speed of drives or arrays signifficantly for highly compressible files, provided a powerfull enough processor or co-processor to handle on-the-fly compression at such speed.

I'll drop DIMM-like SSD and mixed SLC/MLC setups for now, since the post went a bit long.


The last thing i'll mention as a hypothetical is a SSD RAID-box, speciffically made as a 3,5" or 5,25" drive with one or more 4x mini-SAS 6Gbps ports on the back. The back may be pass-through for multiple 3-4 channel SSDs inside the box sharing PCBs.
A quick mental model tells me you could fit roughly 16 such 3-4 channel SSDs in a 5,25" box, 4 PCBs with 4 modules each, each PCB having a 4x mini-SAS 6Gbps port, and all 4 PCBs sharing a 1-2 molex power plugs.
A 3.5" box could probably hold 8 (3-channel) SSD modules and have 2 4x mini-SAS ports.


So, what do you guys think? :P

GullLars - nice article - I will need to read it a couple times more with the backup.
Interesting new metrics - is there a limit to how much can be parallelized within an SSD - how many channels an SSD can include?
Seems like I read that x25-m has 10 channels - is that the most so far?

Nice thread GullLars and as usually some great insights. I’m looking forward to seeing this thread develop. :cool:

Amazing and Great Idea. I did similar on another Site which has been hit well over a million times now and I am not a techy per sae compared to some of you with abs brilliant ideas as seen in the start of this thread. Im here!!!!...if its ok.

Quote:

---

*no SLC SSD has higher read performance density than Async SLC NAND
*many low-capacity MLC SSDs have higher performance density than Async NAND.

---

few things,
1.
the basic purpose of Micron's document is showing the differences and benefits of ONFI 2 standard, adding to it some architecture concerns involving no. of dies per channel and number of channels per die,
asynchronous NAND interface, is the ONFI 1 standard devices,
ONFI 2, support synchronous interface and so is much faster.

2.
what do you want to say by that?
going numbers over numbers can be endless and not really necessary,
calculating the ratio's between read and write bandwidth and MLC vs SLC in respect to that,
fewer channels vs many, many dies vs less, dies vs channels and whatever,
i feel like this freaking robot from star-trek.

now for the rest,

one question,
why not dealing with what is?
what isn't, simply, isn't.
there are many things yet to understand about IC's, signaling, pins, general PCB build, controller architecture, nand quality, companies that are involved with manufacturing and they're agenda, new technologies and the future of nano-electronics, etc. etc.

this post is not about understanding how the technology works, but rather on the out come of it.
the first question you should ask yourself before pointing out such idea's is, why havn't anyone has tried it already, or, possibly that someone is already trying it..?

now,
there are 2 reasons why not sharing any idea with companies,
1.
you would not like anyone to take it, yet, posting it in a known forum does the opposite,
2.
you are unsure whether this is a good idea or not and so, you ask people to share they're thoughts.

and why instead of wasting time talking about these things, not just go over OCZ or mushkin forum and ask people there how feasible your idea's are..
send an e-mail to anand or bug one of the guys from one of the companies by fishing they're mail from some letter they have made or anything,
you will never go forward by drawing a herd of enthusiasts after you,
you will go forward when you march alone.

Onex, I have no financial stake in SSD technology, and the post is just me airing my thoughts and frustrations.
I don't have any problem with companies reading this thread, and finding something interresting or taking ideas from here, if it results in better products, i'd say it's a good thing.
I ask people to share their thoughts, because it's enlightening and fun discussing technology, not because i'm uncertain about my ideas (i won't say i'm certain all are good ideas though, but i find them interresting).
EDIT: like it says at the beginning of the secon post "ONFI 2 rant", it's not a research/technical paper or conventional article ;)


I have another table i feel like sharing, giving an overview of SSD controllers and the tech in them. This is a work in progress, and i've colored the things i don't have info on or are uncertain about yellow. Take the numbers as what they are, rough numbers for an overview.
It's sorted by manufacturers and controller series.
Attachment 104756
If you have or come across info for the yellow squares, pleese post so i can update it :)

i just think you are digging the same ground again and again with no real purpose,
i don't discourage you though.

There is a good link to the specs for the JMF601 here:
http://www.jmicron.com/Product_JMF601.htm

JMF602: http://www.jmicron.com/Product_JMF602.htm

JMF612: http://www.jmicron.com/Product_JMF612.htm

Wiki: http://en.wikipedia.org/wiki/JMicron

That's the site i used for the JMicron specs, but it says nothing about the cache size or spare area, and i didn't find info on capacity range (just number of devices supported).
I didn't find numbers for random read IOPS on the JMF601-602 either. I know it's out there, but it drowned in the searches i did.

You can upgrade the firmware on the X25-E and they do release new versions from time to time, but it doesn't do much (anything?) for performance or otherwise in my experience.

And where did you get 40k 4kb write IOPs for the E? I am hard pressed to get 17k out of mine. 65k-70k for 4 of them in R0.

intels dont have any cache..

From diskusjon.no, ssd benchmark thread, IOmeter single drive results top10 list

Quote:

---

1. Ourasi : Read: 39945.13, Write 43352.82, Workstation 21163,62, X25-E G1 32gb ICH9R

---

I've also seen screenshots of CDM 3.0 and AS SSD where it got >160MB/s for 4KB random write QD 32/64. I couldn't find one right now, i'll check if ourasi cares to run a quick round of CDM3.0 on this 2R0 x25-E ICH10R array as a referance. Intel's data sheet for the x25-E lists it as 3.3K random write IOPS ...
Intel SSDs don't use the external cache for user data, but there is one, and the size is what's listed in their data sheets.

EDIT: anvil hasn't got around to upload results for his LE's or C300's as single drive yet, so ourasi still holds the 1st place at single drive IOmeter score.

OCZ Core Specs:
Read (ATTO Disk Bench) 132.888 MB/s
Write (ATTO Disk Bench) 89.587 MB/s

Reality: http://www.alternativerecursion.info/?p=106

http://www.anandtech.com/show/2614/10

Finding good data on the JMicron JMF602 controller is nearly impossible, but from what I've heard it's got 16KB of on-chip memory for read/write requests. By comparison, Intel's controller has a 256KB SRAM on-die.

The OCZ Core used 8GB NAND chips from Samsung.

http://www.madshrimps.be/vbulletin/f...c-write-56856/

In the case of SSDs, some of the capacity is reserved for formatting and redundancy for wear leveling. These reserved areas on an SSD may occupy up to 5% of the drive’s storage capacity. On the Core V2 Series the new naming convention reflects this and the 30 is equivalent to 32GB, the 60 is equivalent to the 64GB and so on.


Some more bits here: http://hothardware.com/Articles/Four...-Mtron/?page=2

Quote:

---

Originally Posted by One_Hertz

And where did you get 40k 4kb write IOPs for the E? I am hard pressed to get 17k out of mine. 65k-70k for 4 of them in R0.

---

http://www.diskusjon.no/index.php?ap...tach_id=369239
That's a screenshot of an x25-E in use in a webserver for months whitout cleaning. 40K IOPS 4KB random write in CDM 3.0
Ourasi also got >40K IOPS random write in IOmeter over a year ago when he benched one single.

EDIT: thanks for the data audienceofone, i'll add it to a revised version of the table later.
Any thoughts on the rant in post #2, or the meteric MBps/GB? I'm wondering if you think "performance density" could be an appropriate name.
On a related theme, i'll try to make some graphs of IOPS/average accesstime scaling with queue depth for a selection of drives and arrays later, but that will be a big post with nice graps and stuff.

I never tried CDM, but certainly none of the 4 of my drives get anywhere near that. IOMeter?

Quote:

---

I've also seen screenshots of CDM 3.0 and AS SSD where it got >160MB/s for 4KB random write QD 32/64. I couldn't find one right now, i'll check if ourasi cares to run a quick round of CDM3.0 on this 2R0 x25-E ICH10R array as a referance. Intel's data sheet for the x25-E lists it as 3.3K random write IOPS ...

---

first tip:
http://www.google.com/images?um=1&hl...=&oq=&gs_rfai=

gives a great help almost everywhere..:yepp:

Quote:

---

And where did you get 40k 4kb write IOPs for the E? I am hard pressed to get 17k out of mine. 65k-70k for 4 of them in R0.

---

you talk about 4KB write with QD of 1 which stands for ~16K iops,
lars is talking about QD of 32 which stands for ~200MBps or 50K IOPs

The new google layout has had me confused for some time, i didn't think to do an image search... :P I was searching for benches or reviews of x25-E, and didn't find anything that way.
As you can see by the pictures, at QD 32, the x25-E deserves the E in the name, especially if you factor in when it was released (2008). It's still ridiculusly fast compared to any drive from a 3Gbps controller. Only the newest controllers that barely has hit the market can put up a fight, and at a cost. If you use random data (not compressible), the only drive on the consumer market that can beat it is C300 256GB, and for some workloads the 128GB too. The SandForce drives and C300 128GB top out around 130-140MB/s for writes, limiting write IOPS to 30-35K (again, for incompressible data).
This should be nicely reflected in AS SSD total scores. The x25-E gets around 550 points from ICH10R. C300 256GB gets more like 650-700 points, and C300 128GB is around 580.

Quote:

---

Originally Posted by onex

you talk about 4KB write with QD of 1 which stands for ~16K iops,
lars is talking about QD of 32 which stands for ~200MBps or 50K IOPs

---

No actually I am talking queue 32... None of my X25-Es can do over 17k 4kb 100% random writes on ICH10R using IOMeter. Perhaps something is up with my config.

I'd have that checked out. If you can split the RAID whitout too much hazzle, try a single drive in AHCI mode, and post AS SSD results + CrystalDiskInfohere.

Would be very annoying to split as its by boot array :(

I tried googling but I couldn't find any iometer results.

Quote:

---

No actually I am talking queue 32... None of my X25-Es can do over 17k 4kb 100% random writes on ICH10R using IOMeter. Perhaps something is up with my config.

---

boy, that is ODD.
http://img264.imageshack.us/img264/5549/x25e.png
http://www.xtremesystems.org/forums/...&postcount=714

This is all CM... I am talking IOMeter.

One Hertz,

Feel free to download some runs on one of my Es, Link

It includes result files for both read and write. (aligned 4KB)

Hard to tell what could be wrong with your config, the drive in the screenshot linked by GullLars was actually misaligned as well. :shakes:
Both my Es are available for a few benchmarks if there's anything you'd like to see.

Well that is interesting. I've had Es ever since they came out, on multiple computers, and have never gotten anywhere close to those results on any configs. My read results are exactly the same as the ones presented by your guys, but 4k writes are way off.

One Hertz,

On what controller?
NCQ?

Quote:

---

Originally Posted by Anvil

One Hertz,

On what controller?
NCQ?

---

Tried on ICH9R, ICH10R, LSI 9211. The results are all the same.

Yes, NCQ is on. Otherwise, my read results would suck as well and they do not. It seems like my NSQ is off only for writes? Makes no sense how that is possible to me.

One Hertz,

Don't think that's possible wrt NCQ, either it's on or it's off.

I just did a 4KB rw IOmeter test on a single E on the 9211 ~40.000 iops (random write 4KB)
QD1 12740
QD2 22590
QD4 32803
QD8 38356
QD16 40748
QD32 40222
QD64 39396

Can't blame the 9211 :), the ICH test you downloaded was from my UD7. (ICH10R)

It COULD be you have turned off drive write cache... That could result in a major drop in random write performance, but i have no idea how much.

Quote:

---

Tried on ICH9R, ICH10R, LSI 9211. The results are all the same.
Yes, NCQ is on.

---

that's one of the weirdest things i heard,
maybe the MB is problematic..

so actually, it gives the same results with QD of (4KB random read) and QD of 32?
if it is, it is only NCQ/AHCI related, extremely doubtful if the drive is getting QD 1, less then 16K IOPs..
let's see.

Quote:

---

Tried on ICH9R, ICH10R, LSI 9211. The results are all the same.
Yes, NCQ is on.

---

that's one of the weirdest things i heard,
maybe the MB is problematic..

so actually, it gives the same results with QD of 1 (4KB random write) and QD of 32?
if it is, it is only NCQ/AHCI related, extremely doubtful if the drive is getting QD 1, less then 16K IOPs..
let's see.

I disabled the "Write caching" Policy and iops are down to ~2300 so it can't be that. (~9MB/s)
(still on the 9211)

I will show my iometer when I get home tonight...

From my memory - QD 32 4kb random read - ~135k, which is fine. QD 32 4kb random write - 67k, which is low apparently.

Two of my Es are VERY early drives. I got them the week they came out. Perhaps something is gimped with them. All 4 are running latest firmware though,

hey one hertz...did you ever figure out what is the deal with your e's?

Quote:

---

Originally Posted by Computurd

hey one hertz...did you ever figure out what is the deal with your e's?

---

Nope... Mainly because I really do not particularly care if I can do 70k random 4kb write iops or 150k, especially since my reads are fine.

Good point ;)
As long as your sequential write is OK for you too, it shouldn't matter.

good perspective. if it aint broke, dont fix it:up:

Rather: If some aspect is partially broken, but fixing it won't make a difference, don't bother. :)

I have a project i'm working on to show scaling of different storage sollutions (both total performance and compared to itself) by Queue Depth and block sizes 4-32KB for 100% random 100% read. This allows for a deeper look into the effects of parallellity, and the granularity.

Different from earlier suff i've done, this will be with a linear QD stepping of 1 from QD 1 to 32. This is low to medium "parallell load", and will capture the relevant range for desktops and most kinds of workstations.

I'm looking for results from single HDDs and SSDs, and arrays of HDDs and SSDs. Basically any storage type. The more diversity i get the better. Anything from a 10 year old 5400RPM PATA drive, to a first gen JMicron SSD, to a velociraptor, Intel SSD, and 2-8(+) SSD/HDD arrays.

Results will be posted later as nice sorted graphs and diagrams in a dedicated storage/SSD benchmark thread I'll make.

Here's the config, i've included a txt file with info:
Attachment 105130
It will take about 20-25 mins to run and give me 128 datasets/points.
Post result files (csv) with the naming described in the info file in this thread (for now). If you do runs on multiple configurations, you can include multiple csv files in a single zip file with your username (and possible a general description of content).

Any results are greatly appreciated, since this won't be possible without collaboration, and nobody else are doing anything simelar of this detail and work size and sharing results (to my knowledge).
EDIT: it would be great with a comment on what you think, and if you're in, so i can see if there's any interrest in this or if i should just leave it.

OCZ shows off HSDL (High Speed Data Link). A 20Gbps cable interface using a PCIe 2.0 x4 host card. They also have an x16 version with 4 HSDL ports.

it's not about you lars, and never been, actually u and anvil where the ones that actually really helped out since the beginning, answering every question, especially you, and it is really worth mentioning,
i would have never understand SSD's today the way i am without you'r guys help, though you can watch this thread falling back already to the second page in the forum, after only few weeks and it shows you the amount of caring here at the forum, where everyone is occupied with himself, this is how it looks.
when you fired up this thread, first thing which has came up mind was, heck, this would never be the way it was at the Norwegian forum, simply because people don't seem to care and look where it is...
some people do mind, others just keep to they're own, forming groups etc, this is also happening in the WCG forum, by people's 'ranking' etc.
maybe for you it is o.k, you take it, though i think it is just showing the crippleness and actual lack of commitment and caring of people here, this is not how a forum should behave with all due respect to the technical level which is indeed quite high.
that's the point, the main issue.

onex, i think that your complaint could be construed as a basic dislike of society in general :) things in the real world are going to happen just the same as that. it doesn't matter if its work, school or the playground man. that just life. that is how society itself is, and this is just a little cross-section of society here as well. i do not feel that as a whole this forum is worse than any other forum out there. dude there is no lack of commitment here in this sub-forum you have a group of guys who come and collaborate together every day. I challenge you to find a forum that has this nice little utopia that you speak of....it doesnt exist in real life or in forums.
sorry, just my take on it bro.

i can't accept it though, people at anandtech are much different, MUCH different.
yet i'm still investigating it though.
last time with the same happening at a different forum, around 15 people were coming to the thread, posing the same issues, and it was after 2 weeks or so at that site..
here everyone fall silent, why?
because they are just comfortable with it, where people treat them like they are nothing, casually.
i know, this is society, this is what is ugly in society, and every sane person would say that,
where people have to change in order to fit themselves with others, always be sorry, step back, not be too much like that or too much like that, you know, conforming into a pattern,
where others set the tone, where there are no individuals, free thinkers, there is the crippleness of intelligence and freedom of the individual, so lack of creativity.
we have to understand that we as individuals create society, we create all this mess, and it is our responsibility to change that, by stepping out of the line, be different,
not conforming into any pattern, not being socialist or communist or any other form of organization, but people who are free in they're mind and heart,
only then there would be real collaboration, where there is no, me and you, I or him, there is only us, as human beings in this world, working and living,
as one.

hang in there onex

I'm not saying things will change, there will always be cliques.

One of the reasons I haven't gotten much done lately is because there was zero response to the post i'll quote (actually, there was one from audienceofone).

Quote:

---

Originally Posted by GullLars

I have a project i'm working on to show scaling of different storage sollutions (both total performance and compared to itself) by Queue Depth and block sizes 4-32KB for 100% random 100% read. This allows for a deeper look into the effects of parallellity, and the granularity.

Different from earlier suff i've done, this will be with a linear QD stepping of 1 from QD 1 to 32. This is low to medium "parallell load", and will capture the relevant range for desktops and most kinds of workstations.

I'm looking for results from single HDDs and SSDs, and arrays of HDDs and SSDs. Basically any storage type. The more diversity i get the better. Anything from a 10 year old 5400RPM PATA drive, to a first gen JMicron SSD, to a velociraptor, Intel SSD, and 2-8(+) SSD/HDD arrays.

Results will be posted later as nice sorted graphs and diagrams in a dedicated storage/SSD benchmark thread I'll make.

Here's the config, i've included a txt file with info:
Attachment 105130
It will take about 20-25 mins to run and give me 128 datasets/points.
Post result files (csv) with the naming described in the info file in this thread (for now). If you do runs on multiple configurations, you can include multiple csv files in a single zip file with your username (and possible a general description of content).

Any results are greatly appreciated, since this won't be possible without collaboration, and nobody else are doing anything simelar of this detail and work size and sharing results (to my knowledge).
EDIT: it would be great with a comment on what you think, and if you're in, so i can see if there's any interrest in this or if i should just leave it.

---

I would prefer if people say they don't have the time to participate, or have set up their rig in such a way that testing is difficult. I also understand people are thinking of vacation.
I've got a lot of stuff going on IRL ATM, but if i had some result files i could work on them an hour now and then when i get time.

I'll see what drives are available later this evening, should be able to do a few tests.

In which thread did you post this request?

@GullLars - my apologies for not responding, I have been trying to find the last 120 pnts or so in pcmv to overtake Mike one more time -

http://service.futuremark.com/compare?pcmv=333089

Steve,

Hmm, 120 points, not much but not that easy as we know.
SS cooling or what?

@GullLars,
Here's a few runs using the LSI 9211, 1x Seagate 5900LP 2TB and 1x Intel X25-E.
I might be able to do a few more runs later tonight or tomorrow.

still on chilled water - waiting for a cascade

Steve,

OK, that might be what tips the scale.


@GullLars,
WD VelociRaptor
(single and 2R0 using the 9211, 2R0 is software raid, stripe size 4KB)

Quote:

---

Originally Posted by Anvil

Steve,

OK, that might be what tips the scale.

---

That is until Mike hooks up with some ln2 - if he hasn't already.
With the cascade I have on the way I should be able to go to ~-100C - http://www.xtremesystems.org/forums/...d.php?t=253872

ln2 used wisely (Mike is most experienced) goes much lower: -170 or better.

I'm sure he's got something prepared :)

Anyways, you gained 900pts by just tweaking and thats nothing to sneeze at.
I haven't compared your result to Mikes, I'll do that later, I expect you've both got some strengths and weaknesses.

GullLars,

Last run for today.
2R0 C300 9260 stripe size 16KB, test file deviates from the config, _only_ 1GB.

Thanks for the files Anvil. The original request i quoted was posted in this thread on the middle of page 2, i also posted comments in other threads linking to it, and on diskusjon in the benchmark thread (i think).

I haven't got my desktop connected yet since it doesn't have wireless, and i only have wireless internet access ATM, so i'll hang on to the files and look at them, and then make something in a few days. Having the data ready to crunch is nice, so just keep em coming if you guys have more times and available drives/arrays ;)

steve, that's quite brute, man you could've atleast said a few words, some of the things said refer to you, yet you don't seem to mind it, it just fly right near you, quite astonishing,
you will have to step out of that behavior of not dipping your hands into any conflicts, keeping everything smooth and nice,
or rather you are at a different class, you know how to behave, you know the lines and you never cross them,
i have never met anyone like that, that's quite insane.
i'm just struggling to figure that one out.

lars,
a bit off topic from your post, yet there's some interesting papers regarding some more deeper info on SSD's, 3 bit per cell, 2 bit MLC, error rates, general build, the papers are quite complex, yet they take the reader further on into the process, disclaimer: don't read unless you understand.
there is a thread at AT you might like to see, it's in the storage memory forum under 8kB page size on IMFT new 25nm flash?, there are some links there, you can go through even briefly and fish out some facts.
there is also this one, http://www.cyclicdesign.com/whitepap...n_NAND_ECC.pdf dealing with error rates and error correction,
it comes fit with the adoption of 25nm and 8kB pages in NAND technology, as the technology shrinks, the hurdles faced by engineers and research centers are much bigger, making 3BPC MLC a legitimate junction in the road map,
the hurdles are higher error rate due to tighter voltage differences through cell programming and bigger erase pages causing the NAND to drop endurance significantly.
with that coming the development of 100K P/E cycle MLC and 1M cycle SLC for strictly enterprise usage.
some interesting read feeling the mind with some new ideas!
there is also Micron's blog and release press sharing some info, and some nice links to youtube video's too.
there's more and more, so go read read and read :).

Onex, your rants are getting out of line. Every time I read one of your posts I struggle to understand what you are trying to say. I don’t know what you’re trying to say about Steve but you certainly got the 1st bit right. He is a fair person, polite, and (well) mannered. He helps people out and shares. How could you have beef about that? That is a rhetorical question by the way :up:

onex,

I'll have to agree with audienceofone here.
Remember that all of us are doing these activities in our leisure time.

A small note on the testing.
I've done my share of testing and it is very time consuming, especially when I do my own tests and then some variation of my own tests for the forum.
A few days/weeks later there is a new variation of the same test (I'm talking about iometer if anyone wonders), and then it's the same "work" over and over again.

If we could agree on some standard test config files it would make life a lot easier.
(I know we are in the middle of something new here, sort of uncharted territory but we need to agree on where we go from here)

Onex, I too agree with audienceofone and anvil here. I don't see why you would have a beef with Steve.
It's OK that you are trying to get more stuff to happen, but it can be hard to understand what you write, and some of it comes across as overly confrontational, and other parts almost political or philosophical in nature.

I have already read the material you refer to with exception of the Anandtech forum. I've also read all articles on Storagesearch on SSDs and related stuff (about 150 has been published i think), SNIAs papers, ONFI papers, some ECC and wear leveling and garbage collection papers, and other low-level architecture stuff. I follow the micron blog and are subscribed to their youtube channel ;)
If you haven't read it, i would suggest reading the IDF (intel developers forum) whitepaper on the effect of spare area on write amplification, and an IBM research paper on writing algorithms and more in SSDs (Anand used it as a source for his SSD anthology).

2 more tests

X25-E 2R0 on the 9211

One test is using SW raid, 4KB stripe size, the other one is defined in the MSM which gives 64KB stripe size. (hopefully they'll allow for other stripe sizes in future releases)

A note about my test using the C300s, although the testfile is 1GB the test is valid as the test is performed without using the controller cache. (DCD -> Disk Cache Disabled)

Disk cache disabled... Do you mean the cache on each of the C300s, or the RAID controller cache?

Disk cache disabled on the controller, I don't think it's possible to disable the caching on the SSDs.

i think it is disk cache.
cached i/o or direct i/o is how you set it between using cache or not on the controller....so that only leaves one possible explanation for "DISK cache enabled/disabled"

computurd

you may be right about the disk cache policy, it is however a bit more to it.

DIO = for reads all data is bypassing the controllers cache, except when read ahead is enabled, for writes all data bypasses the controllers cache except when WB is enabled
CIO = all data passes through the controllers cache including writes even when WT is enabled.

BTW, CIO is being deprecated.

Given this, one can do iometer read tests on the LSI using 1GB filesize as long as DIO is enabled, for writes WT must be enabled.

Link to LSI document

As for the disk cache policy, the setting does make an impact in some benchmarks (depends on the SSD), I haven't spent much time testing it though.

yes they also have an option in the MSM for running garbage collection manually on the drives in an array, but it is still grayed out....wonder when that will come through? :)
I havent noticed much difference on the disk cache..i have tested it in several configurations and never any noticeable difference.

one thing i have noticed is it seems that in always read ahead and DIO i get the best results as long as write thru is enabled. the latency with always read ahead is the same as well. from running the profiles that gullars gave me, there is no difference basically as long as write-thru is enabled the results are the same, but real world and loading tests always read ahead is better.

Non SSD discussion/rant but hey. Check out 3D video mapping on this site. http://www.urbanscreen.com/usc/category/projects
Well wicked. (Especially P.O.P & 555 Kubik.)

As for storage check out DiskMax. Nice programme. The MEM clean is quite good as well.
http://www.koshyjohn.com/software/diskmax.html

awesome guide, long read but well worth it!

Quote:

---

How could you have beef about that?

---

Quote:

---

I'll have to agree with audienceofone here.

---

Quote:

---

Onex, I too agree with audienceofone and anvil here.

---

just make some order in all of this,
there's no "beef" with anyone really, there's not "bad" feeling, it is just something meant to wake people up,
there are things to be said, yet people would have to be aware at them otherwise it is just seems misplaced.
it is just the behavior at this forum, it is behavior of people, it is how people are used to behave and they take it normal,
when someone point it out, they just don't seem to understand what he\she is talking about and it is also seen through the nature of the comments.

Quote:

---

I have already read the material you refer to with exception of the Anandtech forum. I've also read all articles on Storagesearch on SSDs and related stuff (about 150 has been published i think), SNIAs papers, ONFI papers, some ECC and wear leveling and garbage collection papers, and other low-level architecture stuff. I follow the micron blog and are subscribed to their youtube channel
If you haven't read it, i would suggest reading the IDF (intel developers forum) whitepaper on the effect of spare area on write amplification, and an IBM research paper on writing algorithms and more in SSDs (Anand used it as a source for his SSD anthology).

---

as for that,
i'll note, that these papers are not so easy to figure, one needs some background with mathematics, and has to study deeply in order to understand these papers coherently.
it is not enough to just skim through them or read them briefly, it is not enough either to get the general idea,
it demands a great deal of studying in order to figure them out properly and be able to cross between idea's and views lent through them, figure out things which are not written and so on, it also takes time and thinking.
but if you say you understand them, well, o.k ;).

Quote:

---

Originally Posted by GullLars

I have a project i'm working on to show scaling of different storage sollutions (both total performance and compared to itself) by Queue Depth and block sizes 4-32KB for 100% random 100% read. This allows for a deeper look into the effects of parallellity, and the granularity.

Different from earlier suff i've done, this will be with a linear QD stepping of 1 from QD 1 to 32. This is low to medium "parallell load", and will capture the relevant range for desktops and most kinds of workstations.

I'm looking for results from single HDDs and SSDs, and arrays of HDDs and SSDs. Basically any storage type. The more diversity i get the better. Anything from a 10 year old 5400RPM PATA drive, to a first gen JMicron SSD, to a velociraptor, Intel SSD, and 2-8(+) SSD/HDD arrays.

Results will be posted later as nice sorted graphs and diagrams in a dedicated storage/SSD benchmark thread I'll make.

Here's the config, i've included a txt file with info:
Attachment 105130
It will take about 20-25 mins to run and give me 128 datasets/points.
Post result files (csv) with the naming described in the info file in this thread (for now). If you do runs on multiple configurations, you can include multiple csv files in a single zip file with your username (and possible a general description of content).

Any results are greatly appreciated, since this won't be possible without collaboration, and nobody else are doing anything simelar of this detail and work size and sharing results (to my knowledge).
EDIT: it would be great with a comment on what you think, and if you're in, so i can see if there's any interrest in this or if i should just leave it.

---

Areca 1231ml-4g/5xAcard 9010 R0 results 4k stripe, 4K allocation attached

Quote:

---

Originally Posted by onex

<...>
i'll note, that these papers are not so easy to figure, one needs some background with mathematics, and has to study deeply in order to understand these papers coherently.
it is not enough to just skim through them or read them briefly, it is not enough either to get the general idea,
it demands a great deal of studying in order to figure them out properly and be able to cross between idea's and views lent through them, figure out things which are not written and so on, it also takes time and thinking.
but if you say you understand them, well, o.k ;).

---

I've done 2 years at a university studying chemistry before "dropping out" to study computer science instead, and in that time i did 2 semesters of calculus. I also majored(?) in mathematics, physics, and chemistry in grades 12 and 13 (would you call that high school or college? I'm not sure about the english term for grades 11-13). I may not understand everything in all the papers, but i understand the equations and what they describe, and also how they evolve.

If you were to compare the material and time i've "studied" SSDs (articles, architecture, working on data from benchmarking), it would likely compare to 2-3 courses in a university (at least the same effort on my part compared to the courses i actually took). Still, there's probably a lot i don't know, and i'd like to learn more (both SSDs and computer architecture and workings in general), which is one of the reasons i'm switching to computer science.

EDIT: looking forward to your results steve. I'm getting my gaming rigg / workstation hooked up to the net tomorrow, so then i'll likely be more active with benchmark data again. Looking forward to my 1900x1200 monitor after a couple of weeks on a 15" laptop with 1280x1024 :)

Areca 1231ml-4g/5xAcard 9010 R0 results 4k stripe, 64K allocation attached

curious to see if the 64K allocation shows a difference - I thought I saw a big increase in going from 4k to 64k allocation is pcmv HDD test.

ok so the last run for tonight - good night all.

SteveRo_9211-8i_8R0_Acard9010_8x2GB_4kstripe_4kallocation_980x_ no_oc_pcie100 attached

gullars,
i'm talking about being able to write a thesis about it, writing a research, if you as a person, understand what is being said, you can go forward with it.
if you don't exactly understand what is being said, you skim through the paper or read briefly through it, missing a lot of details that takes away the effect of the paper, takes away the ability to go on from there.
this is why it is extremely important to understand everything that is being said, and if anything is misunderstood, figuring it out at all cost.
that's the only way one can figure out things better.
when one figures out some very complex material, it gives him also better understanding in other fields, if you don't challenge yourself, if you don't push where you cannot understand, where it seems to hard to reach, you never go on properly.
you should query every question raised and every idea to it's end, every confusing acronym or unknown feature.
going through any material without fully understanding it, has very little value,
and it doesn't matter going alone, it doesn't matter having days of not figuring out things and just watching the news, and sometimes you have to think nights in order to figure a certain aspect of the hardware a graph, equation, constant or an analysis,
but then it is all over,
ones it is there, once it springs through your mind, it will stay there, and you understand it so well, you can actually hold it.

Mostly, the stuff in the papers i don't fully understand is the specifics of the photo lithography, low-level circutry (like connector, signal driving details, and transistor workings in detail), and how the ECC and redundant algorithms works in detail (how it calculates data). I'd say i've got a pretty good understanding of the higher level architecture and workings of SSDs, and do a real effort, like you say, to not let go off things i don't get right away in more in-depth papers. Edit: i think this reflects a bit in my long ONFI rant on the first page of this thread ;)

i personally dont know crap about the inner workings of them :) I just reap the rewards! (and abuse them mercilessly)

The data cable goes to the smallish plug and the power to the wider one. :)

the power goes to the wide one!?!?! oh crap!

Quote:

---

Mostly, the stuff in the papers i don't fully understand is the specifics of the photo lithography, low-level circutry (like connector, signal driving details, and transistor workings in detail), and how the ECC and redundant algorithms works in detail (how it calculates data). I'd say i've got a pretty good understanding of the higher level architecture and workings of SSDs, and do a real effort, like you say, to not let go off things i don't get right away in more in-depth papers. Edit: i think this reflects a bit in my long ONFI rant on the first page of this thread

---

personally, i don't think you understand what you don't understand, i tried to help you see things which seemingly you don't, yet you just treat them as if they're just small irrelevant stuff, and you go on.
you don't seem, to leave yourself aside and look without prejudgment on what is happening outside,
and i've told you this already in different ways though it passes next to you and it's a shame.
seems like you already have an opinion on how things should look and showing you something else just interferes with your way of thinking.

i don't think you look deep enough inside yourself.

hopefully, you'll let these words sink in.

Quote:

---

Originally Posted by GullLars

I have a project i'm working on to show scaling of different storage sollutions (both total performance and compared to itself) by Queue Depth and block sizes 4-32KB for 100% random 100% read. This allows for a deeper look into the effects of parallellity, and the granularity.

Different from earlier suff i've done, this will be with a linear QD stepping of 1 from QD 1 to 32. This is low to medium "parallell load", and will capture the relevant range for desktops and most kinds of workstations.

I'm looking for results from single HDDs and SSDs, and arrays of HDDs and SSDs. Basically any storage type. The more diversity i get the better. Anything from a 10 year old 5400RPM PATA drive, to a first gen JMicron SSD, to a velociraptor, Intel SSD, and 2-8(+) SSD/HDD arrays.

Results will be posted later as nice sorted graphs and diagrams in a dedicated storage/SSD benchmark thread I'll make.

Here's the config, i've included a txt file with info:
Attachment 105130
It will take about 20-25 mins to run and give me 128 datasets/points.
Post result files (csv) with the naming described in the info file in this thread (for now). If you do runs on multiple configurations, you can include multiple csv files in a single zip file with your username (and possible a general description of content).

Any results are greatly appreciated, since this won't be possible without collaboration, and nobody else are doing anything simelar of this detail and work size and sharing results (to my knowledge).
EDIT: it would be great with a comment on what you think, and if you're in, so i can see if there's any interrest in this or if i should just leave it.

---

Good morning,

SteveRo_9211-8i_8R0_Acard9010_8x2GB_4kstripe_4kallocation_dynam ic_softraid_980x_no_oc_pcie100 -

highest iops I saw was just under 180k
highest MB/s I saw was 1167.

Hey Gullars, long time old buddy.

I remember I got a lot of good tips from you for getting my Vertex 30GB, soon a couple of years ago. I've killed that little poor thingy now and need a new SSD soon.

I've been looking at Crucial RealSSD C300 64GB, and it seams to be one the best deals for a good SSD right now. Would you recommend hitting on this or wait for Intel G3 or other good SSD that would arrive soon?

If I have to wait, then I want to try some CPR on the Vertex. I've been booting from HDD in a couple of days as a temporary solution, and can't bare it for a long time. I was puling, pushing (and rubbing) the wires inside the HDD-bay and BIOS can't find the Vertex after that. I checked the connections and interface, they look OK, but still looks dead. Is there any good tricks to find out how dead it really is?

guy's you just seem to happen to have a big ego,
at least some of us do seem to pay attention to what is being said,
respecting others, is not a bad way of behavior, rather then lick them upside down with pretty words...

Quote:

---

Originally Posted by onex

guy's you just seem to happen to have a big ego,
at least some of us do seem to pay attention to what is being said,
respecting others, is not a bad way of behavior, rather then lick them upside down with pretty words...

---

WTF? Can someone please tell me what this guys problem is?
Coz I sure as heck can't see how the tone of his posts would be justified :confused:

What ever happened to this thread, miss the old days here :-/

On a side-note, if anyone can add some +ve input here it'd be greatly appreciated:
http://www.xtremesystems.org/forums/...s-for-Mac-Mini