Forum Vs Naplam - Fasted real world storage solution

[NapalmV5]

umm yea...you are.
doesnt take away from the fact that your system IS fast. other things do come into play here for uber loads, etc. nice job
however.
from a storage standpoint, yes that is all you are doing. do it with ich10r that fast and prove him wrong if that is your intention.

it is a nice trick though. very fast!

nice trick?? no tricks here

can you guys get anymore ridiculous/idiotic ??

you guys are clueless to what being cached is

cached means i would have to have firefox running when launching 100x/500x ff.. then you could say its "cached" and even then since im using 512mb cache @ 1231 how the foock can it store 2.2gb ?? can you explain that ??

whether i load 500 ff right after bootup or reload 500 ff over and over the diff is ~2 sec (25-27 seconds)

you guys are total ignorants when it comes to systems/storage systems.. unfortunately you talk/act like you know everything when in fact you know kaka

get this straight: youre confusing my vids with pcmark benches

[Computurd]

napalm read post 400. i could not agree with it more. it is perfect.
you are using your cache, big deal bro. no one is saying it is bad or whatever or that your system is slow. take a chill pill. your cache is there to use, and you are using it with conjunction to a fast cpu gpu and ram. you also have a tweaked operating system and know your stuff, apparently.
i just dont understand why you are arguing about a simple fact!

[Ao1]

Napalm always reminds me that you have to think on a system level and not at a subsystem level.

Will I ever open 300 applications in real life? Na, but it was mildly amusing to try it out.

For me it is interesting to see how insanely fast PC's are in a way that has some kind of perspective, which can easily be lost when dealing purely with benchmark figures.....I think what I try to say is that sometimes it is hard to appreciate how fast something is if you only go by benchmarks.

As for the results.......

Napalm can open 300 ff in 15 seconds with 4 ssd's in raid 0 using a 1231.

With a single SSD I can open 300 ff in 29 seconds.

Currently I use the ASUS U3S6, which bottlenecks my SSD performance and has no cache to compensate. Whilst my pc performed quite well in this task it was clear that my single SSD was the bottleneck as the CPU and RAM did not max out.

Later I will try to run a "multi-thread" batch setup and will monitor the QD when I run it.

[Ao1]

try making a "multi-thread" batch setup, make one batch launching 10 other, and then have those 10 launch N instances of the program each, like 30 for a total of 300. Should be interesting to see if the QD skyrockets.

I made 30 separate bat files, each one opened 10 apps of FF followed by the execution of the next batch file of 10 apps etc.

It took the same amount of time to open this configuration as it did to open a single batch file 300 apps.

As for QD. Weird. I used Performance Monitor> Average Disk Queue Length> Scale 100. The Maximum reading I got for the 30 bat files was 0.004. For the single bat file I got 0.003.

Yesterday it seemed to be saying the QD maxed out at 60. Something must be wrong. Maybe the scales got mixed up or maybe the Performance Monitor can't be trusted.

I've kept the bat files so if you know a better way to get a QD measurement I can re-run.

[Computurd]

i think its cause the QD is being served very quickly from the cache. so there isnt any QD..? or is it that the different threads being activated and requesting info separately is load balancing across cores better? curios as to gullars impressions on that one.

@ NAPALMthe reason it doesnt go over your cache amount mr napalm is that it loads the files into cache once, or maybe a few times, but it is reading the SAME file over and over and over.......
might be time to brush up on your storage basics

[NapalmV5]

^ like i said clueless

belittle me all you want you dont belittle me

Napalm always reminds me that you have to think on a system level and not at a subsystem level.

Will I ever open 300 applications in real life? Na, but it was mildly amusing to try it out.

For me it is interesting to see how insanely fast PC's are in a way that has some kind of perspective, which can easily be lost when dealing purely with benchmark figures.....I think what I try to say is that sometimes it is hard to appreciate how fast something is if you only go by benchmarks.

As for the results.......

Napalm can open 300 ff in 15 seconds with 4 ssd's in raid 0 using a 1231.

With a single SSD I can open 300 ff in 29 seconds.

Currently I use the ASUS U3S6, which bottlenecks my SSD performance and has no cache to compensate. Whilst my pc performed quite well in this task it was clear that my single SSD was the bottleneck as the CPU and RAM did not max out.

Later I will try to run a "multi-thread" batch setup and will monitor the QD when I run it.

thanks man

well audienceofone that 14 sec gap @ 300 ff will widen tomorrow when i receive 4 x25e

really looking forward to all the arguments

rhys rhys rhys what you got me into..

[Ao1]

I picked this up from a Win 7 forum. Not so straightforward.

The Avg. Disk Queue Length counter is derived from the product of Avg. Disk sec/Transfer multiplied by Disk Transfers/sec, which is the average response of the device times the I/O rate. Again, this corresponds to a well-known theorem of Queuing Theory called Little's Law, which states:

N = A * Sr

where N is the number of outstanding requests in the system, A is the arrival rate of requests, and Sr is the response time. So the Avg. Disk Queue Length counter is an estimate of the number of outstanding requests to the (Logical or Physical) disk. This includes any requests that are currently in service at the device, plus any requests that are waiting for service. If requests are currently waiting for the device inside the SCSI device driver layer of software below the diskperf filter driver, the Current Disk Queue Length counter will have a value greater than 0. If requests are queued in the hardware, which is usual for SCSI disks and RAID controllers, the Current Disk Queue Length counter will show a value of 0, even though requests are queued.

Since the Avg. Disk Queue Length counter value is a derived value and not a direct measurement, you do need to be careful how you interpret it. Little's Law is a very general result that is often used in the field of computer measurement to derive a third result when the other two values are measured directly. However, Little's Law does require an equilibrium assumption in order for it be valid. The equilibrium assumption is that the arrival rate equals the completion rate over the measurement interval.

Otherwise, the calculation is meaningless. In practice, this means you should ignore the Ave Disk Queue Length counter value for any interval where the Current Disk Queue Length counter is not equal to the value of Current Disk Queue Length for the previous measurement interval.

Suppose, for example, the Avg. Disk Queue Length counter reads 10.3, and the Current Disk Queue Length counter shows four requests in the disk queue at the end of the measurement interval. If the previous value of Current Disk Queue Length was 0, the equilibrium assumption necessary for Little's Law does not hold. Since the number of arrivals is evidently greater than the number of completions during the interval, there is no valid interpretation for the value in the Avg. Disk Queue Length counter, and you should ignore the counter value. However, if both the present measurement of the Current Disk Queue Length counter and the previous value are equal, then it is safe to interpret the Avg. Disk Queue Length counter as the average number of outstanding I/O requests to the disk over the interval, including both requests currently in service and requests queued for service.

You also need to understand the ramifications of having a total disk roundtrip time measurement instead of a simple disk service time measure. Assuming M/M/1, a disk at 50 percent busy has one request waiting on average and disk response time is 2 times service time. This means that at 50 percent busy--assuming M/M/1 holds--an Avg. Disk Queue Length value of 1.00 is expected. That means that any disk with an Avg. Disk Queue Length value greater than 0.70 probably has a substantial amount of queue time associated with it. The exception, of course, is when M/M/1 does not hold, such as during a back-up operation when there is only a single user of the disk. A single user of the disk can drive a disk to nearly 100 percent utilization without a queue!

[Ao1]

well audienceofone that 14 sec gap @ 300 ff will widen tomorrow when i receive 4 x25e .

Bring it on. I can always switch back to ICH9 to give you a better run.

Seriously though, I'm really looking forward to seeing what you can do with 4 x E's.

[Ao1]

@ comp. I picked this up in my search for how to measure QD and thought of you.

http://sqlblogcasts.com/blogs/christ...eue-depth.aspx

[FlawleZ]

Napalm always reminds me that you have to think on a system level and not at a subsystem level.

Will I ever open 300 applications in real life? Na, but it was mildly amusing to try it out.

For me it is interesting to see how insanely fast PC's are in a way that has some kind of perspective, which can easily be lost when dealing purely with benchmark figures.....I think what I try to say is that sometimes it is hard to appreciate how fast something is if you only go by benchmarks.

As for the results.......

Napalm can open 300 ff in 15 seconds with 4 ssd's in raid 0 using a 1231.

With a single SSD I can open 300 ff in 29 seconds.

Currently I use the ASUS U3S6, which bottlenecks my SSD performance and has no cache to compensate. Whilst my pc performed quite well in this task it was clear that my single SSD was the bottleneck as the CPU and RAM did not max out.

Later I will try to run a "multi-thread" batch setup and will monitor the QD when I run it.

Exactly which version of FF are you guys testing? I'm assuming the latest?

Ya know, this has me thinking...I wonder which is the bottleneck for a heavy request over time on my system. You said 300 FF didn't max out your CPU. Not sure which CPU you're using but I'm just on a lowly 720 BE so I'm interested to see if who will be waiting on who. I'll try and run the same batch file tonight for comparison.

[Ao1]

FF3.6.3 (The latest version) I will post the bat files later, but I use W7/64 so the directory goes to program x86. I have a old QE 6850 @ stock and 6GB of (underclocked) OCZ RAM - PC3 12800, that I want to get rid of ASAP.

FF-300.rar - run from wherever.

Multi thread.rar - Copy all files to the C drive and then run "Open me First"

Both bat files will only work with a 64 bit system. For 32 system bit remove (x86) from the code.

[FlawleZ]

FF3.6.3 (The latest version) I will post the bat files later, but I use W7/64 so the directory goes to program x86. I have a old QE 6850 @ stock and 6GB of (underclocked) OCZ RAM - PC3 12800, that I want to get rid of ASAP.

FF-300.rar - run from wherever.

Multi thread.rar - Copy all files to the C drive and then run "Open me First"

Both bat files will only work with a 64 bit system. For 32 system bit remove (x86) from the code.

That's perfect as my batch files have to include the same paths. I too am on W7 64. Will test later tonight!

[GullLars]

I will post the bat files later

Count TerrorByte went to his castle to search for the bat files!

(image from storagesearch.com's article on "the petabyte SSD", looking ahead some years to what may be)

[FlawleZ]

Is there really any point in running the "Multi Thread" bat list? What's Multithreaded about it? Looks exactly the same as a single bat file except at the end of the execution list it calls a secondary list in which it will process linearly as it did the previous for all 30 bat files. In theory it should actually be slightly slower as the disk is then having to access a separate list per every 10 executions the CPU is performing.

[FlawleZ]

Count TerrorByte went to his castle to search for the bat files!

(image from storagesearch.com's article on "the petabyte SSD", looking ahead some years to what may be)

Great article GullLars! Thanks for posting!

In regards to a multithreaded batch file, I think something like this could be used. Or maybe a better deviation of something similar.

01.@ECHO OFF
02.(set drivestring=J K L)&(set childtemplate=thread.bat)&(set inputfile=computers.csv)
03.Set outputfile=.\log.csv
04.Echo.number,machine,ip> %outputfile%
05.:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
06.:: Set variables above.
07.:: Make sure drivestrings reflect executing machine's available drives.
08.:: If common outputfile is used, make sure it matches in %childtemplate%
09.:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
10. 
11.set /P USERID=Enter your domain\userid:
12.set /P PASSWORD=Enter your password:
13.cls
14. 
15.@ECHO OFF
16.MD logs
17.set strlen=0
18.for %%A in (%drivestring%) do SET /A strlen+=1
19. 
20.SET count=0
21. 
22.FOR /F "delims=" %%A IN (%inputfile%) DO (
23. call SET /A count+=1
24. echo.
25. call set comline=%%A
26. call :processmachine %%A
27.)
28.pause
29.goto :EOF
30. 
31.:processmachine
32.  echo Processing #%count%: %comline%
33.  set newletter=
34.  @for %%D in (%drivestring%) do (
35.   @if NOT exist %%D.bat (
36.    @if NOT exist %%D:\nul (call set newletter=%%D)
37.    @if exist %%D:\nul Echo %%D: already connected. Skipping.
38.   )
39.  )
40.  if NOT DEFINED newletter (
41.   nul
42.   goto processmachine
43.  ) ELSE (
44.   @echo new letter = '%newletter%'
45.   type %childtemplate% > %newletter%.bat
46.   START "%count% -- %newletter%.bat" %comspec% /c newletter%.bat %count% %newletter% %comline%
47. )

Taken from:

http://caseelse.net/2008/05/22/multi...-1-an-example/

[NapalmV5]

making their acquaintances

[Anvil]

Looking forward to your tests

Should be a great setup.

[GullLars]

Looking forward to seeing what those goodies will do.
Can we expect a couple of new videos?

[NapalmV5]

Looking forward to your tests

Should be a great setup.

thanks anvil

well so far total cpu limitation @ 4ghz q9650

as far as jmicron vs intel.. @ some points/in some cases the intels are faster and @ some points/in some cases the jmicrons are faster

as far as loading winxp/left4dead/firefox/notepad/calculator both are equally fast.. sometimes the intels are faster by sec @ firefox.. faster grouping/closing (probably where the iops come in handy) but not @ everything

as far as cache the jmicrons prefer the 512mb over 4gb the intels seem to prefer 4gb over 512mb

still testing.. figuring these intels out but i gotta go to 4.5ghz q9650

looks like getting the jmicrons back in nov 2008 wasnt too bad after all

[Anvil]

NapalmV5,

The 10 channel Intel controller is superior to the jmicrons but it probably won't show that much in simple repetitive tasks or at sequential reads.
For reads you could have gotten the Intel G2's or one of the new SandForce based drives, for writes you've got the best available 2.5" SLC drive on the market in the X25-E. (imho)

Anyways, with your setup you are probably bound by CPU/Memory/MB... and not by storage
(if the jmicrons were holding you back you should have noticed just by making the switch to the Intels)

Going for the Intel E series was a wise decision in any case, they'll outlast any of the current alternatives + it's proven and stable technology.
The next E series is supposed to have MLC so the current E series could be the last chance of getting SLC NAND, within a reasonable price range.

I'll try the FF script this weekend on my UD7 using the Vertex LE's and my current X25-V raid setup and post my results, I'm not expecting anything but we just might be in for a surprise

What was your conclusion on W7, I noticed you're still referring to XP

[SteveRo]

Congrats Mr Napalm!
Excellent choice, I have read that the x25-e's go well with the 1231.
Did you say the x25-e array like 4gb cache an that the old array did not?
My experience has been - everything likes more cache - more noticed rather than measured.

[One_Hertz]

thanks anvil

well so far total cpu limitation @ 4ghz q9650

as far as jmicron vs intel.. @ some points/in some cases the intels are faster and @ some points/in some cases the jmicrons are faster

as far as loading winxp/left4dead/firefox/notepad/calculator both are equally fast.. sometimes the intels are faster by sec @ firefox.. faster grouping/closing (probably where the iops come in handy) but not @ everything

as far as cache the jmicrons prefer the 512mb over 4gb the intels seem to prefer 4gb over 512mb

still testing.. figuring these intels out but i gotta go to 4.5ghz q9650

looks like getting the jmicrons back in nov 2008 wasnt too bad after all

It is all CPU limited...

You need to move on from the Q9650. I should get a new setup with an i7 860 today. It wasn't that expensive so I decided to give it a shot to see how much faster loading things would be with a faster cpu. Will take me all frigging evening to insulate everything again, get a good OC going, and make a new nlite image. Expecting 4.5+. I'll post how much faster 498 ff loads.

I'll probably hook up my Es to the ICH10R this time instead of the 9211.

[NapalmV5]

NapalmV5,

The 10 channel Intel controller is superior to the jmicrons but it probably won't show that much in simple repetitive tasks or at sequential reads.
For reads you could have gotten the Intel G2's or one of the new SandForce based drives, for writes you've got the best available 2.5" SLC drive on the market in the X25-E. (imho)

Anyways, with your setup you are probably bound by CPU/Memory/MB... and not by storage
(if the jmicrons were holding you back you should have noticed just by making the switch to the Intels)

Going for the Intel E series was a wise decision in any case, they'll outlast any of the current alternatives + it's proven and stable technology.
The next E series is supposed to have MLC so the current E series could be the last chance of getting SLC NAND, within a reasonable price range.

I'll try the FF script this weekend on my UD7 using the Vertex LE's and my current X25-V raid setup and post my results, I'm not expecting anything but we just might be in for a surprise

What was your conclusion on W7, I noticed you're still referring to XP

lol i understand its hard for you guys to comprehend but these jmicrons are no slouch

i ran benches and thats what they would have me believe about x25e

repetitive.. i know thats why i loaded left4dead @ 4x and now 3x x25e/512mb and 4gb cache and its not one yota faster than the jmicrons

i expected the sheer x25e iops to help out 1231 a bit more but its not the case

from file extractions what these intels are superior @ is the writes performance which the jmicrons are jcraps @

ive been waiting for sandforce slc drives.. but since i needed extra storage now i had no other choice but the x25e and im not dissapointed

once im @ 4.5ghz q9650 will try win7 bootup

[One_Hertz]

repetitive.. i know thats why i loaded left4dead @ 4x and now 3x x25e/512mb and 4gb cache and its not one yota faster than the jmicrons

Ofc it is not. Game loads are low queue reads. All SSDs are the same at that. ACARDs are a lot faster.

Btw in case you haven't seen it, I did some testing of jmicron vs x25-e over a year ago and came to the same conclusion as you:

http://www.xtremesystems.org/forums/...8&postcount=30

[NapalmV5]

Congrats Mr Napalm!
Excellent choice, I have read that the x25-e's go well with the 1231.
Did you say the x25-e array like 4gb cache an that the old array did not?
My experience has been - everything likes more cache - more noticed rather than measured.

thanks steve.. indeed excellent choice.. i got no problem walking all over you acards @ as ssd

ever since i got the jmicrons 512mb was always faster vs 2gb and later on 4gb @ real apps.. if 2gb/4gb faster thats what i wouldve used.. 2gb/4gb only faster @ benches

yeh they go very well together.. 3x x25e/4gb maxes 1231 out and thats why im @ 3x raid 0