hIOmon SSD Performance Monitor - Understanding desktop usage patterns.

[Anvil]

Ao1,

Are you saying that the transfer size is different on reads and writes for access times?

[Ao1]

Ao1,

Are you saying that the transfer size is different on reads and writes for access times?

Looks that way. There is a also huge difference in the total data transferred between reads and writes.

I only ran the access time bench, so the data is related only to that part of the benchmark.

[Anvil]

Very strange if that is the case, I'll give it a try as well.

I started testing AS SSD yesterday, limited to 4K QD1 only and it looked OK, all I/O was 4K both for reads and writes.

[Ao1]

Yes for 4K QD1 it is 4K for reads and writes.

[Anvil]

I checked access time and to me it looks like it's all 512B, there is a lot of VMM related I/O at 32KB (32768bytes)

I've been using Realtime I/O -> Operation Trace so it might look differently.

rtio_optrace.PNG

[Ao1]

Maybe in the world of Virtual Memory Manager

There are a few unrelated entries after the benchmark, but I left them for completeness. This is on a heavily hammed X25-M.

[johnw]

Access times are calculated on a read data transfer size of 32,768, with a total of 196,608 bytes transferred.

Edit: Whoops messed up. 4K write transfer. Read = 491,520 bytes transferred.

That cannot be the case on most systems. It is easy to find AS-SSD benchmarks for X25-M on the web with 0.07 or 0.08ms read access times. If the latency was zero, and all the time was consumed actually reading 32KB of data, then that implies a read rate from flash memory of over 400MB/s for a 32KB block.

[overthere]

I can see how the hIOmon metrics collected by Ao1 during his AS SSD benchmark run (with only the "Access Time" option selected) can appear puzzling.

I will try to briefly explain what hIOmon observed (as shown in the screenshots provided by Ao1 and Anvil within their prior posts above).

I'll start with the "write" I/O operations (since they are more straightforward). BTW, I ran the same "Access Time option only" with version 1.6.4067.34354 of AS SSD earlier today and monitored the run with hIOmon to further confirm these findings:

1. The "writes" begin by the "AS SSD Benchmark.exe" process sequentially writing out a 1GiB file named "test.bin" that resides within a folder called "AS-SSD-TEST42".
   
   To write/create this test file, this AS SSD process issues 64 separate write I/O operations, with each of these write I/O operations having a data transfer length of 16,777,216 bytes.
   
   These numbers all coincide with those shown within the second "write" table in Ao1's screenshot above (more specifically, the first entry within this table). Please note that the hIOmon software considers the first write I/O operation to be neither a random nor a sequential I/O operation; hence the value of "1,056,964,608" shown within the "WriteSeqAccessDataXfer" column (i.e., 16,777,216 multiplied by 63).
   
   
2. The AS SSD process then performs a single write I/O operation (to the "test.bin" file) with a data transfer length of 4,096 bytes, which matches the value shown within the "Write Xfer Max Size" column for the 5th entry within the "write" metrics table.
   
   
3. The AS SSD process then performs 26,214 random write I/O operations (to the same "test.bin" file) with a data transfer length of 512 bytes, which is the same as shown within Anvil's screenshot above. (Please note that the "Write Xfer Min Size" column is not shown, but would have the value of 512).
   
   These 26,214 random write I/O operations are shown within the "Write Random Access IOP Count" column, with the resulting data transfer amount shown in the "WriteRandomAccessDataXfer" column, which is equal to 512 multiplied by 26,214 (which, of course, is 13,421,568).
   
   So the random data transfer total (13,421,568 bytes) combined with the initial 4,096 bytes of data written equals the 13,425,664 shown within the respective "Write Data Xfer" column.

Before I tackle the "read" I/O operations (which probably are much more puzzling) I have a question to ask Ao1:

Are you perchance running Microsoft Security Essentials (MSE) upon your system?

[Ao1]

Hi overthere. Yes I'm using MS Essentials. Sorry for the confusion on this post.

[overthere]

Hi overthere. Yes I'm using MS Essentials.

Hi Ao1,

As I suspected. Basically the read I/O operations (with a data transfer size of 32,768 bytes) that you see within your "read" table of metrics are due to MSE.

That, of course, leaves the big question of: "Where did all of the AS SSD read I/O operations go?"

The answer is: AS SSD performed the read I/O operations to the physical device directly at the physical device level within the OS I/O stack.

That is, none of these read I/O operations were issued by AS SSD as file I/O operations directed to a "test.bin" file - and so these read I/O operations were not observed by hIOmon at either the file level or at the physical volume level within the OS I/O stack (since the hIOmon software was ostensibly configured to only monitor file I/O operations along with I/O operations at the physical volume level, as shown within your tables of metrics).

Now if the hIOmon software had been configured to (also) monitor the physical device level I/O operations, then the hIOmon I/O Monitor would have observed 26,214 read I/O operations (all but one of which were random) with a data transfer size of 4,096 bytes issued to "\Device\Harddisk0\DR0:\<DASD>" (assuming you ran AS SSD against physical drive 0) for a total combined read data transferred amount of 107,372,544 bytes. (There was actually another read I/O operation with a data transfer length of zero - but that's another story ).

BTW, I cheated and configured the hIOmon software to monitor at all three levels within the OS I/O stack concurrently.

In short, you have brought forth a good example that shows the importance (I believe) of being able to optionally monitor I/O operations at the three different levels within the I/O stack (and, better yet, concurrently if and when needed).

This highlights another nuance of which I suspect many folks at not aware. Just as it is possible for I/O operations to only be performed at the "file-level" (e.g., those that are satisfied by the system file cache), it is also possible for applications to perform I/O operations that essentially bypass the file system (as AS SSD has done with read I/O operations in this case).

[Anvil]

The answer is: AS SSD performed the read I/O operations to the physical device directly at the physical device level within the OS I/O stack.
...
BTW, I cheated and configured the hIOmon software to monitor at all three levels within the OS I/O stack concurrently.
...
In short, you have brought forth a good example that shows the importance (I believe) of being able to optionally monitor I/O operations at the three different levels within the I/O stack (and, better yet, concurrently if and when needed).

This is interesting

AS SSD probably plays a few tricks in trying to exclude the diskcache from the results.

[johnw]

Would someone use hIOmon to look at the 4KB random read QD=1 IO for both AS-SSD and CDM (Crystal Disk Mark) to see if there are any significant differences? For some reason the Vertex 3 SSD seems to bench at about 20MB/s on 4KB QD=1 with AS-SSD, but 35MB/s with CDM. Could the two programs be using different I/O levels?

[Ao1]

CDM with MS Essentials running in the background. Just as well as it picked up that CDM installed "Win32/OpenCandy Adware". Will post AS SSD tomorrow.

[felix_w]

CDM with MS Essentials running in the background. Just as well as it picked up that CDM installed "Win32/OpenCandy Adware". Will post AS SSD tomorrow.

Is this your X25-M on that 4K CDM ?

[Ao1]

Yes. Its been a bit hammered lately. 5 test runs with 50MB test file.

[johnw]

CDM with MS Essentials running in the background. Just as well as it picked up that CDM installed "Win32/OpenCandy Adware". Will post AS SSD tomorrow.

Thanks for looking at that. I still don't understand how to interpret a lot of the results.

Overthere mentioned that on the 512B reads that AS-SSD seemed to be bypassing the filesystem level and reading directly on the physical device level. I was wondering if maybe AS-SSD did the same thing on 4KB read, but that CDM might be using a different level, and perhaps some type of Windows disk cache was speeding up the CDM results? But I do not know how to tell if that is the case from the hIOmon results.

[Anvil]

There are a few factors that confuses the process of monitoring most benchmarks like these.

Most of them are creating one or more testfiles and creating and disposing of the testfile is not a part of the benchmark, so isolating these preparations/cleanups is a bit if a drag.

I haven't analyzed CDM benchmarks as such but I've been watching the amount of reads and writes and this is one of the areas where CDM and AS SSD separates.
AS SSD end's up writing a few GBs of data (~5GB iirc) and it does not depend on the speed of the SSD, it reads/writes a fixed amount of data.
CDM does depend on the speed (looks like it depends on a fixed runtime) of the SSD and so benchmarking a single SandForce based SSD easily writes 15GB+ of data.

I'll see if I can fill inn some of the blanks (if any) when Ao1 finishes his testing.

[johnw]

One difference I noticed between AS-SSD and CDM is that AS-SSD runs each write test first, writing out 1GB test files, then it performs the read tests on those 1GB files.

But CDM does the read tests first. So it seems it must be doing, for each test, writes, then reads, then writes again.

[Ao1]

There are a few factors that confuses the process of monitoring most benchmarks like these.......

Hi Anvil,

Indeed. Quite tricky. I'm not going to post a comparison with AS SSD as a direct comparison is not possible due to my drive now being quite badly degraded. Also I can't work out how the write MB/s are calculated for either CDM or AS SSD, so I don't want to confuse anyone.

For reads AS SSD generates a test file with a 262,145 IOP count with 4K transfers giving a speed of 19.67MB/s. AS SSD gives a score of 17.90MB/s.
CDM generates multiple 28,160 IOP counts with 4K transfers giving an average speed of 19.73MB/s, which is close to AS SSD and very close to the MB's that CDM calculated. (The difference between AS SSD and CDM quite possibly being due to further degradation). The sum of the IOP counts against the temp files generated comes in at 169,120. CDM came out with a score of 19.29MB/s.

Whilst reads can be approximated as being similar, write MB/speeds don't seem to line up with anything, so something strange is going on that I can't work out.

[Anvil]

Hi Ao1

I'll see if I can have a look at it at some later time (unless overthere comes up with some smart advice ), right now I've started playing with iometer again.

IOmeter is much easier to monitor, I've created the Fileserver config, it looks a bit dated (filesizes have changed ) but I'll start off using the original one.
I'll create the rest of the basic iometer confings (webserver, database) and then I'll run them on various hardware, need to get some references

IOmeter is one .... of a tool, so is SQLIO, didn't get quite finished with the SQLIO baselines but it's getting there.

I'll post the configs when I'm done and then we could try comparing some of the metrics using hIOmon.

[Ao1]

I want to test my theory on the lack of IOPS utilisation. I'm convinced now that in real life applications the combination of the OS and programming code of applications is what is limiting the utilisation of IOPs capability. During boot for example multiple loading takes place, but the IOPS are low, so I can only think that the loading is done on a priority basis, when with SSD it could potentially deal with multiple concurrent operations.

For sure SSD is not the bottleneck and this comes to the crux of what I wanted to establish in this thread.

This rings true with Intel's "call to arms" to games developers to start utilising the available speed of SSD.

I just need to work out how to capture this....

[overthere]

I might have inadvertently led some folks to think based upon my prior posts #283 and #285 that AS SSD first performs its write testing followed by its read testing when only its "Access Time" option is selected.

My intent was to discuss write I/O operations first in post #283 (since that discussion seemed more straightforward) and then deal with the read I/O operations within the subsequent post #285.

Here is the actual sequence as observed by hIOmon:

Step 1:

26214 physical device random read I/O operations each with a data transfer length of 4096 for "\Device\Harddisk..." for a total data transferred read amount of 107 372 544 bytes

Step 2:

1. 1 single read file I/O operation with a data transfer length of zero for the "AS-SSD-TEST42\test.bin" test file
   
2. 64 write file I/O operations each with a data transfer length of 16 777 216 sequentially to "AS-SSD-TEST42\test.bin" for a total data transferred write amount of 1GiB

NOTE: These file write I/O operations are subject to "split I/O" operations due to file fragmentation; they can also subsequently be further decomposed into smaller write I/O operations with a data transfer size of 1 048 576 bytes (in this case) at the physical device level (with potential queuing implications).

Step 3:

1. 1 single read file I/O operation with a data transfer length of zero for "AS-SSD-TEST42\test.bin"
2. 1 single write file I/O operations with a data transfer length of 4096 for "AS-SSD-TEST42\test.bin"
3. 26214 random write file I/O operations each with data transfer length of 512 bytes to "AS-SSD-TEST42\test.bin" for a total data transferred write amount of 13 421 568 bytes; each of these file write I/O operations results in a corresponding physical device write I/O operation with a data transfer length of 512 bytes. Also note that the combined amount of data written by this and (2) above is 13 425 664 bytes (and in comparison to the 107 372 544 bytes bytes read via the physical device read I/O operations in Step 1).

Finally, several (perhaps subtle) items for consideration (particularly in light of SSD flash management):

1. The read I/O operations in Step 1 are to random locations upon the physical device; presumably these locations (blocks/sectors) may or may not have been written to previously
   
   
2. Action 2 in Step 2 might be considered to be a "sequential pre-conditioning" of the test area for the subsequent write I/O operations (i.e., action 3 in Step 3)
   
   
3. The 512B random write I/O operations performed in action 3 of Step 3 are presumably to stress the flash media management of the SSD (with potential wear-leveling implications), but to a limited amount of the overall area written by action 2 in Step 2

[johnw]

I'm not sure why you posted that about the "access time" test. Just run AS-SSD normally with all tests active. You can SEE that it first updates the write speed, then it updates the read speed for all tests except "Acc. time". Were you just pointing out that the Acc. time test uses a different order than the others?

[overthere]

I'm not sure why you posted that about the "access time" test. Just run AS-SSD normally with all tests active. You can SEE that it first updates the write speed, then it updates the read speed for each test.

I explicitly stated that the metrics I discussed are in regards to only the "Access Time" test within AS SSD because that was what Ao1 limited his testing to (please see his post #277 where he says that "I only ran the access time bench, so the data is related only to that part of the benchmark.").

Consequently, all of my subsequent comments have been limited to only the "Access Time" test context when run alone (and so I have explicitly indicated same so as not to be confused with/when running all of the tests together).

[johnw]

So, then who was your post directed to? I did not see anyone who was confused.