hIOmon SSD Performance Monitor - Understanding desktop usage patterns.

[Ao1]

A comparison with Black Ops.

[Ao1]

I didn't realise you could use hIOmon as a plug in to Window's Perfom.

[overthere]

... I have tried to follow the user guide to get the Raw Device Extended Feature Option to work, but somehow I must be missing a step...

The hIOmon "Raw Device Extended Feature Option" enables you to collect the "Physical Device Extended Metrics", which basically reflect disk I/O operation activity at the "physical volume" or "physical device" levels within the Windows OS I/O stack for those disks that are partitioned/formatted (e.g., have a filesystem upon them).

Moreover, these metrics automatically correlate file I/O operations with their associated physical volume and/or physical device I/O operations. That is, they essentially reflect those physical volume/device I/O operations that were performed so as to satisfy file I/O operations which were also monitored by the hIOmon I/O Monitor at the associated file level (i.e., they basically reflect “physical volume/device” I/O operations that were required to complete file I/O operations for those files that were also being monitored by the hIOmon I/O Monitor).

Values present within the various hIOmon "PhyDev" metric types indicate that the hIOmon software was successfully configured to include the collection of the "Physical Device Extended Metrics".

The various hIOmon "automated configuration scripts" (such as the hIOmon "Device and File I/O Performance Analysis Add-On" configuration script) provide the option to enable the collection of the "Physical Device Extended Metrics".

So looking at your hIOmon export files and hIOmon Presentation Client screenshots, it appears that you have been successfully collecting these metrics (presumably based upon your use of the various hIOmon "Add-On" configuration scripts).

[Ao1]

[EDIT - I re-ran this to capture all three levels at the same time.]

Here I capture Black Ops SP at the Device level, Volume level and File Level .

To explain some of the figures:

Device Level Reads
• Data transferred = ~406MB (Incurred over 30 minutes of game play).
• Total I/O's = 16,216
• Max IOPs = 22.7.
• FastIOPS 97.4 % (less than one millisecond).
• Min xfer size = 1,024. Average xfer size = 26,317. Max xfer size = 262,144.
• Min response time = 0.0416ms. Avg response time = 0.2179ms. Max response time = 10.4449ms
• Maximum data transfer rate for a single I/O operation. = 268.59MB/s. (If done in less than one millisecond MB/s is technically "inflated").

File/ Logical Disk Level Reads
• Data transferred = 2.35GB
• Total I/O's = 193,016
• Max IOPs = 152.9
• FastIOPS 95.3 % (less than one millisecond).
• Min xfer size = 0. Average xfer size = 13,662. Max xfer size = 262,144.
• Min response time = 0.00416ms. Avg response time = 0.0584ms. Max response time = 10.5207ms
• Maximum data transfer rate for a single I/O operation. = 2,801.833MB/s.

[Ao1]

Here I capture Black Ops MP at the Device Level, Volume Level and File Level . All at the same time

The same map was reloaded around 5 times ~20 minutes. AV enabled.

I only started monitoring AFTER the fist map had loaded. Read xfers related to the first map load at the device level are therefore excluded, but available at the Logical Device Level for subsequent I/O's. (Assuming they were held in cache when the monitoring started).

Any read xfers (at the device level) are subsequently down to dropped cache or reads generated during game play. (Also any other background reads from the OS/ AV etc).

I hope I got that right. It seems to stack up with the highlights below.

Device Level Reads
• Data transferred = ~180MB
• Total I/O's = 1,645
• Max IOPs = 2.7
• FastIOPS 76.8 % (less than one millisecond).
• Min xfer size = 512. Average xfer size = 116,522. Max xfer size = 9,433,088. (~9MB)
• Min response time = 0.0451ms. Avg response time = 0.9008ms. Max response time = 50.154ms
• Maximum data transfer rate for a single I/O operation. = 268.315MB/s.

Logical Disk Reads
• Data transferred = ~3.7GB
• Total I/O's = 839,171
• Max IOPs = 1,502.8
• FastIOPS 97.4 % (less than one millisecond).
• Min xfer size = 2. Average xfer size = 4,973. Max xfer size = 9,957,376.
• Min response time = 0.0034ms. Avg response time = 0.0109ms. Max response time = 50.3104ms
• Maximum data transfer rate for a single I/O operation. = 2,898.44MB/s.

[Computurd]

is there any way to monitor then give us a readout of the file sizes used, at the physical device level only, like say a percentage of each file size used during normal desktop
like a percetage of 4k 8k 16k 64k etc etc???

[johnw]

is there any way to monitor then give us a readout of the file sizes used, at the physical device level only, like say a percentage of each file size used during normal desktop
like a percetage of 4k 8k 16k 64k etc etc???

+1

I was about to ask the same thing. If hIOmon cannot produce a histogram, at least a tabular list or Excel file with breakdowns of physical device I/O frequencies for each block size (number of contiguous LBAs read or written)

[Ao1]

That is top of my next to do list

[Computurd]

thanks A01!

[overthere]

is there any way to monitor then give us a readout of the file sizes used, at the physical device level only, like say a percentage of each file size used during normal desktop
like a percetage of 4k 8k 16k 64k etc etc???

+1

I was about to ask the same thing. If hIOmon cannot produce a histogram, at least a tabular list or Excel file with breakdowns of physical device I/O frequencies for each block size (number of contiguous LBAs read or written)

Hi CT,

I/O operations performed at the physical device level within the OS I/O stack are fairly ignorant of the associated file (if any) and respective file size.

For instance, take a single physical device read I/O operation that has a data transfer length of 4096 bytes (i.e., eight 512-byte sectors). This physical device level read I/O operation could be the result of a read I/O operation for a file whose length ("file size") is one byte, or 511 bytes, or 4096 bytes, or 1MB, etc. - you get the story.

Moreover, the file could be 10MB in size, but the application only accesses (perhaps repeatedly) the first 64 KiB.

Anyway, I suspect that you are interested in capturing a breakdown of the "data transfer length" frequencies (as johnw subsequently mentioned) for I/O operations down at the physical device level within the OS I/O stack.

This is something that the hIOmon software can also handle. I won't go into all of the details here, but one approach is to have the hIOmon software collect an "I/O operation trace" of physical device I/O operations, from which a hIOmon "Transfer Size Summary (TSS)" export file can be generated.

This hIOmon TSS export file is a CSV-file that can include a variety of metrics pertaining to each distinct "data transfer length" observed.

I anticipate that Ao1 with have some empirical data along these lines soon.

[Ao1]

I've got some business to take care of today, but when I get back I will produce some graphs to highlight the outputs from the log files. The log files attached cover metrics from 3 different scenarios.

• Black OPS Single Player - Without AV - (RH-BlackOpsSP-TS)
• Black Ops Multi Player - Without AV - (RH-BlackOpsMP-TS)
• OS with just IE, Word, Excel & Live Mail - with AV - (RH-OS-TS)

Be prepared for a few surprises.

Sincere thanks to overthere for taking the time to help me collect this data.

[Ao1]

Here is summary of device level xfer sizes measured against IOP count occurrences as captured when monitoring OS, Word, Excel, Live Mail (with AV real time on). The OS was fully loaded before monitoring began.

Read xfers

31,299 read I/O operations; total 674,738,176 bytes read
. Largest single I/O xfer size = 1.14MiB (1 occurrence)
. Smallest single I/O xfer size = 512 bytes (303 occurrences).

Write xfers

8,645 write I/O operations; total 98,129,920 bytes written
. Largest single I/O xfer size = 2MiB (2 occurrences)
. Smallest single I/O xfer size = 512 bytes (352 occurrences).

Highest read IOP Count xfer size = 32kB (10,471 occurrences).
Highest write IOP Count xfer size = 4kB (6,134 occurrences).

I know from a mistake I made in post #281 that my AV generates max read xfers of 32,768 bytes so I'd guess a good percentage of 32kB reads are AV related.

More charts to follow.

[johnw]

Interesting data. 4KB and multiples thereof are by far the most frequent access sizes, but there are still a relatively small number of accesses below 4KB in size, and was discussed earlier in this thread. For SSDs that are slowed down a lot by <4KB accesses (most of them, I think, except Intel X25-M), I wonder if having a small number of these <4KB IOs in parallel with the 4KB IOs significantly hurts the performance of the 4KB IOs.

[Ao1]

Hi johnw.
I’m going to do a lot more work on summarising the data in those output files. (Random vs sequential, read IOP max time against different size xfers, read time totals etc.) I’m also going to improve the summary in post #337. I will also re-run the OS session to confirm if AV was the culprit for the 32kB xfers. Time is against me at the moment, but I will try to take care of it as soon as possible.

For me this data provides a way to hone in on what happens in real life. I don’t think any of the synthetic benchmarks provide statistics that reflect the way that SSD’s are actually used in desktop scenarios and to this end I would like, at some stage, to compare stats from other SSD brands to quantify any differences in performance.

[Anvil]

Great info Ao1

I'll do a few small tests on my workpc using one of the other utilities I mentioned, don't know what to expect as it's mostly VM related.

edit:

I've monitored during an windows update (8 updates) on my laptop and here is the result.

It's Top 10 from Writes and the same for Reads

diskmon_top10_rw_windows_update.PNG

4KB is by far the most common request-length on both reads and writes in this particular case.
If one looks at writes it is the 1MB (1048576) transfers that really generates bytes.

There were 37.890 events recorded in 320 seconds, jfyi.

[Ao1]

[Corrected - made a mistake with read IOP/ xfers ]

Here is summary of device level xfer sizes measured against IOP count occurrences as captured when monitoring Black Ops Single Player. (Progressive map loading).

Read xfers

92,623 read I/O operations; total 1,772,378,112 bytes read
. Largest single I/O xfer size = 28.24 MiB (1 occurrence)
. Smallest single I/O xfer size = 512 bytes (305 occurrences).

Write xfers

6,697 write I/O operations; total 113,272,832 bytes written
. Largest single I/O xfer size = 2.75MiB (2 occurrences)
. Smallest single I/O xfer size = 512 bytes (245 occurrences).

Highest read IOP Count xfer size = 4kB (39,023 occurrences).
Highest write IOP Count xfer size = 4kB (5,018 occurrences).

[Ao1]

[Dang, same mistake, now corrected]

Here is summary of device level xfer sizes measured against IOP count occurrences as captured when monitoring Black Ops Multiple Player. (Reloading the same map).

Read xfers

2,648 read I/O operations; total 318,201,856 bytes read
. Largest single I/O xfer size = 0.88 MiB (1 occurrence)
. Smallest single I/O xfer size = 512 bytes (12 occurrences).

Write xfers

6,299 write I/O operations; total 55,623,168 bytes written
. Largest single I/O xfer size = 0.88 MiB (1 occurrences)
. Smallest single I/O xfer size = 512 bytes (167 occurrences).

Highest read IOP Count xfer size = 72kB (1,010 occurrences).
Highest write IOP Count xfer size = 4kB (4,027 occurrences).

[Ao1]

Here I look at the highest IOP count xfer size that occurred for Black Ops Single and Multi Player.

The highest read IOP count xfer size for Black Ops SP was 4kB, with random reads accounting for 142.26 MiB and sequential reads accounting for 10.17MiB. – Total 4kB reads = 152.43MiB.

The total data transferred from all xfers monitored equated to 1, 690.27 MiB, so the total amount of 4kB reads xfers account for ~9% of xfers.

The largest single I/O read xfer was 28.24 MiB. The second largest single I/O read xfer was 8.99 MiB.

Approx 40% of overall xfers are random.

So...........with an avg qd of 1 and a high percentage of small xfers it seems that access time and small xfer transfer performance is key. Now I can see why I only get around ~100MB/s overall.

[Ao1]

It would be great if someone could produce a “benchmark” program using the exact I/0 xfers from each of the device level summaries, that wrote directly to the device. That would provide an easy way to measure real life performance differences between different storage solutions.

[One_Hertz]

142MB of random 4kb reads at QD1 would take the X25-M (and almost all arrays) around 7 seconds to do... That's a lot of disk busy time. 1.5seconds with iodrive.

[Ao1]

Indeed, the iodrive should really shine

I've asked Alex from AS SSD if he would consider making a benchmark based on the exact I/O activity from each of those files. Somehow I doubt if he will do it. I don't know how much work it would require, but for sure it would be a great benchmark that could easily quantify tangible real life performance differences between different storage solutions.

[One_Hertz]

I really need to get going with this software so i can compare my x25-e array to my iodrive.

[Eggcake]

I'm not sure whether you are interested, but I made some traces with Windows Performance Analysis Tools. I traced the boot of Win7 x64. It's a relatively fresh (< 2months) install. I already classified the QDs / file sizes - of course it's also possible to to a histogram. The lower bounds of the classes are exclusive, the upper bounds inclusive (so if you look for 4kB values, they are in 512B-4kB).





Hope you find it at least a bit interesting

[One_Hertz]

Did a quick comparison of my 4x x25-e R0 array versus iodrive. D is iodrive. I launched dragon age 2, loaded a save game, quit, launched css, started video stress test, quit. Rebooted between testing each device of course.

Interesting how most metrics are the same, but iodrive has faster average response time and faster max data transfer rates...

Almost forgot to mention that these results are device level.

[Computurd]

wow interesting fusion v x25e...what controller for the R0? or was that ICH?