View Full Version : hIOmon SSD Performance Monitor - Understanding desktop usage patterns.
Thread Summary
The objective of this thread was to understand how a SSD performed under typical desktop applications. HDD's are perceived as being the bottleneck in the system, so it is logical to assume that a faster storage system would have the potential to significantly improve overall system performance.
Enter the SSD.
Is storage still the bottleneck? What types of demand are put on the SSD and what are the performance characteristics of the SSD that matter? Finally how much of those key performance characteristics are being utilised?
The tools
hIOmon provides a sophisticated platform that can observe the following:
1. Observe I/O operations in action at three different critical points within the OS I/O stack: the file system level (essentially the application/process level), the physical volume level, and the physical device level. This can help provide for a more complete, overall picture of what actually occurs during I/O operation processing.
2. Selectively observe I/O operations at one or more of these three levels.
This can help identify those particular I/O operations that are, for instance, germane to only a single level, e.g., I/O operations that are satisfied directly by the use of the system file cache and thus are effectively limited to the file system level only, I/O operations that are issued by applications/programs directly to a device at the physical device level and without direct interaction with the file system level, etc.
3. Optionally observe I/O operations concurrently at two or more levels, and moreover correlating I/O operations as they traverse the different levels.
To help anyone interested in doing their own monitoring a fully functional trial can be downloaded here (http://www.hyperio.com/) and a quick set up guide to get up and running quickly can be found in post #187 (http://www.xtremesystems.org/forums/showpost.php?p=4633431&postcount=187)
The set up (Unless otherwise stated).
Asus Rampage Extreme. ICH9. QX6850. 8GB RAM. Asus Xonar DX2.4870x2. 160GB X25-M x2 and 1 160GB Seagate Barracuda 7200
Summary of activities monitored
The operating system was Win 7/64. An image was made to compare performance between different storage configurations within the same system to try and provide comparable results. Typical applications included games, office, web browsing, WMP, Power DVD, Traktor and Photoshop.
OS I/O Stack
An example of I/O activity at each of the three levels that hIOmon can monitor can be found in Post # 329 (http://www.xtremesystems.org/forums/showpost.php?p=4790529&postcount=329) and Post # 330 (http://www.xtremesystems.org/forums/showpost.php?p=4791798&postcount=330)
A summary of key device level and logical disk I/O activity monitored during game play with Black Ops SP in post #329 is provided below:
Device Level Reads
Data transferred = ~406MB (Incurred over 30 minutes of game play).
Total I/O's = 16,216
Max IOPs = 22.7.
FastIOPS 97.4 % (less than one millisecond).
Min xfer size = 1,024. Average xfer size = 26,317. Max xfer size = 262,144.
Min response time = 0.0416ms. Avg response time = 0.2179ms. Max response time = 10.4449ms
Maximum data transfer rate for a single I/O operation. = 268.59MB/s. (If done in less than one millisecond MB/s is technically "inflated").
File/ Logical Disk Level Reads
Data transferred = 2.35GB
Total I/O's = 193,016
Max IOPs = 152.9
FastIOPS 95.3 % (less than one millisecond).
Min xfer size = 0. Average xfer size = 13,662. Max xfer size = 262,144.
Min response time = 0.00416ms. Avg response time = 0.0584ms. Max response time = 10.5207ms
Maximum data transfer rate for a single I/O operation. = 2,801.833MB/s.
Device Level I/O Transfers.
I/O xfers at the device level post #336 (http://www.xtremesystems.org/forums/showpost.php?p=4794815&postcount=336)
Metrics from 3 different scenarios were captured.
Black OPS Single Player - Without AV
Black Ops Multi Player - Without AV
OS with just IE, Word, Excel & Live Mail - with AV
OS with just IE, Word, Excel & Live Mail (http://www.xtremesystems.org/forums/showpost.php?p=4795359&postcount=337)
Read xfers
31,299 read I/O operations; total 674,738,176 bytes read
. Largest single I/O xfer size = 1.14MiB (1 occurrence)
. Smallest single I/O xfer size = 512 bytes (303 occurrences).
Write xfers
8,645 write I/O operations; total 98,129,920 bytes written
. Largest single I/O xfer size = 2MiB (2 occurrences)
. Smallest single I/O xfer size = 512 bytes (352 occurrences).
Highest read IOP Count xfer size = 32kB* (10,471 occurrences).
Highest write IOP Count xfer size = 4kB (6,134 occurrences).
*The high occurrences of 32kB xfers are possibly due to AV activity.
Black Ops Single Player (http://www.xtremesystems.org/forums/showpost.php?p=4795887&postcount=341)
Read xfers
92,623 read I/O operations; total 1,772,378,112 bytes read
. Largest single I/O xfer size = 2.75 MiB (1 occurrence)
. Smallest single I/O xfer size = 512 bytes (305 occurrences).
Write xfers
6,697 write I/O operations; total 113,272,832 bytes written
. Largest single I/O xfer size = 2.75MiB (2 occurrences)
. Smallest single I/O xfer size = 512 bytes (245 occurrences).
Highest read IOP Count xfer size = 4kB (39,023 occurrences).
Highest write IOP Count xfer size = 4kB (5,018 occurrences).
Black Ops Multi Player (http://www.xtremesystems.org/forums/showpost.php?p=4795929&postcount=342)
Read xfers
2,648 read I/O operations; total 318,201,856 bytes read
. Largest single I/O xfer size = 0.88 MiB (1 occurrence)
. Smallest single I/O xfer size = 512 bytes (12 occurrences).
Write xfers
6,299 write I/O operations; total 55,623,168 bytes written
. Largest single I/O xfer size = 0.88 MiB (1 occurrences)
. Smallest single I/O xfer size = 512 bytes (167 occurrences).
Highest read IOP Count xfer size = 72kB (1,010 occurrences).
Highest write IOP Count xfer size = 4kB (4,027 occurrences).
Boot up.
The bootup process involves three stages: (Please see post # 255 (http://www.xtremesystems.org/forums/showpost.php?p=4753021&postcount=255) for more details)
1. Hardware enumeration to enable the OS loader to take control
2. Main Boot Path Time - Essential services and drivers to load desktop
3. Post boot time - Drivers, processes and applications that arent critical for user interaction and can be loaded with low-priority I/O that always gives preference to user-initiated actions that execute using Normal I/O priority.
By default the hIOmon software is configured to automatically begin monitoring when the services are started during stage three. There is also an option, however, whereby the hIOmon software can be configured so that it instead begins monitoring very early within the "Main Boot Path Time" (stage 2). Please see post #32 (http://www.xtremesystems.org/forums/showpost.php?p=4595909&postcount=32)for details
The amount of data loading during the boot process can vary significantly depending on the type of applications that have been installed. The time it takes to fully load can also vary significantly depending on user-initiated actions, automatic updates and AV activity.
During boot all the data has to initially be read from the physical disk.
Typical read transfer sizes during boot can be found in post #190 (http://www.xtremesystems.org/forums/showpost.php?p=4638329&postcount=190). The vast amount of read transfer sizes are 6,000 bytes and below. Only 16 read transfer sizes were above 1MB and the largest read transfer sizes was 11MB.
The boot process was monitored on three different storage systems. HDD, SSD & SSD Raid 0. (Please see post # 105 (http://www.xtremesystems.org/forums/showpost.php?p=4606722&postcount=105)).
Key storage load metrics (Approximated)
95% random reads and writes.
20,000 I/O operations overall.
Overall average IOPs = 190
Overall average transfer size 20,000 bytes (19KB)
Total read data transferred 420MB
Total write data transferred 24MB
Key storage performance metrics
HDD Percentage of fast read/ write IOPs performed = 1.2%
SSD Percentage of fast read/ write IOPs performed = 98.3%
SSD Raid 0 Percentage of fast read/ write IOPs performed = 98.9%
HDD busy time = 1min 58s
SSD busy time = 8.54s
SSD Raid 0 time = 6.59s
HDD average response time = 53.40ms
SSD average response time = 0.49ms
SSD Raid 0 average response time = 0.33ms
Whilst the performance difference between HDD & SSD is significant the difference between a single SSD and SSD Raid 0 is marginal.
Anvils post # 118 (http://www.xtremesystems.org/forums/showpost.php?p=4609989&postcount=118) provides a comparison with no apps installed.
i7 920 UD7, 2R0 C300 64GB
Key storage load metrics
88.7% random reads and writes.
12,251 I/O operations overall.
Overall average transfer size 17,546 bytes
Total read data transferred 190MB
Total write data transferred 14MB
Key storage performance metrics
Percentage of fast read/ write IOPs performed 96.4%
Busy time 2.62s
Average response time 0.32ms
AMD 1090T - Samsung F3 HDD
Key storage load metrics
78.9% random reads and writes.
7,179 I/O operations overall.
Overall average transfer size 17,267 bytes
Total read data transferred 98MB
Total write data transferred 20MB
Key storage performance metrics
Percentage of fast read/ write IOPs performed 61.3%
Busy time 12.57s
Average response time 4.47ms
Whilst the total read data transferred differed significantly between the different storage systems that were monitored the random read percentages were all comparable. The overall average transfer sizes were also comparable.
Anvils Samsung F3 was lot faster than the HDD I monitored (Seagate Barracuda). New installs on HDD always seem quite fast at first, but they soon slow down as more data gets loaded over time and the drive becomes fragmented. ;)
The impact of file cache.
After boot up frequently used data can be held in memory-resident file cache to speed up access that would otherwise be dependent on a read form the physical device. Access to files that are resident in memory is significantly faster than having to retrieve the files from the physical disk. This process helps mask the poor random small file access performance associated with HDD.
The impact of file cache can be found in post #223 (http://www.xtremesystems.org/forums/showpost.php?p=4685410&postcount=223) and post #224 (http://www.xtremesystems.org/forums/showpost.php?p=4685475&postcount=224). In post #218 (http://www.xtremesystems.org/forums/showpost.php?p=4683798&postcount=218) it is evident that none of the small file transfers occurred on the physical disk.
An observable impact of cache with a large file transfer can be found in post #222 (http://www.xtremesystems.org/forums/showpost.php?p=4684495&postcount=222). Better clarification of the observations can be found in post #223 (http://www.xtremesystems.org/forums/showpost.php?p=4685410&postcount=223) & post #234. (http://www.xtremesystems.org/forums/showpost.php?p=4691543&postcount=234)
For information on the impact of file cache with typical use a comparison of the top 10 read and write data transfers between those monitored on the physical device and those monitored overall can be found in post # 241 (http://www.xtremesystems.org/forums/showpost.php?p=4706339&postcount=241)(reads) and post # 242 (http://www.xtremesystems.org/forums/showpost.php?p=4706373&postcount=242)(writes).
System top read data transfer size = 24,877 occurrences of 40 bytes
Physical device top read data transfer size = 1,079 occurrences of 4,096 bytes
System top write data transfer size = 7,228 occurrences of 152 bytes
Physical device top write data transfer size = 450 occurrences of 512 bytes
For both reads and writes that occur on the physical device 0.5KB and 4KB file transfers dominate.
The percentage difference between overall writes and reads can be found in post # 244 (http://www.xtremesystems.org/forums/showpost.php?p=4706415&postcount=244)
Overall system reads = 93%
Physical device reads = 88%
The difference between total data transferred between overall data transfer and data transferred on the physical device can be found in post # 243. (http://www.xtremesystems.org/forums/showpost.php?p=4706400&postcount=243)
Gaming Performance
Call Of Duty Black Ops - Single Player
Please see post #159 (http://www.xtremesystems.org/forums/showpost.php?p=4619800&postcount=159) for graphs showing the data summarised below.
Maximum random sequential read speed = 97.92MB/s
Maximum random read speed = 26.33MB/s
Maximum random access IOP count = 13,750
Maximum Sequential Access IOP Count = 3,275
Fast Read IOP count = 98%
Average read queue depth = 1
Maximum read queue depth = 5
Call of Duty Black Ops Multi Player
Please see post #165 (http://www.xtremesystems.org/forums/showpost.php?p=4623489&postcount=165)for graphs showing the data summarised below.
Maximum sequential read speed = 121.11MB/s
Maximum random read speed = 10.63MB/s
Maximum random access IOP count = 11,140
Maximum Sequential Access IOP Count = 10,379
Fast Read IOP count = 98%
Average read queue depth = 1
Maximum read queue depth = 4
For a comparison of key performance metrics between SSD & HDD whilst playing Black ops please see post #48 (http://www.xtremesystems.org/forums/showpost.php?p=4597699&postcount=48)
In this post (http://www.xtremesystems.org/forums/showpost.php?p=4778400&postcount=57)the impact of cached files is observed by reloading the same map in multiplayer mode. The first load incurred ~60% of the total reads. The map had to be reloaded a further 9 times to get the 40% balance of reads.
Windows was able to satisfy a high percentage of I/O's specifically related to the Black Ops folder in cache, regardless of the storage medium
SSD without FancyCache = 86%
HDD without FancyCache = 85%
HDD was much slower at loading the first level but after that it was hard to tell the difference between SSD due the amount of I/O being served in cache.
Page File
The purpose of a page file is described in Wiki (http://en.wikipedia.org/wiki/Paging) as:
"In computer operating systems, paging is one of the memory-management [I]schemes by which a computer can store and retrieve data from secondary storage for use in main memory. In the paging memory-management scheme, the operating system retrieves data from secondary storage in same-size blocks called pages. The main advantage of paging is that it allows the physical address space of a process to be noncontiguous. Before the time paging was used, systems had to fit whole programs into storage contiguously, which caused various storage and fragmentation problems.
Paging is an important part of virtual memory implementation in most contemporary general-purpose operating systems, allowing them to use disk storage for data that does not fit into physical random-access memory (RAM)".
According to a MSDN blog (http://blogs.msdn.com/b/e7/archive/2009/05/05/support-and-q-a-for-solid-state-drives-and.aspx)
...."most pagefile operations are small random reads or larger sequential writes, both of which are types of operations that SSDs handle well.
In looking at telemetry data from thousands of traces and focusing on pagefile reads and writes, we find that
Pagefile.sys reads outnumber pagefile.sys writes by about 40 to 1,
Pagefile.sys read sizes are typically quite small, with 67% less than or equal to 4 KB, and 88% less than 16 KB.
Pagefile.sys writes are relatively large, with 62% greater than or equal to 128 KB and 45% being exactly 1 MB in size.
In fact, given typical pagefile reference patterns and the favorable performance characteristics SSDs have on those patterns, there are few files better than the pagefile to place on an SSD."
In post #160 (http://www.xtremesystems.org/forums/showpost.php?p=4621167&postcount=160)page file was activity monitored. The system had 8GB of RAM.
Key transfer metrics
Total read data transferred = 90.20MB
Total write data transferred = 280.81MB
Max read data transfer = 37MB
Max write data transfer = 118MB
Photoshop generated large writes, which did not appear to be read back. If the large writes generated by Photoshop are excluded the page file activity was representative of the telemetry data collected by MS.
For a comparison between boot up with the page file enabled and disabled please see post #92 (http://www.xtremesystems.org/forums/showpost.php?p=4604628&postcount=92). With the page file disabled response times improved along with the percentage of fast IOPs counts.
Sequential speeds & multitasking
Sequential reads of typical apps can be found in post # 237 (http://www.xtremesystems.org/forums/showpost.php?p=4697501&postcount=237)
Play DVD = 12.67 MB/s
Rip DVD = 2.76 MB/s
Quick MS Essential AV scan = 30.71 MB/s
Zip a file = 6.34 MB/s
Black Op Single Player = 96.30 MB/s
Create HD video = 15.68 MB/s
Copy 600mb AVI file = 252.70 MB/s
Copy 1.3GB folder with various sized jpegs = 251.19 MB/s
An attempt to run multiple tasks to achieve the max sequential read speed of the SSD resulted in the CPU maxing out. (Please see post # 5 (http://www.xtremesystems.org/forums/showpost.php?p=4591883&postcount=5))
For a comparison between multi tasking sequential read speeds obtained between SSD and HDD please see post #47 (http://www.xtremesystems.org/forums/showpost.php?p=4597547&postcount=47)
HDD Black Ops = 67.37MB/s
SSD Black Ops = 81.70MB/s
HDD Photoshop = 14.09MB/s
SSD Photoshop = 28.31MB/s
HDD MS Essentials = 11.36MB/s
SSD MS Essentials = 29.16MB/s
HDD PowerDVD = 11.90 MB/s
SSD PowerDVD = 16.26MB/s
Queue Depths
Queue depths under different loads are summarised in post # 8 (http://www.xtremesystems.org/forums/showpost.php?p=4592025&postcount=8)
Avg Read QD heavy multi tasking - 1.057
Max Read QD heavy multi tasking - 4
Avg Read QD light multi tasking - 1.024
Max Read QD light multi tasking - 35
Black Ops single player read = 1.087
Black Ops single player write = 2.015
A summary of 2 days worth of data can be found in post # 24 (http://www.xtremesystems.org/forums/showpost.php?p=4594083&postcount=24)
Max average QD = 1.418
Max read QD = 18
Max average write QD = 2.431
Max write QD = 70
Intriguingly read QD's were generally lower on HDD when compared to SSD. Conversely write QDs were generally higher on HDD. Please see post # 47 (http://www.xtremesystems.org/forums/showpost.php?p=4597547&postcount=47)
Significantly higher queue depths were only observable when benchmarks were monitored.
IOPs Work in progress
IOPs can be approximated using the following formula:
Queue depth*(1/latency in ms) = IOPs
MB/s can be approximated using the following formula:
BytesPerSec = IOPS * Transfer size in bytes
IOP capability is therfore linked to three key variables:
Transfer size
Queue depth
Latency
Average queue read depths, as observed above, are typically just over one. Most of the transfer sizes are 4,096 bytes or less. If these parameters are taken as a given the impact of latency becomes critical to IOP performance.
Based on the average latency of 0.01ms
Queue depth 1 x (1/0.00001) = 100,000 IOPS
Based on the average latency of 0.1ms
Queue depth 1 x (1/0.0001) = 10,000 IOPS
Based on the average latency of 0.2ms
Queue depth 1 x (1/0.0002) = 5,000 IOPS
Based on the average latency of 0.3ms
Queue depth 1 x (1/0.0003) = 3,333 IOPS
Based on the average latency of 0.49ms
Queue depth 1 x (1/0.00049) = 2,040 IOPS
Overall average IOPs rates observed (Please see post #94 (http://www.xtremesystems.org/forums/showpost.php?p=4605540&postcount=94))
Boot up = 190 IOPs
Multi tasking = 70 IOPs
PCMark HDD Suite (http://www.xtremesystems.org/forums/showpost.php?p=4626211&postcount=178)= 62 IOPs
VMWare (http://www.xtremesystems.org/forums/showpost.php?p=4615688&postcount=131) = 130 IOPs
ATTO (http://www.xtremesystems.org/forums/showpost.php?p=4684460&postcount=221) = 3,552 IOPS
It seems that IOP capability is significantly underutilised, which is a shame as it is perhaps the most significant performance advancement in SSD development.
According to AS SSD Benchmark:
X25-M 4k read random speed at QD 1 = 21MB/s
X25-M 4K random read IOPs at QD1 = 5,322 IOPs
X25-M read access time = 0.077ms
X25-M 4k write random speed at QD 1 = 45.17MB/s
X25-M 4k write IOPS at QD1 = 11,563 IOPs
X25-M write access time = 0.097ms
X25-M sequential reads = 251MB/s
X25-M Sequential writes - 103MB/s
Raptor 4K read random speed at QD 1 = 0.76MB/s
Raptor 4K random read IOPs at QD1 = 195 IOPs
Raptor read access time = 8.408ms
Raptor 4K random write speed at QD 1 = 1.48MB/s
Raptor 4K random write IOPS at QD1 = 378 IOPs
Raptor write access time = 2.677ms
Raptor sequential reads = 77.54MB/s
Raptor Sequential writes - 56.19MB/s
OCZ Core 4K read random speed at QD 1 = 7.07 MB/s
OCZ Core 4K random read IOPs at QD1 = 2,293 IOPs (Estimated)
OCZ Core access time = 0.436ms
OCZ Core 4K random write speed at QD 1 = 1.88 MB/s
OCZ Core 4K random write IOPS at QD1 = 400 IOPs (Estimated)
OCZ Core access time = 2.496ms
OCZ Core sequential reads = 208.44 MB/s
OCZ Core sequential writes - 120.44 MB/s
If Queue depth*(1/latency in ms) = IOPs
OCZ Core - Queue depth 1 x (1/0.002496) = Write 400 IOPs
Raptor - Queue depth 1 x (1/0.002677) = Write 373 IOPs
X25-M - Queue depth 1 x (1/0.000097) = Write 10,309 IOPs
400 4K write IOPs is more than what was observed for typical use yet the OCZ Core would appear to stutter under light load. :shrug:
Native Command Queuing (NCQ)
Intel made a post to describe the benefit of NCQ (http://communities.intel.com/message/104055#104055).
"AHCI is a hardware mechanism that allows software to communicate with SATA drives. To make that transaction smoother, SATA devices were initially designed to handle legacy ATA commands so they could look and act like PATA devices. That is why many motherboards have legacy or IDE modes for SATA devices in that case users are not required to provide additional drivers during OS installation. However, Windows 7 ships with AHCI drivers built in, so soon this mode will no longer be necessary.
But this begs the question: what features does AHCI mode enable? The answer isn't simple, but one of the bigger advantages is NCQ, or native command queuing.
NCQ is a technology that allows hard drives to internally optimize the order of the commands they receive in order to increase their performance. In an SSD everything is different. There is no need to optimize the command queue, but the result of enabling NCQ is the same there is a performance increase. In brief, NCQ in an Intel SSD enables concurrency in the drive so that up to 32 commands can be executed in parallel.
Also take into consideration that the speed of the processor and the RAM also the amount of it will affect the performance of the Solid State Drive."
With NCQ enabled benchmarks show a significant increases in 4k reads/ writes @QD 32.
The impact NCQ on large file transfers is monitored in post #317. (http://www.xtremesystems.org/forums/showpost.php?p=4783626&postcount=317) Here a performance increase can also be seen.
TRIM
Information on TRIM metrics that hIOmon can observe can be found here (http://www.hyperio.com/hIOmon/AddOns/Gadgets/hIOmonSSDTrimMetricsDisplayGadgetHelp.htm)
An example of the TRIM related IO activity can be found in post # 148 (http://www.xtremesystems.org/forums/showpost.php?p=4616928&postcount=148). An explanation of what is being observed can be found in post # 149 (http://www.xtremesystems.org/forums/showpost.php?p=4617012&postcount=149).
In post # 185 (http://www.xtremesystems.org/forums/showpost.php?p=4626691&postcount=185)Anvil was able to observe TRIM related IO activity and establish that TRIM was working in a software Raid 0 configuration.
In post # 131 (http://www.xtremesystems.org/forums/showpost.php?p=4615688&postcount=131)Anvil was able to observe a big increase in the maximum response time due to a TRIM operation. A further explanation on this observation can be found in post # 134 (http://www.xtremesystems.org/forums/showpost.php?p=4616543&postcount=134).
A separate thread on verifying TRIM functionality can be found here. (http://www.xtremesystems.org/forums/showthread.php?p=4662143#post4662143)
I have tried to provide accurate and relevant data in this summary. Please let me know if you see any mistakes or omissions.
A special thanks to overthere for all his help and assistance and for a great piece of software. :up: Thanks also to Anvil for providing comparative data and all other posters that have helped along the way.
Tiltevros
10-18-2010, 04:53 AM
thnx Ao1
Computurd
10-18-2010, 05:47 PM
yes that is an interesting idea. however, i think a big thing to take into consideration here is when you have the high QD situations that we run into.
everyone says we never go over QD of 4 with normal usage. not true. you go over it all the time. you never go over a average of 4QD is a more accurate way of saying it. the high QD spikes are resolved so quickly, that it is not showing up in the average.
in high QD situations that last only seconds, the system will lag or hang. the drives with higher random performance at the hgiher QD will not lag at those points.
I am game, lets test this!
so lets talk methodology here...how are we going to go about it? what will be the baseline?
The 1st thing I learn is that TRIM commands are constantly being executed with an Intel SSD.
http://img543.imageshack.us/img543/7297/88717477.png
Now I will look at QD etc.
The "Solid State Disk (SSD) I/O Performance Analysis Add-On" is quite easy to install if you follow this guide.
http://www.hyperio.com/hIOmon/ScreenShots/hIOmonSSDIOstatsSupport.htm#hIOmonCfgSetup
With the default time period the CVS file will be updated every 10 minutes and a summary of storage activity will be provided.
So on to some monitoring:
Here I am running 75 processes and the CPU is maxed out.
http://img179.imageshack.us/img179/3928/heavymt.png
http://img100.imageshack.us/img100/5376/26245856.png
Here is light use. Around 68 processes running but most are background;nothing heavy, just WMP, office, IE. PS MS messeneger etc.
http://img227.imageshack.us/img227/5532/correctedlu.png
This is a fresh script run
Here is what happens when I run AS SSD Benchmark.
http://img203.imageshack.us/img203/473/82663644.png
To compare more easily
Heavy use (CPU maxed out)
http://img178.imageshack.us/img178/8373/amend.png
Light use (This is a fresh script run)
http://img227.imageshack.us/img227/5532/correctedlu.png
AS SSD Benchmark (This is cummulative to the heavy use script).
http://img203.imageshack.us/img203/473/82663644.png
MW2 Multi Player (Fresh script with only MW2 runing).
http://img440.imageshack.us/img440/7052/mw2ha.png
MW2 Single Player (Fresh script with only MW2 runing).
http://img405.imageshack.us/img405/3879/mw2sp.png
Here I copy the Adobe folder from Program Files to the desktop: 2502 Items in the directory with a combined size of 710MB
http://img841.imageshack.us/img841/3373/fileadobe.png
Here I copy an AVI file from desktop to the C drive. File size 635MB
http://img26.imageshack.us/img26/7766/17293796.png
Here I ran a winsat disk command.
http://img163.imageshack.us/img163/8521/winsat.png
Some observations
If you are multitasking with apps that require processing power you are unlikely to hit maximum sequential speeds because the CPU will most likely max out.
It would seem that the maximum QD length gets quite high at times, but unless you run something like AS SSD, which tests at up to QD64, it seems very hard to get above an average QD of 2 (Just as you said Comp) :)
overthere
10-19-2010, 09:29 AM
I believe that there was an earlier question about the "Read Max Queue Length" and the "Write Max Queue Length" metrics (more specifically, that they appear to stay the same no matter what activity is going on).
First a disclaimer: I work for hyperI/O (the company that has designed/developed the hIOmon software utility package), but I am here only to help answer technical questions if that is of interest to folks.
Now about the two metrics. To keep this post short, a couple of things to consider:
1) The metrics that are being collected/exported are cumulative metrics (as per the particular configuration performed via the use of the hIOmon "SSD I/O Performance Analysis Add-On" script).
That is, the metrics reflect the accumulated values as seen by the hIOmon software since it first began monitoring the specified I/O operations (more technically, since the hIOmon "Filter Selection" configuration file, which specifies what is to be monitored and what is to be collected, was loaded/activated). The configuration and activation of the "Filter Selection" file was automatically performed by the use of the "SSD I/O Performance Analysis Add-On" script.
So then, the "Read Max Queue Length" metric reflects the highest read queue length see by the hIOmon software since it began monitoring the requested device(s). Moreover, this value will not change until the hIOmon software observes a higher value (or until the/a Filter Selection is reloaded/reactivated).
2) The "Read Average Queue Length" reflects the overall average number of read I/O operations as observed by the hIOmon software that were "outstanding" (i.e., issued but not yet completed) for the respective device. Again, this is an overall average since the hIOmon Filter Selection was loaded/activated.
3) The various metrics collected/exported by the hIOmon software are described/defined within the "hIOmon User Guide" document (see "Table 5" within the document for starters).
4) If you are using the "SSD I/O Performance Analysis Add-On" configuration script to compare various scenarios with each other, then you will need to re-run this script before each scenario. This will cause the hIOmon Filter Selection to be reload/reactivated.
Hopefully the above was of help.
^ :welcome:
Thanks for coming to xtreme to help out and thanks for a great piece of software. :up: I'm still trying to get to grips with all the features but I have a feeling some questions will follow.
EDIT:
OK first question.:p: What is being monitored on the TRIM command? Is it recording when the OS sends the command signal or is it when the SSD actually deals with the command?
2nd question. The "Write Max Queue Length" seems quite high even for light use. Is that because the small writes are being cached so that they are written more effectivly to the SSD to reduce wear?
overthere
10-19-2010, 11:19 AM
^ :welcome:
Thanks for coming to xtreme to help out and thanks for a great piece of software. :up: I'm still trying to get to grips with all the features but I have a feeling some questions will follow.
EDIT:
OK first question.:p: What is being monitored on the TRIM command? Is it recording when the OS sends the command signal or is it when the SSD actually deals with the command?
Thanks for the welcome to xtreme and for the kind comment about the hIOmon software.
The hIOmon software does have many features, which are meant to address the wide variety of questions and scenarios that can arise when probing deeper (i.e., "peeling the onion") becomes necessary as an investigation/analysis unfolds.
Anyway, regarding your first question, the short answer is that the hIOmon software can capture TRIM commands from the OS perspective (i.e, the former in your second question above). This is in line with a primary design goal of the hIOmon software: to capture a comprehensive set of I/O operation and performance metrics from the perspective of applications/processes and the OS (in contrast to the device perspective, i.e., as seen by the actual device "at the other end of the cable").
More specifically to your question, the hIOmon software can be configured to capture I/O operation metrics for "control" I/O operations (specifically "Device Control" I/O operations) that specify "Manage Data Set Attributes (MDSA)" requests. These MDSA requests include "TRIM" action requests.
These MDSA "TRIM" action requests are generally issued by the file system related components of the operating system. The hIOmon software can capture the issuance of "TRIM commands" at either the physical volume and/or physical device level within the Windows operating system (the latter is similar to the "PhysicalDisk" level shown within the Windows Performance/System Monitor) regardless of the source.
Additional details about the TRIM command support provided by the hIOmon software can be viewed within the "Background Information" section here:
http://www.hyperIO.com/hIOmon/AddOns/Gadgets/hIOmonSSDTrimMetricsDisplayGadgetHelp.htm
OK thanks for that. So my 4th post should read:
The 1st thing I learn is that TRIM commands are constantly being sent by the OS. ;)
I'm going to do a lot more reading now so I can understand the full potential of this software.
Computurd
10-19-2010, 04:18 PM
well i have been doing some testing with it, getting it figured out, i am mainly using the real time I/O functions, but i do have one big question:
certain apps will have high QD like outlook will get a qd of 12, and Windows Media Player has had QD as high as 90 as the "Read Max Queue Depth" when using the I/O summary.
also, Kaspersky seems to have high max QD values.
My question is this though, is that just the QD of that individual process, or its QD in relation to the system as a whole?
for instance, the queue depth for outlook at the time stamp of 19:13:51 is 2,
but also at that time of 19:13:51 the QD of windows media player is 12
so at that same time, the QD should be 14?
so in essence, the QD is measured per process? how would one go about seeing the cumulative queue depth for all processes, programs, etc?
overthere
10-19-2010, 04:19 PM
2nd question. The "Write Max Queue Length" seems quite high even for light use. Is that because the small writes are being cached so that they are written more effectivly to the SSD to reduce wear?
Unfortunately, there is not a single short, simple answer to your question above.
As I believe you will come to find out, there can be a number of factors that can contribute to incurring a "high" maximum queue length even during ostensibly "light" use.
One approach is to isolate what actually is occurring concurrently when the max queue length is observed down at the physical device level.
The basic idea here is that you can utilize the various configuration options (and "Add-Ons") available with the hIOmon software to further "hone-in" on particular factor(s) that are contributing to high max queue lengths.
For instance, you could use the hIOmon "Process I/O Performance Analysis Add-On" (which is included within the standard hIOmon software installation package) to help identify those specific processes/applications that are experiencing the greatest amount of I/O activity (perhaps with their own respective max queue lengths) that requires actual accesses to the "physical device" when the max queue length is detected by the hIOmon software for the physical device.
I hope that I don't come across as avoiding your question, but the underlying inherent complexity (along with the many subtle interactions amongst concurrent processes and I/O operations, moreover along the overall I/O stack within the operating system, particularly at specific points, i.e., at the "file" level, at the "physical device" level, etc.) can require a more detailed probing of what exactly is going on within the particular system.
And not to belabor the point, but this is where the ability to capture relevant empirical metrics can be quite helpful (e.g., to substantiate or refute speculation and theory).
Lastly, some other questions that you might want to consider in this regard:
http://www.hyperIO.com/hIOmon/hIOmon5StoragePerfQuestions.htm
And a final note: The discussion of the intricacies of the "system file cache" management provided by the Windows operating system is a prolonged one.
In any case, one approach that you could take towards answering your question is to see how much system file cache write activity is actually occurring when under "light" use as shown by the hIOmon "System File Cache" I/O metrics.
Computurd
10-19-2010, 04:38 PM
I did come to the conclusion that the QD are 'per process' by loading iometer, and setting a 4k random file to read at 64QD, while also loading other programs, i ran a test :)
the other programs only showed their own respective QD values in the monitoring. the dynamo.exe (iometer) however showed its own individual qd of 64 , of course.
The main thrust of our testing here is to find out the value of SSD in raid, and its impact upon the actual use of an moderate to heavy user.
There are two basic schools of thought here, the first being that an individual SSD is fast enough that it can handle even heavy use identically to a SSD raid array.
Or, even as an extension of that, that a SSD with a lower random performance at high QD random will perform essentially the same as some of the newer, faster SSD that have much faster high QD performance.
Tools used to measure the QD of a system that we have explored using usually rotate around average QD.
however, even using the windows built in performance monitoring counters you can see there are big QD spikes in heavy usage, but that they are resolved quicker with faster devices/arrays.
since these spikes are resolved quickly, the average QD does little to quantify how high we are hitting.
but the real question arises that we know the average desktop performance rarely goes above 4-6, but I am trying to see how high it spikes, and how fast those spikes are resolved, to compare a single SSD v. an SSD array that can do 160,000 IOPS @ 4k random.
is there a way to show the maximum QD for the individual device?
EDIT: yeah now i see it. :)
overthere
10-19-2010, 05:29 PM
is there a way to show the maximum QD for the individual device?
The hIOmon software can be configured to collect "summary" metrics upon an individual file, process, and/or device basis:
http://www.hyperIO.com/hIOmon/hIOmonSummarizedMetrics.htm
If you are collecting summary metrics upon an individual process basis, then indeed the maximum QD metric within the "summary metrics" for the process will reflect the maximum seen for the respective process.
Similarly, if you are collecting summary metrics upon an individual file basis, then the maximum QD metric within the summary metrics for a particular file reflects the max seen for the respective file.
And of course, the same applies for the summary metrics collected upon an individual device basis. In the case of summary metrics for a specific "physical device" (i.e., at the "physical disk" level within the operating system), the max QD is that seen for/at the respective device (regardless of the "source", i.e., whichever process initiated the I/O operation).
So essentially the summary metrics for a file, process, or device reflect those metrics that pertain just to that file, process, or device.
As noted above, this is all configurable and moreover allows you to observe what is occurring for a particular file, process, or device upon an individual file, process, and device basis respectively.
Oops, just saw your edit. Hopefully the above provides further confirmation.
Also, you raise a very good point about distinguishing between average and maximum QD. :up:
Computurd
10-19-2010, 05:38 PM
wow this tool is very functional! As i am exploring more of it i am seeing that we can do some cool stuff here....
Basically i loaded two I7 920 systems, one having a single Vertex LE (SF based) SSD and the other with my array of 5 vertex (gen 1) SSD on a areca 1880IX-12 w/4gb cache, and began monitoring with a C:\* filter.
Then i turned on programs on each computer one by one...pretty much one after the other....
five instances of firefox, Windows Media player (playing), then acronis image home (did NOT do an image though, just let it sit on) my passsword manager, my sound manager, and a camera feed was turned on on both rigs.
I also have a host of exact same desktop gadgets running on both computers.
Then once i had all this running i turned on the game FarCry2 simultaneously (or very close to :) ) on both rigs, and let the intro run, where you have to ride around in a jeep. however from previous testing i know that during this sequence the program is streaming from the disk pretty heavily.
then i shut down the programs in reverse order (first on one, then on the other), and stopped logging
Vertex LE system
http://i517.photobucket.com/albums/u332/paullie1/QDTesting.png
and then Areca array system
http://i517.photobucket.com/albums/u332/paullie1/QDTesting2.png
Now, i know this is hardly conclusive data here, but it does show a taste of array v single SSD performance of sorts. The array handles the high QD situations much more effectively of course.
Now, to figure out how to record, then replay on a seperate computer, a trace, and monitor it. if that is possible of course....
overthere
10-19-2010, 05:59 PM
Now, to figure out how to record, then replay on a seperate computer, a trace, and monitor it. if that is possible of course....
Not exactly sure what you would like to do as mentioned above.
Are you interested in capturing an "I/O trace" on one machine and then displaying the same "I/O trace" upon another machine?
Or do you want to capture an "I/O trace" on one machine, then replay the same "I/O trace" upon another machine (i.e., use the I/O trace as "input" and actually perform the same I/O operations in the same sequence upon another machine)?
Or am I completely confused?
Computurd
10-19-2010, 06:04 PM
capture an "I/O trace" on one machine, then replay the same "I/O trace" upon another machine (i.e., use the I/O trace as "input" and actually perform the same I/O operations in the same sequence upon another machine)?
yup. but that may be unnecessary. not sure if that could be done with this. but setting up some sort of batch file to initiate programs in the same sequence (possibly with some time intervals between) could be just as effective. the main goal is to be able to compare the two systems' I/O performance under identical load patterns.
overthere
10-19-2010, 06:59 PM
yup. but that may be unnecessary. not sure if that could be done with this. but setting up some sort of batch file to initiate programs in the same sequence (possibly with some time intervals between) could be just as effective. the main goal is to be able to compare the two systems' I/O performance under identical load patterns.
I must admit that I generally try to avoid the use of the "I/O Trace" option.
Please don't get me wrong, there can indeed be situations where using the "I/O Trace" option is definitely needed to capture the particular information required.
But from the outset, the hIOmon software has stressed the use of the "summary" option, which can answer so many questions in a much simpler, easier, quicker, more productive and efficient, etc. manner (e.g., rather than having to deal with tens of thousands if not hundreds of thousands of individual I/O operation trace records - there can often be some troublesome "scaling" issues, for instance, with the I/O trace approach).
Anyway, for perhaps the more "adventurous" folks, the hIOmon software does provide the hIOmon Add-On Support for the "Intel NAS Performance Toolkit (NASPT)".
Basically, this support enables you to capture an I/O operation trace, which can then be exported by the hIOmon software to a disk file within the XML trace input file format that is required by the Intel NASPT tools. In turn, this export file can then be used directly (without conversion) as a NASPT trace input file for both the NASPT Analyzer and the NASPT Exerciser tools (which essentially provide a "replay" option along the lines that you mentioned).
Additional information about this hIOmon Add-On can be found within the "hIOmon Add-On User Guide" document.
But back to the "batch file" approach that you also mentioned. The hIOmon software also includes a "Benchmarking Support Add-On" - surprise, surprise :)
This Add-On helps in efforts to automate the benchmarking process. That is, it can be used to quickly and easily utilize the hIOmon software so as to retrieve real-time, user-specified "summary metrics" and then dump them (i.e., export them) to a CSV file. This is done through the use of several batch files that are included with this Add-On.
More info here:
http://www.hyperIO.com/hIOmon/AddOns/hIOmonBenchmarkingAddOnSupport.htm
Computurd
10-19-2010, 07:11 PM
well awesome :) now to figure this out!
Computurd
10-19-2010, 07:38 PM
well now delving into some more complicated aspects of this i see...
elevated cmd prompt to admin status
then :
http://i517.photobucket.com/albums/u332/paullie1/what2.png
http://i517.photobucket.com/albums/u332/paullie1/what.png
hmmm doing this CLI might not be the easiest way..wondering on how to do this via GUI?
i have gotten it to save the .csv and it has a ton of useful data.
EDIT: the benchmarking set does not actually perform a benchmark? it just records data if i am understanding this correctly. so hows about we use it in conjunction with everyones fave??
YuP! some PCMark Vantage action is inbound fellas. lets see what a trace summary of PCMV looks like :)
btw overthere we are big time PCMV junkies in here..:)
@steve-ro..been quiet on the PCMV front, thought the aussies would come gunning for ya for sure steve-o...seems the dust has settled on the hall of fame for the time being...
overthere
10-19-2010, 08:39 PM
;)
well now delving into some more complicated aspects of this i see...
elevated cmd prompt to admin status then :
It might be better to troubleshoot this "Access Denied" issue offline so that we don't encumber this thread with stuff that folks might consider off-topic.
In any case, I suspect that there was more than one open hIOmon client attempting to concurrently change the operation/configuration of the hIOmon software. Could you please take a quick look into the Application Event Log and see if there is an EVENT ID 785, Source: hIOmonClientComm entry?
I just ran the hIOmonBenchmarkExport batch file on a Windows 7 x64 system (both from an admin and a guest account) without any problems, so I'm a bit puzzled.
Thanks
Also, you are correct: the hIOmon "Benchmarking Support Add-On" does not actually perform/drive any benchmarking itself. That is left up to the user's particular tastes/desires ;)
Rather the primary intent of the Add-On is to help "clear" the summary metrics before the actual benchmark run and then dump the collected summary metrics during and/or the benchmark run as part of an automated process.
Here is a summary of a couple of days worth of cumulative data monitoring statistics with normal use.
• Max Av QD Reads = 1.653
• Max Av QD Writes = 2.431
• Max Av QD Reads & Writes = 1.838
• Max Read QD = 18
• Max Write QD = 70
Now I will try the more advanced features to see what causes the peak QD's.
I'm also going to swap back to HDD (when I get time) to see how QD's compare to SSD.
http://img837.imageshack.us/img837/6514/16643157.png
Hi overthere,
I have a few questions and apologies if they are a bit dumb.
Read Data Xfer = the amount of data transferred?
Read Xfer Max size = the maximum block size transfer?
Here I ran a winsat disk command. The QD's went through the roof!
http://img163.imageshack.us/img163/8521/winsat.png
Here I copy the Adobe folder from Program Files to the desktop:
2502 Items in the directory with a combined size of 710MB
http://img841.imageshack.us/img841/3373/fileadobe.png
Here I copy an AVI file from desktop to the C drive.
File size 635MB
http://img26.imageshack.us/img26/7766/17293796.png
overthere
10-22-2010, 08:19 AM
Hi overthere,
I have a few questions and apologies if they are a bit dumb.
Read Data Xfer = the amount of data transferred?
Read Xfer Max size = the maximum block size transfer?
You are on the right track. :up: The values as exported by the hIOmon software are:
Read Data Xfer = The accumulated total (in bytes) of the data transferred by the read I/O operations as observed by the hIOmon I/O Monitor.
Read Xfer Max Size = The largest data transfer size (in bytes) associated with an observed read I/O operation.
A couple of other further explanations of the metrics shown that might be of interest to folks following this thread:
1) The "FastIOPcounts" (e.g., the "ReadFastIOPcount"); these counts (for either read or write I/O operations) reflect the accumulated number of I/O operations respectively that were successfully completed in less than one millisecond as observed by the hIOmon I/O Monitor.
2) The "Read Time Total" and the "Write Time Total" reflect the accumulated total (in seconds) of the time durations associated with the respective I/O operations.
Consequently, you can derive the "average" response time by dividing the "time total" by the respective "IOP Count". So, for example, a "Read Time Total" of 11.9739501 seconds for an associated "Read IOP Count" of 37322 indicates an average Read I/O operation response time of .3208 milliseconds.
3) Similarly, the "Read IOP Max Time" and the "Write I/O Max Time" reflect the maximum time duration (in seconds) for a read or write I/O operation respectively as observed by the hIOmon I/O Monitor. So a "Read IOP Max Time" of 0.1502034 reflects a maximum read I/O operation response time of 150.2034 milliseconds.
Finally, there are many more I/O operation metric types that can be collected/exported by the hIOmon software.
For example, there is the "ReadDataXferRateMax" metric, which reflects the maximum data transfer rate, i.e. the maximum amount (in bytes) of data transferred by I/O operations of the respective type (i.e., either read or write) as detected by the hIOmon I/O Monitor during a one-second interval.
The various hIOmon clients can be used to add additional metrics to an export file. The hIOmon Add-On scripts can also be modified with a simple change so as to include additional metric types for export.
Thanks for the clarification. :)
So, outside of the monitoring results that included benchmarks the vast majority of read/ write IOPS are concluded in less than one millisecond. :cool:
When I copied a single 635MB file the max largest transfer size of 1.02MB is what occurred with a single read I/O operation? Now I'm struggling. The 635MB file is split into small "chunks" each with its own I/O operation? The larger the file the larger the "chunk" per I/O and the faster the write? Why are my Read/ Write Xfer Max Size typically comming in at around 1MB? Is that the optimum size for an I/O operation? (Sorry for being dumb again).
overthere
10-22-2010, 10:39 AM
When I copied a single 635MB file the max largest transfer size of 1.02MB is what occurred with a single read I/O operation? Now I'm struggling. The 635MB file is split into small "chunks" each with its own I/O operation? The larger the file the larger the "chunk" per I/O and the faster the write? Why are my Read/ Write Xfer Max Size typically comming in at around 1MB? Is that the optimum size for an I/O operation? (Sorry for being dumb again).
There is not necessarily a one-to-one correspondence between a given file I/O operation and any actual I/O operation to the disk drive itself.
As simple examples:
1) A single file I/O operation can incur two or more I/O operations to the disk drive (e.g., file fragmentation). BTW, the hIOmon software can capture I/O operation metrics that help empirically identify the performance impact of file fragmentation:
http://www.hyperIO.com/hIOmon/ScreenShots/hIOmonScreenshotsFileIOsummarySplitIOWMIbrowser.ht m
2) Two or more file I/O operations can be satified by a single device I/O operation. This is largely the result of the "system file cache" usage.
For example, some data is read from the device into the system file cache (which is maintained by the operating system) and multiple file I/O operations are subsequently satisfied from the system file cache (rather than directly from the device).
See, for instance, the second screenshot on the following web page:
http://www.hyperIO.com/hIOmon/hIOmon5StoragePerfQuestions.htm
Overall, the interactions between the I/O operation activity at the "file level" within the I/O stack within the operating system and the lower physical volume and physical device levels within the I/O stack can be complex.
This is a major reason why the hIOmon software provides the ability to capture/collect I/O operation performance metrics at all three levels (and, moreover, optionally do so concurrently so that you get a much clearer picture of what exactly is going on).
But back to your questions, there are a number of factors that can be involved in determining the particular types of I/O operations that will actually result from a "simple" copy operation (e.g., the nature of the copy program itself, buffering nuances, system resources available/used at the time of the copying, any other concurrently running processes, etc.).
So unfortunately there are no simple answers to your questions, but indeed the copying of a "large" file will likely entail multiple write I/O operations to the actual disk device.
Boy storage is complicated stuff. The more I think I've learnt the more I realise I don't know. :shakes:
HIOmon is really helping me though and I really appreciate your insights.
I've noticed that the ReadSystemCacheIOPcount and ReadSystemCacheIOPmissCount (same with writes) are always 0. I think the only exception was when I ran the winsat disk benchmark.
What does that tell me?
EDIT:
Also is there a way to monitor what happens when the PC boots?
overthere
10-22-2010, 11:59 AM
Boy storage is complicated stuff. The more I think I've learnt the more I realise I don't know. :shakes:
HIOmon is really helping me though and I really appreciate your insights.
I've noticed that the ReadSystemCacheIOPcount and ReadSystemCacheIOPmissCount (same with writes) are always 0. I think the only exception was when I ran the winsat disk benchmark.
What does that tell me?
I certainly agree: storage in general is inherently pretty complex. :)
The wide range of summary metrics made available by the hIOmon software provides an explicit reflection of this complexity in large measure.
Of course, the complexity involved in actually capturing/collecting/exporting/etc. these summary metrics is yet another matter! ;)
Anyway, regarding the ReadSystemCacheIOPcount and the ReadSystemCacheIOPmissCount metrics, these metrics reflect read I/O operations that, as observed by the hIOmon I/O Monitor, were directed towards a file and which explicitly requested the use of the system file cache. The same applies to the write variants of these metrics as regards write I/O operations.
As such, they only pertain to file-level I/O operations. Accordingly, these metrics are not applicable to I/O operations that are observed by the hIOmon I/O Monitor at either the physical volume or the physical device level within the operating system (and so, of course, their value will be zero in these cases).
overthere
10-22-2010, 12:19 PM
Also is there a way to monitor what happens when the PC boots?
The hIOmon software does provide an option to collect I/O trace and/or summary metrics during the system boot/start process.
This can get a bit tricky given, for instance, the volume of I/O operation activity that generally occurs during the system boot/start - along with the mechanics required to collect the requested metrics while the system itself is not fully operational yet.
If you would like to pursue this, please see the "Boot Logging" section (starting on page 68) and section "7.5 Boot Logging" within the "hIOmon User Guide" document.
Here I use the WMI browser to give me a live feed of the top 10 processes, whilst running all the typical programs that I use, namely: Power DVD, WMP, Photoshop, Modern Warfare 2 SP, Tracktor, Office, IE9, Messenger & MS Essentials (Quick scan).
Bandwidth
Interestingly MW2 SP only loaded at a max read transfer rate of 81.78MB/s
If I add up all the top ten maximum max read transfer rates I get a total of 327.12MB/s, so if I wanted to run all of the them at the same time I would need a bandwidth of 327.12MB/s, although that would not be physically possible to do. If I took out MW2 SP I would need a bandwidth of 245.35MB/s, which is possible with the X25-M, but again it would still be physically impossible to run them all at the same time. (Without running a batch file).
The fastest write speed I obtained was 42MB/s using Photoshop. If I add up all the top ten maximum max write transfer rates I get a total of 98.88MB/s, which is within the specs of an X25-M, but again impossible to implement together at the same time.
IOPS
The IOPS constantly change when being monitored so between selecting specific filters it keeps increasing, however it is clear to see that virtually 100% of IOPS are done in less than one millisecond. :cool:
Queue depths
The maximum QD's for writes surprised me but the averages (not shown below) are always below 2. The same with reads. I ran more proceses togther than what I would do normally to try and push the QD up.
Now I see exactly why a SSD raid 0 array was not any faster for my particular usage patterns in comparison to a single SSD. I didn't need the bandwidth or an increase in IOPS.
http://img109.imageshack.us/img109/8601/resultsf.png
Computurd
10-23-2010, 11:04 AM
those QD are for each individual process separately, correct? so you have some different processes, for instance the wmplayer, with 3. it has its own QD of 3, but that doesnt include the QD of other system programs at the same time. they each have their own QD. That QD is only its own, not the cumulative of the system as a whole.
you should configure it to read the QD of a entire drive, so that you get a cumulative number, and not numbers that are for each process separately. what is the entire QD for the drive? there are tons of processes/prgrams that are active in the background while you are running these individual programs. what about those? and how does that affect overall performance?
if you use the same setup as here with top ten monitoring you might get different results :)
as you see it is logging the entire C:\ drive, you would use the "top ten devices" instead of "top ten processes" to get this view.
http://i517.photobucket.com/albums/u332/paullie1/QDTesting.png
also, when you are looking at the reads of each of these programs separately, it does little to tell you the read activity of the entire disk. there are other file accesses going on that are independent of each respective program...a program for instance is going to cause OS activity, etc.
you arent quantifying the 'collateral' access that is going on during your use, and that is just as important, or even more important, than the individual program.
you list the five processes that you put in bold on the Data Transfered_MB/s_Maximum Read graphic, but what of the other five above those? and what of the myriad of other things going on in the background that arent being tracked ?
testing only the singular, as opposed to the whole :)
Anvil
10-23-2010, 01:08 PM
Looks like an interesting tool :)
Haven't had the time to do any testing lately :(
Computurd
10-23-2010, 01:14 PM
yea currently on vacation myself :)
will be back to do some more testing in about 9 days :)
Anvil
10-23-2010, 01:31 PM
Hi CT :)
Enjoy your vacation :)
I'll be back in a week or two, a lot of things to try :) when I get back.
BTW, my first SSD (Vertex 250GB) bricked Friday a week ago, no data was lost though. (it died literally while I was doing the weekly backup)
I'm the 1st to admit....my storage use is not xtreme, but I wanted to monitor how I typically use the PC. If anything I had too many apps open to what I would normally run.
those QD are for each individual process separately, correct? so you have some different processes, for instance the wmplayer, with 3. it has its own QD of 3, but that doesnt include the QD of other system programs at the same time. they each have their own QD. That QD is only its own, not the cumulative of the system as a whole.
Max and AVG QD's are what I have more extensively monitored in previous posts, but here is the summary of the total QD's
http://img696.imageshack.us/img696/76/qdmax.png
.....also, when you are looking at the reads of each of these programs separately, it does little to tell you the read activity of the entire disk. there are other file accesses going on that are independent of each respective program...a program for instance is going to cause OS activity, etc.
That is true. I'm only capturing the 10 ten processes, so there where many other processes running in the background. Totals can be seen in previous posts, which were obtained over a much longer period.
you arent quantifying the 'collateral' access that is going on during your use, and that is just as important, or even more important, than the individual program.
This set of monitoring was more to try and understand individual process demands. More monitoring is to follow ;) Next up the boot log. :)
you list the five processes that you put in bold on the Data Transfered_MB/s_Maximum Read graphic, but what of the other five above those? and what of the myriad of other things going on in the background that aren't being tracked ?
Again this is only monitoring the top 10 processes, but I was surprised by how much the OS takes up. Some of those processes I don't recognise. I think MsMpEng.exe is MS Essentials. I ran a quick scan whilst I was monitoring. I have no idea what svchost.exe was doing, but I typically have quite a few of those running at any one time if I look in the task manager. There were also dozens of tiny traces from silly things related to my Asus D2X Audio Centre and Logitech apps. I did monitor all individual program files but the excel sheet was to detailed to post here.
Hi Anvil :wave:
Computurd
10-23-2010, 01:42 PM
@Anvil -ouch sorry to hear that :( lucky so far with my gen 1's, other than the three i lost when the 1500w psu blew up....still waiting for word from SilverStone as to whether they will replace the drives or not. they sent the psu and defective cable back to taiwan to confirm that was the root cause...
by no data lost, you mean you were able to get everything off of it? did it just cease to write?
(sorry for derail but i gotta know :) )
@A01-very nice work, does take time!
I would be curious to see the max read of all processes combined, even though i am sure the number will not be too fantastic, it is the nature of the accesses that will be important of course.
250 mb/s of sequential reads is a whole different animal than a 250 mb/s mixed read/write with varying levels of random and sequential, as the real accesses are.
of course it is an inherent weakness of all solutions, both HDD and SSD that they will not handle the mixed patterns very well. of course, that is the vast majority of real usage accesses!
that is another aspect that i find that arrays handle very easily, the mixed read/write accesses that are killer to single devices. of course, throwing on cache with a hardware raid card changes the ballgame entirely for the mixed read/write performance! makes life easy :)
You can see a summary of my highest totals in post #24
I don't know if I will do a boot trace next or if I should swap over to a HDD and run the same apps to see what happens. I somehow doubt that all the IOPS will be done in less than one millisecond. ;) Swapping to HDD is going to be a pain, but I'll do it on a system image so everything is the same apart from the SSD. Ehr.... loads of work, but I think the result will be interesting.
By the way its much easier to use hIOmon after you have installed Java. When will I learn to read the manual first. :rofl:
Anvil
10-23-2010, 02:02 PM
@CT
It just died within minuttes having performed the backup.
I'm not able to read or write to the drive (not recognized by any utility/mb), a pity as it was a great "all purpose" SSD.
edit
Hi Ao1 :)
I'll try to keep up with your findings, comparing SSD vs HDD using this tool would be interesting.
I've been using an HDD as a temporary replacement for the Vertex for a few days but it's not an option, at all :)
Comp, Here are the total reads/ writes for the top ten processes during that session:
http://img2.imageshack.us/img2/4117/read.png
http://img704.imageshack.us/img704/7239/write.png
Here is the outcome of monitoring a HDD. I used a system image so the HDD was configured exactly as the SSD and I used the same hardware system.
As I am monitoring general use I can't do an exact copy of how I ran various programs when I monitored the SSD, but I used the same programs as before. The multi task element included watching a DVD with Power DVD, whilst playing an mp3 with WMP and running a MS Essentials quick scan. I also open Tractor (Audio mixing) at the same time. I also did this for the SSD monitoring.
The hard drive was a Seagate Barracuda 160GB. I had a Rap, but it was too small for the system image.
Bandwidth
The a max read transfer rates were all lower than an SSD.
IOPS
I have now included max response times and here a big difference can be seen.
Queue depths
I expected much higher on the HDD, both peak and average. The max read QD was actually lower on HDD but the average QD for writes was much higher.
Having not used HDD for a while the obvious observations were slow boot and a lack of responsiveness when Windows first loaded. MW2 loaded noticeably slower, and during the multi task exercise the DVD playback became a bit choppy.
http://img835.imageshack.us/img835/636/hddr.png
Computurd
10-23-2010, 03:04 PM
here is a comparison of the areca 1880IX-12 on the left, and the 9260-8i on the right, just kind of illustrating the difference in low QD speed from two uber cards.
The point is the killer speeds at the low QD, these are speeds that a single ssd cannot muster.
as you see, with throughput goes latency. the faster you serve the requests, even at low QD, the lower your latency is.
lower latency=faster performance. one thing that is agreed upon is that lower latency equals more speed :)
the low QD numbers for the array are miles ahead of single ssd, beginning at anything over 1 with the 9260, and just in total with the areca.
so...even at the lower QD, the arrays own. with the tremendous throughput of the array, it is easy to see why it is so hard to raise the QD. these are 4k random.
if there were mixed read/write patterns mixed in i would venture to say that the differences between the array and a single device would be even more of a mismatch.
EDIT: did a 4k test @ qd 1 with 75%read 25%write 100% random.
vertex LE single drive.................. 19 mb/s
areca w/5 vertex no read ahead:.. 47 mb/s (6gb test file)
http://i517.photobucket.com/albums/u332/paullie1/titlt.png
One_Hertz
10-23-2010, 03:15 PM
Would love to play this but the software completely messes up my OS after the initial install :(. BSOD at bootup. Had to safemode in and remove the service to get it to boot up again. I don't think it likes my nlited windows.
Here is a side by side comparision to compare more easily.
http://img177.imageshack.us/img177/8386/hddvsssd.png
From The SSDPerfAnalysisFS Filter
SSD
• Max Av QD Reads = 1.653
• Max Av QD Writes = 2.431
• Max Read QD = 18
• Max Write QD = 70
HDD
• Max Av QD Reads = 1.318
• Max Av QD Writes = 8.203
• Max Read QD = 13
• Max Write QD = 64
iw4sp.exe = Modern Warfare 2 Single player
MsMpEng.exe = MS Essentials (AV)
Photoshop.exe = Photoshop CS 5 64 Bit
Traktor.exe = Traktor Scratch Pro (Audio mixing)
wmplayer.exe = Windows Media Player
trustedInstaller.exe = I think I uninstalled Windows Gadget and then re-installed it.
SetPoint.exe = Logitech wireless mouse and keyboard
The other processes are either self explanatory or I don't know.
Hi One_Hertz :wave: Yeah, most likely a nlited problem.
Here I directly compare the performance between HDD and SSD for Modern Warfare 2 SP. Fast IOP = less that one ms.
http://img834.imageshack.us/img834/7422/89003701.png
Here I directly compare the performance between HDD and SSD using Photoshop.
When I monitored this I only created and saved a 1.57MB file.
To compare what happened when working with a much larger file I created one @144MB. Max write speed increased to 82.10MB/s. (Using SSD)
Next I created a 316MB large format Photoshop.psb file. Max write speed remained the same. Here however my RAM and CPU were working much harder so the writes were probably a lot slower becuase I was waiting on the CPU/ RAM.
http://img207.imageshack.us/img207/4397/73438049.png
Computurd
10-24-2010, 05:35 PM
strange results tbh, dont know what to make of that, especially the QD ssd v hdd...maybe the writing was slowing down the reading with the SSD?
the fact that the max read and write are faster with the SSD makes perfect sense, and the fast IOP numbers all look in line with what i would suspect.
the QD though, is like the numbers are...dunno...unexplainable!
are the QD going higher on the ssd simply because it is reading faster, and can handle it better? i noticed the max reads are much faster for the processes with the ssd, and the fast iop count is faster, so common sense would say the QD would be lower...or is the system treating the SSD differently and requesting information faster....we know that the Win7 environment is optimized for SSD and that it handles them differently. but that much difference? wondering if the results would be the same with vista? headscratcher for sure. what about NCQ and ACHI, are they accessing information in a different manner from the SSD? when you think about it, faster read/write SHOULD equal lower QD!!
doesnt make any sense either, how can the drives have different amounts written/read from them in the total sections..the write iop count comparison is wrong, you have 40K writes compared to 11k writes, so that isnt the same...would throw the numbers off. the max data writes are way different too...even though you wrote the same amount of info for example:
photoshop test:
max data transferred HDD; 283,496,448
max data transferred SSD; 778,919,936
but you ran the same test on them? why would it write 2.7 times more data to the SSD for the same size file?
for the modern warfare 2 the results are much the same...tremendously more amount written to the SSD with the exact same usage?
seems like all of the write comparisons across the board follow this same pattern here. your read numbers for amount transferred look very close though. are you sure that you cleared the test before you ran them again on ssd?
are the QD going higher on the ssd simply because it is reading faster, and can handle it better?
This is what I am starting to suspect. Maybe high peak QD's are occuring because the SSD is dealing with requests so quickly. Maybe that is why the average QD stays around the same. :shrug:
photoshop test:
max data transferred HDD; 283,496,448
max data transferred SSD; 778,919,936
but you ran the same test on them? why would it write 2.7 times more data to the SSD for the same size file?
for the modern warfare 2 the results are much the same...tremendously more amount written to the SSD with the exact same usage?
seems like all of the write comparisons across the board follow this same pattern here. your read numbers for amount transferred look very close though. are you sure that you cleared the test before you ran them again on ssd?
I picked this up as well. Maybe I just had Photoshop open longer and it auto saved when I was monitoring the SSD. For MW2 I loaded a level, played the level, let the next level load and then closed it. I did the same on HDD and SSD so I have no idea why writes were so much higher.
Later today I am going to install a game on both HDD and SSD. That way I will know for sure that I am writting the exact amount of data to both HDD and SSD.
The monitoring is recording per process, but it seems TRIM instructions are being sent out all the time by the OS and maybe that is part of the reason that SSD in general is seeing a lot more writes.
The clear conclusion for me at this stage is that the vast majority of IOP read/ writes are done in less than a ms with SSD, which is obviously why a SSD feels so snappy and I'm not sure if raid could improve that. What I don't know is if max MB/s speeds would increase in a raid set up.
Comp, can you run a game on a raid array and show the results as I have shown on the last couple of posts?
Here is what happens when I install Bad Company 2. First I reset hIOmon, rebooted and then I installed the game.
It's not that straightforward as it generated multiple exe files.
http://img408.imageshack.us/img408/3433/revisedsj.jpg
From The SSDPerfAnalysisFS Filter using HDD. This is a summary of the game install, patch update and general use whilst waiting for the patch to download and install.
Now to reset for the game load comparison.
http://img841.imageshack.us/img841/6091/76839854.png
Here is what happens when I play Bad Company 2 for the first time. I waited for the intro to play out and then immediately proceeded to get myself killed. Waited for the reload and then closed the program. It looks like this game does small incremental loads. Maybe later on in the game the loads are more significant. :shrug:
http://img834.imageshack.us/img834/6476/gameload.jpg
Boy glad to be back on SSD. That HDD sucked. Anyway that is it now for HDD vs SSD comparison, although I can check anything out if of interest to anyone.
Next up I look at the bootlog, but what I really want to see is some results from anyone with an SSD raid array. In particular MB/s max times for game and application loading. Clearly my MB/s max times are well below my drives capabilities so it would be interesting to see if a raid array made much difference.
overthere
10-25-2010, 08:09 AM
A quick word of caution when monitoring I/O operations at either the physical volume level or the physical device level as regards the process-related metrics:
As previously noted, the hIOmon software can (concurrently) capture I/O operation performance metrics at three distinct levels within the I/O stack within the operating system:
1) The file-level (logical drive/disk)
2) The physical volume level, and
3) The physical device level (which is similar to the "PhysicalDisk" level in the Windows Performance/System Monitor).
It is important to note that I/O operation performance metrics at the file-level are observed by the hIOmon I/O Monitor within the context of the particular process that initiated the respective I/O operation. Consequently, the process-related metrics (e.g., the "process name") associated with the other respective metrics do reflect what the particular process did/experienced in terms of I/O activity.
However, it is another story further down at the physical volume and physical device levels. In the case of I/O operations at either of these levels within the operating system, the hIOmon I/O Monitor observes the I/O operations within an essentially indeterminate context.
That is, the "process/thread" currently running at the time during which the hIOmon I/O Monitor observes the I/O operation could be for any process (and not necessarily for the particular process that initiated the "originating" I/O operation up at the file-level).
What this all means is that if you only configure the hIOmon software to monitor I/O operations at either the physical volume and/or physical device levels, then you need to be cautious about which particular process is identified with a respective set of I/O operation metrics.
Now in these cases the metrics reported do accurately reflect the particular process that was "running" when the hIOmon I/O Monitor observed the respective I/O operation (but again, this process is not necessarily the one that initiated/originated the I/O operation).
So overall then, if you are interested in capturing I/O operation performance metrics related to specific processes in general, then you need to collect I/O operation performance metrics at the file-level (so that the hIOmon I/O Monitor can initially observe the I/O operations within the context of the particular process that initiated the I/O operations).
Note too that the hIOmon software also provides a "Physical Device Extended Metrics" option, whereby if you have configured the hIOmon software to capture such metrics and to concurrently monitor also at the physical volume and/or physical device levels, then it can automatically correlate the I/O operations performed at these levels with those at the upper file-level. Basically this enables you to associate the physical volume/device I/O operations with their respective process (which again, cannot be done - in general - by monitoring only at the physical volume and/or physical device levels as noted above).
The reason I said "in general" is that there are applications that essentially bypass performing their I/O operations at the file-level, but instead initiate them directly at either the physical volume or physical device level within the operating system - but this is another whole story. :)
Hopefully the above is clear.
And perhaps another key take-away here is that the "flow" of I/O operations through the I/O stack of the operating system is not as "simple" as maybe some presume.
^ thanks for the clarification and continued guidance. I can understand what you are saying about the file-level importance to get the right metrics, but hopefully there is something to learn from what I have done so far. There are so many ways to do things and so many things to monitor it's a bit overwhelming.
Here I post the device top ten devices using SSD. The summary includes what I monitored above plus an install of Left for Dead 2. Once Left 4 Dead was installed I played an mp3 with WMP, a dvd with Power DVD, ran a quick scan with MS Essentials, played a track on Tractor - all at the same time, whilst working on a word document and downloading a huge patch file for left 4 Dead 2. Way OTT to what I would normally run at the same time.
http://img840.imageshack.us/img840/9301/deviced.jpg
OK here is the same as above with HDD. I noticed the CPU was maxing out quite a lot when I ran all the programs. I didn't check that with the SSD but I'd guess it was the same.
So what can I conclude? It's impossible to use the SSD to its max sequential read speeds with the programs I used? Maybe faster sequential writes would not go amiss? I'm guessing if most IOPS are fast (less than a ms) there would not be much benefit if they got even faster?
http://img835.imageshack.us/img835/1511/devicex.jpg
overthere
10-25-2010, 10:59 AM
^ thanks for the clarification and continued guidance. I can understand what you are saying about the file-level importance to get the right metrics, but hopefully there is something to learn from what I have done so far. There are so many ways to do things and so many things to monitor it's a bit overwhelming.
Happy to be of help.
Looking at what you have done so far, you have shown some interesting data points.
These observations can help elicit a discussion of which particular metrics along with which specific scenarios are perhaps of primary interest.
In addition, it might be good to revisit Computurd's comment in post #3 of this thread (i.e., his comment: "so lets talk methodology here...how are we going to go about it? what will be the baseline?").
Regarding the "overwhelming" part of your comment, again there is an inherent, underlying complexity as to how I/O operations are actually handled at the "host" end of the cable (i.e., from the application down through the operating system).
The hIOmon software makes these dynamics much more transparent to folks - for better or worse! :eek:
In any case, one approach to consider is to formulate specific questions for which the hIOmon software can make available empirical metrics, and given that this is often an iterative process (especially as the "peeling of the onion" unfolds).
Lastly, a brief explanation of the "<DASD>" stuff that appears within your prior screen shots (in case some folks are curious).
The hIOmon software provides the option to collect summary metrics upon an individual, specific device basis (whether it be a logical drive/device, a physical volume, or a physical device within the operating system).
The "device summary" metrics reflect the overall aggregate of all summarized I/O operation metrics for I/O operations that were monitored by the hIOmon I/O Monitor and which were directed to the respective device.
In the case of a Device Summary for a logical drive/device (e.g., the "C:" drive), the "device summary" metrics reflect the overall aggregate of all summarized I/O operation metrics for all files (and only those files) that were monitored by the hIOmon I/O Monitor and which reside upon the respective device.
This approach enables you to obtain an overall picture of I/O activity to a particular logical device limited to only those particular files (and optionally processes) that are of specific interest to you.
Now I/O operation activity at the logical drive/device level is normally directed towards a specific file/path.
However, in those cases where a specific file/path name is not associated with the I/O operation, a "pseudo file name" of "<DASD>" is reported (since there was no actual file/path specified as part of the I/O operation). This can occur when an I/O operation for a device that is being monitored is being performed to access filesystem-related data structures (meta-data) present upon the device (although more recent versions of Windows have begun to provide explicit "file names", e.g. "$Mft", for these meta-data files).
In addition, <DASD>\VMM" is displayed when the system flags associated with the I/O operation indicate involvement by the system Virtual Memory Manager (VMM).
Two last points:
1) In the case of physical volumes and physical devices, the I/O operations are directed to blocks/sectors (i.e., Logical Block Addresses) and not offsets within files (since at these levels the I/O operations are directed to specific blocks/sectors residing upon the respective volume/device). Consequently, only the "pseudo file names" are reported within the metrics collected by the hIOmon software (and mainly so that one can differentiate between I/O operations involving the VMM and those that do not).
2) The summary metrics for the device proper is, as noted above, the overall aggregrate for all of the associated files (including pseudo files).
And if you are wondering why the read max QD for DR0 is 21 in your prior screenshots, but the highest read max QD for an associated (pseudo) file is only 20 (i.e., for <DASD>"), this is because at the time that read max QD for <DASD>\VMM" was 20, there was also presumably one I/O operation queued for <DASD>\VMM" - thus bringing the aggregated total to 21 for the "DR0" device overall. (This could be substantiated if an I/O trace were also collected).
I got excited by what hIOmon could do so admittedly I rushed in a bit ;) A bit of methodology is now in order :)
First here are the specs for my SSD:
• Sustained Sequential Read: up to 250MB/s
• Sustained Sequential Write: up to 100MB/s
• Read latency •65 microseconds
• Write latency •85 microseconds
• Random 4KB Reads: up to 35,000 IOPS
• Random 4KB Writes: 160GB X25/X18-M: up to 8,600 IOPS
What would I really like to know? Primarily I'm trying to understanding how well my SSD performs in relationship to my typical desktop use. Is it adequate or could it be faster. What needs to be faster? This raises 4 questions for me, although maybe they are not the right questions. ;)
1. How much am I utilising the available max sequential read/ write speeds overall?
(From what I've seen so far it would seem that read speeds are underutilised but, surprisingly for me, maybe faster sequential writes would speed things up (?) If I've got that wrong what is the best way to find that out?)
2. How much am I utilising the max sequential read speeds for loading a game? What is limiting the max MB/s read speed?
3. Would an increase in IOPS capability give me any advantage?
4. Would an SSD raid 0 array improve responsiveness and make my apps work faster in comparison to a single SSD?
Building a SSD raid 0 array is possible, but it's going to be loads of work as the other drive I have is only 80GB so I could not use an image and therefore I'd have to install everything from scratch. (Not to mention my wife is currently using it).
Somehow I don't think that it would be necessary anyway with hIOmon :)
overthere
10-25-2010, 05:59 PM
I got excited by what hIOmon could do so admittedly I rushed in a bit ;) A bit of methodology is now in order :)
Good to hear of your enthusiasm! :)
Certainly various benchmark tools can be used to explore the limits of your system.
As to the extent to which your particular applications, overall system, and normal, everyday usage actually reach (or require) those limits is, of course, another matter.
But capturing empirical metrics (that reflect your actual usage) can be a important step in establishing, for example, a baseline understanding of your system and usage - as you have begun to do.
And the quantitative nature of the metrics can be valuable in evaluating and determining, for instance, the actual merits of system/application changes and improvements.
Your four questions are good ones. But I defer to the experts in this forum to hear their suggestions and thoughts as to the possible answers. ;)
As to the extent to which your particular applications, overall system, and normal, everyday usage actually reach (or require) those limits is, of course, another matter.
That in a nut shell is where my interest lies.:)
There are some very knowledgeable people on the storage forum so I'm hoping they can give some input at some stage.
hIOmonn has showed me that I am well out my depth, but I will try to work through it regardless, although I now realise I don't even understand the basics concepts of storage. This can be summarised well in my query below. ;)
If I look at the data transfer speeds per single IOP read I get 260MB/s.
http://img534.imageshack.us/img534/6205/19345433.png
Here I get a data transfer speed for a single IOP write of 251MB/s.
http://img109.imageshack.us/img109/9295/79468666.png
Here is a summary of the top ten processes
http://img530.imageshack.us/img530/4395/23786144.png
http://img716.imageshack.us/img716/9343/77470784.png
How could I be getting a those speeds for a single IOP?
I don't understand the relationship between the single IOP MB/s speeds and the speeds below, which I assume are sequential speeds?
http://img251.imageshack.us/img251/7462/58386385.png
http://img222.imageshack.us/img222/4522/27146871.png
http://img204.imageshack.us/img204/3792/17752335.jpg
http://img5.imageshack.us/img5/5504/47120492.png
Today I am using the Presentation Client to monitor my normal use and I'll post a screen shot of that later.
Here is a Device I/O summary from the Presentation Client. Here I use my PC as I would do normally.
http://img266.imageshack.us/img266/9544/77409412.png
overthere
10-26-2010, 01:49 PM
If I look at the data transfer speeds per single IOP read I get 260MB/s.
Here I get a data transfer speed for a single IOP write of 251MB/s.
How could I be getting a those speeds for a single IOP?
I don't understand the relationship between the single IOP MB/s speeds and the speeds below, which I assume are sequential speeds?
Here is a short explanation for each of these two MB/s metric types:
Read Single IOP Maximum MB/s:
This metric basically reflects the highest data transfer rate observed by the hIOmon I/O Monitor for a single read I/O operation.
This MB/s rate value is the amount of data transferred by the single read I/O operation divided by the amount of time that it took to perform the operation (i.e., the time duration of the I/O operation) as observed by the hIOmon I/O Monitor.
Read Maximum MB/s:
This metric represents the maximum MB/s rate for read I/O operations as detected by the hIOmon I/O Monitor during a one-second interval since the hIOmon I/O Monitor began collecting I/O operation performance data for the file (or device or process).
This metric reflects the combined total amount of data transferred by all of the read I/O operations during the one-second interval.
There is perhaps a subtle distinction between these two metric types.
The first (i.e., the Read Single IOP Maximum MB/s) is based upon the time duration of the single I/O operation itself, whereas the second metric is based upon the total amount of data transferred (by all of the respective I/O operations) over the course of a one-second time interval.
In terms of an example, say that a read I/O operation transferred one byte within one millisecond.
By the first metric type noted above, this would represent a "one thousand bytes per second" data transfer rate.
But such a "single IOP" MB/s rate is not to say that one thousand bytes were transferred during the course of a one second interval.
On the other hand, the second metric type does reflect the actual maximum amount of data transferred during a one-second interval.
I guess the conclusions to your experiments will be fascinating for all of us, Ao1.
It did strike me, though, that this software might be able to give us the answer to an age-old question that's dogged this forum and many others before - the need for and actual OS usage of pagefiles in a system with large installed RAM. It's not unrelated, since those of us with SSDs are are all keen to reduce writes where possible, even if they are ideally suited for pagefiles as MS claims.
Given the entrenched views on this, anyone interested in doing that research would be something of a hero. :p:
Anvil
10-26-2010, 03:57 PM
I had to give it a try :)
This is a small sample, a bit of surfing using Chrome, not much more than that.
108853
Interesting results.
e.g. random vs sequential io
108854
Computurd
10-26-2010, 06:02 PM
@Anvil is that with the 9260 or 9211 or areca? you have all the goodies :)
also, which drives, etc?
i noticed your write qd is much lower than A01 has been getting, which leads me to believe you are on a controller with cache, ruling out the 9211...but then again with an array the writes will be much better even without cache :)
@A01, yes this has raised more questions than it has answered so far, but good questions nonetheless...this exercise is definitely going to broaden our horizons a bit ;)
my plan is taking shape in my mind:
i plan on creating one system image, then testing that image on a single vertex, then a array, and also a caviar black. win7 x64 of course.
there will be some timing done of game loads, and various apps for comparison.
all tests will be run at the same clock. i think 4.4 outta be about right. i feel that many of the lack of superior load times/etc are due to systems not being fast enough to hang with it, so i want to test with a high baseline clock.
What i am thinking of is not 'load testing' in its entirety though. as a matter of fact i am less interested in the loading metrics than most would be.
i am going to run a 'suite' of programs, and not begin monitoring until they are all loaded, in an effort to see the load of the system 'in flight' and compare the three.
load testing: games to test for load times will be my usual, crysis and left 4 dead. i also plan to run a metric while those games are loading the same levels on each of the three setups.
in-flight performance testing: my standard gaming config: two instances of firefox, video chat on MSN, game, and various desktop gadgets, also running AIDA64 for monitoring, FRAPS for fps, and kaspersky antivirus.
any combination of programs that you would like to see, let me know. i have technet so i can get all MS stuff, and i also have PS, anything else i can get off the 'Binz just as well as anyone else :)
i can get any os i need off technet so i might run a vista as well, to see if there are significant differences in QD handling with a single SSD, if i see any significant difference i will expand upon that testing with an ssd array. what, if any, major difference with SSD and vista/win7 intrigues me...to save time though if there is no difference i will abandon that.
sometimes with so-called gen 1 ssd (specifically vertex) the win7 environment will not detect the ssd until you run winsat disk, either via cmd line or via WEI> there is a means of telling if it has detected the SSD, you can check in the defrag schedule, and if it hasnt detected the SSD it will be in the schedule list. if win7 HAS detected it, the SSD will be removed from that list. hopefully i can replicate this issue and get the OS not to recognize the single ssd. this would speak volumes to run the same tests, with and without the OS being 'optimized' for it. maybe it can help to quantify the actual amount of filesystem optimizations, if any.
im not as interested in monitoring processes/programs individually as much as i am interested in the device usage as a whole. i am not programming here, i just wanna see the device level metrics. however, i will make a standard list of metrics that will be captured, so let me know of any you would like included. i will make a list up when i get set up to post before testing.
im sure i will add to this before i begin in roughly seven days :) any and all opinions/thoughts of tests to be added will be appreciated. this is gonna take some time and setup, so i want to do it right the first time :)
bored waiting on intel G3 anyway so lets do this.
Anvil
10-27-2010, 12:19 AM
It's the good old ICH running 3R0 Intel X25-M 160GB G2
I'll try to do some iometer/sqlio tests later tonight, so far i'm just playing with the different config options
Guys,
I think we need to very careful about what we are doing or we will end up with a lot of data that does not seem to make sense.
First off we need to establish the metrics that are of interest. Right there is the first big challenge.
Next we need to be able to compare the exact same workloads and that is also a big challenge.
We also need to be careful about how we monitor, as per post #56 in response to what I discovered in post #55.
How do we establish if the overall system is impacting the storage performance? Already I have seen my CPU max out when I multitasked, which (I assume) must have impacted storage performance.
Looking at Anvil's results in post #66 his average QD is 1,514 with just a bit of web browsing and his maximum response times seem quite high. Clearly there are going to be some large variations in certain metrics between a single SSD, onboard SSD raid and hard raid SSD, so we need to understand what these are going to be and more importantly what to look for in terms of the performance output. Here I am struggling as I am out of my depth. :(
With regards to IanB's post #65 that is something that I think hIOmon could do quite easily. In the default filter it monitors the page file, so I think it could isolate and monitor the page file activities and then access the overall system impact with the page file either on or off. Again we need a clear methodology on how to go about doing that to save wasting time.
So....first off what are the metrics that are of interest and why?
Anvil
10-27-2010, 02:06 AM
Ao1
I agree, I need to play a bit with the metrics to try to fully understand and possibly limit the IO to specific files. e.g. an iometer test file.
I'm not comfortable yet with the metrics, I need to know what is actual IO, meaning DRn vs DASD vs VMM
The only way of getting the same workload would be using benchmarks like iometer, all other approaches would lead to non comparable data.
(we all have different setups, drivers, background tasks, ...)
I think we need to play a bit to get to know what to look for, QD, access times are metrics that are of general interest imho.
I'll try comparing last nights session as to get results without using the pagefile.
Anvil, in post #23 Overthere explains that it is possible to use a benchmark as a plug in. This would give a fixed data workload. I was hoping that I could monior my normal use and come to some conclusions about how well my SSD was utilised and if any improvements in performance would see real life performance gains, but on the other hand at least a benchmark provides a comparable workload that we could all work with.
Anvil
10-27-2010, 03:41 AM
Ao1,
One can't run a benchmark as a plug-in, the point is that one can have a clean start/stop for the metrics, which is very important. (which I was looking for)
I'm currently thinking in line with CT, meaning that I'll first try monitoring the system as a whole and then later monitor files, i.e. VMWare files.
(I'd love to know how IO looks like for VM's, random Vs sequential in particular)
I've got separate "devices" for VM's so that shouldn't be too hard to monitor.
overthere
10-27-2010, 08:19 AM
I'm not comfortable yet with the metrics, I need to know what is actual IO, meaning DRn vs DASD vs VMM
There is a brief explanation of the "<DASD>" stuff back on post #59.
To hopefully be clearer, basically:
1) The hIOmon software generates "pseudo file names" for those I/O operations that are observed by the hIOmon I/O Monitor for which there was no file name/path specified (or applicable).
2) Two different "pseudo file names" are used: "<DASD>" and "<DASD>\VMM"
3) "<DASD>\VMM" represents collectively those I/O operations whose associated system flags indicated involvement by the Windows system "Virtual Memory Manager (VMM)".
4) "<DASD>" represents collectively all other I/O operations (i.e., other than the "<DASD>\VMM" I/O operations) for which there was no file name/path specified (or applicable).
5) All I/O operations performed at either the physical volume or physical device level specify basically a block/sector address (rather than a file name/path) upon the volume/device to which the I/O operation is directed.
Consequently, all I/O operations to a physical volume/device will be relegated by the hIOmon software to either the "<DASD>" or the "<DASD>\VMM" pseudo file name.
6) The DRn represents the "Device summary" metrics for the associated physical device.
The "device summary" metrics reflect the overall aggregate of all summarized I/O operation metrics for I/O operations that were monitored by the hIOmon I/O Monitor and which were directed to the respective device/volume.
Accordingly, the "device summary" metrics for DRn reflect the overall aggregate of both of the associated pseudo files (i.e., "<DASD>" and "<DASD>\VMM").
Sorry for any confusion.
We'll get there in the end ;)
I've got another X25-M 160GB arriving tomorrow so I will soon be able to post some results in raid 0. Can't wait.
overthere
10-27-2010, 09:39 AM
We'll get there in the end ;)
I've got another X25-M 160GB arriving tomorrow so I will soon be able to post some results in raid 0. Can't wait.
It's often an iterative process, and I am happy to answer questions about the hIOmon software as they arise. :up:
And looking forward to your upcoming results :cool:
Cleaned up, as superseded in post #88.
Anvil
10-27-2010, 11:03 AM
Give it a shot using the summary from the Presentation Client.
There are of course other metrics that are of interest :)
Do the test as you've planned, the outcome might need second runs and then you can include other metrics.
Anvil
10-27-2010, 01:41 PM
Ok, so I tried the hIOmonBenchmarkExport batch file and captured an AS SSD run.
Is there some way to display the csv file using hiomon or should I just open it in Excel?
(this output is generated using the wmi browser,
it's not accurate as there were activities recorded while generating the output)
108863
corresponding AS SSD screenshot
108865
You can open it with excel. It will only open as a read only file but you can still work with it and "save it as" if you want to keep the changes.
You know the other thing I'm really intereted to see is if the read/ write speeds for a single IOP go up when I get the raid 0 array going.
Edit Anvil is that with 3xRO
overthere
10-27-2010, 01:59 PM
Ok, so I tried the hIOmonBenchmarkExport batch file and captured an AS SSD run.
Is there some way to display the csv file using hiomon or should I just open it in Excel?
Hi Anvil,
I gather that you used the "Save" option with the hIOmonBenchmarkExport batch file, which resulted in the current accumulated summary metrics being written to the CSV file.
The short answer is that you will need to use some other program (e.g., Excel as you mentioned) to display this CSV file.
Depending upon how the hIOmon software was configured to collect the I/O operation performance metrics, the hIOmon Presentation Client can be used to display the collected metrics from the hIOmon "File I/O Log" file - but this is another story! :)
In any case, none of the hIOmon clients can be currently used to directly view a CSV file.
The basic idea has been that once the metrics were exported/contained within a CSV file, then a variety of other existing tools (e.g., Excel) could be used to display, process, etc. these metrics in a manner familiar to the user and leveraging the features of such tools.
Anvil
10-27-2010, 02:10 PM
Hi overthere :)
I used the Save option and I expected Excel was the answer :)
I've included the result file if anyone would like to have a look at it.
It looks like the collected metrics are really close to what one would expect from an AS SSD run, looking great so far :)
Anvil
10-27-2010, 02:13 PM
Ao1
Yes, that using 3R0, strip size 16KB, RST 10.0.0.1026
I've downloaded the latest RST, I'll install it this weekend.
Anvil, have you tried using the default SSDPerfAnalysisFS script? (post #5) It has quite a few good performance metrics on it as standard. (It's also possible to add more).
If you didn't select the automated monitoring when you installed hIOmon its easy to do it afterwards.
Start>hIOmon>Getting started help>hIOmon automated monitoring configuration set up. Then select option 3
When I compare results from my As Benchmark run I see from your cvs file that your avg and max read QD were much higher than mine but conversely my IOP count was much higher.
Edit: What strip size shall I use? I was going to use 128k, but I think a much lower strip will be more interesting (?)
Anvil
10-27-2010, 02:46 PM
Ao1,
I'm currently using the SSDPerfAnalysisFS script and I've been playing a bit with the automated configs.
I havent loaded the csv file yet, my bench-pc is for benchmarks only :), I'll load it on my laptop shortly.
Most of the time I've been using 8-16KB, for the Intels 16KB is looking really good, especially using RST 9.5 or later.
Computurd
10-27-2010, 03:17 PM
very interesting stuff here. interested in your high QD numbers, thinking here of reasons... 10 channels per device, three devices, total of 30 channels. that would be a QD of 50 per-channel roughly...might be explainable?? but doubt it. i think the system sees the devices as one drive.
NCQ allows the system to ask for I/O in a different manner, optimized and designed for HDD, it still works phenomenal wonders with SSD.
The shopping analogy applies very well here, for those who aren't sure of its functions:
Imagine you have a shopping list of 50 items, and you go to the grocery store. if you would go and find the first item, then go to the checkout lane and pay for it, then go back, get another item, then go pay for it, repeat repeat repeat 50 times, that is in essence how I/O operations work without NCQ. the system waits for the I/O request to 'return' before it asks for another. it has its list, but it wants it one at a time.
NCQ gives you the list, then tells the device to go get it, in any order, all at once. it doesn't care what order the "items" are obtained, it just wants them all to return at once. it does not wait for return verification.
so....much like taking the 50 item list shopping, go get all 50 items in the most efficient order, THEN go check out.
much faster, and more efficient.
NCQ works exceedingly well with SSD. So...maybe the higher QD are the result of these 'lists' being served so quickly, or being dumped on to the controller of the SSD so quickly. what if it is counting each request on the list as a QD? so one list of 50 would be a 50 QD spike? maybe longer 'lists' are served to devices that can return the 'lists' faster....maybe these higher QD numbers are the result of VERY efficient NCQ being run by the system.
god who knows. interesting any way you slice it :)
how would you prove such a theory though?
i guess that has become my side-mission here. maybe my main mission! to find how NCQ 'lists' are monitored as QD would be very interesting. what if there is a curveball here though...what if one 50 item list, only counts as ONE QD???
overthere
10-27-2010, 03:39 PM
It looks like the collected metrics are really close to what one would expect from an AS SSD run, looking great so far :)
Certainly good to hear! :up:
overthere
10-27-2010, 03:59 PM
i guess that has become my side-mission here. maybe my main mission! to find how NCQ 'lists' are monitored as QD would be very interesting. what if there is a curveball here though...what if one 50 item list, only counts as ONE QD???
CT,
The basic mechanism behind the "queue depth (QD)" metric within the hIOmon software is fairly straightforward:
When the hIOmon I/O Monitor observes the start of an individual monitored I/O operation, then the "queue depth" counter is incremented.
And when the hIOmon I/O Monitor observes the completion of a monitored I/O operation, then it decrements the respective "queue depth" counter.
There is a separate "queue depth" counter maintained for each monitored device, file, and process, along with associated "subset" QD counters (e.g., separate "Read" and "Write" QD counters, again upon an individual monitored device, file, and process basis).
Also, the above applies separately at each level of the I/O stack being monitored by the hIOmon software, i.e., at the file (logical drive/disk), physical volume, and physical device levels.
Here I compare raid 0 against a single drive.
The monitoring period started up after boot up. I ran the following one after the other with only IE & MSM left running throughout.
I tried to replicate the use of apps as close as possible on each run.
• Play MW2 SP (Intro and then a couple of level loads).
• IE
• MS Messenger
• Play DVD
• Play WMC
• Photoshop - Open 4 jpeg files from the desktop simultaneously. File sizes: 1MB, 2.14MB, 22.3MB, 127MB. Run a "save as" on each file back to the desktop. (CPU maxed out a bit on the larger files sizes).
• Tracktor Scratch Pro
• MS Essentials quick scan
I avoided multi taking because I didn't want the CPU to get maxed out and possibly skew the results, but I will try it later. The strip was set at 128K. I might try a really small strip to see what that does.
General observation.
Unlike the big difference between ssd and hdd I did not notice any difference in boot time or game loading. Photoshop did seem to save the files quicker, but the CPU struggled a bit on larger files sizes, so file saves weren't that fast anyway.
http://img19.imageshack.us/img19/5804/deviceg.jpg
http://img594.imageshack.us/img594/1285/ssfp.jpg
http://img149.imageshack.us/img149/1334/raidvssingle.jpg
Computurd
10-28-2010, 07:22 PM
when you ran these did you get any plain ol C:\ drive stats? as opposed to the app monitoring?
Hi Comp,
The 1st image shows drive summaries. The other images drill down into the top ten processes. As Overthere has stated an exe file is not necessarily the only process associated with an application and to get that I would need to monitor the specific folder of interest.
iw4sp.exe is Modern Warfare 2 SP. The max MB/s read was actually marginally slower on the raid 0 configuration!
The big surprise for me (based on my usage patterns) so far is that MB/s read speeds don't come in anywhere close to the capability of a single drive, never mind raid 0, but conversely faster MB/s writes speeds are actually quite useful.
Here is a comparision of what happens during boot up.
http://img502.imageshack.us/img502/3959/bootlog.jpg
http://img714.imageshack.us/img714/4886/pagefiled.jpg
http://img705.imageshack.us/img705/3136/bootlog3.jpg
From now on: page file off\
Computurd
10-29-2010, 05:17 PM
ok so i want to start with the results on the post 88 and tell you what i see.....
RAID.............................................. ................................................Si ngle drive
http://i517.photobucket.com/albums/u332/paullie1/a-1.png
the most glaring here...latency is much faster for reading. .0058 v .0263 (22 percent faster) and .3925 v .8114 (48 percent faster)
http://i517.photobucket.com/albums/u332/paullie1/1-1.png
Fast write IOPS around twenty percent better with raid. this is just the fast writing, not the overall btw,.
http://i517.photobucket.com/albums/u332/paullie1/b.png
onto overall writing, damn near twice as fast. 478 raid v 253 single device
http://i517.photobucket.com/albums/u332/paullie1/2-1.png
mixed read/write fast iops are much better,, around ten percent, with the raid
http://i517.photobucket.com/albums/u332/paullie1/D.png
overall response times, again a big difference here... .0065 v .0256 (25 percent faster) and also .6828 v 1.1426 (60 percent faster)
R/W Fast IOPS 93.7 v 84.4 (10 percent better)
---------------------------------------------------------------------------
had to steal this from the SSD raid o "bootlog" vs single device "bootlog" section since the data was missing from post 88...
so the format is different....
single device............................................ ..........................................raid 0
http://i517.photobucket.com/albums/u332/paullie1/E.png
there is a tremendous difference here....
l
--------------------------------------------------------------------------------------------
the other metrics are quantitative...I.E. they aren't good for comparisons sake...should be close to same amount of data transferred (x amount) etc...really the rest of them are not going to help with comparing.
the other testing with the ssd raid 0 vs single device boot logging mirror these results pretty clearly.
Whether or not your system is fast enough as a whole to capitalize on these differences is another thing entirely. imagine when programs are coded for SSD...jesus the difference would be phenomenal.
Here I monitor statistics, post boot, for 30 minutes with a single SSD. I multitasked as I would normally do and opened and used the typical applications/ games that I use.
http://img408.imageshack.us/img408/2643/43009214.jpg
Now I try to consider the IOPS that occurred in use vs the IOPS capability of my drive. To do this I consider average queue depth and average latency readings to get some ball park figures using the following formula:
Queue depth*(1/.latency in ms) = IOPS (in seconds)
Based on the average latency of 0.49ms
• Queue depth 1 x (1/0.00049) = 2,040 IOPS
• Queue depth 64 x (1/0.00049) = 130,612 IOPS
My average queue depth is 1.57
• 1.57 x (1/0.0049) = 3,204 IOPS
My maximum IOP count occurred during boot @ 190 IOPS, but my SSD is capable of 3,204 IOPS based on average latency and queue depth.
I know there could be a lot of variables to the averages but have I got that correct as a ball park figure?
http://img28.imageshack.us/img28/2262/iosummary.jpg
Now I consider the average response times (latency) between a single SSD and Raid 0.
http://img177.imageshack.us/img177/4589/latency.jpg
What I discover when I turn off the page file is that the average response time for a single SSD drops to the average level of an SSD Raid 0 array.
Turning off the page file in the Raid 0 array had no impact on average response times.
The average response times between a single SSD with no page file and an SSD raid 0 array are therefore of no significant difference.
My average queue depths remain in the same ball park figure regardless if I use a single SSD or a Raid 0 array.
Consequently there is no significant average variable to generate more IOPS between the two configurations, although as per post #94 I can't even use anywhere close to the IOPS I get from a single drive.
So no improvement on average latency or average queue depth with Raid 0 based on my storage demands.
Looking at the individual processes I ran none of them could utilise the max sequential read speed capability of my SSD. When running multiple processes together to try and reach the maximum sequential read speed capability of my SSD the CPU would max out.
Using Raid 0 did not improve the maximum sequential read speed of games or most other processes.
Using Raid 0 did however improve the maximum sequential write speeds for applications such as Photoshop.
Does all this sound right or am I making a mistake to focus on averages?
Looking at the individual processes I ran none of them could utilise the max sequential read speed capability of my SSD. When running multiple processes together to try and reach the maximum sequential read speed capability of my SSD the CPU would max out.
This does underscore the things we need to think about when looking at SSD marketing numbers for our relatively undemanding desktop systems. ;)
10,000 IOPs or more may scream "better!" but if the highest IOPs actually demanded by the system is in the hundreds there's no practical gain from that. Sucking in huge quantities of data faster (higher streaming speed) from disks is better, surely? But the processor then has to do something with that vast quantity of data, which places the bottleneck back on processing speed beyond a certain point.
For server situations, high IOPs are significant since you have multiple users making multiple requests and higher streaming speeds may mean the individual requests can complete faster. That's obviously an advantage.
But the bottom line you've demonstrated, if my skim of the results and your analysis is correct, is that the read/write latency is the only significant factor affecting speed for most single-user desktop systems that do not have specialised requirements. That should be the obvious metric we need to compare between disks. And if you think about it... that's what we've always done with storage! The mechanical disk with a lower track seek has always had the overall speed advantage. ;)
And yes, the pagefile result is fascinating! I don't think anyone would have expected that turning it off competely affects speed. As a guess, maybe Windows is dumping some reads speculatively into pagefile simply because it's there, wasting time if the physical memory in the system is enough to handle the total VM requests.
Here is an AS SSD benchmark of an OCZ Core on the left and an X25-M on the right.
http://img214.imageshack.us/img214/8819/19808237.jpg
The OCZ Core has no NCQ support, so IOPS can not increase with queue depth.
• Both drives have fast sequential read/ write speeds. (Compared to HDD).
• Read response times are low on both drives. (Compared to HDD).
The write response time is however very high on the OCZ Core when you start writing lots of small files randomly.
This is due to the write penalty. The OCZ Core had a 4KB page file. The 4KB page files sit in a 512KB block. Once a 4KB page has been written, it can’t be overwritten, it must be erased first before you can write to it again. So, a 4KB write can require a 512KB block to be erased before it can be written. A delay is therefore caused by the deletion of a 512KB block before the 4KB file can be written, which is why the write access time becomes very high.
From what I have monitored large [single files like avi's] are undertaken with only a few IOPS, hence there should be no problem copying large [single] files with the OCZ Core. Lots of small writes are however a problem.
When the OCZ Core is hammered with 4K writes the access time reached 2.5 seconds.
• Queue depth 1 x (1/0.002496) = Write 400 IOPs
When the X25-M is hammered with 4K writes the access time reached 0.091ms.
• Queue depth 1 x (1/0.000091) = 10,989 IOPS (Which is above "up to" max specs of 8,600 IOPS)
I hope I have understood this correctly. Please jump in if I am making incorrect assumptions.
This does underscore the things we need to think about when looking at SSD marketing numbers for our relatively undemanding desktop systems. ;)
10,000 IOPs or more may scream "better!" but if the highest IOPs actually demanded by the system is in the hundreds there's no practical gain from that. Sucking in huge quantities of data faster (higher streaming speed) from disks is better, surely? But the processor then has to do something with that vast quantity of data, which places the bottleneck back on processing speed beyond a certain point.
For server situations, high IOPs are significant since you have multiple users making multiple requests and higher streaming speeds may mean the individual requests can complete faster. That's obviously an advantage.
But the bottom line you've demonstrated, if my skim of the results and your analysis is correct, is that the read/write latency is the only significant factor affecting speed for most single-user desktop systems that do not have specialised requirements. That should be the obvious metric we need to compare between disks. And if you think about it... that's what we've always done with storage! The mechanical disk with a lower track seek has always had the overall speed advantage. ;)
And yes, the pagefile result is fascinating! I don't think anyone would have expected that turning it off competely affects speed. As a guess, maybe Windows is dumping some reads speculatively into pagefile simply because it's there, wasting time if the physical memory in the system is enough to handle the total VM requests.
I am trying to muddle through this with a very limited understanding. The monitoring results are what they are. I really hope the conclusions I try to draw are not misleading.
It would be great to get some feedback if I am looking at it incorrectly. :up:
Computurd
10-30-2010, 08:29 AM
Now I try to consider the IOPS that occurred in use vs the IOPS capability of my drive. To do this I consider average queue depth and average latency readings to get some ball park figures using the following formula:
Queue depth*(1/.latency in ms) = IOPS (in seconds)
Based on the average latency of 0.49ms
Queue depth 1 x (1/0.00049) = 2,040 IOPS
Queue depth 64 x (1/0.00049) = 130,612 IOPS
My average queue depth is 1.57
1.57 x (1/0.0049) = 3,204 IOPS
that cannot be correct, because you are assuming that the latency is always the same here. it isnt. it changes with QD, and also with the type of access that you are doing. mixed read/write kills latency
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
What I discover when I turn off the page file is that the average response time for a single SSD drops to the average level of an SSD Raid 0 array.
So no improvement on average latency or average queue depth with Raid 0 based on my storage demands.
what are you talking about? :rolleyes: look at the numbers here...how can you say that it is the same level? they arent even close!! this is from the graphic above where you posted this, if you look at the numbers...
http://i517.photobucket.com/albums/u332/paullie1/F-1.png
you have .0239 v .0051 that is over twice as fast! also, you have 474.8354ms v 132.0012ms that is a 3x difference!
can you highlight the numbers that show where average response time for a single SSD drops to the average level of an SSD Raid 0 array?? i mean not to be incredulous or anything, but i am :)
Consequently there is no significant average variable to generate more IOPS between the two configurations, although as per post #94 I can't even use anywhere close to the IOPS I get from a single drive.
well you have two programs requesting the same amount of information, correct? it is the latency of the IOPS that counts.
Looking at the individual processes I ran none of them could utilize the max sequential read speed capability of my SSD.
of course not. do the processes even max out the sequential read capability of your HDD? nope. again, not the amount, the latency is why they are faster. there are many programs that do not max out the speed of your hdd which are way faster with SSD. latency.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
It would be great to get some feedback if I am looking at it incorrectly.
your thoughts on post 93? and this post? they both cover much the same info. maybe we are looking at different numbers.
Hi Comp, I'll get back to you on your posts soon, but 1st I thought some more info on the device summary statistics would be useful.
http://img826.imageshack.us/img826/8199/repair1.jpg
http://img149.imageshack.us/img149/821/iopsclarif.jpg
http://img819.imageshack.us/img819/4456/repair2.jpg
http://img219.imageshack.us/img219/6892/datatransfered.jpg
http://img143.imageshack.us/img143/4687/readwriteratio.jpg
http://img146.imageshack.us/img146/2914/fastiops.jpg
that cannot be correct, because you are assuming that the latency is always the same here. it isnt. it changes with QD, and also with the type of access that you are doing. mixed read/write kills latency.
I only consider average latency as this was the only easy way I could compare. There are of course fluctuations on either side.
what are you talking about? :rolleyes: look at the numbers here...how can you say that it is the same level? they arent even close!! this is from the graphic above where you posted this, if you look at the numbers....
Again I only consider average values.
you have .0239 v .0051 that is over twice as fast! also, you have 474.8354ms v 132.0012ms that is a 3x difference!
can you highlight the numbers that show where average response time for a single SSD drops to the average level of an SSD Raid 0 array?? i mean not to be incredulous or anything, but i am :).
http://img813.imageshack.us/img813/1390/faceoff.jpg
With page file on my average response time was 0.5101.ms
of course not. do the processes even max out the sequential read capability of your HDD? nope. again, not the amount, the latency is why they are faster. there are many programs that do not max out the speed of your hdd which are way faster with SSD. latency.
Before I started this I did not think my CPU being bottled neck was a problem, but now I relise that the CPU would clap out before the hard drive when pushed hard.
your thoughts on post 93? and this post? they both cover much the same info. maybe we are looking at different numbers.
I think I covered everything?
Comp, I do not try to say that raid is not faster (it clearly is ;) ). What I try to establish is how well do I utilise my SSD based on my usage patterns. My CPU is a bit dated now (QX6850) and different results might arise with a faster set up. (hint, hint ;) )
josh1980
10-30-2010, 10:46 AM
Got some fascinating data in this thread. Very interesting read to say the least. I will say that this thread has proven ALOT of my observations. I've been disabling pagefiles for every computer that I have installed an x64 OS on. I've always built computers with at least 8GB of RAM when building a computer for Windows 7. I never used Vista since it was such a POC. Even my laptop has 8GB of RAM. I bought my Asus G51jx laptop solely because it has 4 RAM slots. First laptop I have ever seen that has 4 memory slots. In fact, I just ordered 4GB sticks for it 3 days ago.
Ao1 - Your formula for using queue depth and latency to get total IOPS is accurate, but also inaccurate. There's some tricks that the SSD controllers do to boost the IOPS. My guess is caching and pre-retrieving data. But from the most fundamental level, your formula is correct. I believe that the method SSDs use to increase IOPS is the same used to increase the sequential speeds. Theoretically, the random reads and writes should be the same as sequential, with a few exceptions(that being not having TRIMmed space available). The fact that we can get 200MB/sec read sequential but not 200MB/sec read random is interesting to me.
Anyway, good thread. I'll definitely be watching to see what others say in this thread. I'd be interested to see what the same OS imaged onto a single drive and RAID of 2 different model drives would show. I'm not sure what the tested drives were, but testing Intel versus Sandforce would be interesting.
Overall, I would say that the data tends to support my comments in another thread about Intel G3s.. and that is that the most important thing that needs to happen to SSDs is lower price and increased size. Performance-wise, we're doing very good at keeping the CPU full when requesting data from the hard drive.
And if you REALLY want performance, get LOTS of RAM and ditch the pagefile. It is interesting that there isn't as big of a performance change going to RAID0 from a single drive. It would appear that RAID0 is good for increasing total drive size, but very little else unless you are reading/writing massive amounts of data sequentially. There's also a latency penalty that is pretty significant for RAID0. I've noticed the RAID0 penalty when using RAMDrives in RAID0 too.
I don't know about anyone else, but the more RAM slots the better when it comes to buying/building computers. There will be an increase in the usage of the hard drive (±10% or so) but the latency improvement is HUGE. I've always spent the extra $$$ to get X58 instead of X55 motherboards because those 2 ramslots can add 8GB of RAM if you buy 4GB sticks. I've had 24GB in my gaming desktop since 2009 ;).
Kudos to Ao1 for spending the time to gather the data. It's nice to have cold hard facts to back up my observations ;).:clap::clap::clap:
Thank you. It's been hard work, but the real thanks should go out to Overthere for providing a great piece of software to make it possible. :up:
Just a few clarifications. I used two X25-M's. The I/O operation and performance metrics are "from the perspective of applications/processes and the OS (in contrast to the device perspective, i.e., as seen by the actual device "at the other end of the cable")". [Overthere post #16]
overthere
10-30-2010, 12:29 PM
Thank you. It's been hard work, but the real thanks should go out to Overthere for providing a great piece of software to make it possible.
I too would like to thank Ao1 for all of his time and hard work. :up:
And thanks also to Ao1 for his kind compliment again about the hIOmon software.
But as with any tool, the actual use of the tool - and moreover, the user of the tool - are also very important.
Otherwise it's just another tool on the shelf, no matter its merits.
So thanks again to Ao1 - and to the other folks in this forum (such as CT, Anvil, IanB, josh1980, and others) for their comments about and interest in the various topics discussed (and uncovered) in this thread. :clap:
Here is a side by side comparison of HDD vs SSD vs SSD Raid 0 during boot up. (All done with the page file on). It's easy to see why HDD sucks during boot up and I think it puts a perspective on the difference between SSD and SSD raid 0.
I don't think there is anything more I can show now, but it would be nice to see other results from faster set ups.
http://img684.imageshack.us/img684/8996/summaryu.jpg
http://img169.imageshack.us/img169/8851/50465085.jpg
By the way I have used the second SSD as a storage drive. This has allowed me to keep my static data on SSD, so I can at last get rid of my storage HDD. :cool:
I can already see notible beneifts to this. For example I have 60GB of mp3's. When they were on HDD it took forever to retrieve media information and find the playlists. Now it just screams through it and this leaves my OS fully responsive in a way that did not happen when they were on HDD.
Also all my personal folders are defaulted on to the SSD and that also really speeds thing up as well. Signicant noticable improvements over using a Raid 0 with a HDD storage drive.
Now I try to look at the best SSD’s and configurations for the key desktop performance metrics. I dug out the WEI thread to re-look at the WEI sub-scores based on the outcome of the monitoring with hIOmon.
The four key metrics for desktop use that seem to stand out are:
• Responsiveness: Disk Random 16.0 Read (Higher is better)
• Responsiveness: Average IO Rate (Lower is better)
• Responsiveness: Overall (Lower is better)
• Average Read Time with Random Writes (Lower is better)
Unsurprisingly SteveRO’s Acards and Lowfat’s ioXtreme really shine, however it’s interesting that the same SSD’s perform very differently with these metrics on different chipsets. For example my X25-M on ICH9 performed very differently to SteveRO’s X25-M on the 680i chipset.
Also looking at Tiltevros degraded vs fresh scores on the LSI 9211 it is these metrics that get hit the hardest.
Anvil’s post comparing SATA 2.0 & SATA 3.0 with the C300 would seem to suggest that for desktop use the SATA 3.0 would be better despite the fact that maximum latency is through the roof. The total run time is also lower.
CRUCIAL C300 256GB (Marvell 9128 6Gb/s on-board controller GB X58A-UD7)
> Disk Sequential 64.0 Read 353.08 MB/s 7.9
> Disk Random 16.0 Read 258.52 MB/s 7.9
> Responsiveness: Average IO Rate 0.73 ms/IO 7.9
> Responsiveness: Grouped IOs 8.57 units 7.4
> Responsiveness: Long IOs 1.52 units 7.9
> Responsiveness: Overall 13.02 units 7.9
> Responsiveness: PenaltyFactor 0.0
> Disk Sequential 64.0 Write 213.23 MB/s 7.4
> Average Read Time with Sequential Writes 0.887 ms 7.7
> Latency: 95th Percentile 1.546 ms 7.9
> Latency: Maximum 373.757 ms 5.5
> Average Read Time with Random Writes 0.900 ms 7.9
> Total Run Time 00:01:07.91
C300 256GB (ICH10R)
> Disk Sequential 64.0 Read 270.03 MB/s 7.6
> Disk Random 16.0 Read 207.44 MB/s 7.8
> Responsiveness: Average IO Rate 0.70 ms/IO 7.9
> Responsiveness: Grouped IOs 8.70 units 7.4
> Responsiveness: Long IOs 1.58 units 7.9
> Responsiveness: Overall 13.78 units 7.9
> Responsiveness: PenaltyFactor 0.0
> Disk Sequential 64.0 Write 216.43 MB/s 7.4
> Average Read Time with Sequential Writes 0.936 ms 7.7
> Latency: 95th Percentile 1.601 ms 7.9
> Latency: Maximum 1.747 ms 7.9
> Average Read Time with Random Writes 0.934 ms 7.9
> Total Run Time 00:01:09.42
http://img838.imageshack.us/img838/1853/excelupdate.jpg
Anvil
11-01-2010, 08:39 AM
Great work Ao1
I'm still a bit busy but I'll try to find some time to compare C300 64GB single vs raid-0 on my AMD rig.
How did you go on to get the bootlog metrics?
Hi Anvil, to get the bootlog I reset the SSDPerfAnalysisFS script with a report period of 2 minutes. I then rebooted, waited two minutes without doing anything and then took the results from the Presentation Client as soon as they appeared.
I had to use 2 minutes as the HDD took that long after Windows first opened before everything fully loaded. With SSD 30 seconds should be more than enough.
It will be great to see some comparative results. Im over the moon with my set up now. I have the page file switched off and all my data is on SSD. Its really made a difference.
Anvil
11-01-2010, 12:40 PM
Hmm,
It doesn't look like it works on AMD systems?
It installs but I can't get the manager to start!
Nizzen
11-01-2010, 01:28 PM
Is it possible to fill me in to the Winsat chart?
Areca 1880ix-24-4GB + 1x 128gb C300
overthere
11-01-2010, 01:49 PM
Hmm,
It doesn't look like it works on AMD systems?
It installs but I can't get the manager to start!
Hi Anvil,
Please take a look in the Application Event Log to see what entries have been added by the hIOmon software.
In particular, please check to see if there is an entry with a Source of "hIOmonGr" and an Event ID of "499" (which indicates that the hIOmon software has expired).
Is it possible to fill me in to the Winsat chart?
Areca 1880ix-24-4GB + 1x 128gb C300
Insane. :cool: Done.
Anvil
11-01-2010, 03:10 PM
...
"hIOmonGr" and an Event ID of "499" (which indicates that the hIOmon software has expired).
Hi overthere :)
You're spot on, I found that exact entry.
What do I do?
Anvil, the reason I did the bootlog was because it is a relatively fixed work load that is very hard to replicate between different drives with normal use. I think it is also the hardest workload that occurs on most PC’s. Anything exciting post boot is likely to be more about sequential read/ write speeds.
When the pc boots ~ 400MB of mostly random data gets read in a very short time. (Well on SSD at least ;) ) That kind of load is hard to achive with normal use. Special applications are obviously different like VM’s etc.
Edit: When I monitored 30 minutes of using my typical apps I read and wrote loads more data than occured during boot up, but the IOPS were way down. I guess that was because most of what I was doing involved sequential reads, like game loading, music etc.
overthere
11-01-2010, 03:39 PM
Hi overthere :)
You're spot on, I found that exact entry.
What do I do?
Probably the best route (rather than tie up the discussion within this thread) is for me to send a PM to you with instructions.
Sorry for the inconvenience. :(
Anvil
11-01-2010, 04:15 PM
Hi overthere
Thanks for the PM.
No problem at all, will continue testing tomorrow.
Anvil
11-02-2010, 12:23 PM
2 min monitoring
i7 920 UD7, 2R0 C300 64GB, no apps installed
109097 109100
Installing Security Essentials right now and will do a new run shortly. (Done)
AMD w/HDD
Ao1, it looks like the F3 outperforms your HDD.
Not sure about Data Transferred, seems a bit low.
109102
The hard drive I used was a Seagate Barracuda so its going to be slower than the F3 but I think the biggest difference is that my system image was of a fully loaded OS (Win 7/ 64) with apps and games installed. That seems to make big difference as I incurred 400MB reads vs 100MB on you F3.
The total reads/ IOPS on your SSD raid are also around half of mine.
Trying to compare my results with yours is going to be difficult/ impossible without using the same system image. The key however is do you see similar outcomes between different configurations as I did.
Anvil
11-03-2010, 01:49 AM
My setup is a full W7 x64 install + the basic necessities. (all windows updates, AV, java)
The AMD software setup is 100% identical to the Intel setup, except for some drivers for the PERC 6/i and the LSI 9260.
Thats why I found the AMD metrics a bit strange, it's a single drive but the number of MBs should have been closer to the Intel system.
What "startup" programs do you have installed?
I know we won't be able to compare 100% but we could have been much closer if I installed some or most of your apps. (most apps don't/shouldn't make a difference to the bootup process)
Intriguing. I made the image from an OS that I had been using for around 6 months. As you say most apps shouldn’t make a difference to the bootup process. In post 5 I show the Windows Task Manager, which shows most of what is running on that image.
Apart from the difference of what got loaded during boot on my system image maybe the 2GB pagefile helped your HDD? When I ran the boot up on HDD I had the pagefile set to be managed by Windows.
overthere
11-03-2010, 08:49 AM
The AMD software setup is 100% identical to the Intel setup, except for some drivers for the PERC 6/i and the LSI 9260.
Thats why I found the AMD metrics a bit strange, it's a single drive but the number of MBs should have been closer to the Intel system.
For starters, here are a couple of factors that might be considered:
One factor is when the hIOmon software begins monitoring the requested I/O operations and collecting the requested I/O operation performance metrics.
As part of the hIOmon software installation process, the hIOmon software is configured so that the hIOmon Manager service component will, by default, automatically start when the system is started.
In addition, the hIOmon software is configured so that the hIOmon Manager, when started, will automatically load/activate the "default" Filter Selection. (A "Filter Selection" is a hIOmon configuration file that specifies the types of I/O operations that are to be monitored for which particular files, devices, and/or processes, the types of I/O operation performance metrics to collect, etc.)
So in short when using the "SSDPerfAnalysisFS" script, the hIOmon Manager begins capturing the summary I/O operation performance metrics (based upon the "SSDPerfAnalysisFS" Filter Selection) when the hIOmon Manager service is started.
Another factor to consider is that (looking at the screenshots above for the SSD 2R0 C300 and the F3 HDD systems), the summary metrics shown are for the first 2 minutes after the hIOmon software began its collection of the I/O operation performance metrics.
Indeed, the SSD system transferred 199,837,696 read bytes during this 2 minute period (as opposed to the 102,762,496 read bytes transferred with the F3 HDD system) - nearly double.
And the SSD system also performed 10,868 read I/O operations (in order to transfer the 200MB of data), which was nearly twice the number of read I/O operations (5,662) performed upon F3 HDD system.
Moreover, a much higher percentage (98%) of the read I/O operations upon the SSD system were fast I/O operations (i.e., completed in less than one millisecond), where upon the F3 HDD system the fast I/O operation percentage was only 57.4%
Overall, the read I/O operation activity (e.g., minimum and average response times) were an order of magnitude (or more) faster upon the SSD system.
So the SSD system was basically able to read in twice the amount of data during the 2 minute time period.
A deeper analysis would be required to determine what exactly was being read in (e.g., which files) and by whom (e.g., which processes) during the 2 minute period.
This also applies to the write I/O operation activity, where there was more write data transferred upon the F3 HDD system (21MB) than the SSD system (15MB).
Anvil
11-03-2010, 02:52 PM
Hi overthere
I know that there's more to it :), the OS can't boot off the "hIOmon software" and the hIOmon software can't be the first service to start.
I expect that the bootup metrics would generally be misleading when comparing installs across different computers.
It should be consistent for that exact pc/install though. (still there will be activities that are not captured/monitored due to the nature of booting)
Booting is not what brought me to trying the software though.
It does bring quite a lot of interesting points to the table (also while booting), like QD, comparing: seq. Vs random io, pagefile vs no pagefile, single drives vs raid and I would expect for determining optimal strip sizes.
I've found hIOmon to be a great tool for capturing/monitoring tasks.
I'll be comparing i/o on VMs on both HDDs and SSDs on different controllers this weekend, should be interesting and probably very time consuming :)
Regarding my HDD vs SSD bootup metrics
The F3 boots in about 21-22 seconds (using boottimer) and so the period to monitor could easily have been set to 60 seconds (or less), I don't think that would have changed anything, that is unless there were some random scheduled activities that started during the monitoring period. (i.e. windows update, AV scan, ...)
Like you've already mentioned, it's all down to when the monitoring starts.
Both the SSD and the HDD are capable of reading/writing TBs of data during the 2 minute monitoring period so that's a moot point wrt to comparing total i/o during bootup.
Looking at the idle/busy time percentages confirms that most of the time it's just idling during the 2 minute period.
(and that the idle part is in favour of the SSDs :))
overthere
11-03-2010, 03:59 PM
I know that there's more to it :), the OS can't boot off the "hIOmon software" and the hIOmon software can't be the first service to start.
I expect that the bootup metrics would generally be misleading when comparing installs across different computers.
It should be consistent for that exact pc/install though. (still there will be activities that are not captured/monitored due to the nature of booting)
...
I'll be comparing i/o on VMs on both HDDs and SSDs on different controllers this weekend, should be interesting and probably very time consuming :)
...
Looking at the idle/busy time percentages confirms that most of the time it's just idling during the 2 minute period.
(and that the idle part is in favour of the SSDs :))
Hi Anvil,
I would agree that there's more to it! ;)
BTW, the hIOmon software does provide an option to collect I/O trace and/or summary metrics during the actual system boot/start process.
I briefly mentioned this option in a prior post (#32), but let me know if you need any additional information:
http://www.xtremesystems.org/forums/showpost.php?p=4595909&postcount=32
In any case, the booting process should be fairly consistent for the exact same pc/install as you said (given the same login timing, no "extraneous" activities performed by the startup tasks, etc.).
Good observation by you about the idle/busy time percentages. :up:
And the various metrics captured by the hIOmon software (e.g., the random/sequential access metrics) can indeed be helpful in a variety of tasks (such as VM performance analyses as you mentioned).
Looking forward to hearing about your VM I/O comparisons!
My setup is a full W7 x64 install + the basic necessities. (all windows updates, AV, java)
The AMD software setup is 100% identical to the Intel setup, except for some drivers for the PERC 6/i and the LSI 9260.
Thats why I found the AMD metrics a bit strange, it's a single drive but the number of MBs should have been closer to the Intel system.
What "startup" programs do you have installed?
I know we won't be able to compare 100% but we could have been much closer if I installed some or most of your apps. (most apps don't/shouldn't make a difference to the bootup process)
Thanks to Lsdmeasap (http://www.xtremesystems.org/forums/showthread.php?t=261791) I found a useful little app that monitors the boot up process. Im now running a fairly fresh OS and I have not installed all the apps yet, however my CPU is already maxing out.
When I get a chance I will reinstall the original system image to see the impact on the CPU. It would seem that the amount and types of apps installed actually have quite a big impact on the OS boot up process.
http://img709.imageshack.us/img709/5584/ssdl.jpg
Regarding my HDD vs SSD bootup metrics
The F3 boots in about 21-22 seconds (using boottimer) and so the period to monitor could easily have been set to 60 seconds (or less), I don't think that would have changed anything, that is unless there were some random scheduled activities that started during the monitoring period. (i.e. windows update, AV scan, ...)
Like you've already mentioned, it's all down to when the monitoring starts.
Both the SSD and the HDD are capable of reading/writing TBs of data during the 2 minute monitoring period so that's a moot point wrt to comparing total i/o during bootup.
Looking at the idle/busy time percentages confirms that most of the time it's just idling during the 2 minute period.
(and that the idle part is in favour of the SSDs :))
I found with HDD that there is a BIG difference between Windows first appearing and being fully loaded. With HDD you can hear the drive clunking away well after Windows first appears and if I made the test period less than 2 minutes I could not get a comparable data load.
Not related but when I used hard raid with cache it appeared that Windows reports cache speed. It reports that data is transferred when in fact the data is still being transferred from cache to the hard drive. That is one benefit of HDD; you can listen to what is going on.
I'm really looking forward to seeing your analysis with VM's :up:
Here I run Winbootinfo using the system image from my previous monitoring and compare it to a new OS install with less apps loaded.
Winbootinfo reports less read data than hIOmon - ~337MB vs 420MB.
I'm not really worried about this because what I did with hIOmon was comparable in terms of data transfer between SSD and HDD, which was my primary objective.
What this does show however is that the apps you have loaded make a huge difference to boot times.
The CPU graph was way too long to post due to the time it took the HDD to load, but interestingly the CPU was maxed out a lot more over a much longer duration.
http://img820.imageshack.us/img820/9060/hddvsssdnewandold.jpg
Anvil
11-05-2010, 02:46 AM
I noticed the boot analysis app in the C300 review, it looked OK :)
I'll give it a try and I might also follow up on overtheres comment about the hIOmon bootup monitoring.
I thought I was already using the correct method for monitoring bootup using hIOmon :)
My install on the F3 was about 10 days old at the time of testing so it's a pretty fresh/clean install compared to your setup.
edit:
I've downloaded the app, will try later today.
Looking at Lsdmeasap's result I'd guess it was done on a fresh install; only 52MB read data. I used to think it was fragmentation that slowed down HDD boot times over time, but it seems that its more related to the increase in extra files that get loaded over time. That doesn't seem to make much difference with SSD because they are so fast in comparison.
http://img534.imageshack.us/img534/3224/benchmark.jpg
Edit:
hIOmon provides a much more sophisticated boot analysis tool to the way I under took the comparison. I was not so interested in the boot process but the difference in performance with the same workload. In that context I think what I did was valid as it managed to replicate a very similar work load for comparative analysis between HDD/ SSD and SSD raid 0.
Anvil
11-05-2010, 11:49 AM
AMD rig, Samsung F3, 1090T
According to WinBootInfo the system boots in 35,7 sec
Enabling all cores in MSConfig shaves off ~3 seconds
109169
Anvil
11-06-2010, 03:26 PM
VMWare hIOmon testing.
Setup
X58-UD7, Ci7 920@4, 12GB RAM, boot drive 2R0 C300 64GB
Controllers : ICH, PERC 6/i, LSI 9260-8i
Drives used for testing VMs
2R0 Intel G2 160GB G2 (LSI)
4R5 Samsung F3 1TB (PERC)
WD20EARS (ICH)
Vertex 2 60GB (ICH)
VM used : W7 Enterprise, a development VM using 3 drives, a total of 23,5GB
The test consists of starting the VM, loading the development environment and performing a complete build of an application (~1.1M codelines), then shutdown the VM.
hIOmon is set for capturing every 8 minutes. (using the Presentation Client)
The results are somewhat close to what I expected.
I'll compile a spreadsheet for the most interesting metrics.
For now, here are the screenshots.
109205 109206
109207 109208
Summary
109225
A note about the 1.2798s max response time on the Vertex 2
The response time is due to TRIM operations logged in the Control I/O section of hIOmon, refer to read/write I/O for max responsetime during normal operations.
Youve got to hand it to ICH when it comes to SSD. Its standing its ground with a single SSD against a hard raid array with 512MB DDRII cache.
Anvil
11-07-2010, 05:48 AM
Ao1,
I've uploaded a summary.
I knew the Vertex 2 would be very close to the LSI array, as long as the V2 is not in steady state it simply rocks :)
I don't have any SF drives that are in steady state right now, I've recently cleaned all of them.
Some comments about the runs.
I think the numbers speak for themselves.
QD never gets to build up on SSDs vs HDDs, the rest is as expected.
The Busy time is a great metric, it clearly shows the time spent on I/O.
The HDDs are simply terrible compared to any SSD I've got.
Loading a VM using an SSD is a joy compared to any HDD setup, whether it's a single HDD or an array, SSDs are in a different league.
The Vertex 2 and the LSI 9260 setup feels very close in terms of responsiveness, however this is running 1 VM only.
I normally run 3-4 VMs simultaneously, the difference between running 1 VM vs 2-3 VMs on the ICH is huge compared to running the same number of VMs on the LSI or Areca.
overthere
11-07-2010, 08:51 AM
Hi Anvil,
Great job on your VMware hIOmon testing and on your reporting of the result! :up:
One quick observation on the results that you posted:
Within the summary table, the "Totals I/O resp Max" for the "Vertex2 60GB" is shown as "1.2798s".
This is technically correct since there was a control I/O operation with the "Vertex2 60GB" that took 1.2798563 seconds as observed by the hIOmon software.
I assume that you used the "SSDPerfAnalysisFS" script to configure the hIOmon software. This script configures the hIOmon software to collect I/O operation performance metrics related to the "TRIM" control I/O operation.
As a result, the metrics shown within the hIOmon Presentation Client summary displays under the "Control I/O Operations" section reflect the TRIM control I/O operations as observed by the hIOmon software. (Please note that there are display options available that enable additional TRIM-specific metrics to be shown within the hIOmon Presentation Client display).
In any case, the observations of the "Control" I/O operations are included within the "Total I/O Operations" section shown near the bottom of the hIOmon Presentation "summary" display.
And consequently the "Vertex2 60GB" has a "1.2798s" maximum response time (since the "totals" include the control I/O operations together with the read and the write I/O operations).
So technically the "1.2798s" is correct for the "totals" maximum response time for the "Vertex2 60GB".
However from the perspective of only read and write I/O operations, the maximum response time for the "Vertex2 60GB" is 32.9417ms (as also shown within your summary table).
Thanks again for your time and efforts on this! :up:
Anvil
11-07-2010, 09:37 AM
Hi overthere,
Yes, I used the SSDPerfAnalysisFS on all drives, including HDDs.
I assumed that the Control I/O was related to TRIM and was going to ask you about that later.
In any case, it's relevant and has to be included, TRIM does interfere with other i/o. (read slowdown)
In case I do more testing using the same VM it will be reflected on any TRIM capable SSD, as long as it's used in single mode.
I believe Ao1 disabled TRIM on his W7 install and rather did the cleaning using the SSD Toolbox on the Intels.
Where do I enable the additional TRIM metrics?
Hi overthere,
If you use the fsutil behavior query disabledeletenotify command to disable the OS from issuing TRIM commands would this also disable hIOmon from registering TRIM control I/O operations?
I guess is you are in raid configuration (and cant pass the TRIM command) it would be better to disable it (?)
Hi overthere,
I believe Ao1 disabled TRIM on his W7 install and rather did the cleaning using the SSD Toolbox on the Intels.
I left TRIM on and didn't use the toolbox as this is how I normally use my system.
Anvil, I don't doubt that 2 or more VM's would start to slow down on one SSD, but I intrigued to see what load can be thrown at a single drive before it starts to slow down and where the slow down occurs. Would it be loads of work to run two then three, maybe fours VM's? Does the CPU have to work a lot harder on a single SSD when compared to hard raid?
It would also be great to see how steady state performance differs from fresh when comparing an X25-M, Vertex 2 & C300. I know I'm a cheeky bu**er ;)
Here's a quick side by side of my observations from the boot up monitoring.
http://img714.imageshack.us/img714/3982/comparative.png
overthere
11-07-2010, 10:19 AM
Yes, I used the SSDPerfAnalysisFS on all drives, including HDDs.
I assumed that the Control I/O was related to TRIM and was going to ask you about that later.
In any case, it's relevant and has to be included, TRIM does interfere with other i/o. (read slowdown)
In case I do more testing using the same VM it will be reflected on any TRIM capable SSD, as long as it's used in single mode.
I believe Ao1 disabled TRIM on his W7 install and rather did the cleaning using the SSD Toolbox on the Intels.
Where do I enable the additional TRIM metrics?
I can see where you would want to include the TRIM control I/O operation metrics.
My main concern in my prior post was that some folks might be puzzled as to the "1.2798s" value shown for the "Vertex2 60GB" within the "Totals I/O resp Max" entry in your summary table.
I thought that it might be helpful to provide a brief explanation as to where this value came from (and that it was not some bogus value).
Anyway, to enable the display of the additional TRIM metrics within the hIOmon Presentation Client "summary" display, please perform the following steps:
1) Down in the lefthand corner at the bottom of the hIOmon Presentation Client "summary" display you will find two buttons: "Table View" and "Options".
2) Click upon the "Options" button (located immediately to the right of the "Table Views" option), which will cause a new hIOmon "Display List Options" window to be displayed.
3) Near the bottom of this "Display List Options" window you will see a section called "I/O Summary metric types to be displayed:"
There is a checkbox for "Control I/O Operations", under which there will be an additional option called "Include MDSA metrics".
Enable/select the "Include MDSA metrics" checkbox so that the additional metrics related to "Manage Data Set Attributes (MDSA)" control I/O operations (which include TRIM commands) will also be displayed within the hIOmon Presentation Client "summary" displays.
Anvil
11-07-2010, 10:23 AM
Ao1
I'll see what I can do about the steady-state performance, it takes a while to get the SF drive into steady state (compression) and to provoke steady state on the SF would also invoke DuraWrite.
The issue with comparing more than 1 VM (on any number of drives) is to get comparable results, not that easy to achieve for such a test, I might give it a try though.
I'm almost done with the WD5000BEKT (2.5" 7200rpm drive), I'll post the metrics shortly.
Anvil
11-07-2010, 10:45 AM
I can see where you would want to include the TRIM control I/O operation metrics.
My main concern in my prior post was that some folks might be puzzled as to the "1.2798s" value shown for the "Vertex2 60GB" within the "Totals I/O resp Max" entry in your summary table.
...
I'll make a note about the 1.2s response time :)
I found the MDSA option, I'll do a similar test on the V2 with the option enabled.
overthere
11-07-2010, 10:49 AM
Hi overthere,
If you use the fsutil behavior query disabledeletenotify command to disable the OS from issuing TRIM commands would this also disable hIOmon from registering TRIM control I/O operations?
I guess is you are in raid configuration (and cant pass the TRIM command) it would be better to disable it (?)
Hi Ao1,
The short answer is that the use of the "fsutil" utility does not change the configuration of the hIOmon software.
Consequently, if the hIOmon software is configured to capture I/O operation performance metrics for MDSA control I/O operations (e.g., TRIM commands), then the hIOmon software will continue to collect such metrics if it observes the TRIM commands.
But if fsutil is used to disable the OS from issuing TRIM commands, then the hIOmon software should not, of course, observe any TRIM commands.
Regarding whether to use fsutil in RAID configurations so as to disable the issuance of TRIM commands by the OS, leaving TRIM enabled is basically negligible.
That is, the OS will issue a single TRIM command to the physical device (if TRIM commands are enabled to be issued by the OS). If this TRIM control I/O operation is not successfully completed by the physical device, then the OS will not subsequently issue any other TRIM commands to the physical device.
You can see this effect in Anvil's hIOmon Presentation Client "summary" display excerpts in those cases where a HDD/RAID was used.
Anvil
11-07-2010, 10:58 AM
I've added the "fastest?" 2.5" non-hybrid HDD to the mix.
109248
It does pretty well compared to the 5400rpm EARS drive.
A new summary is on it's way.
overthere
11-07-2010, 11:09 AM
I'll make a note about the 1.2s response time :)
I found the MDSA option, I'll do a similar test on the V2 with the option enabled.
Sounds good. :up:
Just to be clear to other folks that might be interested in the use the "MDSA" display option provided by the hIOmon Presentation Client "summary" display:
This option only pertains to the display of the MDSA-related metrics within the hIOmon Presentation Client "summary" metrics display.
That is, this option does not change the collection (by the hIOmon software) of the MDSA-related I/O operation performance metrics (e.g., those associated with TRIM commands).
Basically, this MDSA display option is provided to help reduce the overall size of the hIOmon Presentation Client "summary" metrics display.
In other words, the MDSA-related metrics can be displayed only when you want them included within the hIOmon Presentation Client "summary" metrics display.
Hopefully I haven't made things confusing. :)
Anvil
11-07-2010, 11:23 AM
New summary that includes the WD5000BEKT
109250
I've started testing Seagate Momentus XT.
(the last drive to be tested this time)
Anvil, what I was interested in seeing is if the degradation that can be seen in a benchmark has any impact on real life applications that are not pushing the drive to its limits. In other words whilst degradation can be quantified easily with a benchmark can hIOmon quantify the impact in something like a single VM. I suspect average response times would increase but I think overall the impact would be minimal. i.e nowhere near as scarry as a benchmark might indicate.
Thanks for all the testing by the way. :) It’s nice to be able to see some comparative results. :up:
Looking forward to seeing the Momentus XT
Anvil
11-07-2010, 12:15 PM
Ao1,
It is quite noticeable on the Vertex 2, once settled in (steady state) one doesn't need to run a benchmark to notice the change.
I have to say that I haven't experienced this using the latest firmware 1.23, my latest experience with this phenomenon was using FW 1.10.
(I do run 1.23 on all SF drives)
Momentus XT coming up now :)
Anvil
11-07-2010, 12:17 PM
Momentus XT (the so called Hybrid drive)
I've included both the 1st and 2nd run as there is a remarkable change in "Busy time" and the fact that 2nd runs is what this drive is "all" about.
109251 109252
Summary
109253
Anvil
11-07-2010, 01:17 PM
overthere,
This is the Control I/O w/MSDA metrics
I expect this could be the snapshot that's deleted.
109254
overthere
11-07-2010, 02:39 PM
This is the Control I/O w/MSDA metrics
For those curious about what these MDSA-related metrics reflect:
There were 5 MDSA control I/O operations that requested a TRIM
The total combined number of "Data Set Ranges (DSR)" specified by these 5 TRIM control I/O operations was 19.
A DSR identifies the starting offset (i.e., essentially the starting block address for the particular set of blocks/sectors) along with the overall combined length of the blocks/sectors to be "trimmed".
The total combined length of the DSRs specified by these 5 TRIM commands was 1,611,800,576 bytes.
The smallest DSR length observed amongst all of these DSRs was 4,096 bytes
The largest DSR length observed amongst all of these DSRs was 1,610,612,736 bytes
The lowest starting address (in bytes) observed amongst all of these DSRs was 1,232,896
The highest starting address (in bytes) observed amongst all of these DSRs was 3,180,806,144
The minimum total number of DSRs specified by a single TRIM control I/O operation was 1
The maximum total number of DSRs specified by a single TRIM control I/O operation was 8
The minimum total combined lengths of the DSRs specified by a single TRIM control I/O operation was 53,248 bytes
The maximum total combined lengths of the DSRs specified by a single TRIM control I/O operation was 1,610,862,592 bytes
Anvil
11-07-2010, 03:15 PM
VMWare creates a 1.5GB snapshot (matches the size of the largest DSR) + 4 LCK files @ 4KB each.
I added the Seagate 7200.12 1TB and the WD VelociRaptor (WD VR300) to the chart. Unless someone wants to have a look at the screenshots I won't upload the screenshots.
New summary
109257
Interesting. The Momentus XT was faster than the VelociRaptor, even on the first run.
Anvil
11-08-2010, 01:44 AM
There is a run 0 where I just boot the OS.
It's needed as VMWare reacts to the VM being "moved" to a new drive.
The VR isn't what it used to be, the new 600GB edition is certainly faster than the old one. (up to 20-30% on some tasks)
Tiltevros
11-08-2010, 01:51 AM
45 days old OS
For those curious about what these MDSA-related metrics reflect:
The largest DSR length observed amongst all of these DSRs was 1,610,612,736 bytes
So in this instance the OS issued a single command to the SSD to wipe ~ 1.5 gigabytes of “deleted” data? I assume the SSD controller then waits for a quiet period and then deletes the data when it will have no/ less impact.
[Tiltevros] 16 Cores :cool:
I have created a separate thread for Winbootinfo otherwise this thread will end up way off topic.
http://www.xtremesystems.org/forums/showthread.php?t=261924
Anvil
11-08-2010, 11:59 AM
The WD SiliconEdge is no slouch :)
It doesn't scale with QD, and so it's on par with HDDs in that regard, still it is a great drive imho.
109293
New chart...
109294
overthere
11-08-2010, 02:49 PM
So in this instance the OS issued a single command to the SSD to wipe ~ 1.5 gigabytes of deleted data? I assume the SSD controller then waits for a quiet period and then deletes the data when it will have no/ less impact.
Yes, and that is certainly one way that the SSD controller could handle the "backend" processing of the TRIM commands from the OS. :)
By "backend", I mean the actual manner in which (and when) the SSD controller performs various flash management tasks (e.g., wear-leveling) based upon the DSRs that it has received from the OS.
As you know, essentially the TRIM commands currently provide a way for the OS to notify the SSD that a specific set of (logical) blocks/sectors upon the SSD are considered by the file system to be "deleted data blocks" (that is, sectors upon the SSD that the file system considers to no longer contain valid data).
But as to what exactly the SSD does with these "trimmed" blocks/sectors (and when), this is, I believe, basically "vendor-specific" at the current time.
Anvil
11-08-2010, 03:27 PM
This "old-timer" is still doing great :)
(it was formatted prior to the test and so it is not in steady state)
109296
Anvil
11-08-2010, 04:19 PM
2R0 Intel X25-M 160GB G2
Hmmm :)
109298
109300
Here I monitor Call of Duty Black Ops Single Player for 30 Minutes. I monitored the specific folder Call of Duty at a 1 minute interval. The first surprise is the amount of processes that were generated, which are summarised below. All contributed to numerous random and sequential read requests over the 30 minute duration to varying degrees. 64 processes generated 1,127 individual reads.
Games are not all about sequential reads, random read speeds seem to also be important.
http://img221.imageshack.us/img221/2621/processes.jpg
Read speeds
http://img831.imageshack.us/img831/3639/blackopsv.jpg
IOP Count
http://img577.imageshack.us/img577/7136/iops.jpg
Fast Read IOP count. Worst case percentage = 1%. Average 98%.
http://img818.imageshack.us/img818/2651/fastreadsiops.jpg
Queue Depth - Maximum Max = 5
http://img213.imageshack.us/img213/8509/qdepth.jpg
Here I show around 1 ½ hours of activity on the page file with the size set automatically by Windows.
During the monitoring period I ran most of the typical programs I use all the time, including:
Large and small Photoshop files (2MB to 2.3GB)
Web browsing with 7 pages open. (IE9 and Chrome).
Traktor Scratch
WMP
Checked emails and cleaned up the in box
Black Ops MP
Worked on an Word document
Worked on an Excel document
On average Windows Task Manager recorded around 64 process running.
OS = Win 7/64 with 8GB RAM.
http://img826.imageshack.us/img826/4349/pagefileactivity.jpg
The figures tell the story. :clap:
You have 8GB of memory, the biggest file you opened in Photoshop was around a quarter of that, yet despite all that spare RAM the system saw fit to dump a piddly 280Mb of data to pagefile - just in case? But this data clearly wasn't needed again, since it only read back in 90Mb tops. :ROTF:
The pagefile activity you showed was totally redundant. The system wastes time using it if it is there, so it's simpler and more efficient to monitor your maximum Commit Charge for your working set of apps and data over time, then make sure your RAM is more than enough to cover that and turn the pagefile off instead.
Plus, anyone who follows the archaic Microsoft dogma of sizing the pagefile as 1.5 x RAM (with large amounts of installed RAM) is a fool who's simply wasting disk space. :shakes: Just turn off the system error dump since you'll never use it if you do ever bluescreen.
EDIT: I admit, I ran out of memory the other day. :( But that was a fault with Media Player Classic buffering a large, damaged video when I tried to skip through it. I watched the Virtual Memory Commit Charge spiral out of control in Process Explorer, so it was a bug in Media Player or one of its components, not a fault in my choice of turning off pagefile, since likely it would have continued on to use whatever pagefile I'd provided too and hit the limit of that.
There will always be exceptions, but this doesn't change the obvious best practise that Ao1's example proves.
EDIT: I admit, I ran out of memory the other day. :(
That is the only reason I can think that MS would still recommend using the pagefile. What I could see over 1 1/2 hours is exactly as MS describe for why you should use the pagefile with SSD. A predominance of small random reads and large sequential writes, which an SSD can handle very well. The bottom line however is that RAM can hangle it better and it saves data being pointlessly writtten to the SSD.
Ive left the pagefile.sys script running so later today I will post the Excel summary sheet for anyone that is interested.
On a separate issue here is a summary of Anvils monitoring.
http://img5.imageshack.us/img5/9882/anvilsummary.jpg
Attached is the pagefile log after 7 hours of use.
That is the only reason I can think that MS would still recommend using the pagefile.
To cope with buggy programs that develop a 4GB memory leak in seconds, and likely wouldn't have stopped leaking if I'd had gigs more? :shrug:
Here I monitor Call of Duty Black Ops Multi player.
http://img213.imageshack.us/img213/1705/blackopsmpprocesses.jpg
http://img243.imageshack.us/img243/3025/summary.jpg
Anvil
11-13-2010, 05:39 AM
Thanks for the chart(s) Ao1,
I've started testing on a single C300 256GB, looks like we might have a new champion :)
(not by much though)
I might try using the onboard Marvell controller (6Gb/s), should be an interesting compare to the ICH.
Anvil
11-13-2010, 06:27 AM
109440
109445
Anvil, here is a quick summary of the key performance metrics, including the C300. It’s easy to see where the G3 DRAM will come in handy. ;)
http://img35.imageshack.us/img35/6917/88195000.png
Anvil
11-13-2010, 11:09 AM
Thanks again Ao1, great charts!
I've almost given up on getting the Marvell 6Gb/s onboard controller working and so I popped in the 9211 to get some 6Gb/s "results" using the C300
Max response times aren't that good but all in all it is the best combo, so far :)
109449
I'll try another driver for the Marvell controller, if that doesn't work out that is the end of testing the C300 as a single drive.
edit:
Finally found a driver for the 9128 that works, looks OK for a single drive.
109452
Nice one Anvil. :up: It’s a real shame that there is not a good SATA 3.0 controller out yet for the C300. :(
I’ve updated the chart above to include the latest runs. Marvell really hit the fast IOP count and added loads to the response times.
SteveRo, any chance of persuading you to do a PCMV run and posting the log file? It would be great to see the log file entry that PCMV generated. :surf:
Anvil
11-13-2010, 03:22 PM
I had to try 3R0, the Intels are the first ones to try.
There's no doubt about it, nR0 really does improve performance.
109458
109459
Anvil
11-14-2010, 03:27 AM
Ao1
I see you have updated the charts.
I tried running 3R0 software raid on the 9211 and everything was OK with the exception of one thing.
HIOmon wouldn't accept the volume as a valid device.
Not a big issue but it would have been an interesting run.
Anvil
11-14-2010, 03:46 AM
There are a lot of variables for the pcmv score.
Raid-0 would have helped a lot.
Could you try running the hdd suite?
I might have a go with the pcmv on my 2R0 C300 64GB later today.
Anvil
11-14-2010, 04:38 AM
Here is the standard PCMark Suite test.
It does include booting and so it's not a "pure" pcmv run.
109482
QD is somewhat high on reads, max responsetime is sky high for some reason.
Anvil, on the trial version of PCMV it is not possible to select the storage bench :(
To monitor folders I have attached a quick guide to help get up and running quickly. I set the monitoring period to 1 minute for PCMV, but it can be set to whatever you want. Depending on what you are monitoring however it will generate a huge amount of data entries due to the amout of processes that get generated when the app being monitored is executed.
Anvil
11-14-2010, 06:19 AM
I've recorded an HDD Suite using the PM, I'll have a look at you guide later and do a re-run
Max QD was 77 which is quite high. (read)
Busy time was merely 16-17 seconds
overthere
11-14-2010, 08:39 AM
Here is a run monitoring PCVM with only the storage benchmark. No writes were generated at all and there was not much reading, especially random reading.
Hi Ao1,
Perhaps PCVM is using some other temporary directory as part of its actual benchmarking I/O operation activity.
Anvil
11-14-2010, 09:05 AM
Here's my HDD suite test. (for reference)
The test is performed on drive I. (not the OS drive)
109488
Anvil
11-14-2010, 09:33 AM
Here is the log file from the HDD suite test.
(I don't have Excel installed on this computer and so I haven't looked at it yet)
Ao1,
As overthere noted, there are other directories that needs monitoring, not sure of where they are located, they are created on the fly during the benchmark.
Monitoring C:\* for a single run should do the trick. (for finding where it processes the data)
Dang, your right. It creates a temp file that does not get monitored from the folder. I've consequently deleted post 173 & 178.
OK, this looks more like it. I ran the storage benchmark only and only changed the target drive setting. The drive tested was my storage drive with very little free space. The score dropped to 27,722 compared to 30,963 on the C drive. Now we can start to see a real difference between a single ssd and a raid array ;)
http://img263.imageshack.us/img263/2586/pcvmstorage.png
Anvil
11-14-2010, 12:09 PM
This is my latest run, let's see if the numbers compare
HDD Suite, target drive : I: (3R0 Intel G2 SS16KB)
Score : 83356
Anvil, the data transferred is not an exact match, especially on the writes, but its clear enough to see a story.
http://img26.imageshack.us/img26/4701/anvilcomp.png
Anvil
11-14-2010, 01:49 PM
Yes, there's a huge difference on writes, quite a bit on reads as well but not that spectacular.
If you look at the seq/random write distribution on my run vs yours there is something odd going on, most of my writes are sequential.
QD was quite the surprise, there is nothing normal about QD ~80
Was this my latest run?
Anvil
11-14-2010, 02:48 PM
I had an issue monitoring software raids a few post back.
overthere informed me about using drive letter + ? (i.e I? for drive I) for software raids and it worked :)
Problem solved!
Notice that I've highlighted Control I/O (TRIM related)
109502
TRIM does work on Software RAIDS in W7 :)
109503
overthere
11-14-2010, 08:19 PM
I had an issue monitoring software raids a few post back.
overthere informed me about using drive letter + ? (i.e I? for drive I) for software raids and it worked :)
Problem solved!
Notice that I've highlighted Control I/O (TRIM related)
TRIM does work on Software RAIDS in W7 :)
Hi Anvil,
Good to hear that the "workaround" helped. :)
In the case of the hIOmon "SSD Performance Analysis" script, basically what this "workaround" does is to configure the hIOmon software so that the "physical volume" (rather than the "physical device") associated with the specified Logical Drive/Disk (i.e., logical disk "I" in your case) is monitored by the hIOmon software.
This is why you see "\Device\HarddiskVolume5" shown within the hIOmon Presentation Client display (rather than, say, "\Device\Harddisk1\DR1", which reflects "physical device/disk number 1").
There is another nuance, specifically with TRIM, that I would also like to mention here.
Basically, the software driver at the physical volume level within the I/O stack can honor a TRIM command for a physical volume whose associated physical device does not successfully process the TRIM command.
That is, the hIOmon software can observe the completion status of the TRIM I/O operations performed at the physical volume level as having been successfully performed by the software driver (and even though these TRIM commands were not actually passed downed to the physical device).
You can verify this TRIM nuance with a HDD running under Win 7. The hIOmon software can observe the "successful" completion of the TRIM commands at the physical volume level, but with the (single) attempt of a TRIM command at the physical device level observed by the hIOmon software as being unsuccessful - so consequently no additional TRIM commands will be passed down to the physical device.
BTW, this is also why the hIOmon "SSD Performance Analysis" script - by default - always configures the hIOmon software to monitor the specified "physical device" (rather than the "physical volume").
A few people have said they can't open the quick set up guide I posted in post #175, so I have re-attached in Word docx format in a new zip file. Due to attachment limit I can't zip in a pdf format, but if you still can't open it feel free to PM me and I will email it to you.
You can verify this TRIM nuance with a HDD running under Win 7. The hIOmon software can observe the "successful" completion of the TRIM commands at the physical volume level, but with the (single) attempt of a TRIM command at the physical device level observed by the hIOmon software as being unsuccessful - so consequently no additional TRIM commands will be passed down to the physical device.
Interesting. I thought that Win 7 only sent a TRIM command if the storage device reported a non spinnning status :shrug:
overthere
11-22-2010, 08:49 AM
Interesting. I thought that Win 7 only sent a TRIM command if the storage device reported a non spinnning status :shrug:
Hi Ao1,
I recall seeing something somewhere along the lines of your comment above.
But using the SSDPerfAnalysis script along with the "x?" option (to request that the physical volume associated with the Logical Disk "x" be monitored rather than the associated physical device), I have re-confirmed (using a VMware virtual machine running Win7-x64 with virtual HDDs) that TRIM commands are seen by the hIOmon software as being successfully processed at the physical volume level within the operating system I/O stack.
I will continue to revisit this some more ...
Here I monitor boot up with a little more precision. (For me anyway ;) ) Total boot up read transfers = 53MB. One and a half hours later doing nothing but work with Excel I had 1,573MB of read transfers and 101MB of writes. :eek:
This chart shows the read transfer sizes. Less than 200 bytes = 920. More than 200, but less than 400 = 233 etc.
http://img84.imageshack.us/img84/1254/44550014.jpg
In total 2,772 read transfers were recorded. Only 16 were above 1MB:
http://img821.imageshack.us/img821/3882/73640896.jpg
Anvil
11-22-2010, 02:14 PM
I knew that writes would be low but 101MB is almost nothing, maybe W7 is super efficient after all.
Doing fw updates on my C300s, if all goes well I might post some info tonight about monitoring software raid, each drive individually :) and the "drive" as a whole.
(got a special delivery from overthere for monitoring the drives individually while in raid :up:)
overthere
11-23-2010, 11:09 AM
... using the SSDPerfAnalysis script along with the "x?" option (to request that the physical volume associated with the Logical Disk "x" be monitored rather than the associated physical device), I have re-confirmed (using a VMware virtual machine running Win7-x64 with virtual HDDs) that TRIM commands are seen by the hIOmon software as being successfully processed at the physical volume level within the operating system I/O stack.
I will continue to revisit this some more ...
To help dispel the notion/concern that the observation by hIOmon (see my post #189 above) that "Win 7 successfully issues TRIM commands to HDD-backed physical volumes" was somehow due to running within a virtual machine environment, yesterday evening I ran the hIOmon software upon a "native" Windows 7-x64 system (i.e., a laptop with a single HDD and only Win 7-64x installed) just to make sure nothing has changed (i.e., from prior testing by hyperI/O).
The results were the same: multiple "successful" TRIM commands observed at the physical volume level within the operating system I/O stack (but none at the physical device level).
This re-confirmed once again (as was first noticed well over a year ago :)) that Windows 7 does successfully issue TRIM commands at the physical volume level even for a HDD.
The ability to discretely monitor I/O operations at the physical volume level (in addition to the physical device level) helps bring this nuance of TRIM within Win7 to light.
(got a special delivery from overthere for monitoring the drives individually while in raid :up:)
That should be interesting. others have posted that writes are far from evenly distributed on 2 drives on a raid 0 array.
Ive been wondering why Windows had a big variation in the boot up data that got loaded from my previous posts so I ran a check on the HDD image I kept that was made from an old OS.
It seems that the boot up processes consists of core files only. Once the boot process is complete significantly more files get read and they are mostly very small random reads. Within around 15 minutes of the desktop first appearing a further ~1.5GB of data is read, which consists of program, user and Windows files. This is without doing anything and it happened on a relatively new OS and an old OS.
Previously when I monitored boot up I did it on a time basis after I thought Windows had finished loading, but clearly things keep getting loaded well after the Windows desktop is launched.
This explains why I was getting 400MB reads, as I was capturing data a couple of minutes after Windows had first loaded.
On a seperate issue the Intel mod recently posted this on their forum:
Intel SSDs do not have idle time garbage collection (ITGC) or background garbage collection (BGC) Some SSD controllers like Indilinx Barefoot, SandForce SF-1200, or Samsung S3C29RBB01-YK40 do have this feature. This is an internal feature of the drive. The controller determines when it is idle for long enough and then does a file system compare against what is actually valid on the drive. It then cleans up the dirty pages. The actually implementation of garbage varies between controllers though. The downside to idle/background garbage collection is write amplification as it may perform unnecessary writes.
While Intel SSDs do not idle/background garbage collection, they have are very resilient to dirty pages. I believe this is what you are referring to when you say realtime garbage collection. Then impact of deletes varies between controllers and has to do with how the drive decides manages data.
This re-confirmed once again (as was first noticed well over a year ago :)) that Windows 7 does successfully issue TRIM commands at the physical volume level even for a HDD.
The ability to discretely monitor I/O operations at the physical volume level (in addition to the physical device level) helps bring this nuance of TRIM within Win7 to light.
Excellent observation :up:
When I had my mp3’s on an external USB HDD WMP would, on occasion, become unresponsive. This seemed to happen when the media information was synchronising.
To look into this I decided to compare what happened when I played mp3’s with WMP from a standalone SSD I now use for storage and an external USB HDD.
I played the same 8 mp3’s from the SSD (E drive) and on USB HDD (F drive) The combined mp3 size on disk was 77.5MB with 49 minutes of music playtime.
In total I have 59.6GB of MP3’s. They are all in one folder with 1,009 sub-directories. Content indexing on the E & F drive is enabled when just playing mp3's.
During the monitoring period I did not access the E or F drive other than load the MP3’s.
Entries were generated by the following processes:
• MsMpEng.exe = (MS Essentials)
• System
• Wmplayer.exe (Windows Media Player)
• Explorer.exe
• Dllhost.exe
Before running the mp3 folder synchronisation on the USB drive I disabled file indexing, but I did not disable file indexing on the SSD when I ran the same task. Maybe that is why the reads were so much lower when synchronising the mp3 folder with the SSD drive.
Either way it’s easy to see why I prefer using SSD for storage over a USB HDD.
http://img574.imageshack.us/img574/8162/wmpcomp.png
Anvil
12-03-2010, 05:06 AM
I haven't forgotten about this thread :)
(I'm on a short vacation)
Ao1,
Interesing and expected results for the USB drive, get yourself an eSATA docking station :)
I went one better and got an SSD ;) Access times for HDD now includes me getting off my butt to pull it off the shelf and dust it down.
Ive been doing more work trying to find out what happens pre boot and after boot. Im close to working it all out so soon I will be able to post some results.
Anvil
12-03-2010, 01:58 PM
Can't argue with your decision :)
WD SiliconDrive N1x (SLC) were on sale a few weeks ago on one of the main computer shops in Norway, more than 50% off and so I had to order a few of them.
(they perform almost exactly like the SiliconEdge Blue and so they are quite nice for active storage)
Hi Anvil,
Ive had a bit of time to look into the boot process and I can share some of my observations.
If you only monitor the boot processes related to Windows the data transferred is quite small, although it is almost exclusively small random reads. What I have established however is that Windows keeps on loading well after the desktop 1st appears. Ive also established that non Windows apps get loaded before the desktop appears, so Windows files are getting loaded in parallel with non-Window based applications.
What you have installed on the OS can therefore make a big difference in the period before the desktop appears. The amount of data that gets loaded after the desktop appears is highly variable. Ive seen between 400MB and 1.7GB of read transfers within 15 minutes of the desktop first appearing without doing anything.
The data that gets loaded after the desktop is also predominately small random reads, although larger data transfers typically appear. The processes that generate read transfers after the desktop appears are highly variable.
Ive kept a couple of Excel files that show pre and post desktop process that generate read transfers. They are too large to post here, but if you are interested I can email them to you.
Here I run a manual TRIM with Intels Toolbox. I had run it a few times before recording the outcome below.
This is in AHCI mode. Intriguing that some of the temp files are called TRIM Raid. I got the same with the Toolbox version 1.3 and 2.01 and with MSACHI drivers and the latest RST drivers.
One_Hertz thread on the FusionIO got me thinking about file transfer speeds.
Here I experiment copying a 712,394KB file from the C drive to the E drive (Both X25-M160GB drives).
In the first image I use Window’s Explorer to copy the files from the C drive to the E drive and then I use TeraCopy.
In the second image I copy the same file from the E drive to the E drive, first using TeraCopy and then with Explorer.
I’m not 100% sure how to interpret the results, but it seems a file copy operation using Window’s is done with cache. I’ve seen, or more accurately heard; that the transfer speed Windows reports/displays is cache speed, as a hard drive can be heard clanking away long after Windows reports the file transfer as being complete.
Here I look at the write performance from C drive to E drive.
hIOmon and TerraCopy report more or less exactly the same. Write speeds of ~105MB/s.
Windows shows a transfer speed of 222MB/s, which seems to be reflective of the cache read speed not the write speed.
EDIT: Looking a little more closly Windows reports 1 item remaining with a file size of 29.6MB. That is the exact size of the Write Xfer Max size that hIOmon reports, so it seems that when you copy a file with Windows it is split into smaller file sizes during the copy processes. When using TerraCOPY the file transfer is made in one sequential Xfer.
This article helps explain the copy process in Windows:
File Cache Performance and Tuning
http://technet.microsoft.com/en-us/library/bb742613.aspx
1 = The Copy interface copies data from the system cache into an application file buffer
2 = A page fault occurs when the Cache Manager tries to access data not in cache memory
3 = Dirty pages in the system cache are written to disk by lazy write system worker threads Physical disk updates
The CacheSet Utility A tool to provide a simple control for setting the minimum and maximum system working set size:
http://technet.microsoft.com/en-gb/sysinternals/bb897561
There was probably a much easier way of do this, but anyway. In the first shot I monitored what occurred during a 4K AS SSD Benchmark. (I disabled the other tests from running). Theory being that this would show the max IOP capability with 4K reads and writes.
The maximum IOP read rate was 5,037, write 9,872.
Next I monitor a couple of hour’s general usage. I ran a SP & MP game and most of the programs I typically use.
The maximum IOP read rate was 1,788, write 236.
Next I update the AV data base and run a full AV scan, without resetting the general use metrics.
The maximum IOP read rate was 3,484, write 375.
It seems that even with an average queue depth of one I can’t get close to using the available read IOP’s capability of my SSD. As for write IOPs….
One_Hertz
12-27-2010, 02:34 PM
It seems that even with an average queue depth of one I can’t get close to using the available read IOP’s capability of my SSD. As for write IOPs….
Keep in mind these measurements are in a per second basis. I.e. 5k I/Os served in 0.5s (perhaps that was all which was required) and then 0 i/os in the next 0.5s would show as 2500 IOPs total. This does not necessarily mean that your SSD is doing the best that can possibly be done. Not that I disagree, it is just that your particular test does not prove it.
Keep in mind these measurements are in a per second basis. I.e. 5k I/Os served in 0.5s (perhaps that was all which was required) and then 0 i/os in the next 0.5s would show as 2500 IOPs total. This does not necessarily mean that your SSD is doing the best that can possibly be done. Not that I disagree, it is just that your particular test does not prove it.
Good point. I already know that most IOPS (from monitoring the percentage FastIOPcounts) are less than one millisecond. The mistake with SSD is to think in seconds. ;) It’s the same with MB/s. For write speeds it’s possible with the X25-M to see 250MB/s for a single I/O write operation when it is done in less than a second. Here is where it would be interesting to see how other SSD’s perform.
What I try to establish above is the percentage of IOPs utilisation compared to what is available.
Anvil
12-27-2010, 03:31 PM
I think hIOmon is as good as it gets, at least for now.
I've seen the same "flaw" where throughput is greater than the SSD can possibly deliver, not a big issue imho, still it is there.
Wrt copy operations, I've tried both TeraCopy and Total Commander.
Total Commander (bigfile + a decent buffer :)) is the one to use, it outperforms TeraCopy in most situations.
Hi Anvil, what is the highest file copy speed you have been able to achieve? (EDIT) and how did it compare to the theoretical speed?
Anvil
12-27-2010, 03:45 PM
I haven't really tried pushing the envelope, most copy operations are 250-350MB/s (from SSD to HDD where I do my daily "backups")
As a test case I was thinking of trying 4-5 SF drives as the destination and 4 Intels as the source drive, if that doesn't break the 500MB/s "limit" nothing will :)
(using a large file that's easily compressible the SF array should in theory be able to write close to 1GB/s)
One_Hertz
12-27-2010, 04:07 PM
Just reiterating what I found:
ICH X25-E array can read at 640MB/s in iometer
IOdrive can write at 582MB/s in iometer
Using total commander I can copy from ICH to IOdrive at 470MB/s
Anvil
12-27-2010, 04:39 PM
Is that using 4 E's? and what about the other way around? (from the IOdrive to the ICH array)
I was thinking of doing a test from raid-controller to raid-controller but I'll throw in the ICH10R as well (as a source "drive").
overthere
12-27-2010, 05:09 PM
Regarding several of the "maximum" I/O operation performance metrics collected by the hIOmon software, here are a couple of "nuances" that folks might want to consider.
The "maximum" IOPS metric reflects the actual number of I/O operations (e.g., read I/O operations obviously in the case of the "max read IOPS" metric) observed by the hIOmon software during a one-second interval since the start of the time duration when the hIOmon software began collecting I/O operation performance data for the respective file, device, or process.
For example, if the hIOmon software saw 5000 read I/O operations performed during a one-second interval (and this was the maximum number of read I/O operations observed so far within a one-second interval), then the reported value would be 5000.
As One_Hertz suggested above, it could be that all 5000 of these read I/O operations were performed during the "first" half-second of the one-second interval. One could subsequently argue that the "potential" maximum rate is really 10000 (since if the device did 5000 in 1/2 second, then it "should" be able to do 10000 within a full second).
In any case, the reported maximum IOPS values reflect that which was actually observed by the hIOmon software based upon actual device usage - and not necessarily the maximum that the device (for example) can perform.
Likewise, the maximum MB/s metric (i.e., the maximum amount of data transferred during a one-second interval) and the "maximum MB/s for a single I/O operation" metric are both based upon actual observations of the hIOmon software during the course of monitored file/device usage. [EDIT] A short description of each of these throughput metrics (and an important distinction between them) can be found at this prior post:
http://www.xtremesystems.org/forums/showpost.php?p=4600803&postcount=64
One other quick note. The maximum metrics described above can sometimes appear to exceed the capability of the device.
Take, as an example, a read I/O operation whose data transfer can be entirely satisfied by the OS without requiring a data transfer directly with the device. In this case, the data transfer is basically a memory transfer operation performed "internally" by the OS without interaction with the device (and so consequently not subject, for instance, to the bandwidth limitations of the actual device interface).
And as a result, the performance characteristics of the read I/O operation (as observed by the hIOmon software based upon the starting time and completion time of the I/O operation along with the amount of data transferred) can appear "exaggerated" in light of the performance expectations/limitations of the corresponding device.
Anvil
12-28-2010, 03:53 AM
...
In any case, the reported maximum IOPS values reflect that which was actually observed by the hIOmon software based upon actual device usage - and not necessarily the maximum that the device (for example) can perform.
...
This is why I like the hIOmon software, it reports what is observed.
If I want to get the maximum iops I'd run iometer or sqlio for e.g. 10 seconds, that would give me the maximum iops.
For monitoring tasks that aren't synthetic one needs to pair several metrics in order to try to "fully" understand what took place.
As for the observations that may be "polluted" by cache, well, if the cache works it works for your benefit, cache is part of the system and it doesn't invalidate the result.
The effect of system file cache was something I had not really understood, but as I re-read posts I can begin to understand it better.
A block diagram showing a data flow path at each level that hIOmon can monitor would be really helpful to provide a visual concept of what occurs where and why it is important.
Anvil
Im looking forward to seeing your copy speeds. :up:
Anvil
12-28-2010, 10:43 AM
I havent installed hIOmon yet and so this is without any monitoring.
Copy 5GB file (iometer test file) using Total Commander (bigfile 8MB buffer)
Target : 4R0 Vertex 2 60GB on the 9260
Source : 3R0 Vertex 2 100GB on the ICH
110660
Source : 2R0 C300 64GB on the ICH
110661
I'd say the 3R0 Vertex 2 array is about as fast as it gets using the ICH as the source.
I'll install hIOmon before I continue...
edit:
Rerun of the 3R0 to 4R0 copy operation, I didn't capture the transfer speed this time but it was more or less exactly the same. (607.nn)
110664
110665
Anvil
12-28-2010, 12:30 PM
Now, the other way around
Source : 4R0 Vertex 2 60GB on the LSI 9260
Target : 3R0 Vertex 2/LE 100GB on the ICH
110666
hIOmon summaries....
110667
110668
overthere
12-28-2010, 03:09 PM
As for the observations that may be "polluted" by cache, well, if the cache works it works for your benefit, cache is part of the system and it doesn't invalidate the result.
The system file cache is certainly an integral part of the overall system. Unfortunately it seems that its performance impact - and implications - can be neglected and/or misunderstood when considering an application's I/O activity.
And then there is that old (database) adage: "The best I/O is no I/O" :)
overthere
12-28-2010, 03:29 PM
A block diagram showing a data flow path at each level that hIOmon can monitor would be really helpful to provide a visual concept of what occurs where and why it is important.
Perhaps the figure at the bottom of the following web page is along the lines that you have suggested:
http://www.hyperIO.com/hIOmon/hIOmonArch.htm
Following from this figure, it might also be worth pointing out the ability of a single software tool to:
Observe I/O operations in action at three different critical points within the OS I/O stack: the file system level (essentially the application/process level), the physical volume level, and the physical device level.
This can help provide for a more complete, overall picture of what actually occurs during I/O operation processing.
Selectively observe I/O operations at one or more of these three levels.
This can help identify those particular I/O operations that are, for instance, germane to only a single level, e.g., I/O operations that are satisfied directly by the use of the system file cache and thus are effectively limited to the file system level only, I/O operations that are issued by applications/programs directly to a device at the physical device level and without direct interaction with the file system level, etc.
Optionally observe I/O operations concurrently at two or more levels, and moreover correlating I/O operations as they traverse the different levels.
This can help identify, for example, the nature and extent to which a particular application's file I/O operation activity actually incurs I/O operations at the device itself.
overthere
12-28-2010, 06:49 PM
I thought that it might be helpful to provide several brief comments about the summary I/O operation performance metrics shown within Ao1's prior post #201 (http://www.xtremesystems.org/forums/showpost.php?p=4678240&postcount=201), where he compared file transfer speeds between TeraCopy and Windows Explorer.
Basically, the summary metrics (which were collected upon a periodic basis for each of the respective files) displayed by the hIOmon Presentation Client for the overall summary metrics reflect the combined total of the opened "instances" of the file, whereas the displayed tabular metrics (presumably from the CSV-formatted hIOmon Manager Export File) reflect a separate row for each opened instance of the respective file (which is why there are sometimes multiple entries for the same file within the displayed tables, with each entry for the same file basically representing a separate "instance" of the file).
So what's an "instance" of a file? The hIOmon software can optionally collect and maintain summary I/O operation performance metrics upon an individual specific file, device, and/or process basis.
In the case of files, it is possible that a given file will be concurrently accessed by, for example, more than one process (e.g., Windows Explorer, the System process, and/or an AntiVirus program).
Normally upon "opening" the file, each such process will receive a "file handle", which essentially represents its particular open "instance" of the same file. Accordingly, the hIOmon software collects and maintains separate summary I/O operation performance metrics for each such separate "instance", with these summary metrics reflecting the monitored I/O operations directed towards the respective instance of the file. (Please note that the hIOmon software also maintains some summary metrics upon an overall file basis, i.e., regardless of the particular instance of the file).
As a second comment, Ao1 is correct about the Windows Explorer file copy operation involving the system file cache. In particular, the hIOmon Presentation Client display illustrates that the data to be copied was written to the system file cache (q.v., the write "SystemCache" data transfer amount), then transferred to the device proper (q.v., the overall amount of data written was twice the size of the "TEST file.mpg" file). This is in accordance with the information Ao1 presented in his post #203 (http://www.xtremesystems.org/forums/showpost.php?p=4679973&postcount=203)
As a third comment, the sum of the random I/O operation count and the sequential I/O operation count will be one less than the associated reported total I/O operation count for the respective file/device (since the first I/O operation to the file/device is considered to be neither a random nor a sequential I/O operation). In turn, the sum of the random data transferred amount and the sequential data transferred amount will be less than the total data transferred amount (i.e., the data transferred by the first I/O operation to the file/device is likewise considered to be neither random or sequential data transferred).
Lastly, the hIOmon software presents its throughput metrics in terms of MB/s (i.e., megabytes-per-second, with one megabyte = 1 000 000 bytes) rather than MiB/s (i.e., mebibytes-per-second, with one mebibyte = 1 048 576 bytes).
Anvil
12-29-2010, 04:00 AM
I've been doing a bit of tests using the LSI 9260 and the PERC 6/i.
When using the 9260 as the target, copy speed is 700+MB/s but the other way around the results are varying a lot, from 400-450MB/s.
110676
The PERC is perfectly capable of reading at full speed (1GB/s) but writing is slower than expected.
I'll swap the PERC for an LSI 9211 using SW raid.
Anvil
12-29-2010, 07:13 AM
While preparing for the copy tests I've made a few checks using ATTO for sequential bottlenecks.
Here's the 9260 array
110677
110678
...
I'll updload the 9211 SW test a little later.
Hi Overthere. You have the patience of a saint. I can see that some of the items Im now looking at were covered in much earlier posts, but I did not twig on to what you were saying, it went right over my head. ;)
Let me see if I can test your patience further :up:
To better understand the three monitoring levels I thought a simple file copy would help elaborate what happens where and when.
Ive noticed that if you right click to copy a file, then delete the copied file and then immediately recopy it, the transfer appears to be instantaneous.
This scenario would therefore seem to give me a way of tracking what happens in the file system and what actually happens on the disk and when.
To do this I selected:
1 (Device and file I/O Performance Analysis).
E:\*
1 (Enable collection of Physical Device Extended Metrics)
Any which way its going to be covered.:up:
E drive is static data only. The Test File is 731,082,752 bytes.
The copy/ delete/ copy process was done as follows:
Copy file (right click) Test.avi from E:\Test File to E:\ (paste)
Move copied file to recycle bin (right click, delete)
Re paste file to E:\ (right click paste)
The above was carried out within a minute, but interesting it would appear that reads and writes generated by the file transfer occurred sometime later on the actual device, but back to that later.
I like the diagram from the TechNet article as it shows the flow of data. If I use that diagram to try and understand what is happening from the monitoring results below:
All uncoloured entries are activities occurring between stages 1 and 5.
Assumptions:
Performance in these stages is down to the OS and RAM. Nothing is occurring on the device.
The system cache IOP miss counts are page faults from RAM, which then require a re-read from the file system.
Most of the Read/ Write FastIOPcounts are done in cache and when this happens there are independent of the device speed. If I look at the results from post #48 it is likely that the 87.6% FastIOPcounts with HDD were predominantly achieved by cache. The SSD managed 100% because the speed of the SSD was such that it could satisfy non cached read IOP counts in less than a millisecond. [Edit] So it seems activities from 1 to 5 are designed to mitigate the slow speed impact of a HDD. I wonder if the file system driver/ cache manager could be better optimised for SSD. (?)
\Device\HarddiskVolume3 is when data actually gets read and written to the device. I assume stage 7 - Disk Driver.
Assumptions:
This is when the speed of the device counts, however when this actually occurs is typically after the event. The speed of the device only therefor counts if it is a direct read/ write that bypasses the cache manager. (Assuming no further load when the read/ write occurs)
E: is a summary of all activity within stages 1 and 7
Assumptions:
This is showing all activity be it in cache or on the device.
\Device\HarddiskVolume3:\<DASD>\VMMFrom previous posts this has been explained as system flags associated with the I/O operation indicate involvement by the system Virtual Memory Manager (VMM).
Assumptions:
Entry 21 is why the re-paste appeared to be instantaneous. It was copying data from the VMM to the file system driver. The actual read/ write on the device occurred a minute later.
Ive attached the script, which contains hidden cells (that can be un-hidden if required) that are not shown on the image below. I only wanted to show relevant data to what I try to describe above, but if I missed anything it can be un-hidden.
Please feel free to wade into mistakes in my understanding and I will correct the post accordingly.
Here is a summary of the top largest read transfers after a couple of hours of normal general use. The entries are arranged by size, not by when they occurred.
Max read cache xfer speed = 3,219MB/s.
Next up I look at the smallest read transfers.
Here are the bottom lowest read transfer sizes, again arranged by transfer size. There are no physical disk reads. Looking at the rest of the log file physical disk reads only start to occur when read transfers are 4,096 bytes or above….hmm. Now I check writes.
Here are writes, same method as reads. Fastest cache write speed = 2,046MB/s
Out of 25,288 write entries only 1,079 had a write to the physical device.
So it would seem that the SSD does not see anything like the small writes that are generated, as they resolved in the file system without going to disk. I guess on top of that the SDD itself can further cache small writes to reduce wear.
overthere
12-31-2010, 05:19 PM
In post #221 (http://www.xtremesystems.org/forums/showpost.php?p=4684460&postcount=221) Anvil provided a hIOmon Presentation Client screenshot that showed various metrics collected during an ATTO Disk Benchmark run upon his 9260 array. Here are some considerations and further observations that might be of interest:
Consideration A: Certain metrics shown within the hIOmon Presentation Client display reflect an average calculated for either the latest Observation period (or an overall average calculated for the time period since the start of the first Observation period up until the end of the latest Observation period).
For example, the Read IOPS value of 3552.4 (as shown within the screenshot) is simply the total Read I/O operation count (1 066 249) divided by the time duration of the latest Observation period (5 minutes and 0.1453561 seconds, i.e., 300.1453561 seconds), which was the only Observation period (so far).
So basically what the Read IOPS represents is the average IOPS rate over the course of the Observation period (i.e., an average of 3552 read I/O operations per second during the 5 minute Observation period).
Of course, the ATTO benchmark performed read I/O operations at varying transfer sizes (and accordingly varying IOPS rates) during the overall benchmark test period. In any case, the hIOmon Presentation Client calculates/reports the average IOPS rate based upon the time duration of the particular Observation period.
One might also have noticed that the last read I/O operation completed approximately 1 minute and 6 seconds before the end of the 5 minute Observation period (and so if this 1 minute and 6 second time duration, within which no read I/O operations were performed, were excluded from the average read IOPS calculation, then the calculated average read IOPS value would be greater than the 3552.4 IOPS value).
The hIOmon Presentation Client calculates the "average" IOPS value in this manner so that users can perform, for instance, various capacity planning and performance analysis studies over extended time frames (e.g., to identify those particular time periods of intense and/or low I/O operation activity.
Please note that the length of an "Observation period" is essentially the user-specified periodic time interval for which the hIOmon software can be configured to collect the summary I/O operation performance metrics for display, export, etc.
As an alternative to collecting summary I/O operation performance metrics upon a user-specified periodic basis, the hIOmon software can also be configured to collect the summary metrics for offload either when the respective file is closed and/or upon an "alert" basis (i.e., when a user-specified threshold - for instance, some maximum read I/O operation response time value, some total amount of write data transferred, etc. - has been detected/observed by the hIOmon I/O Monitor for the respective file).
Consideration B: The hIOmon "SSD I/O Performance Analysis" script option configures the hIOmon software so that I/O operation activity performed by any process will be included within the summary I/O operation performance metrics that are collected.
Consequently, the collected summary metrics displayed by the hIOmon Presentation Client (as shown within the screenshot) might include I/O operations that were performed by processes other than the ATTO benchmark program. These processes could include system processes that update various file system metadata (which can occur since the read and write I/O operation activity performed by the ATTO benchmark tool is directed to a file).
In any case, the actual amount of I/O operation activity performed by such processes is likely negligible (especially since the bulk of the I/O operation activity was due to the ATTO benchmark program and granted that there was no other appreciable I/O operation activity performed by other processes, such as an antivirus program, directed to the logical drive that was the target of the ATTO benchmark program). However, any such I/O operation activity that is extraneous to that performed directly by ATTO can skew (however slightly) the reported metrics from those expected to be attributable to the ATTO program alone.
Consideration C: I configured the hIOmon software to take a closer (although still very cursory) look at how the ATTO Disk Benchmark actually performs its "benchmarking" I/O activity.
One thing that I first noticed is that ATTO begins by first generating (i.e., writing) its "benchtst.$$$" test file. If the selected value of the ATTO "Total Length" option is 2GB (as in the case of Anvil's screenshot), then ATTO will generate/write a 2GB test file using 256 write I/O operations each with a data transfer length of 8388608 bytes, which happens to be the largest "write I/O operation" data transfer size shown with the hIOmon Presentation Client screenshot and also the largest selectable ATTO "Transfer size" option value.
Incidentally, it appears (based upon the I/O operation performance metrics that I collected using hIOmon) that if you select an ATTO "Total Length" option value smaller than 8MB (technically 8MiB), then ATTO will nevertheless write out an initial test file that is 8 MiB in size.
Anyway, some of the key points here are: (1) ATTO always starts out by generating/writing its test file - so there is some initial write I/O operation activity performed by ATTO before it actually begins its various "transfer size" I/O operation activity, which is the basis of its "advertised" benchmarking; and (2) the write I/O operation activity by ATTO that occurs when initially generating its test file is included within the "Write I/O operation" metrics collected/shown within Anvil's hIOmon Presentation Client screenshot.
Consideration D: Based once again upon my collection of I/O operation performance metrics using hIOmon, I noticed that ATTO apparently performs varying amounts of read (and write) I/O operations. The difference between successive ATTO runs was sometimes on the order of thousands (e.g., between 10 000 and 20 000 overall total read I/O operation count difference between successive ATTO runs). Perhaps this difference is due to some system-dependent phenomenon. In any case, please note that I configured the hIOmon software to collect summary metrics for the "benchtst.$$$" file only and only for the ATTO "bench32.exe" process (using ATTO Disk Benchmark version 2.46).
This read and write I/O operation variance is disconcerting to me (at least based upon my working with folks defining benchmarking standards). To my understanding, one of the tenets of sound benchmark design is the "consistent repeatability of the stimulus". In other words, the benchmarking tool should perform the same prescribed activity each time the benchmark tool is run using the same configuration options (otherwise it is difficult to perform, for example, an "apples-to-apples" comparison).
Consideration E: The ATTO benchmarking tool does not necessarily read (nor write) the entire "Total Length" for (at least) the 512 byte transfer size. This in evident in Anvil's hIOmon Presentation Client screenshot. For a transfer size of 0.5 KiB and a "Total Length" of 2 GiB, a total of 4 194 304 sequential read I/O operations (each with a data transfer size of 512 bytes) would be required to read the entire 2 GiB test file.
As shown in Anvil's screenshots, ATTO was configured to include a transfer size of 0.5 KiB. The hIOmon Presentation Client shows the smallest read data transfer size of 512 (along with a maximum read data transfer size of 8 388 608 - which confirms that the ATTO "Transfer size" span that Anvil specified was in fact performed). However, the hIOmon Presentation Client shows an overall total of 1 066 249 read I/O operations were performed; as noted above, 4 194 304 sequential read I/O operations would be required to read the entire 2 GiB test file for a 512 bytes data transfer size alone. Anvil might have been pointing this out with his highlighting of portions of the hIOmon Presentation Client screenshot.
OK, now I try to draw some conclusions for discussion. Please bear in mind that I discuss typical desktop usage patterns, which are relatively undemanding.
From what I have been able to observe Raid 0 has made no noticeable difference to the speed of applications, but I have never understood why. Also a lot of people have reported no observable difference between different brands of SSD, when some are technically faster than others.
The only real differences that I have noticed between SSD & HDD is boot up, responsiveness immediately after boot up and the way applications open instantly. I guess game loading is also faster, but I can't say game playing is any different. (I only play COD, so maybe there are exceptions).
The impact of system file cache has been a revelation for me, but thinking about it more the access times for HDD are so high that you would be waiting forever for anything to happen unless the system file manager could speed things up.
The system file manager seems to do a very good job at making things run fast without putting a load on the device or deferring the load to reduce impact. In instances where the system file manager cannot cope with the demand the speed of the device becomes the determining factor.
For arguments sake if it was assumed that the system file manager could deal with 80% of I/O demand by either bypassing or deferring load to the device the remaining 20% of I/O's are when you would see the performance penalty of the HDD in comparison to SSD. The higher the load (like bootup) the slower things get as they end up being dealt with by the device. With SSD you can get around 98% fast IOP counts. With raid it goes up to around 99% but it seems it does this via more effective cache management not via faster speeds from the device. This would change under heavier loading, but for typical desktop use the demand is not sufficient to see the benefit of the array device speed.
Below I capture typical use and isolate the device speeds in raid 0 and raid 1. I'm not using anything like the available sequential read speeds and I copied over a couple of large files, which should have been a lot faster. When I get time I will do the same for a single device to compare.
Assuming I'm on the right track this makes me wonder why you need a raid array to give you more/ better managed system cache. It would seem that with more/ better managed system cache an SSD's overall performance could be a lot better for desktop use. :shrug:
Next up I'm going to drop down to 2GB of RAM (from 8GB) and under clock it to see the impact on the storage performance. I'm also trying to think of a way to see if the system file manager is actually slowing down SSD performance in some instances.
saint-francis
01-01-2011, 06:25 PM
Ao1, what you are doing here is the most interesting and important SSD work I've seen in probably a year or more.
Anvil
01-02-2011, 10:17 AM
Hi Overthere
I've been swamped with a head cold and so I haven't read all of your analysis, will do later tonight.
I did notice that the number of I/Os seemed low compared to what one would expect, it might just do a "sample" number of I/Os, I'll redo the test using a smalller testfile just to see what happens.
It's not an issue as long as ATTO is performing the same number of IOs for that exact testfile size, or it might just be that ATTO isn't built for SSD speeds :)
Ao1,
Your assumption about the SSD (most SSDs) isn't exactly stressed while doing normal desktop tasks is as expected, the X25 is not even close to being stressed.
At the same time, there is a difference between SSD controllers, application loadtime can differ quite a lot, you should have had a SF or C300 to compare with the Intel :)
The difference between single drives and raid should show up once the task is starting to stress the SSD controller.
You could try monitoring an AS SSD run using sinlge, raid-1 and raid-0.
AS SSD, being a synthetic benchmark would of course not reflect normal desktop usage but it will show when there is a point in adding an extra SSD for more throughput/iops.
My VM tests does show improvement going from a single drive to raid, at the same time the number of seconds used by storage compared to the runtime length shows that most of the time the SSD is just waiting.
(meaning there are now other bottlenecks in the system)
Here I use Cacheman 7 to manage the file cache system. I've only tweaked a few settings, but already there are some noticable improvements.
overthere
01-02-2011, 11:59 PM
...I did notice that the number of I/Os seemed low compared to what one would expect, it might just do a "sample" number of I/Os, I'll redo the test using a smalller testfile just to see what happens.
It's not an issue as long as ATTO is performing the same number of IOs for that exact testfile size, or it might just be that ATTO isn't built for SSD speeds :)
Hi Anvil,
I believe you are right that "it's not an issue as long as ATTO is performing the same number of IOs for that exact testfile size..." - although I suspect many/most folks might assume that the total number of I/O operations to be performed will be the exact number required to access the entire testfile size (presumably once) as indicated by the selected "Total Length" value.
In any case, a variation between ATTO runs in the total number of I/O operations (for the same given "Transfer Size" and the same associated testfile size) means, of course, a variation in the total amount of data transferred for the particular "Transfer Size" test.
And varying the total amount of data transferred under such circumstances (i.e., given the same transfer size and testfile size) in a benchmark test whose focus is on "data transfer throughput" (specifically "maximum sequential MB/s") seems suspect to me.
I can see that in those cases where a larger testfile size (e.g., 2 GiB) is selected, ATTO might want to reduce the total number of I/O operations performed (and correspondingly the amount of data transferred) for the smaller "Transfer Sizes" so as to help reduce the overall time required to run the benchmark test.
But, as I think that you would agree, such a reduction in the total number of I/O operations performed should be consistent (i.e., the same) between ATTO runs.
Hi Ao1,
Sorry that I have not yet replied to your post #222 (http://www.xtremesystems.org/forums/showpost.php?p=4684495&postcount=222); I have already looked over the post and have several comments in mind, which I hope to get posted sometime later today.
overthere
01-03-2011, 05:09 PM
In post #222 (http://www.xtremesystems.org/forums/showpost.php?p=4684495&postcount=222) Ao1 chose a good scenario/example that can help illustrate some of the dynamics involved with the operation of the "system file cache" mechanism within the Windows OS:
To better understand the three monitoring levels I thought a simple file copy would help elaborate what happens where and when. ...
E drive is static data only. The Test File is 731,082,752 bytes.
The copy/ delete/ copy process was done as follows:
Copy file (right click) Test.avi from E:\Test File to E:\ (paste)
Move copied file to recycle bin (right click, delete)
Re paste file to E:\ (right click paste)
The above was carried out within a minute, but interesting it would appear that reads and writes generated by the file transfer occurred sometime later on the actual device, but back to that later.
I like the diagram from the TechNet article as it shows the flow of data. If I use that diagram to try and understand what is happening from the monitoring results below:
All uncoloured entries are activities occurring between stages 1 and 5.
The TechNet diagram that Ao1 mentions (and which is also included within his post) basically reflects the case where the data to be accessed by a file read I/O operation is not presently located within the system file cache and so must first be retrieved from the actual storage media (i.e., the physical disk device).
This case/diagram is the second scenario/diagram shown amongst the three diagrams that Ao1 included within his prior post #203 (http://www.xtremesystems.org/forums/showpost.php?p=4679973&postcount=203), where the first TechNet diagram is the case in which the data to be transferred by the file read I/O operation is present within the system file cache (i.e., a "system file cache read hit") and the third TechNet diagram is the case where the data written to the system file cache is subsequently (at some later time) written/transferred out to the actual storage media.
It is important to note that the file "copy / delete /copy" example chosen by Ao1 actually involves all three of these TechNet cases/diagrams and not simply the second diagram alone/only. This can be substantiated by the summary I/O operation performance metrics that were collected by hIOmon and which are included/shown within Ao1's post.
For example, the "system file cache read hit" I/O operation count can be derived by subtracting the hIOmon "ReadSystemCacheIOPmissCount" metric value from its associated "ReadSystemCacheIOPcount" metric value. The hIOmon "ReadDataXferSystemCacheHit" metric value shows the corresponding amount of data transferred from the system file cache for the "system file cache read hit" I/O operations. In any case, these metrics reflect those read I/O operations that fall into the TechNet diagram/scenario number one.
Similarly, the hIOmon "ReadSystemCacheIOPmissCount" and the "ReadDataXferSystemCacheMiss" metric values can be associated with the second TechNet diagram. In addition, the hIOmon "PhyDev Read Data Xfer" metric can generally be associated with the retrieval of the data from the actual physical device in the case of a "system file cache read miss" scenario (i.e., the second TechNet diagram).
Finally, the hIOmon "WriteSystemCacheIOPcount", the "WriteDataXferSystemCacheHit", and the "PhyDev Read Write Xfer" metrics can be associated with the third TechNet diagram (where the data associated with file write I/O operations performed as a result of the "file copy" actions are first written to the system file cache and then subsequently transferred to the actual storage media).
As an aside, unhiding the hIOmon "Read IOP Count" metric column within the displayed hIOmon metrics table might be helpful for the sake of completeness.
One other brief comment here. Folks will likely notice several occasions where there are multiple rows with the same file name and same time period within the displayed hIOmon metrics table. This is due to the "multiple opened instances" of the same file (as briefly described within my prior post #219 (http://www.xtremesystems.org/forums/showpost.php?p=4683926&postcount=219)).
I suspect that these multiple instances of the same file are related to processes such as Windows Explorer (e.g., performing the actual copy of the file), the System process (which can be involved with "pre-fetching" portions of the file into the system file cache and also with subsequently writing out the file from the system file cache to the storage media), and perhaps an Antivirus program (such as Microsoft Security Essentials) that are (concurrently) accessing the same file. Please note that the names of those specific processes involved with a particular instance of a file are included amongst the summary I/O operation metrics collected by hIOmon for the particular file instance.
This post is already getting a bit long, so I'll defer my other comments to another subsequent post.
overthere
01-04-2011, 12:09 AM
Continuing from my prior post #233 (http://www.xtremesystems.org/forums/showpost.php?p=4691194&postcount=233) in regards to Ao1's post #222 (http://www.xtremesystems.org/forums/showpost.php?p=4684495&postcount=222):
Assumptions:
Performance in these stages is down to the OS and RAM. Nothing is occurring on the device.
The system cache IOP miss counts are page faults from RAM, which then require a re-read from the file system.
Most of the Read/ Write FastIOPcounts are done in cache and when this happens there are independent of the device speed. If I look at the results from post #48 it is likely that the 87.6% FastIOPcounts with HDD were predominantly achieved by cache. The SSD managed 100% because the speed of the SSD was such that it could satisfy non cached read IOP counts in less than a millisecond. [Edit] So it seems activities from 1 to 5 are designed to mitigate the slow speed impact of a HDD. I wonder if the file system driver/ cache manager could be better optimised for SSD. (?)
Performance throughout the TechNet diagram/scenario number one (i.e., system file cache "read hit") is essentially related to that of the OS and RAM, with the actual device not being a factor within this scenario.
Performance in the case of the TechNet diagram/scenario number two (i.e., system file cache "read miss") is also essentially related to that of the OS and RAM, with the device only becoming a factor when a system file cache read miss actually occurs (since the data for the read I/O operation must then be retrieved from the actual storage media).
The set of "SystemCache" summary I/O operation performance metrics collected by the hIOmon software reflects those I/O operations that explicitly requested the use of the "system file cache"; moreover, these I/O operations are handled by means of an expedited path through the OS I/O stack.
The hIOmon software considers an I/O operation to be a "system cache hit" when the I/O operation, as observed by the hIOmon I/O Monitor, was successfully performed using the "system file cache" by explicit request and which completed within less than one millisecond.
Similarly, "SystemCache" I/O operations that completed within one or more milliseconds are regarded to be "system cache misses" and are considered to have incurred an I/O operation directed to the corresponding actual device.
It is also imporant to note that:
Those I/O operations that are identified by the hIOmon I/O Monitor as "Fast I/O operations" include both the "System Cache Hit" I/O operations (as described above) as well as those I/O operations that did not explicitly request the use of the "system file cache" but which nevertheless completed in under one millisecond.
"Fast I/O operations" that do not explicitly request the use of the "system file cache" can nevertheless be satisfied by the system file cache under the direction of the OS. That is, there can be I/O operations that complete in less than one millisecond (and so are considered to be "Fast" I/O operations) that the OS satisfied by using the system file cache even though these I/O operations did not explicitly request the use of the system file cache.
"Fast I/O operations" also include those I/O operations for which there was no system file cache involvement but which neverthesless completed in less than one millisecond; these I/O operations can include physical volume and/or physical device I/O operations.
In regards to the FastIOPcount metric values shown in post #48, I am not sure whether the metrics shown were collected by hIOmon at the file level (i.e., logical disk level) or at the physical volume or physical device level.
In any case, HDD physical volume (or physical device) I/O operations that are observed by hIOmon to be "Fast I/O operations" are often the result of caching further down the I/O stack (e.g., a cache upon a RAID controller card, a cache upon the device itself, etc.). SSDs are, of course, another matter given their inherent fast read access times as Ao1 mentioned.
\Device\HarddiskVolume3 is when data actually gets read and written to the device. I assume stage 7 - Disk Driver.
Assumptions:
This is when the speed of the device counts, however when this actually occurs is typically after the event. The speed of the device only therefor counts if it is a direct read/ write that bypasses the cache manager. (Assuming no further load when the read/ write occurs)
The I/O operation performance metrics collected by hIOmon at either the physical volume level (e.g., \Device\HarddiskVolume3) or the physical device level within the OS do reflect data that is actually transferred with the storage media proper.
This is where the performance characteristics of the storage media (along with the other components along the I/O operation path, e.g., the transport interface type, HBA, etc.) especially come into play and can have a significant impact upon the overall performance of the I/O operation.
Please note that the "speed" of the device (and the other elements down the path to the device) is a factor not only in the case of "system file cache misses" (i.e., how quickly the data can be retrieved to satisfy a file I/O operation that encounters the system file cache miss), but can also be a factor in anticipatory pre-fetching by the OS into the system file cache.
That is, the "faster" the OS can pre-fetch the data from the device, then presumably the more likely that the data will already be resident within the system file cache when a subsequent file read I/O operation is performed to retrieve this pre-fetched data. Of course, there are also a number of other factors to be considered in such a scenario (e.g., the extent to which the file I/O operations are sequential read I/O operations, the concurrent I/O activity of other, perhaps competing processes/applications, etc.) along with a slew of other caching, system, and resource issues.
E: is a summary of all activity within stages 1 and 7
Assumptions:
This is showing all activity be it in cache or on the device.
The hIOmon software can be configured to collect "summary" I/O operation performance metrics upon an individual device basis.
In the case of a "device summary" for a logical drive/disk (e.g., E:), the summarized I/O operation performance metrics represent the sum of the performance metrics for all of the files being monitored by the hIOmon I/O Monitor for the specified logical disk.
That is, the "Device Summary" for the logical disk represents the overall aggregate of all summarized I/O operation metrics for all files (and only those files) that were being monitored by the hIOmon I/O Monitor and which reside upon the respective device.
Also please note that the "Device Summary" reflects cumulative metrics that have been collected by the hIOmon I/O Monitor since it began (or re-started) monitoring I/O operations for the respective device.
In contrast, the summary I/O operation performance metrics that are collected by the hIOmon I/O Monitor upon a periodic basis for files reflect only those I/O operations that were observed by the hIOmon I/O Monitor during the duration of the corresponding summary time period (e.g., if the summary metrics are being collected upon a "one minute" periodic basis, then the summary metrics reflect only those I/O operations that were observed during the respective one-minute period).
In any case, the "Device Summary" metrics for a logical device include both "SystemCache" and "PhyDev" I/O operation performance metrics for those monitored files associated with the logical device.
\Device\HarddiskVolume3:\<DASD>\VMMFrom previous posts this has been explained as system flags associated with the I/O operation indicate involvement by the system Virtual Memory Manager (VMM).
Assumptions:
Entry 21 is why the re-paste appeared to be instantaneous. It was copying data from the VMM to the file system driver. The actual read/ write on the device occurred a minute later.
Entry 21 within the hIOmon metrics table is certainly a key entry. It shows not only the data read from the device (Read Data Xfer), including that for the "E:\Test File\Test.avi" file to be copied (some of which was copied into the system file cache) but also the data written to the device from the system file cache (Write Data Xfer), including both times the file was copied.
In addition, entries 33 and 36 show the copied file being written to the system file cache (q.v., WriteSystemCacheIOPcount and WriteDataXferSystemCacheHit) and written to the device (PhyDevWriteDataXfer). :)
Anvil
01-04-2011, 04:47 PM
Overthere
I've performed a few more tests using ATTO and there is definately something strange going on :)
First I tried 2 runs using 512B only, one run using 256MB testfile and the next run using 512MB.
Here are the results of those two runs
110851
As you can see, the one thing that differs is Write I/O.
(mainly due to the size of the testfile)
--
Finally I ran the full range (512B-8192KB) using a 256MB testfile
110852
I haven't compared it to the 2GB testfile run yet.
overthere
01-04-2011, 06:29 PM
Overthere
I've performed a few more tests using ATTO and there is definately something strange going on :)
First I tried 2 runs using 512B only, one run using 256MB testfile and the next run using 512MB.
Here are the results of those two runs
110851
As you can see, the one thing that differs is Write I/O.
(mainly due to the size of the testfile)
--
Finally I ran the full range (512B-8192KB) using a 256MB testfile
110852
I haven't compared it to the 2GB testfile run yet.
Hi Anvil,
Thanks for running the additional ATTO tests. :up:
Several items immediately come to mind when comparing the two 512B runs (i.e., the one with the 256MiB testfile size and the other with the 512MiB testfile size):
I believe that you are right about the additional write data transferred (for the 512MiB testfile) being attributable largely to the increased testfile size. Adding the additional 268 435 456 bytes (incurred when ATTO initially wrote out the 512MiB testfile, in comparison to the 256MiB testfile, before performing the actual benchmarking tests) to the 329 531 392 (i.e., the total write data transferred for the 256MiB testfile) gets you into the ballpark of the total write data transferred amount for the 512MiB testfile (with the actual sum of the two being 597 966 847, and keeping in mind that there is also a small amount of other extraneous read and write I/O operation activity occurring in the background due to, for instance, file system metadata I/O operations).
Now the writing of the additional 256MiB for the 512MiB testfile required an additional 32 write I/O operations (assuming a write data transfer size of 8 388 608). So if you subtract these additional 32 write I/O operations from the grand total of write I/O operations (119 097) for the 512MiB testfile, I believe you get a total of 119 065 - which is less than the grand total of write I/O operations (119 069) for the 256MiB testfile size! :confused:
It seems then that ATTO did not even touch the upper 256MiB of the 512MiB testfile (especially since it is performing sequential I/O operations with each transfer size test beginning at zero). :)
And the same pretty much applies to the read I/O operations (i.e., the 512MiB testfile had only 1 000 more read I/O operations than the 256MiB testfile, which of course was one-half the size of the 512MiB testfile).
BTW, the difference in the total read data transfer amounts for the 256MiB testfile and the 512MiB testfile runs is 512 000 bytes (which just so happens to be the 1 000 read I/O operations of 512B each). :)
Regarding the comparison of the full range (512B-8192KB) using a 256MiB testfile run with the 2GB testfile, are you referring to your prior run shown within your prior post #221 (http://www.xtremesystems.org/forums/showpost.php?p=4684460&postcount=221)?
If so, that prior ATTO run used your 9260 array, in which case did you also use the same 9260 array for your full range / 256MiB testfile run above?
In any case, it appears that your latest full range / 256MiB testfile run shows a rather balanced total amount of read and write I/O operations (and, of course, total amounts of data transferred - about 33GB each).
But your prior 9260 full range / 2GiB run not only shows an "imbalanced" total amount of data transferred (about 41GB read to 57GB write), it also seems that the increased testfile size (which is 8 times as large) only increased the amount of data actually transferred by a relatively small percentage.
I am not suggesting that there necessarily should be 8 times more data transferred (or 16 times total for both read and write combined) for the 2GiB testfile size, but I am not sure in any case what scaling factor is used by ATTO for the various testfile sizes.
Yep, but there does appear to be something strange going on ... :)
Here I capture metrics using the WMI Browser/ top ten devices. [EDIT: To show the MB's speeds for various tasks.]
Data Xfer Rate Max =Max MB/s observed.
Each task that has been monitored was run on its own with only OS tasks running in the background.
(Overthere, thanks for your comments :up:)
overthere
01-09-2011, 09:09 AM
Here are the bottom lowest read transfer sizes, again arranged by transfer size. There are no physical disk reads. Looking at the rest of the log file physical disk reads only start to occur when read transfers are 4,096 bytes or above
.hmm. Now I check writes.
Just to help clarify, the "transfer size" values discussed within the post above (as well as in the prior post 223 (http://www.xtremesystems.org/forums/showpost.php?p=4685410&postcount=223)) reflect the combined total amount of data transferred by all read I/O operations for the respective file (as observed by the hIOmon software).
The "summary" I/O operation performance metrics collected by hIOmon also include the minimum and the maximum data transfer size associated with a single I/O operation (each separately for reads and writes) - that is, the size of the smallest data transfer performed by a single read I/O operation (and similarly, the size of the largest data transfer performed by a single read I/O operation).
Ao1 also makes a keen observation about there being no physical device I/O operations associated with the various files shown within the post above (i.e., post #224 (http://www.xtremesystems.org/forums/showpost.php?p=4685475&postcount=224), which shows the files with the lowest amount of data transferred).
This phenomenon can be the result of these files having been read into the system file cache prior to their being monitored by the hIOmon software in accordance with the currently-active hIOmon "Filter Selection" configuration file (that indicates which particular files, devices, and/or processes are to be monitored by hIOmon).
For example, one such scenario:
The hIOmon Filter Selection is activated; this Filter Selection says that the hIOmon I/O Monitor is to monitor and collect summary I/O operation performance metrics for file "x"
The hIOmon I/O Monitor subsequently observes the occurrence of a read file I/O operation directed towards file "x"; as a result, the corresponding data is transferred from the physical device into the system file cache as part of processing this file read I/O operation. The various "summary" metrics (including the "physical device metrics" reflecting the transfer of the data from the device) are accordingly updated by the hIOmon I/O Monitor
The hIOmon Filter Selection is subsequently re-activated (e.g., by running the hIOmon "SSD I/O Performance Analysis" script); as a result, all of the summary I/O operation performance metrics collected by the hIOmon I/O Monitor are re-initialized (i.e., reset to zeroes).
The hIOmon I/O Monitor subsequently observes the occurrence of a read I/O operation directed towards file "x". This read I/O operation does not explicitly request the use of the system file cache (consequently, it will not be recorded by the hIOmon software as a "SystemCache" type of I/O operation).
However, the data transferred by this read I/O operation still resides within the system file cache (from step 2 above) and so the data transferred by the read I/O operation will be satisfied from that contained within the system file cache (and without directly incurring an associated read I/O operation involving the physical device; accordingly the "PhyDev" summary metrics remain unchanged).
What you describe is how I undertook the monitoring, although I re-booted occasionally to clear the system cache.
I should have perhaps explained what I was trying to demonstrate in post #237. I wanted to show the maximum MB/s speeds that occurred on the device when undertaking typical tasks within a typical environment.
I captured the data Xfer statistics only to show that the maximum observed MB/s speeds were based on a reasonable amount of data being generated during the observation period.
The data Xfer totals are not necessarily derived solely from the task I indicated. Background OS tasks and cache movements would have had an impact. i.e. a read on the device could have been instigated before the monitoring period started.
Excluding file copying the MB's speeds are surprising low, although all of the tasks require processing time, so it should not be that surprising I guess.
With regards to system cache I've noticed that although I don't stress out the SSD, Cacheman 7 can improve the fast IOP count on both the system and the device. With a single device using Cacheman 7 I can achieve a higher fast IOP count than what I could with a Raid 0. (And it's a lot cheaper :cool:).
TBH it is system cache that I find more interesting now in terms of looking for performance improvement with SSD. (Napalm (if you read this thread) it's taken me a long time but I'm getting there now :) )
I don't know how to test or prove it, but I would bet that something like Cacheman 7 that was designed specifically for SSD could really help to improve SSD performance. I say this as system cache has presumably been developed to mask the slow speed of a HDD rather than take advantage of the speed of a SSD.
overthere
01-09-2011, 03:29 PM
I should have perhaps explained what I was trying to demonstrate in post #237. I wanted to show the maximum MB/s speeds that occurred on the device when undertaking typical tasks within a typical environment.
Thanks for the explanation - it confirms what I suspected was your intent.
Just to remind folks, the maximum MB/s rates that are included within your hIOmon metrics tables reflect the maximum amount of data that was actually transferred during a one-second interval as observed by the hIOmon software.
So for example, a 252.707 maximum read MB/s indicates that the hIOmon I/O Monitor observed 252.707 megabytes of data being actually transferred by read I/O operations during a particular one-second interval (with the hIOmon software also recording a timestamp for this specific, one-second interval).
After monitoring for a couple of days I have saved a log file to pull off some stats.
First off I establish the top 10 read data xfer sizes. The chart on the left includes all data xfer events (cache and physical). The chart on the right only includes data xfer events that occurred on the physical device.
The top figures on the x axis are bytes. On the left chart there were 24,877 occurrences of 40 byte data xfers. 16,977 occurrences of 2,187 byte data xfers etc.
EDIT:
Some points of clarification. No Page file. Superfetch/prefetch disabled. Cacheman 7 running with auto optimised profile set to maximum performance.
Here I compare the amount of write data transferred at the logical device level (C:\) with the corresponding amount of write data transferred at the associated physical volume level (\Device\HarddiskVolume1:\).
Percentage reads to writes