Sandforce Life Time Throttling

[zads]

...
I will say this quickly though. There is MUCH confusion about the "time calculations" being used to determine when lifetime throttling will be implemented. Time has absolutley nothing to do with it(at least at this level of throttle, although "hammered states" may very well rely on the time/data size correlation). It has to do strictly with the capacity of the drive in question. EVERY first gen Sandforce drive WILL throttle IMMEDIATELY when all nand has been touched because the required GC map has been fully formed.
...

If you SE a drive that was previously throttled and the drive were to slow down before all capacity had been rewritten once more?.. then you have other issues at play. Won't even begin to speculate as to why that would occur(though I did a bit in the above link)
...

Your "who" is wrong here, although the "how" is right-
There is a performance drop(also can be called a 'hammered state') when going from a blank slate (brand new or secured erased drive) to a 'semi-dirty' state where all flash LBA have been written to, especially random 4k uncompressed data (worst case).
This isn't 'lifetime throttling'.
The lifetime/warranty throttling is entirely based on power-on time versus flash writes.

[Ao1]

Your guess would probably be correct as far as the small lags you guys are seeing is concerned. The more the drive must do?.. the more overhead and losses would result.

Everyone interested in Sandforce should also keep in mind that just beecause TRIM commands are sent/marked immediately?.. has absolutely squat to do with "when" the controller decides to recover/release those back into the fresh block pool. TRIM released blocks are more often then not recovered during low activity idle times. Often called "lazy TRIM" by some users. Hence, the excellent GC recovery that's found with ocassional logoff idles with power remaining to the drive(S1 sleeps).

Incorrect to my observations. TRIM occurs straight after a delete. It does not wait for the drive to become idle.

[sergiu]

It is not so wild I proved the hang was TRIM related over in the Endurance thread.

I have not expressed my thoughts well, my mistake. What I would suggest would be disabling trim (+OS restart to make sure everything is applied), do a secure erase with all methods you know and start the endurance again. Maybe writing at a lower speed due to lack of trim will trigger the Duraclass much later.

[One_Hertz]

Incorrect to my observations. TRIM occurs straight after a delete. It does not wait for the drive to become idle.

What he is saying is that the command is sent right after a delete to mark the relevant cells as empty... We do not know when the cells are actually erased on the NAND level.

[One_Hertz]

I doubleposted somehow.

[Ao1]

What he is saying is that the command is sent right after a delete to mark the relevant cells as empty... We do not know when the cells are actually erased on the NAND level.

When an application hangs you can be fairly sure the TRIM command is being executed in some form at the SSD level. Even if its not being executed at the NAND level the drive is inaccessable just after a large delete.

A wild guess. It is not the TRIM operation that takes the time, it is SF processing time. TRIMing a drive full of Ofil data takes the same or more time than TRIMing a drive full of non compressible data.

[Ao1]

I have not expressed my thoughts well, my mistake. What I would suggest would be disabling trim (+OS restart to make sure everything is applied), do a secure erase with all methods you know and start the endurance again. Maybe writing at a lower speed due to lack of trim will trigger the Duraclass much later.

Will do.

[johnw]

Its only the lowest 4 binary nibbles of that raw value that are power-on hours.
The upper 4 binary nibbles of that raw value are the number of milliseconds since last Power-on hours update.

Thanks! I had to look up "nibble". So the low 16-bits is the power-on hours.

90237262890248 = 0x521200000508
0x0508 = 1288

So the 1288 hours agree with the figure shown near the top in CT's post. But I still do not understand why the power-on hours are only 1288 when CT says the SSD has been on for about a year, and has not had the firmware upgraded.

[johnw]

What I would suggest would be disabling trim (+OS restart to make sure everything is applied), do a secure erase with all methods you know and start the endurance again. Maybe writing at a lower speed due to lack of trim will trigger the Duraclass much later.

So far, everything I have seen with Ao1's data is consistent with Sandforce's explanation of lifetime throttle as a cumulative linear write limit -- basically, a line (bytes written vs. power-on hours) that it will not allow you to exceed (at least, not for long). Based on Ao1's data, total bytes written / power-on time gives the slope of that line as about 20 MB/s for Ao1's SSD. Once you cross that line, it throttles the write speed back to about one-third of the line slope, 6 - 7 MB/s, until you have recrossed the line onto the safe side. Then it releases the throttle. It must throttle the write speed to less than the slope of the line, otherwise you would not cross back to the other side of the line.

Given this explanation, what does TRIM have to do with it?

[groberts101]

@ groberts101

#1 question According to my guestimates in post #27 you will need to write at least 10.75GB per "power on hour" to see throttling in the form I have experienced. You are nowhere close to that based on your SMART data. (I actually wrote ~28,352 GB in 168 power on hours)

#2 If a SE has restored the performance of that particular drive it is unlikely it had anything to do with this form of throttling that I have experienced.

#3 Also throughout running Anvils app there was not a significant slow down in performance. Overall performance was constant UNTIL throttling kicked in. When that happened write speeds dropped very quickly. (Over a few seconds)

#4 Now you might say that for whatever reason my numerous attempts to SE using three different apps across 2 PC's was a failure.

#5 Here is the kicker. Immediately after a SE write speeds were throttled. If the drive was degraded or throttled because the SE failed, why were speeds restored with nothing more than 1 hour of idle time?

sigh.. I'm trying very hard to stay with you here but you seriously need to stop thinking that you have this controller figured out here. Wipe some of the slate clean and you will be much further ahead.

It seems that I'm just repeating myself constantly. There are some known facts about these controllers and any other findings are pointing to user error or just plain bad "fact finding". Not trying to be ty at all and simply have limited time and often shoot from the hip with my responses so absolutley no offense is intended. Lets just drop the ego and pride out of it and use some facts that have been proven by many. IOW's,.. let's just make it about figuring out the issue with your drive and cut through all the other crap.

from above quote:

#1 QUIT GUESSTIMATING OFF THESE FIGURES! seriously dude... they have absolutley NOTHING to do with the "settled state" throttles(which is my preferenced term over the "lifetime thottle" being used for these drives. I already expalined it over on OCZ forums for you. Tony helped my figure most of this crap out by testing and studying them right along beside me with me being the "Terabytes of incompressible data written" guinea pig for many others. Hundreds upon hundreds have cleared their drives with an SE and is SOP when issues or heavy throttles(hammered state) arise.

And to be clear here.. I have had my drives(singles and arrays) throttled to all known stages(which I've also explained previously, as well). SE recovers/restores factory fresh speeds each and EVERY time. I may have passed the 100 SE mark but lost track long ago when I often did 7-8 per day when heavily testing them. I know these drives internal algorithms like a bad dream and even know EXACTLY when settled states hit my array. Not to be confused with hammered states. Small lags when heavily multitasking(and I mean HEAVY) and even logoffs/logons/shutdowns/reboots can be measurably longer by about 10 seconds.

It's like clockwork(settled states hitting me) and an SE/reimage(even using the same saved image as currently in use since I did try and exclude the old used W7 install as the culprit) will bring me back to fresh speeds with well over 100MB/s incompressible write speeds. After all.. I should know by now when my video transfers are down by that much, right?

#2 not even a remote possibility as I know my drives and their limitations very well for some of the reasons listed above in #1.

#3 This is where you are having difficulty getting your head wrapped around the stages of throttle on this controller. That massive slowdown would be considered a "hammered state throttle" and NOT a settled state. A settled state throttle will limit incompressible write speeds to about 70% of fresh speed. Hammered states would be about 50% of fresh when writing incompressible data. Why you are getting the drive drops way down to that 6-7MB/s range?.. I have no idea but can help you to more effectively test within the parameters of the lifespan test you are trying to achieve in the other thread.

Would need a lot more trust and opening of the mind before that would be possible, though. I am seriously interested in those endurance tests and coming around here(even trying over on the OCZ forum with you) to help and post what I know has nothing at all to do with trying to strut around and show off my Sandforce understanding. It's far more to do with what I think of myself and the need to share.. than what anyone else thinks of me as I;m not nearly that shallow. If you and everyone else keep putting pride and ego in the way then I'll just sit back and watch you all flounder around and develop even more false pride in "what you know" about these controllers. Again.. I'm trying to be patient with some here(you mostly) and will take some time to help you along as best I can but constantly revisiting things that have already been explained will surely just dry up my unthusiasm to do so. Sounds harsh and ultimatum like, I know.. but again with the "from the hip" attitude.

#4 I did say that early on but have since many times reworded that to say that there is surely soem other underlying issue at play. Over at OCZ forums I did acyually invite you to PM me as it's against the "forum recommendation" grain for me to recommend the way to do proper degradation/recovery tests on first gen Sandforce drives. I've helped many in PM environments and that should/has gone a ways towards letting those involved realize it's not about strutting around to act the "know it all" that some think I am. If I know it to be fact with quatifiable data?.. I'll tell you or even link it. If not?.. I'll clearly speculate and give detailed reasoning based on logic or past members/users experiences. I do know for fact that SE's can and often do go wrong with the ability to replicate failures at will. I now doubt that's the issue at play here.

#5 This is where the picture gets muddled and lost in translation. I see some clear issues with your perception of the throttled states starting to form as linked/discussed here in post #14.
http://www.ocztechnologyforum.com/fo...etime-Throttle

So, if the speeds were throttled immediately following an SE?.. there are a few possibilities from least likely to most.
1. The SE command did not initiate the drives internal ATA secure erase protocol. Seems unlikely at this point.

2. The drive has some form of firmware corruption or is just plain damaged. Also seems unlikely as these controllers don't just slow down and will panic lock quite easily if something is not right along those lines.

3. The testing methods to determine these states of throttle is flawed. Not sure if you were only relying on the AS SSD benchmark or if using Anvil's app? IOW, I need more clarification as to how you measured those speeds and after what amount of data written.

As for the "speeds being restored by an hour of if=dle time"? Well that right there will tell anyone that's familiar with a degraded Sandforce drive that there is ABSOLUTELY no way that a drive would recover from a degraded state in such a short timeframe. I've done oodles of degradation/recovery tests and seen more than my fair share of posters with speed issues over at OCZ forums and everywhere else for that matter, and none are able to invoke recovery in such a short timeframe. This controller is simply not aggressive enough to allow much moe than just a short little burst until the little fresh block recovery has been quickly spent again. And if the drive were actually in a "hammered state throttle"?.. there is NO, ZERO, ZIP, NADA.. chance that it would release any throttle whatsoever. The controller is actually coded to NOT release it that quickly and any form of recovery would definately mean that you were NOT in that state.

A lot to absorb and I can come off as a , I realize. But if you're seriously into learning and possibly fixing.. or at least modifying the testing parameters to achieve the desired result in the nedurance test?.. I may have some idea's to get you back into the running with this drive. Let me know.

[sergiu]

So far, everything I have seen with Ao1's data is consistent with Sandforce's explanation of lifetime throttle as a cumulative linear write limit -- basically, a line (bytes written vs. power-on hours) that it will not allow you to exceed (at least, not for long). Based on Ao1's data, total bytes written / power-on time gives the slope of that line as about 20 MB/s for Ao1's SSD. Once you cross that line, it throttles the write speed back to about one-third of the line slope, 6 - 7 MB/s, until you have recrossed the line onto the safe side. Then it releases the throttle. It must throttle the write speed to less than the slope of the line, otherwise you would not cross back to the other side of the line.

Given this explanation, what does TRIM have to do with it?

Normally, this is what I would also expect. Yet I find trim behavior to be strange. If no trim is available, the drive will erase blocks as it needs while writing data. What we know is that triggering a trim command will mark blocks for erasing which will be effective later just as with no trim. Marking the blocks would probably be writing a few pages which should not take too much so I suspect with trim there are other unknown things that are happening. That's unless trim is just a synchronous erase (operation which should probably have been implemented asynchronously)

[johnw]

And to be clear here.. I have had my drives(singles and arrays) throttled to all known stages(which I've also explained previously, as well). SE recovers/restores factory fresh speeds each and EVERY time.

Then you clearly have NOT had your drive in the state where it is throttled from writing too many GBs in too few power-on hours. Ao1 has. You are wrong, Ao1 is correct. You might try writing less and reading (and thinking) more.

[Ao1]

I have not expressed my thoughts well, my mistake. What I would suggest would be disabling trim (+OS restart to make sure everything is applied), do a secure erase with all methods you know and start the endurance again. Maybe writing at a lower speed due to lack of trim will trigger the Duraclass much later.

Sergiu, it's probably best to leave the V2 to idle for a week or so. Anything I do now will most likely be cut quite short.

I can do it now if you think that would be better. Your shot

[Ao1]

lol, groberts. I've described everything I am doing. AS SSD & CDM for performance testing. I'm using Anvils app to generate the writes. The very same app and workload that is being used for the X25-V & 320.

I test & post results. You can believe what I post or not.

Your SMART result shows you have not written anything close to a limit where life time throttling would have been deployed.

[sergiu]

Sergiu, it's probably best to leave the V2 to idle for a week or so. Anything I do now will most likely me quite short.

I can do it now if you think that would be better. Your shot

I believe a few hours of idle would be enough, let's say 5-8h.
I also have another theory based on behavior. Assuming that lifetime throttle is indeed around 50% then what we see is a completely abnormal throttling which might be related to some partial hardware failure. Another guess (this time completely wild) is that there is some electronic component that, due to extensive stress generated by writing, is heating up and no longer behaves normally thus inducing some kind of errors internally. These errors are either corrected successfully in a longer time, either, controller itself is switching to a lower internal speed. This unfortunately could be confirmed by running same tests on a second identical drive, preferably from another batch.

[Ao1]

I believe a few hours of idle would be enough, let's say 5-8h.
I also have another theory based on behavior. Assuming that lifetime throttle is indeed around 50% then what we see is a completely abnormal throttling which might be related to some partial hardware failure. Another guess (this time completely wild) is that there is some electronic component that, due to extensive stress generated by writing, is heating up and no longer behaves normally thus inducing some kind of errors internally. These errors are either corrected successfully in a longer time, either, controller itself is switching to a lower internal speed. This unfortunately could be confirmed by running same tests on a second identical drive, preferably from another batch.

It would be great if someone else could duplicate what I have done. Then its not me saying it again. I really doubt however that the drive is performing in any way different to how it was designed to behave.

Anyway, I'll start a new run tomorrow.

From the 1TB of 0fill experiment there were 128GB of writes

131,072MB in 2.85hours

45,990MB per hour

Avg write speed to NAND = 12.77MB/s

Anvils app was showing Avg ~100MB/s whilst it was running during this period.

[groberts101]

jesus friggin' christ!.. some people's ignorance here seems to know no bounds! If this is the constant I have to put up with here?.. then you guys can have your godamned "expert opinions. Come over to the OCZ forums and post some of this stupid ! You'll get schooled(unless of course they too relaize you're just wasted time and breath) and laughed at.

I've tested these Sandforce controllers more(timewise and specifically for degradation/recovery) than all here combined(aside from this type of Terabytes worth lifespan testing) and might just have more of a clue than you realize, eh? I won't be helping anyone here any further and will only help the OP over at OCZ forums.

@johnw.. You're so full of ego about you about things you don't know or have any firsthand experience with?.. that you belong over on the Anandtech forums. That place is FULL of experts like you. LOL

So, have at er' then guys... cause I'm done wasting my breath here. Some of you are so far off and poking holes in Sandforce reality here.. that you're gonna swamp the boat for others who actually want to stay afloat with this drive being in the test. Your loss but most will never know it due to thickly formed skulls blocking the whole ing learning process.

Here's some "data" for those who seem to have it all "figured out" as one final egotistical " you very much". Been a real pleasure.

Too damned lazy to dig through all the gigs worth of my tests so here's a quick'y real time test of a 50GB V2 that was destructive flashed by a "friend" due to a panic lock caused while running 9 x 4000MB sized CDM3 degradation tests.

stats for a 50GB V2

Fresh new drive from the package over 1 year ago.


Settled State today


Smart data for that same settled state drive


Fresh SE


Smart data after an SE


Now imagine doing this dozens upon dozens of time's with the addition of a hammered state being pushed on the drive. SE does in fact clear ALL throttling. Tony tried to tell you, Ryder tried to tell you, and I'm done wasting my time to help educate here. Like talking to a brick wall.

[zads]

sigh.. I'm trying very hard to stay with you here but you seriously need to stop thinking that you have this controller figured out here.
...
Lets just drop the ego and pride out of it and use some facts that have been proven by many.
...

Hm, I wonder if you could you do the same..?

...
#1
"settled state" throttles(which is my preferenced term over the "lifetime thottle" being used for these drives.
...

I think this is why you are a little bit mistaken.
Settled/steady state performance is much different from the Lifetime/Warranty throttle.
Their result might seem similar on the surface (much reduced write performance), but their causes are different. The settled performance degradation can be reset by SE, the Lifetime/Warranty throttle can not.
Settled/steady state performance is a consequence of the pages being filled with random incompressible data, rather than an actual firmware feature/built-in "throttle".
If you don't believe me, ask your trusted source Tony@OCZ.

...
Hundreds upon hundreds have cleared their drives with an SE and is SOP when issues or heavy throttles(hammered state) arise. And to be clear here.. I have had my drives(singles and arrays) throttled to all known stages(which I've also explained previously, as well). SE recovers/restores factory fresh speeds each and EVERY time.
...

Again, SE is clearing a steady state performance degradation.
How could you tell that the "hundreds upon hundreds" were experiencing a lifetime/warranty/hammered throttle state and not a steady state performance degradation?
If the SE restored performance; either the lifetime throttle released after sitting for some idle time,
or they are NOT experiencing a lifetime/warranty throttle!

...
I know these drives internal algorithms like a bad dream and even know EXACTLY when settled states hit my array. Not to be confused with hammered states.
...

Well, knowing test data in and out of a black box doesn't mean you know the internal algorithms.

What is your personal definition differentiation between settled and hammered state?
I assume your hammered state is referring to lifetime/warranty throttling..

...
#3 This is where you are having difficulty getting your head wrapped around the stages of throttle on this controller. That massive slowdown would be considered a "hammered state throttle" and NOT a settled state. A settled state throttle will limit incompressible write speeds to about 70% of fresh speed. Hammered states would be about 50% of fresh when writing incompressible data.
...

Please don't call it a "settled state throttle".
Please call it steady state/settled performance degradation.
Throttle implies that there is something in the firmware causing a reduction in performance on purpose.
This is not such a case.
Be careful with those numbers too. You aren't stating specific conditions for those performance numbers (70%), which will make it vary widely.
This is not absolute for steady state performance degradation.
It depends on existing data and transaction history, compression, etc.
I can make run steady state performance on any SF-1200/1500/2100/2200/2500 all over the board.
The 2500 starts off at 90k random 4k write IOPS, and I can run it steady state (>24 hours) from 8k to 60k IOPS.

Would need a lot more trust and opening of the mind before that would be possible, though.
...
It's far more to do with what I think of myself and the need to share.. than what anyone else thinks of me as I;m not nearly that shallow. If you and everyone else keep putting pride and ego in the way then I'll just sit back and watch you all flounder around and develop even more false pride in "what you know" about these controllers. Again.. I'm trying to be patient with some here(you mostly) and will take some time to help you along as best I can but constantly revisiting things that have already been explained will surely just dry up my unthusiasm to do so. Sounds harsh and ultimatum like, I know.. but again with the "from the hip" attitude.
....
lalalala
...

I understand you are trying to help here, and you have many of the basic concepts correct.
But there is some confusion/mixup in your definitions and understanding of what's going on in the controller.

Then you clearly have NOT had your drive in the state where it is throttled from writing too many GBs in too few power-on hours. Ao1 has. You are wrong, Ao1 is correct. You might try writing less and reading (and thinking) more.

I think you are correct that his drives never hit the lifetime/warranty throttle.
It can be hard to hit the lifetime throttle barrier if you have significant idle time on the drive.
For example, assuming steady state uncompressible sequential performance of 100MB/sec (just a made up number) on a 120GB SF-1200 drive with 34nm flash (5k PE cycles) with the life curve set at 1 year:
If you let it idle for a week before trying to hammer the drive, you have to hit it for more than 38 hours nonstop before it hits the theoretical limit line.
And then the controller will allow a little bit longer than that before it seriously cuts your writes.

[groberts101]

Your SMART result shows you have not written anything close to a limit where life time throttling would have been deployed.

LOL.. is right on taget there. Go ask Tony over at OCZ WHAT "lifetime throttling" actually is. It's another term for "settled state" and is looosely used and obviously misunderstood by most. He's already said here that it can and will be reset with an SE.
http://www.ocztechnologyforum.com/fo...etime-Throttle

you're the one's who are reading between the lines and are mistaken. Have at er'. Just wish I could say that you guys'll figure it out.. but doesn't look too promising so far.

[zads]

.. some people's ignorance here seems to know no bounds!
...
Come over to the OCZ forums and post some of this stupid ! You'll get schooled(unless of course they too relaize you're just wasted time and breath) and laughed at.

I've tested these Sandforce controllers more(timewise and specifically for degradation/recovery) than all here combined(aside from this type of Terabytes worth lifespan testing) and might just have more of a clue than you realize, eh?
...

Ever consider someone here's job is to do tier 1 OEM/enterprise-level validation on Sandforce SSDs?
That person would be getting paid quite a lot to know the Sandforce SSDs and their algorithms, internal mechanisms, modes of operation, limits of performance...
That person would also have access to the confidential datasheets, whitepapers, test data...
That person would have access to hundreds of sacrificial engineering test samples, done performance testing in all sorts of case loads, would have run lots of units through their entire PE cycles, would have access to analyze flash BER/UBER at end of warrantied PE cycles, and done countless other types of testing you haven't even heard about..
Yes, it is possible that maybe possibly someone on here has that resume.


Oh and by the way, OCZ might be number 1 or whatever in the SF retail market, but its just a joke in the enterprise segment. [/cheapshotlulz]

[groberts101]

have to call "BULL" on that one there buddy.. if that "were" the case and they were privy to that internal NDA info?.. they wouldn't touch this thread with a 10 foot pole or their 2 inch weiner. No one under NDA would be stupid enough to do that.

[sergiu]

I believe we need to have a common base:
100% Speed = FRESH State
70% Speed = SETTLED State
15% Speed = XXXXX State

If XXXXX is not "BROKEN" state, then it is a new state for us which is:
- unknown by OCZ or,
- well kept secret due to possible bad publicity or,
- something new introduced with latest firmwares, or
- any combination of the above

[zads]

have to call "BULL" on that one there buddy.. if that "were" the case and they were privy to that internal NDA info?.. they wouldn't touch this thread with a 10 foot pole or their 2 inch weiner. No one under NDA would be stupid enough to do that.

I'm not divulging anything proprietary or anything more than your buddies@OCZ are,
I'm just explaining it clearly and plainly for the benefit of people on this forum.
I also like how your feeling of insecurity in the matter has made you regress into a penis size insult- you mad?

[gojirasan]

It seems groberts101 is simply denying the existence of life/warranty throttling completely despite the repeated admissions of OCZ and boasts of this "performance throttling" right on Sandforce's web site. OCZ, Sandorce, and Ao1's testing are all obviously wrong. Ao1's data must be some kind of drive malfunction that strangely seems to mimic a data/time curve .

He seems to be asserting the existence of 3 and only 3 states:

1. Fresh. Drive is brand new or has been secure erased. No block assignments in the FTL and all blocks = 1.

2. Garbage collection begins. All blocks have been written too. So Sandforce must reluctantly start recycling used blocks with GC thus theoretically slowing down the drive if they ignored GC initially. This is what he is calling "settled". Why does the controller wait until there is absolutely no choice to start GC? Possibly in order to get higher benchmarks in drive reviews from reviewers who didn't write to every block on their drive before benchmarking. Or it could just be the way the GC algorithms were written. The algorithms may just ignore the "fresh" state because it is so brief.

Ao1 hasn't seen any difference between state 1 and 2. Maybe because most reviewers aren't being sent those small drives for review. So there is no advantage in delaying the normal GC operation. Or maybe at such slow right speeds there isn't any initial speed advantage to temporarily disabling GC.

3. Garbage collection failure. Inadequate GC cannot keep up with sustained writes and the drive has run out of clean blocks to write to. In order to prevent the kind of application interrupting stuttering that Ao1 has been seeing after a TRIM, you throttle the drive down to a speed which the GC process can keep up with. This is a good sort of throttling. Just enough to smooth out the curve. Ao1 hasn't seen this behavior. Probably due to a firmware bug. With sufficiently aggressive GC this short of throttling may not be necessary at all, but such aggressive GC may not be optimal for typical SSD usage patterns in terms of both peak speed and write amplification. So this sort of GC failure is not necessarily a bad thing.

[Ao1]

^ The workload from Anvil's app is not very aggresive. Mostly sequential. If it was 4K random full span, it would no doubt have seen performance degradation.

When I run the app with TRIM deactivated I expect to see performance degradation.

There is also one other form of throttling (and possibly others). Burst throttling.