IOMMU virtualization

[zir_blazer]

I have been researching about this matter from quite a bit of time ago, but there are some things that I simply didn't found anywhere, reason why I am asking here. But first you have to understand what I'm trying to accoplish.


I had some years ago an idea about how to set up and use a platform. As I like to have my computer running 24/7, one of the things that most irritates me about computers these days, is all the times that you have to restart your machine due to Drivers installs or Windows updates, or anything else, so basically you have annoying downtimes here and there. Sometimes you may also need a reset due to some application crashing, where albeit the machine is still working and you technically have full control of it, you can't do anything practical, like for example, when an unstable game that freezes leaving the game screen on top of all, even on top of the Task Manager, so you can't see what Process you are forcing to close and end up screwing it anyways. This also applies to multiOS support, because while dual booting is still somewhat popular, I find it annoying. Specially if for some reason you have an specific need only covered by the other OS (For example, if I wanted to play games, I would need to restart to boot into Windows). Basically, there are a lot of events that brings downtime with them.
These reasons are why even thorough I am a simple desktop user (Bah, I'm closer to a power user), I find virtualization very interesing. The actual idea would be to have a solid Hypervisor like Xen, and everything else running on top of it. This means that instead of having to restart the computer to switch from Windows to Linux or viceversa, I could simply alt tab from one to another because I could run them simultaneously. It would also allow me to make full backups copies rather easy, as dealing with virtual hard disk is quite similar to have a CD collection in ISOs, and even I could use them simultaneously. Anyways, I find virtualizing everything a solid approach for 24/7 availability, and removes part of the lazyness of experimentation. On a world that is always-on, you feel than that hour or two that you may be installing Windows, downloading the last version of the Drivers, etc, is time you missed from doing actual things or at least being connected to MSN/Skype, or being able to browse a Forum to check if you got a reply to a Thread you made.
Heck, I even suppose that as while virtualizing you could deal with each hardware piece as if it was a simple resource, you could have two Keyboards and Mouses plugged into a single machine, then make a Virtual Machine that uses exclusively only a single Keyboard and Mouse, and effectively have a multiuser computer.


Indeed, virtualization seems extremely useful as you suddently become a sort of OS overseer. This has two downsides: One is performance. Virtualization adds some overhead, but as for most task you still have more than enough performance in any decent modern machine, you can pay for that overhead happily if it does something useful for you. The second, however, was a show stopper: Video Card was always a difficult matter when dealing with virtualization, so you couldn't expect to install Windows and play games on it like if you were running natively. Although even three years ago I recall hearing that it was possible to do gaming via very specific Hypervisor features (Xen supported VGA passthorough, that allowed a VM to use the Video Card), a new, proper feature appeared: IOMMU virtualization. Basically, it would also you to virtualize add-in cards, that could possibily allow each VM OS to use its own Drivers for it and the like. And it would, finally, allow full support of GPUs inside a VM, as everything would run virtualized. Even BitCoin miners found this useful (http://forums.anandtech.com/showpost.php?p=33413433&postcount=6).


That IOMMU virtualization, is called AMD-Vi on AMD side, and on Intel, VT-d. Now, the problem comes to Hardware support. This is the actual reason why I made this Thread.

As far that I know, in AMD Socket AM3 platform you need Chipset and BIOS support for AMD-Vi. Only the 890FX, 970, 990X and 990FX Chipsets supports it. However, besides Chipset support, you also need BIOS support in the form of an ACPI IVRS table. This is what many Motherboards seems to be missing, so even if you have Hardware support, you can NOT use it.
This is the sad truth, but it makes tons of sense. When AMD-Vi was released for the 890FX, the Chipset itself was aimed at the high end "enthusiast" market. As it seems that "enthusiast" these days means "gamer or overclocker that likes to spend big bucks for the looks" (Definition that does NOT include true power users), many Motherboard makers seems to not have spend development time on make sure than the BIOS got a proper ACPI IVRS table for a feature that few, if no Desktop user, will use. The result is that you google around, you will see several Threads with people asking if X or Y Motherboard got it, and most of them will never ever get a response. I suppose that with the proper tools and knowledge of what you're searching for, you could simply download the BIOS of the Motherboard you want, then check the BIN file yourself to see if it is there and got data on it. This is better that hopelessly spend time making Threads expecting that someone else will have tried it with your Motherboard.
What I didn't found info about, however, is what happened on APU platforms regarding AMD-Vi support. I don't know if it is supported by a specific Chipset, CPU, and much less regarding actual BIOS support. I also don't really know what is going on at the Intel side. I know that VT-d support is a CPU feature. It is supported by most Core i5, with the notable exception of series K models that are intended for "enthusiasts", that does NOT have this feature (While normal Core i5 2500 or 2600 have VT-d, 2500K and 2600K do not. This also applies to vPro and TXT support). I consider one of the most retarded market segmentation schemes ever: Instead of just paying more for the Unlocked Multiplier, that is the main feature of a K model, you're actually doing a tradeoff bewthem that and VT-d, so Intel thinks that if you're an overclocker you don't do serious virtualization. However, I have no idea about if you need Chipset/BIOS support for it and how widespread it is.

Any insight or experiencies?

[MadMike261]

I run an ESXi box using direct hardware passthrough of a HBA controller to a VM. (Full story of my machine here (http://www.xtremesystems.org/forums/showthread.php?285347-Good-deal-low-price-ESXI-server&p=5176661&viewfull=1#post5176661))
You need hardware support, but also it needs to be enabled in the BIOS. As you said some hardware configs support it, but they left out BIOS support to enable it.
Best bet is looking for proven setups online and choose from that. Less risks. A lot of people use this feature like me, with a VM that runs ZFS. So it might be worth it to google in that direction.

As far as I know ESXi is the only hypervisor that supports this. And except for very very specific use cases no Enterprise will use this feature because it removes flexibility from the VM (eg. you cant vmotion it etc etc).

Theoretically what you want is possible. But you will need 1 Graphics Card per VM. As you can only assign a hardware passthrough device to 1 VM and this also makes it in-accesible to the ESXi host.
Also I am not sure how USB passthrough works for ESXi, so that might be tricky. On my own box I can see 2 internal usb controllers that I could passthrough directly to a VM if I would want. But I never tried this.

There are tools for attaching one set of keyboard/mouse to multiple machines. Hardware KVM or for example this -> http://synergy-foss.org/

Edit:
As for the K processors not supporting this. I am guessing they don't want people to run OC on hypervisors as it will probably introduce a whole world of trouble.
So probably just a support kind of thing.

[zir_blazer]

Theoretically what you want is possible. But you will need 1 Graphics Card per VM. As you can only assign a hardware passthrough device to 1 VM and this also makes it in-accesible to the ESXi host.[/url]
As far that I know, Xen also supported VGA passthorough from around 3 or 4 years ago, however, you didn't need neither AMD-Vi or Intel VT-d for doing that. The point is what you COULD do with either of those after you virtualize the Video Card. I don't know about the proper limitations of either method. I suppose that if you virtualize it instead of doing passthorough, you can let several VMs use the same GPU like you do with the CPUs, but I would need to confirm. This isn't exactly a well documented topic so is hard to come by with exact facts about what you can, or can't.



Edit:
As for the K processors not supporting this. I am guessing they don't want people to run OC on hypervisors as it will probably introduce a whole world of trouble.
So probably just a support kind of thing.
The Core i7 3930K does have both features (A Sandy Bridge-E, the lone exception of the K series), so there shouldn't be issues about overclocking and virtualizing (What is the worst that could happen? Resets? BSODs? Data loss? Any overclocker that runs stuff out-of-spec should be ready to face that. In a virtualized enviroment the risk could be much higher, but its still your responsability if it happens if your overclock isn't rock solid). Indeed, it also forces you to buy a more expensive LGA 2011 Motherboard, and twice the price tag on the Processor itself. It all points to be a market segmentation tactic more that any type of technical limitation.


I still didn't come by regarding info about how Socket FM1/FM2 supports AMD-Vi. As chances are that if I buy a new cheap machine, I would go for a Trinity (Or wait for Richland), how AMD-Vi support works on that platform is something purchase-defining.

[Andi64]

I find this post very interesting.

I've made some tests with ESXi (5.0 and 5.1) with AMD and Intel desktop boards. I work at IBM and I have worked with different environments using IOMMU, but on desktop boards, as you already stated, it kinda hit or miss.

I have and FM1, and can confirm that it does not support AMD-Vi, but I read in official documents that FM2 and A85X supports it. I wanted to migrate my FM1 hardware to FM2, but I don't have the money right now. So, I do not know of any board actually supporting AMD-Vi, but I know the platform has support for it. We could ask several of the users in this forums who have FM2 boards to search the option to enable or disable the IOMMI virtualization within BIOS, and make a list. That would be very helpful, 'cause as you said, a lot of people search for this info on forums or the internet and nobody can give a straight answer.

On AM3, I did have some kind of success playing with AMD-Vi. One of my other systems is a Phenom II X6 1055T on an Asrock 890FX Deluxe 4. I didn't have much time, because I had to make only non-destructive tests on that machine, and only tried once. There is (was?) a bug with Pass-through on ESXi 5.1 at that time, the USB controller failed to pass to the VM, but a NIC was working. Didn't have time to test anything else.

As you know, the only way to use VT-d with Intel is giving up on overclocking, because K series on LGA1155 does not support it, or going for LGA2011. I have two Intel rigs, a 1156 with a Core i3 530 and my primary system with an i7 3820. With LGA2011 you need C2 stepping to have VT-d, if you go for a 3930K or 3960X.

With the 3820 I've made a lot of tests, with my HD7950, USB controllers, and even with the Creative X-Fi XtremeGamer, which barely works in a non-virtual environment. I have a Gigabyte X79-UD3, which supports VT-d, and everything works perfectly, except for the X-Fi.

The i3 530, as you know, does not support VT-d, but the Core i5 series does. I can't get my hands on a LGA1156 i5, that isn't expensive as a brand new i5, so I couldn't test?it, but the motherboard that I have, BIOSTAR TH55B HD, supports VT-d in the BIOS.

[WrigleyVillain]

Yeah "VT-d", "d" for "directed I/O". There is also standard "VT-x" which most if not all modern desktop Intel CPUs support, I believe.

As for 1156 i5s you got me looking at the Intel ARK and interestingly enough the i5 7xx does not support VT-d but the i5 6xx does. It's all marketing decisions as to what CPU has or doesn't have what features like this, man, not that that really helps those particulars make any more sense sometimes as we have just seen.

[[XC] Synthetickiller]

I have and FM1, and can confirm that it does not support AMD-Vi, but I read in official documents that FM2 and A85X supports it. I wanted to migrate my FM1 hardware to FM2, but I don't have the money right now. So, I do not know of any board actually supporting AMD-Vi, but I know the platform has support for it. We could ask several of the users in this forums who have FM2 boards to search the option to enable or disable the IOMMI virtualization within BIOS, and make a list. That would be very helpful, 'cause as you said, a lot of people search for this info on forums or the internet and nobody can give a straight answer.


I can confirm that there's IOMMU support with FM2: (sorry for the horrible cell phone pic)
http://i48.tinypic.com/181yjt.jpg


This is an A10-5800K & SUS F2A85-V (A85X). Awesome Board!


I downloaded the booklet (as I don't feel like hooking it up) another FM2 rig I have built. A75 chipset & lacks support.

I'm working if only A85 chipsets have IOMMU support?

[Andi64]

Yes, as far as I know only the A85 chipset supports IOMMU. FM1 doesn't support it, so, any chipset that was launched for FM1 will not support IOMMU on FM2. You need CPU, Chipset and BIOS support.

[madman2233]

I just ran into this problem on my board. No ACPI IVRS table. I was running Arch Linux with KVM and had everything setup to work with IOMMU, but on boot linux says it had to disable IOMMU because of faulty motherboard firmware and some other error about IVRS values.

[NEOAethyr]

I get this error with xen 4.1:
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) FATAL PAGE FAULT
(XEN) [error_code=0000]
(XEN) Faulting linear address: 0000000000000000
(XEN) ****************************************

After disabling iommu it gets past that point, gets to the login.
At this point, tty7 and above is busy doing nothing, they don't work.
Sometimes tty0 gets busy and won't finish loading up to the login prompt.
However tty1-6 work.
Startx not working, wth..., gdm3 start is working though, even though I prefer kde plasma lol.
When trying gdm3, it loads up a bit, goes to a blank screen and just sits there like tty7+.

I'm using kali linux, which has given me nothing but probs.
Backtrack 5r3 was better in my opinion, less bugs, and much easier to setup.

I got many thing working again, but of all things, I have to install deb packs from the freakin cmd line.
Apt-get server list thingy had to be redone to get it to download anything at all...
I had to get rid of that darned nouveau thing, that made it a pain to get the real drivers working.
I could go on and on lol.

I'm still dl'ing ubuntu and debian.
From my understanding though they already have that nouveau crap installed :\.
I need to roll my own again like way back in the day, but I need good experience with a working os 1st.

My board is missing the acpi iommu rom as well.
Kind of ticks me off because that feature is something I wanted and figured I was getting when I bought this board+cpu a few years back now.
It's no wonder that I never got any pass through working on a windows baised os vm.
Real shame too, 6 core's, not really put to use when it comes to the os it's self.

Ohwell, at least eventually, hopefully, I'll get xen wokring to play with, even without passthrough.
Just without it it won't be the same, I won't beable to use that setup, just play with it as a toy.
I got this grid gpu and nothing to use it with lol, thanks asus ^^ :\.
Hopefully since the setting for it is in the bios, just no rom, maybe it'll be an easy fix by just inserting the module form another board.
But then again, being apart of acpi, it might be a real pain to insert and get it hooked up right in the bios.
This is like the sli mods, except it's a new thing to mess with :(.

But anyways I'm in the same situation as the thread op.
I want to get a hypervisor working with passthrough, and in the end, my grid gpu working.
So each os can share the same gpu.

[zir_blazer]

I posted this (http://lists.xen.org/archives/html/xen-users/2013-04/msg00115.html) on Xen Mailing List. Got some useful replies, mainly this (http://lists.xen.org/archives/html/xen-users/2013-04/msg00126.html) and this (http://lists.xen.org/archives/html/xen-users/2013-04/msg00148.html).


The resumen:
* In order to do passthrough, you need to have that Hardware part virtualized. So virtualization and passthrough are not two different methods, but one is a prerequisite for the other to work. And that continues to the next point:
* VT-d was introduced on Wolfdale supporting Chipsets. It seems to be a Chipset feature on that platform, unrelated to the Processor, as even on an older Conroe based Core 2 Duo that doesn't officialy includes VT-d support (On some Wolfdales is mentioned as a feature on Intel Ark), VGA passthrough worked. I originally though that on the VGA passthrough video with the Core 2 Duo, the GPU wasn't virtualized because I didn't know that it was actually using VT-d.
* There was another Thread (http://lists.xen.org/archives/html/xen-users/2013-04/msg00139.html) asking about IOMMU virtualization with a Trinity. Got pretty much no replies. So it seems that its REALLY unknow how widespread AMD-Vi support is on that platform, or the actual Chipsets requeriments (I know some guys here said that it was an A85X feature, but I didn't found anything else that claims that only it supports it and the other Chipsets do not).
* Motherboard/BIOS is still the most important piece of the puzzle. One of the guys that replied me said that some extra chips on a Motherboard, like the PCI Bridges used to split PCIe lanes, doesn't seem to work with IOMMU virtualization and can render unusable for this purpose any PCIe slot connected to it. This means that you're more likely to get all your Motherboard components working properly virtualized the more simple it is, so maybe you want to avoid high end Motherboards that got an extra SATA Controller, another USB Controller, etc. Basically, you need to get the most out of the existing Northbridge and Southbridge chips. This means that many high end Motherboards could be discarded on sight due to them being filled with all those extra features that you will not be able to virtualize on the first place.



I have yet to resend AMD the ticket asking about AMD-Vi support on Fusion platforms, and also pay a visit to Coreboot Mailing List. I don't have idea about how much Coreboot progressed on the virtualization area, but they claim that its possible (http://www.coreboot.org/Benefits#Possible_Advantages) for it to support features that the shipping Motherboard's BIOS doesn't have. I don't know if they already managed to get working VT-d or AMD-Vi. Assuming they do, you have at least a minor hope of somehow getting those features working on your Motherboard in case your Motherboard manufacturer doesn't provides the proper support.
Now, making Coreboot to work on your Motherboard is yet another nightmare, worse that finding one that works properly out of the box, because you need a lot of tools and knowledge to get it running. Failure means high chances of bricking your Motherboard, unless you have a spare Flash chip with the original BIOS or a Flash reprogrammer at hand. It seems to be a mammoth task, maybe of moderate difficulty for someone with programming experience, but don't expect any type of Joe Average to succed here. Assuming I could manage to get a hand with that matter, the idea would be to purchase a simple Socket FM2 Motherboard (High quality construction with no extra controllers or fancy features, like the old Biostar TForce style) expecting that I could make Coreboot work on it and thus the missing virtualization features.

[NEOAethyr]

I found out you're not supposed to use xen server as an os, just as a server.
Supposedly it can be done but I don't know.
What I expected to do was to use the same system as multiple systems with multiple screens and inputs hooked up.
So I could multiplayer on a virtual lan on the same pc.

I haven't messed with linux in a few days, I'll get back to that eventually soon.
I'de like to get this xen server setup the way I want it, as an os with gui.
Hopefully sooner or later I'll be booting straight to that os and always that os :).
I already know now it's not supposed to be done but hopefully it's still possible.

[zir_blazer]

Well, my Ticket got answered. Not with the responses I wanted to...



You may want to read this Forum Thread
http://www.xtremesystems.org/forums/showthread.php?285408-IOMMU-virtualization

Question 1: Basically, I'm aware that in order to have AMD-Vi support, if I am using a Socket AM3/AM3+ platform, the only Chipsets that support it are AMD 890FX, 970, 990X and 990FX. However, I didn't manage to get any data regarding on support on Socket FM1/FM2 platforms. Do I need an specific Chipset/APU to have that feature?

Question 2: I'm also aware that in order to have AMD-Vi support working properly, besides one of those Chipsets, I also need BIOS support for it, in the form of an ACPI IVRS table. Many Motherboards makers seems to have skipped adding support for it, so even thorough at least in Socket AM3 I know what Chipsets I need to have Hardware support, the firmware support isn't there. So it is basically non functional. As AMD-Vi support doesn't seems to be an important feature on Desktop platforms, there seems to be little info regarding an specific Motherboard support for it, as it isn't clearly specified on a Motherboard specifications if has BIOS supports for it or not. Is there any way to know if a BIOS should have support for it without resorting to test it? Like, for example, downloading that Motherboard's Manual to check if it has a AMD-Vi/IOMMU setting in the BIOS that could be enabled.

Question 3: A bit unrelated, but I would like to know if for Turbo Core 2.0 support on Bulldozer/Piledriver for Socket AM3/AM3+ platform I need to have a specific Chipset, or it just works with BIOS support. I'm asking because there are many Motherboards that are for Socket AM3+ (The black socket launched pre-Bulldozer) yet uses old Chipsets like AMD 760G. Do I miss any feature advertised as part of the CPU if I use it in an older Chipset?
Question 3 is related to this Thread (http://www.xtremesystems.org/forums/showthread.php?285505-Requeriments-for-Turbo-Core-2.0-support-on-Bulldozer-Piledriver).

Anyways, the response:


I apologize for the delay in this response, as there were issues with our system that seemingly had this ticket lost for a while. Please see my answers below to your 3 questions.

ONE.
> It is my understanding that AMD-Vi (Virtualization) is supported by all current AMD processors. It is not dependant upon chipset.

TWO.
> At the end of the day, I suppose the motherboard manufactuer can disable/enable any amount of features and functionality with their boards and the BIOS setttings provided. I think it is in your best interest to contact the motherboard manufacturer (and check manuals) prior to purchse to ensure the board will do what you want.

THREE.
> Other chipsets may limit the type of memory able to be used, and will also use older integrated graphics chipset and components. I see very little point in building a system with a current CPU, but using an older chipset.

100% sure the guy totally confused AMD-Vi with AMD-V. Third question also doesn't make me confident, because Chipset is totally unrelated to supported memory type from the K8 era, and that the guy is also unaware that many manufacturers sells low end AM3+ Motherboards for Bulldozer with older 7xx or 8xx Chipsets. Guess I will have to reply the Ticket and wait three weeks. Again.


I also PMed The Stilt. Not going to copypaste it without asking, but he said basically that AMD-Vi is unrelated to the Chipset in Fusion platforms and that it was oficially introduced in the Bulldozer die, so I suppose that you can get it working on FM2 with Trinity but absolutely not with FM1, as Llano doesn't have it neither on itself nor on the Chipsets. What I understanded is that you could use ANY FM2 Motherboard with a Trinity and you should have Hardware AMD-Vi support, albeit you still need to sort things regarding BIOS support. I hope that he posts here for clarification.

[zir_blazer]

I received reply from AMD Customer Care some days ago but didn't took time to post it here due to the seemingly lack of interest of people in this Thread. Anyways, I replied explaining that I wanted to know about AMD-Vi, not AMD-V, and got this reply:


I apologize for not entirely understanding your questions. As you can see, the AMD GCC group is not the best group of people within AMD to address your questions. We deal more specifically with end-users of Radeon graphics products. I would recommend you contact AMD's Embedded Support Group to ensure you get detailed and accurate answers to your questions and concerns.
If you are a Developer - building software or a hardware platform or implementation of some sort, you may also consider AMD's Developer Support.

Embedded:
http://wwwd.amd.com/amd/devsite.nsf/home/welcome.htm?opendocument&login

Developer:
http://developer.amd.com/support/


Anyways. Googling around I found a guide (http://www.overclock.net/t/1205216/guide-create-a-gaming-virtual-machine) by a guy that says how to use Xen to make a VM with Windows for gaming. Check this 60 page Thread. THIS is specifically what I want IOMMU virtualization for. Its the future of the true power users.

[Andi64]

I posted this (http://lists.xen.org/archives/html/xen-users/2013-04/msg00115.html) on Xen Mailing List. Got some useful replies, mainly this (http://lists.xen.org/archives/html/xen-users/2013-04/msg00126.html) and this (http://lists.xen.org/archives/html/xen-users/2013-04/msg00148.html).


The resumen:
* In order to do passthrough, you need to have that Hardware part virtualized. So virtualization and passthrough are not two different methods, but one is a prerequisite for the other to work. And that continues to the next point:
* VT-d was introduced on Wolfdale supporting Chipsets. It seems to be a Chipset feature on that platform, unrelated to the Processor, as even on an older Conroe based Core 2 Duo that doesn't officialy includes VT-d support (On some Wolfdales is mentioned as a feature on Intel Ark), VGA passthrough worked. I originally though that on the VGA passthrough video with the Core 2 Duo, the GPU wasn't virtualized because I didn't know that it was actually using VT-d.
* There was another Thread (http://lists.xen.org/archives/html/xen-users/2013-04/msg00139.html) asking about IOMMU virtualization with a Trinity. Got pretty much no replies. So it seems that its REALLY unknow how widespread AMD-Vi support is on that platform, or the actual Chipsets requeriments (I know some guys here said that it was an A85X feature, but I didn't found anything else that claims that only it supports it and the other Chipsets do not).
* Motherboard/BIOS is still the most important piece of the puzzle. One of the guys that replied me said that some extra chips on a Motherboard, like the PCI Bridges used to split PCIe lanes, doesn't seem to work with IOMMU virtualization and can render unusable for this purpose any PCIe slot connected to it. This means that you're more likely to get all your Motherboard components working properly virtualized the more simple it is, so maybe you want to avoid high end Motherboards that got an extra SATA Controller, another USB Controller, etc. Basically, you need to get the most out of the existing Northbridge and Southbridge chips. This means that many high end Motherboards could be discarded on sight due to them being filled with all those extra features that you will not be able to virtualize on the first place.



I have yet to resend AMD the ticket asking about AMD-Vi support on Fusion platforms, and also pay a visit to Coreboot Mailing List. I don't have idea about how much Coreboot progressed on the virtualization area, but they claim that its possible (http://www.coreboot.org/Benefits#Possible_Advantages) for it to support features that the shipping Motherboard's BIOS doesn't have. I don't know if they already managed to get working VT-d or AMD-Vi. Assuming they do, you have at least a minor hope of somehow getting those features working on your Motherboard in case your Motherboard manufacturer doesn't provides the proper support.
Now, making Coreboot to work on your Motherboard is yet another nightmare, worse that finding one that works properly out of the box, because you need a lot of tools and knowledge to get it running. Failure means high chances of bricking your Motherboard, unless you have a spare Flash chip with the original BIOS or a Flash reprogrammer at hand. It seems to be a mammoth task, maybe of moderate difficulty for someone with programming experience, but don't expect any type of Joe Average to succed here. Assuming I could manage to get a hand with that matter, the idea would be to purchase a simple Socket FM2 Motherboard (High quality construction with no extra controllers or fancy features, like the old Biostar TForce style) expecting that I could make Coreboot work on it and thus the missing virtualization features.

I have a A75 platform, and can confirm it doesn't support AMD-Vi. There is no BIOS option, and I'm almost sure that the CPUs doesn't support it either. FM2 does support it, but you need a good board an a A85X chipset.

I can't help you with Xen, never tried it. I work with VMware, and I've implemented solutions with VT-d. Also in my home lab I have VT-d working. My i7 3820 has a VM with Windows 7, using passthrough I've managed to make USB3, HD7950 and the Marvell SATA3 controllers to work, but failed with the Creative X-Fi XtremeGamer.

[zir_blazer]

I have a A75 platform, and can confirm it doesn't support AMD-Vi. There is no BIOS option, and I'm almost sure that the CPUs doesn't support it either. FM2 does support it, but you need a good board an a A85X chipset.
I was just checking some AsRock Motherboards, and at least the FM2A75M-DGS (http://www.asrock.com/mb/overview.asp?cat=Download&os=Beta&Model=FM2A75M-DGS) with A75 Chipset AND even the FM2A55 Pro (http://www.asrock.com/mb/AMD/FM2A55%20Pro/?cat=Beta) with an A55, got recent beta BIOS updates to "add IOMMU option". It looks like they do support it, but your Motherboard manufacturer choose to not enable it. I hear comments googling around that AsRock is one of the best supporters of IOMMU around, so this didn't look surprising.


BTW, The Stilt said in a PM that the IOMMU was incorporated into the Bulldozer die. As he didn't replied my next PM with my theory (Going to re-send another one), at least I will expose it here:

It is, at least confirmed, that support on AM3 platform for the IOMMU appeared first with the AMD 890FX Chipset (Tons of google info about it). However, for an official AMD tech doc of the 900 series Chipsets here (http://support.amd.com/us/ChipsetMotherboard_TechDocs/48691.pdf), they say on Page 9, 1.1 the following:


The RD990 also supports a revision 1.26 compliant IOMMU (Input/Output Memory Management Unit) implementation
for address translation and protection services. This feature allows virtual addresses from PCI Express endpoint devices to
be translated to physical memory addresses. On-chip caching of address translations is provided to improve I/O
performance. The device is also compliant with revision 1.0 of the PCI Express Address Translation Services (ATS)
specification to enable ATS-compliant endpoint devices to cache address translation. These features enhance memory
protection and support hardware-based I/O virtualization when combined with appropriate operating system or hypervisor
software. Combined with AMD Virtualization™ (AMD-V™) technology, these features are designed to provide
comprehensive platform level virtualization support.

Basically, they're stating that only the 990FX (RD990) Chipset has an IOMMU, while the other 3 (970, 980G, 990X) does NOT. However, according to a Xen support list (http://wiki.xensource.com/xenwiki/VTdHowTo), the 970 and 990X does have IOMMU support...

What I'm thinking is that you actually got two IOMMUs on Socket AM3: The one in the Chipset (That is ONLY available on 890FX and 990FX), and the one of the Bulldozer CPUs. As you're supposed to use Bulldozer on a 900 series Chipset, you actually have IOMMU, just that its not coming from the Chipset but the CPU. Only thing that doesn't fit in that theory is why 980G is not included has supporting IOMMU, maybe it wasn't tested on a Motherboard with proper BIOS. So basically, the following combinations should be available:

Socket AM3: K10 with 890FX Chipset
Socket AM3+: K10 with 990FX Chipset
Socket AM3+: Bulldozer with ANY Chipset (I would say that you could use one of those budget, old 700 series Chipset for AM3+ and still got IOMMU support with proper BIOS)
Socket AM3+: Bulldozer with 990FX Chipset (You effectively have 2 IOMMUs, but use only one)

In case that you use a K10 on 970, 980G and 990X, you shouldn't be able to get the IOMMU working as there simply isn't one. I just need to get confirmation that someone did get the IOMMU working with a Bulldozer on a 980G Chipset.
Now, the fact that the IOMMU is part of Bulldozer, means that Trinity should have it build-in on the APU itself, regardless of Chipset (The Stilt also said that AMD-Vi on FM2 isn't related to Chipset). It simply means that as the lower Chipsets are considered for budget, most Motherboard manufacturers didn't even bothered to spend time add support for it. Maybe it means too much work for a cheap Motherboard.

Meanwhile, FM1 support should never get IOMMU support: Its a K10 so it is not on the APU. And there is none on the Chipset, either.

[zir_blazer]

*BUMP*

I had some research progress on this area, but its slow as hell. I am currently looking for a LGA 1150 Motherboard that has full VT-d support for Haswell for my soon-to-be new build. I don't care if its a Desktop Motherboard or a Server one, for as long as I am aware of what feature win/loss I may expect (I don't like Server Motherboards because that had absolutely NO control on Processor Voltage and the like, but as far that I am aware, most are like that. The fact that I will not be able to overclock with a non-K Haswell to have VT-d, doesn't means that I will not try to get the best possible undervolt at default Frequency).

AsRock confirmed that they got VT-d support on most Haswell Chipsets here (http://lists.xen.org/archives/html/xen-users/2013-06/msg00083.html), while ASUS says (http://www.xtremesystems.org/forums/showthread.php?286345-ASUS-Z87-Motherboards-Overview-Guides-and-Official-Support&p=5191955&viewfull=1#post5191955) that they don't include "pseudo support" because there are some test from their test battery that fails on non-Q Chipsets (However, as far that I am aware, just by looking at other manufacturers working support I would assume this to be false. Besides, at least on AMD platforms, there are many that complains that ASUS DO include a BIOS option to enable IOMMU, even if support is incomplete and not working).

Anyone that got this working on a Haswell platform is welcomed.

[pcunite]

zir_blazer,

I'm very interested in having a system as you've described. To make sure I understand, I'll describe in my own words what I want.

* I turn on my computer and it boots to some OS ... Linux, Windows, whatever.
* An option to load a VM image from a menu screen (vhd drives or whatever) is accessible.
* I select a Windows 7 image and it loads (optionally) full screen.
* ALT-TAB'ing I can load another image of say, Linux.
* Both images are now running with complete access to the local hardware.
* The two running images are actually only writing back to their respective .vhd files.

This would make it awesome to reinstall a VM image as I make changes to it over time. With SSD and new PCIe drives coming out, writing back to a single file will be plenty fast enough.