multi-gpu under linux (ubuntu)?

[stevecs]

I figured this may be the best place for this post since there's a higher chance of others here running into the same type of config. If wrong area, please move.


Ok, I've been 'playing' with BTC mining for a bit, though historically I only had a single gpu on a MB. I now have a nice collection of radeon hd5770's (single slot 5-dp versions) and was thinking about having this serve as a mining system at least for off-hours, though am running into a problem with AMD's opencl not seeing all devices.

Anyway, I mainly run ubuntu 10.04LTS 64bit server kernel (2.6.32-33) with desktop extensions as this is mainly a multi-monitor setup. I have 4 cards per system with (currently) two monitors attached per card (8 monitors). This is mainly for office applications and remote desktop so real-low gpu use.

I have all monitors working just fine:
$ aticonfig --list-adapters
* 0. 08:00.0 ATI Radeon HD 5700 Series
1. 06:00.0 ATI Radeon HD 5700 Series
2. 07:00.0 ATI Radeon HD 5700 Series
3. 03:00.0 ATI Radeon HD 5700 Series


However with clinfo:
Number of platforms: 1
Platform Profile: FULL_PROFILE
Platform Version: OpenCL 1.1 AMD-APP-SDK-v2.5 (684.213)
Platform Name: AMD Accelerated Parallel Processing
Platform Vendor: Advanced Micro Devices, Inc.
Platform Extensions: cl_khr_icd cl_amd_event_callback cl_amd_offline_devices


Platform Name: AMD Accelerated Parallel Processing
Number of devices: 2
Device Type: CL_DEVICE_TYPE_GPU

---

Only the first GPU & CPU for the two devices.


I've tried catalyst drivers 11.6; 11.7; 11.8 and SDK versions 2.1; 2.2; 2.4 & 2.5


Basic question is that does anyone have multi-gpu (no crossfire (which by the way I tried that as well but that didn't get set up correctly under linux either but it DOES work under windows 7 on the same MB) working under linux?

[Aerou]

$ sudo aticonfig --initial --adapter=all

[stevecs]

Thanks, that was already done/ initialized with each driver package. I don't think it's a catalyst issue here as like I mentioned I can use all cards as for a normal desktop and they show up correctly with aticonfig --list-adapters. I really think it's an issue w/ the opencl SDK (or the right driver with the right SDK).

Though as I mentioned I have tried quite a few combinations. At this point I've temporarily loaded up win7 and am using that (which sucks as cygwin is no-where as good as a true linux environment).

I've also tested ubuntu's (all64bit) 10.04 desktop & server+desktop; 11.04 desktop & server+desktop; and 10.10 desktop. To see if it was some type of kernel issue.

[[XC] gomeler]

Only issue I've ever run into with 4 to 6 video cards on a single board was solved by what Aerou posted above. For reference I am using Ubuntu 10.04.2 x64 Desktop. Don't remember the driver revision but I think I'm using OpenCL 2.1 or 2.2, whichever is the last HD5K series exclusive that doesn't have the performance penalty.

[stevecs]

Only issue I've ever run into with 4 to 6 video cards on a single board was solved by what Aerou posted above. For reference I am using Ubuntu 10.04.2 x64 Desktop. Don't remember the driver revision but I think I'm using OpenCL 2.1 or 2.2, whichever is the last HD5K series exclusive that doesn't have the performance penalty.

So you haven't upgraded to 10.04.3 then? If you can get what version of kernel you have (uname -a); what driver you're running and what SDK that may help. I can hopefully use that as a new starting point.

[Jen]

i have used a 5 card system under linuxcoin with smartcoin application without problems , may want to look in to this as a alternative


Jen

[p2501]

What's the output of fglrxinfo?

[stevecs]

@jen- thanks but this is not a dedicated miner system. it's my work desktop for numerous RDP's and other displays (all 2D/low gpu performance). Just wanted to do something more with the cards then just sit there. Right now until I can get this working under linux, I'm using W7 w/ cygwin on the same system and that works.

@p2501:
--
$ export DISPLAY=:0 ; fglrxinfo
display: :0.0 screen: 0
OpenGL vendor string: ATI Technologies Inc.
OpenGL renderer string: ATI Radeon HD 5700 Series
OpenGL version string: 4.1.10834 Compatibility Profile Context
----

note, like I mentioned above, the drivers themselves are working (I have 8 monitors on this system, 2 per card). I really think it's a problem between the opencl crap and hooks into the drivers. This is ubuntu 10.04.3LTS fyi.

[p2501]

There is something wrong and it doesn't look like it's OpenCL. Fglrxinfo should show all supported and enabled devices, which it doesn't following what you posted. Meaning, your drivers are working alright but X doesn't know about it.

What happens if you sudo dpkg-reconfigure xorg followed by sudo service gdm restart (not from a gnome shell) then sudo aticonfig --initial --adapter=all reboot.

[stevecs]

@p2501, see above. fglrxinfo never shows all adapters (tried 10.04, 10.10, 11.04 all fresh installs ubuntu's with above list of atidrivers). In /ALL/ cases the drivers work (compiz; multi-displays; and aticonfig --list-adapters will show all adapters). I've seen numerous reports that this is not abnormal with ATI (fglrxinfo not showing everything) on many sites. Since X is working (what teh aticonfig --initial --adapter=all is supposed to do which is set up the xorg.conf file) that's why I'm looking at the combination of driver & opencl.

I know the hardware works as I've also loaded win7 on the same hardware and have used 11.6, 11.7, and 11.8 drivers all which work under windows. though in this case the opencl is installed (and my guess compiled with) the windows driver.

[p2501]

Okay, as you noticed I didn't know that fglrxinfo is not always showing all devices. When that happened to me what I posted always fixed the situation.

What's the output of

ls /etc/OpenCL/vendors?

Is your APP path correctly set in /etc/ld.so.conf.d/local.conf

Did you maybe forgot to run ldconfig after setting the path?

[stevecs]

OpenCL has:
etc/OpenCL/
etc/OpenCL/vendors/
etc/OpenCL/vendors/atiocl64.icd
etc/OpenCL/vendors/atiocl32.icd


(from v2.1 sdk). FYI, this is the location for up to 2.4, at 2.4 (when the name change happened to AMD from ATI, the location for these is not in /opt/ ) I've tried 2.1, 2.2., 2.3, 2.4, and 2.5 for the SDK.

At this point I'm now trying the supplied divers w/ ubuntu (opposed to downloading 11.x from amd) and seeing if that works. Are you running 10.04 64bit as well? if so perhaps it would save some time if you could iterate what driver and sdk you are using?

edit:
also you can see that CLInfo (or cliinfo w/ 2.4+) does work so libraries and permissions are correct (otherwise since the program is dynamically linked it would not run) it is only showing just one card under openCL and the cpu. Not the 4 cards that I have in the system. So it looks like it's just not enumerating all the cards.

----
<code>
/opt/ati-stream-sdk-v2.1-lnx64/samples/opencl/bin/x86_64$ ./CLInfo
Number of platforms: 1
Platform Profile: FULL_PROFILE
Platform Version: OpenCL 1.0 ATI-Stream-v2.1 (145)
Platform Name: ATI Stream
Platform Vendor: Advanced Micro Devices, Inc.
Platform Extensions: cl_khr_icd


Platform Name: ATI Stream
Number of devices: 2
Device Type: CL_DEVICE_TYPE_CPU
Device ID: 4098
Max compute units: 4
Max work items dimensions: 3
Max work items[0]: 1024
Max work items[1]: 1024
Max work items[2]: 1024
Max work group size: 1024
Preferred vector width char: 16
Preferred vector width short: 8
Preferred vector width int: 4
Preferred vector width long: 2
Preferred vector width float: 4
Preferred vector width double: 0
Max clock frequency: 1596Mhz
Address bits: 64
Max memory allocation: 1073741824
Image support: No
Max size of kernel argument: 4096
Alignment (bits) of base address: 1024
Minimum alignment (bytes) for any datatype: 128
Single precision floating point capability
Denorms: Yes
Quiet NaNs: Yes
Round to nearest even: Yes
Round to zero: No
Round to +ve and infinity: No
IEEE754-2008 fused multiply-add: No
Cache type: Read/Write
Cache line size: 0
Cache size: 0
Global memory size: 3221225472
Constant buffer size: 65536
Max number of constant args: 8
Local memory type: Global
Local memory size: 32768
Profiling timer resolution: 1
Device endianess: Little
Available: Yes
Compiler available: Yes
Execution capabilities:
Execute OpenCL kernels: Yes
Execute native function: No
Queue properties:
Out-of-Order: No
Profiling : Yes
Platform ID: 0x7fe0ba1d1228
Name: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
Vendor: GenuineIntel
Driver version: 1.1
Profile: FULL_PROFILE
Version: OpenCL 1.0 ATI-Stream-v2.1 (145)
Extensions: cl_khr_icd cl_amd_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_byte_addressable_store cl_khr_gl_sharing cl_ext_device_fission cl_amd_device_attribute_query cl_amd_printf
Device Type: CL_DEVICE_TYPE_GPU
Device ID: 4098
Max compute units: 10
Max work items dimensions: 3
Max work items[0]: 256
Max work items[1]: 256
Max work items[2]: 256
Max work group size: 256
Preferred vector width char: 16
Preferred vector width short: 8
Preferred vector width int: 4
Preferred vector width long: 2
Preferred vector width float: 4
Preferred vector width double: 0
Max clock frequency: 850Mhz
Address bits: 32
Max memory allocation: 268435456
Image support: Yes
Max number of images read arguments: 128
Max number of images write arguments: 8
Max image 2D width: 8192
Max image 2D height: 8192
Max image 3D width: 2048
Max image 3D height: 2048
Max image 3D depth: 2048
Max samplers within kernel: 16
Max size of kernel argument: 1024
Alignment (bits) of base address: 32768
Minimum alignment (bytes) for any datatype: 128
Single precision floating point capability
Denorms: No
Quiet NaNs: Yes
Round to nearest even: Yes
Round to zero: No
Round to +ve and infinity: No
IEEE754-2008 fused multiply-add: No
Cache type: None
Cache line size: 0
Cache size: 0
Global memory size: 268435456
Constant buffer size: 65536
Max number of constant args: 8
Local memory type: Scratchpad
Local memory size: 32768
Profiling timer resolution: 1
Device endianess: Little
Available: Yes
Compiler available: Yes
Execution capabilities:
Execute OpenCL kernels: Yes
Execute native function: No
Queue properties:
Out-of-Order: No
Profiling : Yes
Platform ID: 0x7fe0ba1d1228
Name: Juniper
Vendor: Advanced Micro Devices, Inc.
Driver version: CAL 1.4.1417
Profile: FULL_PROFILE
Version: OpenCL 1.0 ATI-Stream-v2.1 (145)
Extensions: cl_khr_icd cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_byte_addressable_store cl_khr_gl_sharing cl_amd_device_attribute_query cl_amd_media_ops


Passed!
</code>