Results 1 to 12 of 12

Thread: multi-gpu under linux (ubuntu)?

  1. #1
    Xtreme Addict
    Join Date
    Jul 2006
    Posts
    1,124

    multi-gpu under linux (ubuntu)?

    I figured this may be the best place for this post since there's a higher chance of others here running into the same type of config. If wrong area, please move.


    Ok, I've been 'playing' with BTC mining for a bit, though historically I only had a single gpu on a MB. I now have a nice collection of radeon hd5770's (single slot 5-dp versions) and was thinking about having this serve as a mining system at least for off-hours, though am running into a problem with AMD's opencl not seeing all devices.

    Anyway, I mainly run ubuntu 10.04LTS 64bit server kernel (2.6.32-33) with desktop extensions as this is mainly a multi-monitor setup. I have 4 cards per system with (currently) two monitors attached per card (8 monitors). This is mainly for office applications and remote desktop so real-low gpu use.

    I have all monitors working just fine:
    $ aticonfig --list-adapters
    * 0. 08:00.0 ATI Radeon HD 5700 Series
    1. 06:00.0 ATI Radeon HD 5700 Series
    2. 07:00.0 ATI Radeon HD 5700 Series
    3. 03:00.0 ATI Radeon HD 5700 Series


    However with clinfo:
    Number of platforms: 1
    Platform Profile: FULL_PROFILE
    Platform Version: OpenCL 1.1 AMD-APP-SDK-v2.5 (684.213)
    Platform Name: AMD Accelerated Parallel Processing
    Platform Vendor: Advanced Micro Devices, Inc.
    Platform Extensions: cl_khr_icd cl_amd_event_callback cl_amd_offline_devices


    Platform Name: AMD Accelerated Parallel Processing
    Number of devices: 2
    Device Type: CL_DEVICE_TYPE_GPU

    ---

    Only the first GPU & CPU for the two devices.


    I've tried catalyst drivers 11.6; 11.7; 11.8 and SDK versions 2.1; 2.2; 2.4 & 2.5


    Basic question is that does anyone have multi-gpu (no crossfire (which by the way I tried that as well but that didn't get set up correctly under linux either but it DOES work under windows 7 on the same MB) working under linux?

    |.Server/Storage System.............|.Gaming/Work System..............................|.Sundry...... ............|
    |.Supermico X8DTH-6f................|.Asus Z9PE-D8 WS.................................|.HP LP3065 30"LCD Monitor.|
    |.(2) Xeon X5690....................|.2xE5-2643 v2....................................|.Mino lta magicolor 7450..|
    |.(192GB) Samsung PC10600 ECC.......|.2xEVGA nVidia GTX670 4GB........................|.Nikon coolscan 9000......|
    |.800W Redundant PSU................|.(8x8GB) Kingston DDR3-1600 ECC..................|.Quantum LTO-4HH..........|
    |.NEC Slimline DVD RW DL............|.Corsair AX1200..................................|........ .................|
    |.(..6) LSI 9200-8e HBAs............|.Lite-On iHBS112.................................|.Dell D820 Laptop.........|
    |.(..8) ST9300653SS (300GB) (RAID0).|.PA120.3, Apogee, MCW N&S bridge.................|...2.33Ghz; 8GB Ram;......|
    |.(112) ST2000DL003 (2TB) (RAIDZ2)..|.(1) Areca ARC1880ix-8 512MiB Cache..............|...DVDRW; 128GB SSD.......|
    |.(..2) ST9146803SS (146GB) (RAID-1)|.(8) Intel SSD 520 240GB (RAID6).................|...Ubuntu 12.04 64bit.....|
    |.Ubuntu 12.04 64bit Server.........|.Windows 7 x64 Pro...............................|............... ..........|

  2. #2
    Crunch-Fu Adept
    Join Date
    Sep 2005
    Location
    Czech Rep.
    Posts
    1,484
    $ sudo aticonfig --initial --adapter=all
    Sometimes a good slap in the face is all you need

    Bios my arss.....
    I can fix this problem with a hardware mod....
    Hipro5


    "Overclock till death. Overclocking is life." Hipro5

  3. #3
    Xtreme Addict
    Join Date
    Jul 2006
    Posts
    1,124
    Thanks, that was already done/ initialized with each driver package. I don't think it's a catalyst issue here as like I mentioned I can use all cards as for a normal desktop and they show up correctly with aticonfig --list-adapters. I really think it's an issue w/ the opencl SDK (or the right driver with the right SDK).

    Though as I mentioned I have tried quite a few combinations. At this point I've temporarily loaded up win7 and am using that (which sucks as cygwin is no-where as good as a true linux environment).

    I've also tested ubuntu's (all64bit) 10.04 desktop & server+desktop; 11.04 desktop & server+desktop; and 10.10 desktop. To see if it was some type of kernel issue.

    |.Server/Storage System.............|.Gaming/Work System..............................|.Sundry...... ............|
    |.Supermico X8DTH-6f................|.Asus Z9PE-D8 WS.................................|.HP LP3065 30"LCD Monitor.|
    |.(2) Xeon X5690....................|.2xE5-2643 v2....................................|.Mino lta magicolor 7450..|
    |.(192GB) Samsung PC10600 ECC.......|.2xEVGA nVidia GTX670 4GB........................|.Nikon coolscan 9000......|
    |.800W Redundant PSU................|.(8x8GB) Kingston DDR3-1600 ECC..................|.Quantum LTO-4HH..........|
    |.NEC Slimline DVD RW DL............|.Corsair AX1200..................................|........ .................|
    |.(..6) LSI 9200-8e HBAs............|.Lite-On iHBS112.................................|.Dell D820 Laptop.........|
    |.(..8) ST9300653SS (300GB) (RAID0).|.PA120.3, Apogee, MCW N&S bridge.................|...2.33Ghz; 8GB Ram;......|
    |.(112) ST2000DL003 (2TB) (RAIDZ2)..|.(1) Areca ARC1880ix-8 512MiB Cache..............|...DVDRW; 128GB SSD.......|
    |.(..2) ST9146803SS (146GB) (RAID-1)|.(8) Intel SSD 520 240GB (RAID6).................|...Ubuntu 12.04 64bit.....|
    |.Ubuntu 12.04 64bit Server.........|.Windows 7 x64 Pro...............................|............... ..........|

  4. #4
    Diablo 3! Who's Excited?
    Join Date
    May 2005
    Location
    Boulder, Colorado
    Posts
    9,440
    Only issue I've ever run into with 4 to 6 video cards on a single board was solved by what Aerou posted above. For reference I am using Ubuntu 10.04.2 x64 Desktop. Don't remember the driver revision but I think I'm using OpenCL 2.1 or 2.2, whichever is the last HD5K series exclusive that doesn't have the performance penalty.

  5. #5
    Xtreme Addict
    Join Date
    Jul 2006
    Posts
    1,124
    Quote Originally Posted by [XC] gomeler View Post
    Only issue I've ever run into with 4 to 6 video cards on a single board was solved by what Aerou posted above. For reference I am using Ubuntu 10.04.2 x64 Desktop. Don't remember the driver revision but I think I'm using OpenCL 2.1 or 2.2, whichever is the last HD5K series exclusive that doesn't have the performance penalty.
    So you haven't upgraded to 10.04.3 then? If you can get what version of kernel you have (uname -a); what driver you're running and what SDK that may help. I can hopefully use that as a new starting point.

    |.Server/Storage System.............|.Gaming/Work System..............................|.Sundry...... ............|
    |.Supermico X8DTH-6f................|.Asus Z9PE-D8 WS.................................|.HP LP3065 30"LCD Monitor.|
    |.(2) Xeon X5690....................|.2xE5-2643 v2....................................|.Mino lta magicolor 7450..|
    |.(192GB) Samsung PC10600 ECC.......|.2xEVGA nVidia GTX670 4GB........................|.Nikon coolscan 9000......|
    |.800W Redundant PSU................|.(8x8GB) Kingston DDR3-1600 ECC..................|.Quantum LTO-4HH..........|
    |.NEC Slimline DVD RW DL............|.Corsair AX1200..................................|........ .................|
    |.(..6) LSI 9200-8e HBAs............|.Lite-On iHBS112.................................|.Dell D820 Laptop.........|
    |.(..8) ST9300653SS (300GB) (RAID0).|.PA120.3, Apogee, MCW N&S bridge.................|...2.33Ghz; 8GB Ram;......|
    |.(112) ST2000DL003 (2TB) (RAIDZ2)..|.(1) Areca ARC1880ix-8 512MiB Cache..............|...DVDRW; 128GB SSD.......|
    |.(..2) ST9146803SS (146GB) (RAID-1)|.(8) Intel SSD 520 240GB (RAID6).................|...Ubuntu 12.04 64bit.....|
    |.Ubuntu 12.04 64bit Server.........|.Windows 7 x64 Pro...............................|............... ..........|

  6. #6
    Lil' Red Devil
    Join Date
    Jun 2002
    Posts
    258
    i have used a 5 card system under linuxcoin with smartcoin application without problems , may want to look in to this as a alternative


    Jen
    Beat on it till its BROKEN

    Then you can say you did it right



    Heatware 405-0-0

  7. #7
    Xtreme Addict
    Join Date
    Jul 2007
    Location
    Germany
    Posts
    1,592
    What's the output of fglrxinfo?
    The XS Folding@Home team needs your help! Join us and help fight diseases with your CPU and GPU!!


  8. #8
    Xtreme Addict
    Join Date
    Jul 2006
    Posts
    1,124
    @jen- thanks but this is not a dedicated miner system. it's my work desktop for numerous RDP's and other displays (all 2D/low gpu performance). Just wanted to do something more with the cards then just sit there. Right now until I can get this working under linux, I'm using W7 w/ cygwin on the same system and that works.

    @p2501:
    --
    $ export DISPLAY=:0 ; fglrxinfo
    display: :0.0 screen: 0
    OpenGL vendor string: ATI Technologies Inc.
    OpenGL renderer string: ATI Radeon HD 5700 Series
    OpenGL version string: 4.1.10834 Compatibility Profile Context
    ----

    note, like I mentioned above, the drivers themselves are working (I have 8 monitors on this system, 2 per card). I really think it's a problem between the opencl crap and hooks into the drivers. This is ubuntu 10.04.3LTS fyi.

    |.Server/Storage System.............|.Gaming/Work System..............................|.Sundry...... ............|
    |.Supermico X8DTH-6f................|.Asus Z9PE-D8 WS.................................|.HP LP3065 30"LCD Monitor.|
    |.(2) Xeon X5690....................|.2xE5-2643 v2....................................|.Mino lta magicolor 7450..|
    |.(192GB) Samsung PC10600 ECC.......|.2xEVGA nVidia GTX670 4GB........................|.Nikon coolscan 9000......|
    |.800W Redundant PSU................|.(8x8GB) Kingston DDR3-1600 ECC..................|.Quantum LTO-4HH..........|
    |.NEC Slimline DVD RW DL............|.Corsair AX1200..................................|........ .................|
    |.(..6) LSI 9200-8e HBAs............|.Lite-On iHBS112.................................|.Dell D820 Laptop.........|
    |.(..8) ST9300653SS (300GB) (RAID0).|.PA120.3, Apogee, MCW N&S bridge.................|...2.33Ghz; 8GB Ram;......|
    |.(112) ST2000DL003 (2TB) (RAIDZ2)..|.(1) Areca ARC1880ix-8 512MiB Cache..............|...DVDRW; 128GB SSD.......|
    |.(..2) ST9146803SS (146GB) (RAID-1)|.(8) Intel SSD 520 240GB (RAID6).................|...Ubuntu 12.04 64bit.....|
    |.Ubuntu 12.04 64bit Server.........|.Windows 7 x64 Pro...............................|............... ..........|

  9. #9
    Xtreme Addict
    Join Date
    Jul 2007
    Location
    Germany
    Posts
    1,592
    There is something wrong and it doesn't look like it's OpenCL. Fglrxinfo should show all supported and enabled devices, which it doesn't following what you posted. Meaning, your drivers are working alright but X doesn't know about it.

    What happens if you sudo dpkg-reconfigure xorg followed by sudo service gdm restart (not from a gnome shell) then sudo aticonfig --initial --adapter=all reboot.
    The XS Folding@Home team needs your help! Join us and help fight diseases with your CPU and GPU!!


  10. #10
    Xtreme Addict
    Join Date
    Jul 2006
    Posts
    1,124
    @p2501, see above. fglrxinfo never shows all adapters (tried 10.04, 10.10, 11.04 all fresh installs ubuntu's with above list of atidrivers). In /ALL/ cases the drivers work (compiz; multi-displays; and aticonfig --list-adapters will show all adapters). I've seen numerous reports that this is not abnormal with ATI (fglrxinfo not showing everything) on many sites. Since X is working (what teh aticonfig --initial --adapter=all is supposed to do which is set up the xorg.conf file) that's why I'm looking at the combination of driver & opencl.

    I know the hardware works as I've also loaded win7 on the same hardware and have used 11.6, 11.7, and 11.8 drivers all which work under windows. though in this case the opencl is installed (and my guess compiled with) the windows driver.

    |.Server/Storage System.............|.Gaming/Work System..............................|.Sundry...... ............|
    |.Supermico X8DTH-6f................|.Asus Z9PE-D8 WS.................................|.HP LP3065 30"LCD Monitor.|
    |.(2) Xeon X5690....................|.2xE5-2643 v2....................................|.Mino lta magicolor 7450..|
    |.(192GB) Samsung PC10600 ECC.......|.2xEVGA nVidia GTX670 4GB........................|.Nikon coolscan 9000......|
    |.800W Redundant PSU................|.(8x8GB) Kingston DDR3-1600 ECC..................|.Quantum LTO-4HH..........|
    |.NEC Slimline DVD RW DL............|.Corsair AX1200..................................|........ .................|
    |.(..6) LSI 9200-8e HBAs............|.Lite-On iHBS112.................................|.Dell D820 Laptop.........|
    |.(..8) ST9300653SS (300GB) (RAID0).|.PA120.3, Apogee, MCW N&S bridge.................|...2.33Ghz; 8GB Ram;......|
    |.(112) ST2000DL003 (2TB) (RAIDZ2)..|.(1) Areca ARC1880ix-8 512MiB Cache..............|...DVDRW; 128GB SSD.......|
    |.(..2) ST9146803SS (146GB) (RAID-1)|.(8) Intel SSD 520 240GB (RAID6).................|...Ubuntu 12.04 64bit.....|
    |.Ubuntu 12.04 64bit Server.........|.Windows 7 x64 Pro...............................|............... ..........|

  11. #11
    Xtreme Addict
    Join Date
    Jul 2007
    Location
    Germany
    Posts
    1,592
    Okay, as you noticed I didn't know that fglrxinfo is not always showing all devices. When that happened to me what I posted always fixed the situation.

    What's the output of

    ls /etc/OpenCL/vendors?

    Is your APP path correctly set in /etc/ld.so.conf.d/local.conf

    Did you maybe forgot to run ldconfig after setting the path?
    The XS Folding@Home team needs your help! Join us and help fight diseases with your CPU and GPU!!


  12. #12
    Xtreme Addict
    Join Date
    Jul 2006
    Posts
    1,124
    OpenCL has:
    etc/OpenCL/
    etc/OpenCL/vendors/
    etc/OpenCL/vendors/atiocl64.icd
    etc/OpenCL/vendors/atiocl32.icd


    (from v2.1 sdk). FYI, this is the location for up to 2.4, at 2.4 (when the name change happened to AMD from ATI, the location for these is not in /opt/ ) I've tried 2.1, 2.2., 2.3, 2.4, and 2.5 for the SDK.

    At this point I'm now trying the supplied divers w/ ubuntu (opposed to downloading 11.x from amd) and seeing if that works. Are you running 10.04 64bit as well? if so perhaps it would save some time if you could iterate what driver and sdk you are using?

    edit:
    also you can see that CLInfo (or cliinfo w/ 2.4+) does work so libraries and permissions are correct (otherwise since the program is dynamically linked it would not run) it is only showing just one card under openCL and the cpu. Not the 4 cards that I have in the system. So it looks like it's just not enumerating all the cards.

    ----
    <code>
    /opt/ati-stream-sdk-v2.1-lnx64/samples/opencl/bin/x86_64$ ./CLInfo
    Number of platforms: 1
    Platform Profile: FULL_PROFILE
    Platform Version: OpenCL 1.0 ATI-Stream-v2.1 (145)
    Platform Name: ATI Stream
    Platform Vendor: Advanced Micro Devices, Inc.
    Platform Extensions: cl_khr_icd


    Platform Name: ATI Stream
    Number of devices: 2
    Device Type: CL_DEVICE_TYPE_CPU
    Device ID: 4098
    Max compute units: 4
    Max work items dimensions: 3
    Max work items[0]: 1024
    Max work items[1]: 1024
    Max work items[2]: 1024
    Max work group size: 1024
    Preferred vector width char: 16
    Preferred vector width short: 8
    Preferred vector width int: 4
    Preferred vector width long: 2
    Preferred vector width float: 4
    Preferred vector width double: 0
    Max clock frequency: 1596Mhz
    Address bits: 64
    Max memory allocation: 1073741824
    Image support: No
    Max size of kernel argument: 4096
    Alignment (bits) of base address: 1024
    Minimum alignment (bytes) for any datatype: 128
    Single precision floating point capability
    Denorms: Yes
    Quiet NaNs: Yes
    Round to nearest even: Yes
    Round to zero: No
    Round to +ve and infinity: No
    IEEE754-2008 fused multiply-add: No
    Cache type: Read/Write
    Cache line size: 0
    Cache size: 0
    Global memory size: 3221225472
    Constant buffer size: 65536
    Max number of constant args: 8
    Local memory type: Global
    Local memory size: 32768
    Profiling timer resolution: 1
    Device endianess: Little
    Available: Yes
    Compiler available: Yes
    Execution capabilities:
    Execute OpenCL kernels: Yes
    Execute native function: No
    Queue properties:
    Out-of-Order: No
    Profiling : Yes
    Platform ID: 0x7fe0ba1d1228
    Name: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
    Vendor: GenuineIntel
    Driver version: 1.1
    Profile: FULL_PROFILE
    Version: OpenCL 1.0 ATI-Stream-v2.1 (145)
    Extensions: cl_khr_icd cl_amd_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_byte_addressable_store cl_khr_gl_sharing cl_ext_device_fission cl_amd_device_attribute_query cl_amd_printf
    Device Type: CL_DEVICE_TYPE_GPU
    Device ID: 4098
    Max compute units: 10
    Max work items dimensions: 3
    Max work items[0]: 256
    Max work items[1]: 256
    Max work items[2]: 256
    Max work group size: 256
    Preferred vector width char: 16
    Preferred vector width short: 8
    Preferred vector width int: 4
    Preferred vector width long: 2
    Preferred vector width float: 4
    Preferred vector width double: 0
    Max clock frequency: 850Mhz
    Address bits: 32
    Max memory allocation: 268435456
    Image support: Yes
    Max number of images read arguments: 128
    Max number of images write arguments: 8
    Max image 2D width: 8192
    Max image 2D height: 8192
    Max image 3D width: 2048
    Max image 3D height: 2048
    Max image 3D depth: 2048
    Max samplers within kernel: 16
    Max size of kernel argument: 1024
    Alignment (bits) of base address: 32768
    Minimum alignment (bytes) for any datatype: 128
    Single precision floating point capability
    Denorms: No
    Quiet NaNs: Yes
    Round to nearest even: Yes
    Round to zero: No
    Round to +ve and infinity: No
    IEEE754-2008 fused multiply-add: No
    Cache type: None
    Cache line size: 0
    Cache size: 0
    Global memory size: 268435456
    Constant buffer size: 65536
    Max number of constant args: 8
    Local memory type: Scratchpad
    Local memory size: 32768
    Profiling timer resolution: 1
    Device endianess: Little
    Available: Yes
    Compiler available: Yes
    Execution capabilities:
    Execute OpenCL kernels: Yes
    Execute native function: No
    Queue properties:
    Out-of-Order: No
    Profiling : Yes
    Platform ID: 0x7fe0ba1d1228
    Name: Juniper
    Vendor: Advanced Micro Devices, Inc.
    Driver version: CAL 1.4.1417
    Profile: FULL_PROFILE
    Version: OpenCL 1.0 ATI-Stream-v2.1 (145)
    Extensions: cl_khr_icd cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_byte_addressable_store cl_khr_gl_sharing cl_amd_device_attribute_query cl_amd_media_ops


    Passed!
    </code>
    Last edited by stevecs; 09-21-2011 at 04:33 AM.

    |.Server/Storage System.............|.Gaming/Work System..............................|.Sundry...... ............|
    |.Supermico X8DTH-6f................|.Asus Z9PE-D8 WS.................................|.HP LP3065 30"LCD Monitor.|
    |.(2) Xeon X5690....................|.2xE5-2643 v2....................................|.Mino lta magicolor 7450..|
    |.(192GB) Samsung PC10600 ECC.......|.2xEVGA nVidia GTX670 4GB........................|.Nikon coolscan 9000......|
    |.800W Redundant PSU................|.(8x8GB) Kingston DDR3-1600 ECC..................|.Quantum LTO-4HH..........|
    |.NEC Slimline DVD RW DL............|.Corsair AX1200..................................|........ .................|
    |.(..6) LSI 9200-8e HBAs............|.Lite-On iHBS112.................................|.Dell D820 Laptop.........|
    |.(..8) ST9300653SS (300GB) (RAID0).|.PA120.3, Apogee, MCW N&S bridge.................|...2.33Ghz; 8GB Ram;......|
    |.(112) ST2000DL003 (2TB) (RAIDZ2)..|.(1) Areca ARC1880ix-8 512MiB Cache..............|...DVDRW; 128GB SSD.......|
    |.(..2) ST9146803SS (146GB) (RAID-1)|.(8) Intel SSD 520 240GB (RAID6).................|...Ubuntu 12.04 64bit.....|
    |.Ubuntu 12.04 64bit Server.........|.Windows 7 x64 Pro...............................|............... ..........|

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •