-
Notifications
You must be signed in to change notification settings - Fork 490
OpenMPI and OpenSHMEM installation with UCX
Requirements: Autoconf 2.63 and above.
- Get latest version of the UCX code
$ git clone https://github.com/openucx/ucx.git ucx
$ cd ucx- Run autogen:
$ ./autogen.sh- (This step is only required for OpenPOWER platforms) Power 8 On Ubuntu platform the config.guess file is a bit outdated and does not have support for power. In order to resolve the issue you have to download an updated config.guess. From the root of the project:
$ wget https://github.com/shamisp/ucx/raw/topic/power8-config/config.guess- Configure:
$ mkdir build
$ cd build
$ ../configure --prefix=/your_install_pathNOTE: For best performance configuration, use
../contrib/configure-release.
This will strip all debugging and profiling code.
NOTE: For debug/development build configuration, use
../contrib/configure-devel.
This will enable all debugging and profiling code.
- Build and install:
$ make
$ make install- Get latest-and-greatest Open MPI version:
$ git clone https://github.com/open-mpi/ompi.git- Autogen:
$ cd ompi
$ ./autogen.pl- Configure with UCX:
$ mkdir build
$ cd build
$ ../configure --prefix=/your_install_path/ --with-ucx=/path_to_ucx_installationNOTE With Open MPI 4.0 and above, there could be compilation errors from "btl_uct" component. This component is not critical for using UCX; so it could be disabled this way:
$ ./configure ... --enable-mca-no-build=btl-uct ...
- Build:
$ make
$ make installExample of the command line (with optional flag to select IB device mlx5_0 port 1):
$ mpirun -np 2 -mca pml ucx -x UCX_NET_DEVICES=mlx5_0:1 ./app
IMPORTANT NOTE: Recent Open MPI versions contain a BTL component called 'uct', which can cause data corruption when enabled, due to conflict on malloc hooks between OPAL and UCM. In order to work-around this, use one of the following alternatives:
Alternative 1: Disable btl/uct in Open MPI build configuration:
$ ./configure ... --enable-mca-no-build=btl-uct ...Alternative 2: Disable btl/uct at runtime
$ mpirun -np 2 -mca pml ucx -mca btl ^uct -x UCX_NET_DEVICES=mlx5_0:1 ./app
- By default Open MPI enables build-in transports (BTLs), which may result in additional software overheads in the Open MPI progress function. In order to workaround this issue you may try to disable certain BTLs.
$ mpirun -np 2 -mca pml ucx --mca btl ^vader,tcp,openib,uct -x UCX_NET_DEVICES=mlx5_0:1 ./app
-
UCX current tarball: https://github.com/openucx/ucx/archive/master.zip
-
The table of MPI and OpenSHMEM distributions that are tested with the HEAD of UCX master
| MPI/OpenSHMEM project | |
|---|---|
| Open MPI/OSHMEM | 4.0 and above |
| MPICH | Latest |
In order to run UCX on Cray systems a user has explicitly enabled UCX unified mode: UCX_UNIFIED_MODE=1.
For example:
$ mpirun -np 2 -mca pml ucx --mca btl ^vader,tcp,openib,uct -x UCX_UNIFIED_MODE=1 ./app