Contents

Dask-MPI

Dask-MPI

Easily deploy Dask using MPI

The Dask-MPI project makes it easy to deploy Dask from within an existing MPI environment, such as one created with the common MPI command-line launchers mpirun or mpiexec. Such environments are commonly found in high performance supercomputers, academic research institutions, and other clusters where MPI has already been installed.

Dask-MPI provides two convenient interfaces to launch Dask, either from within a batch script or directly from the command-line.

Batch Script Example

You can turn your batch Python script into an MPI executable with the dask_mpi.initialize function.

from dask_mpi import initialize
initialize()

from dask.distributed import Client
client = Client()  # Connect this local process to remote workers

This makes your Python script launchable directly with mpirun or mpiexec.

mpirun -np 4 python my_client_script.py

This deploys the Dask scheduler and workers as well as the user’s Client process within a single cohesive MPI computation.

Command Line Example

Alternatively you can launch a Dask cluster directly from the command-line using the dask-mpi command and specifying a scheduler file where Dask can write connection information.

mpirun -np 4 dask-mpi --scheduler-file ~/dask-scheduler.json

You can then access this cluster either from a separate batch script or from an interactive session (such as a Jupyter Notebook) by referencing the same scheduler file that dask-mpi created.

from dask.distributed import Client
client = Client(scheduler_file='~/dask-scheduler.json')

Use Job Queuing System Directly

You can also use Dask Jobqueue to deploy Dask directly on a job queuing system like SLURM, SGE, PBS, LSF, Torque, or others. This can be especially nice when you want to dynamically scale your cluster during your computation, or for interactive use.

next

Installing