dask_mpi.core.initialize

dask_mpi.core.initialize(interface=None, nthreads=1, local_directory='', memory_limit='auto', nanny=False, dashboard=True, dashboard_address=':8787', protocol=None, worker_class='distributed.Worker', worker_options=None, comm=None, exit=True)[source]

Initialize a Dask cluster using mpi4py

Using mpi4py, MPI rank 0 launches the Scheduler, MPI rank 1 passes through to the client script, and all other MPI ranks launch workers. All MPI ranks other than MPI rank 1 block while their event loops run.

In normal operation these ranks exit once rank 1 ends. If exit=False is set they instead return an bool indicating whether they are the client and should execute more client code, or a worker/scheduler who should not. In this case the user is responsible for the client calling send_close_signal when work is complete, and checking the returned value to choose further actions.

Parameters
interfacestr

Network interface like ‘eth0’ or ‘ib0’

nthreadsint

Number of threads per worker

local_directorystr

Directory to place worker files

memory_limitint, float, or ‘auto’

Number of bytes before spilling data to disk. This can be an integer (nbytes), float (fraction of total memory), or ‘auto’.

nannybool

Start workers in nanny process for management (deprecated, use worker_class instead)

dashboardbool

Enable Bokeh visual diagnostics

dashboard_addressstr

Bokeh port for visual diagnostics

protocolstr

Protocol like ‘inproc’ or ‘tcp’

worker_classstr

Class to use when creating workers

worker_optionsdict

Options to pass to workers

comm: mpi4py.MPI.Intracomm

Optional MPI communicator to use instead of COMM_WORLD

exit: bool

Whether to call sys.exit on the workers and schedulers when the event loop completes.

Returns
is_client: bool

Only returned if exit=False. Inidcates whether this rank should continue to run client code (True), or if it acts as a scheduler or worker (False).