SLURMCluster(queue=None, project=None, walltime=None, job_cpu=None, job_mem=None, job_extra=None, config_name='slurm', **kwargs)¶
Launch Dask on a SLURM cluster
Destination queue for each worker job. Passed to #SBATCH -p option.
Accounting string associated with each worker job. Passed to #SBATCH -A option.
Walltime for each worker job.
Number of cpu to book in SLURM, if None, defaults to worker threads * processes
Amount of memory to request in SLURM. If None, defaults to worker processes * memory
List of other Slurm options, for example -j oe. Each option will be prepended with the #SBATCH prefix.
Name of Dask workers.
Total number of cores per job
- memory: str
Total amount of memory per job
Number of processes per job
Network interface like ‘eth0’ or ‘ib0’.
Seconds to wait for a scheduler before closing workers
Dask worker local directory for file spilling.
Additional arguments to pass to dask-worker
Other commands to add to script before launching worker.
Directory to use for job scheduler logs.
Path to desired interpreter for your batch submission script.
Python executable used to launch Dask workers.
Section to use from jobqueue.yaml configuration file.
Additional keyword arguments to pass to LocalCluster
>>> from dask_jobqueue import SLURMCluster >>> cluster = SLURMCluster(processes=6, cores=24, memory="120GB", env_extra=['export LANG="en_US.utf8"', 'export LANGUAGE="en_US.utf8"', 'export LC_ALL="en_US.utf8"']) >>> cluster.scale(10) # this may take a few seconds to launch
>>> from dask.distributed import Client >>> client = Client(cluster)
This also works with adaptive clusters. This automatically launches and kill workers based on load.
__init__(self, queue=None, project=None, walltime=None, job_cpu=None, job_mem=None, job_extra=None, config_name='slurm', **kwargs)¶
__init__(self[, queue, project, walltime, …])
adapt(self[, minimum_cores, maximum_cores, …])
Turn on adaptivity For keyword arguments see dask.distributed.Adaptive Instead of minimum and maximum parameters which apply to the number of worker, If Cluster object implements worker_spec attribute, one can use the following parameters: Parameters ———- minimum_cores: int Minimum number of cores for the cluster maximum_cores: int Maximum number of cores for the cluster minimum_memory: str Minimum amount of memory for the cluster maximum_memory: str Maximum amount of memory for the cluster Examples ——– >>> cluster.adapt(minimum=0, maximum=10, interval=‘500ms’) >>> cluster.adapt(minimum_cores=24, maximum_cores=96) >>> cluster.adapt(minimum_memory=‘60 GB’, maximum_memory= ‘1 TB’)
Stops all running and pending jobs and stops scheduler
Write job submission script to temporary file
Construct a job submission script
scale(self[, n, cores, memory])
Scale cluster to n workers or to the given number of cores or memory number of cores and memory are converted into number of workers using worker_spec attribute.
scale_down(self, workers[, n])
Close the workers with the given addresses
scale_up(self, n, \*\*kwargs)
Brings total worker count up to
Start workers and point them to our local scheduler
Stops all running and pending jobs
Stop a list of jobs
Stop a list of workers
Jobs that have finished
Jobs pending in the queue
Jobs with currently active workers
The scheduler of this cluster