Development version

  • all cluster classes: fix a bug that would allow to pass any named parameter without an error (GH#398)

  • all cluster classes: fix a bug where security was not correctly passed through (GH#398)

  • all cluster classes: scheduler_options parameter can be set through the config file in the scheduler-options section (GH#405).

  • MoabCluster: fix bug where MoabCluster was using the jobqueue.pbs config section rather than the jobqueue.moab section. (GH#450)

0.7.1 / 2020-03-26

  • all cluster classes: add scheduler_options allows to pass parameters to the Dask scheduler. For example scheduler_options={'interface': 'eth0', dashboard_addresses=':12435') (GH#384). Breaking change: using port or dashboard_addresses arguments raises an error. They have to be passed through scheduler_options.

  • all cluster classes: processes parameter default has changed. By default, processes ~= sqrt(cores) so that the number of processes and the number of threads per process is roughly the same. Old default was to use one process and only threads, i.e. proccesses=1, threads_per_process=cores. (GH#375)

  • all cluster classes: interface was ignored when set in a config file. (GH#366)

  • LSFCluster: switch to use_stdin=True by default (GH#388).

  • LSFCluster: add use_stdin to LSFCluster. This switches between bsub < job_script and bsub job_script to launch a LSF job (GH#360).

  • HTCondorCluster: support older HTCondor versions without -file argument (GH#351).

  • OARCluster: fix bug (forgotten async def) in OARCluster._submit_job (GH#380).

0.7.0 / 2019-10-09

  • Base Dask-Jobqueue on top of the core dask.distributed.SpecCluster class (GH#307)

    This is nearly complete reimplementation of the dask-jobqueue logic on top of more centralized logic. This improves standardization and adds new features, but does include the following breaking changes:

    • The cluster.start_workers method has been removed. Use cluster.scale instead.

    • The cluster.stop_all_jobs() method has been removed. Please use cluster.scale(0) instead.

    • The attributes running_jobs, pending_jobs, and cancelled_jobs have been removed. These have been moved upstream to the dask.distributed.SpecCluster class instead as workers and worker_spec, as well as .plan, .requested, and .observed.

    • The name attribute has been moved to job_name.

  • You can now specify jobs in .scale and .adapt: for example cluster.scale(jobs=2) and cluster.adapt(minimum_jobs=0, maximum_jobs=10). Specifying scaling in terms of jobs is generally more intuitive than in terms of Dask workers. This was part of GH#307.

  • Update .scale() and .adapt() docstrings (GH#346)

  • Update interactive docs (GH#340)

  • Improve error message when cores or memory is not specified (GH#331)

  • Fix Python 3.5.0 support in (GH#317)

0.6.3 / 2019-08-18

  • Compatibility with Dask 2.3.0: add scheduler_info from local_cluster (GH#313)

  • Remove lingering Python 2 specific code (GH#308)

  • Remove __future__ imports since we depend on Python >3.5 (GH#311)

  • Remove Python 3 check for black in CI (GH#315)

0.6.2 / 2019-07-31

  • Ensure compatibility with Dask 2.2 (GH#303)

  • Update documentation

0.6.1 / 2019-07-25

  • more fixes related to distributed >= 2 changes (GH#278, GH#291)

  • distributed >= 2.1 is now required (GH#295)

  • remove deprecated threads parameter from all the Cluster classes (GH#297)

  • doc improvements (GH#290, GH#294, GH#296)

0.6.0 / 2019-07-06

  • Drop Python 2 support (GH#284)

  • Fix adaptive compatibility with SpecificationCluster in Distributed 2.0 (GH#282)

0.5.0 / 2019-06-20

0.4.1 / 2018-10-18

  • Handle worker restart with clearer message (GH#138)

  • Better error handling on job submission failure (GH#146)

  • Fixed Python 2.7 error when starting workers (GH#155)

  • Better handling of extra scheduler options (GH#160)

  • Correct testing of Python 2.7 compatibility (GH#154)

  • Add ability to override python used to start workers (GH#167)

  • Internal improvements and edge cases handling (GH#97)

  • Possibility to specify a folder to store every job logs file (GH#145)

  • Require all cores on the same node for LSF (GH#177)

0.4.0 / 2018-09-06

  • Use number of worker processes as an argument to scale instead of number of jobs.

  • Bind scheduler bokeh UI to every network interfaces by default.

  • Adds an OAR job queue system implementation.

  • Adds an LSF job queue system implementation.

  • Adds some convenient methods to JobQueueCluster objects: __repr__, stop_jobs(), close().