CLI Docs

Warning

The CLI is experimental and may change between versions

dask-yarn

Deploy Dask on Apache YARN

usage: dask-yarn [--help] [--version] command ...
--help, -h

Show this help message then exit

--version

Show version then exit

dask-yarn kill

Kill a Dask application

usage: dask-yarn kill [--help] APP_ID
app_id

The application id

--help, -h

Show this help message then exit

dask-yarn services

Manage Dask services

usage: dask-yarn services [--help] command ...
--help, -h

Show this help message then exit

dask-yarn services client

Start a Dask client process

usage: dask-yarn services client [--help] script [args...]
script

Path to a Python script to run.

args

Any additional arguments to forward to script

--help, -h

Show this help message then exit

dask-yarn services scheduler

Start a Dask scheduler process

usage: dask-yarn services scheduler [--help]
--help, -h

Show this help message then exit

dask-yarn services worker

Start a Dask worker process

usage: dask-yarn services worker [--nthreads NTHREADS]
                                 [--memory_limit MEMORY_LIMIT] [--help]
--nthreads <nthreads>

Number of threads. Defaults to number of vcores in container

--memory_limit <memory_limit>

Maximum memory available to the worker. This can be an integer (in bytes), a string (like ‘5 GiB’ or ‘500 MiB’), or 0 (no memory management). Defaults to the container memory limit.

--help, -h

Show this help message then exit

dask-yarn status

Check the status of a submitted Dask application

usage: dask-yarn status [--help] APP_ID
app_id

The application id

--help, -h

Show this help message then exit

dask-yarn submit

Submit a Dask application to a YARN cluster

usage: dask-yarn submit [--name NAME] [--queue QUEUE] [--user USER]
                        [--tags TAGS] [--environment ENVIRONMENT]
                        [--deploy-mode DEPLOY_MODE]
                        [--worker-count WORKER_COUNT]
                        [--worker-vcores WORKER_VCORES]
                        [--worker-memory WORKER_MEMORY]
                        [--worker-restarts WORKER_RESTARTS]
                        [--worker-env WORKER_ENV]
                        [--client-vcores CLIENT_VCORES]
                        [--client-memory CLIENT_MEMORY]
                        [--client-env CLIENT_ENV]
                        [--scheduler-vcores SCHEDULER_VCORES]
                        [--scheduler-memory SCHEDULER_MEMORY]
                        [--temporary-security-credentials] [--help]
                        script [args...]
script

Path to a python script to run on the client

args

Any additional arguments to forward to script

--name <name>

The application name

--queue <queue>

The queue to deploy to

--user <user>

The user to submit the application on behalf of. Default is the current user - submitting as a different user requires proxy-user permissions.

--tags <tags>

A comma-separated list of strings to use as tags for this application.

--environment <environment>

Path to the Python environment to use. See the docs for more information

--deploy-mode <deploy_mode>

Either ‘remote’ (default) or ‘local’. If ‘remote’, the scheduler and client will be deployed in a YARN container. If ‘local’, they will be run locally.

--worker-count <worker_count>

The number of workers to initially start.

--worker-vcores <worker_vcores>

The number of virtual cores to allocate per worker.

--worker-memory <worker_memory>

The amount of memory to allocate per worker. Accepts a unit suffix (e.g. ‘2 GiB’ or ‘4096 MiB’). Will be rounded up to the nearest MiB.

--worker-restarts <worker_restarts>

The maximum number of worker restarts to allow before failing the application. Default is unlimited.

--worker-env <worker_env>

Environment variables to set on the workers. Pass a key-value pair like --worker-env key=val. May be used more than once.

--client-vcores <client_vcores>

The number of virtual cores to allocate for the client.

--client-memory <client_memory>

The amount of memory to allocate for the client. Accepts a unit suffix (e.g. ‘2 GiB’ or ‘4096 MiB’). Will be rounded up to the nearest MiB.

--client-env <client_env>

Environment variables to set on the client. Pass a key-value pair like --client-env key=val. May be used more than once.

--scheduler-vcores <scheduler_vcores>

The number of virtual cores to allocate for the scheduler.

--scheduler-memory <scheduler_memory>

The amount of memory to allocate for the scheduler. Accepts a unit suffix (e.g. ‘2 GiB’ or ‘4096 MiB’). Will be rounded up to the nearest MiB.

--temporary-security-credentials

Instead of using a consistent set of TLS credentials for all clusters, create a fresh set just for this application.

--help, -h

Show this help message then exit