CLI Docs¶

Warning

The CLI is experimental and may change between versions

dask-yarn¶

Deploy Dask on Apache YARN

usage: dask-yarn [--help] [--version] command ...

--help, -h¶: Show this help message then exit

--version¶: Show version then exit

dask-yarn kill¶

Kill a Dask application

usage: dask-yarn kill [--help] APP_ID

app_id¶: The application id

--help, -h¶: Show this help message then exit

dask-yarn services¶

Manage Dask services

usage: dask-yarn services [--help] command ...

--help, -h¶: Show this help message then exit

dask-yarn services client¶

Start a Dask client process

usage: dask-yarn services client [--help] script [args...]

script¶: Path to a Python script to run.

args¶: Any additional arguments to forward to script

--help, -h¶: Show this help message then exit

dask-yarn services scheduler¶

Start a Dask scheduler process

usage: dask-yarn services scheduler [--help]

--help, -h¶: Show this help message then exit

dask-yarn services worker¶

Start a Dask worker process

usage: dask-yarn services worker [--nthreads NTHREADS]
                                 [--memory_limit MEMORY_LIMIT] [--help]

--nthreads <nthreads>¶: Number of threads. Defaults to number of vcores in container

--memory_limit <memory_limit>¶: Maximum memory available to the worker. This can be an integer (in bytes), a string (like ‘5 GiB’ or ‘500 MiB’), or 0 (no memory management). Defaults to the container memory limit.

--help, -h¶: Show this help message then exit

dask-yarn status¶

Check the status of a submitted Dask application

usage: dask-yarn status [--help] APP_ID

app_id¶: The application id

--help, -h¶: Show this help message then exit

dask-yarn submit¶

Submit a Dask application to a YARN cluster

usage: dask-yarn submit [--name NAME] [--queue QUEUE] [--user USER]
                        [--tags TAGS] [--environment ENVIRONMENT]
                        [--deploy-mode DEPLOY_MODE]
                        [--worker-count WORKER_COUNT]
                        [--worker-vcores WORKER_VCORES]
                        [--worker-memory WORKER_MEMORY]
                        [--worker-restarts WORKER_RESTARTS]
                        [--worker-env WORKER_ENV]
                        [--client-vcores CLIENT_VCORES]
                        [--client-memory CLIENT_MEMORY]
                        [--client-env CLIENT_ENV]
                        [--scheduler-vcores SCHEDULER_VCORES]
                        [--scheduler-memory SCHEDULER_MEMORY]
                        [--temporary-security-credentials] [--help]
                        script [args...]

script¶: Path to a python script to run on the client

args¶: Any additional arguments to forward to script

--name <name>¶: The application name

--queue <queue>¶: The queue to deploy to

--user <user>¶: The user to submit the application on behalf of. Default is the current user - submitting as a different user requires proxy-user permissions.

--tags <tags>¶: A comma-separated list of strings to use as tags for this application.

--environment <environment>¶: Path to the Python environment to use. See the docs for more information

--deploy-mode <deploy_mode>¶: Either ‘remote’ (default) or ‘local’. If ‘remote’, the scheduler and client will be deployed in a YARN container. If ‘local’, they will be run locally.

--worker-count <worker_count>¶: The number of workers to initially start.

--worker-vcores <worker_vcores>¶: The number of virtual cores to allocate per worker.

--worker-memory <worker_memory>¶: The amount of memory to allocate per worker. Accepts a unit suffix (e.g. ‘2 GiB’ or ‘4096 MiB’). Will be rounded up to the nearest MiB.

--worker-restarts <worker_restarts>¶: The maximum number of worker restarts to allow before failing the application. Default is unlimited.

--worker-env <worker_env>¶: Environment variables to set on the workers. Pass a key-value pair like --worker-env key=val. May be used more than once.

--client-vcores <client_vcores>¶: The number of virtual cores to allocate for the client.

--client-memory <client_memory>¶: The amount of memory to allocate for the client. Accepts a unit suffix (e.g. ‘2 GiB’ or ‘4096 MiB’). Will be rounded up to the nearest MiB.

--client-env <client_env>¶: Environment variables to set on the client. Pass a key-value pair like --client-env key=val. May be used more than once.

--scheduler-vcores <scheduler_vcores>¶: The number of virtual cores to allocate for the scheduler.

--scheduler-memory <scheduler_memory>¶: The amount of memory to allocate for the scheduler. Accepts a unit suffix (e.g. ‘2 GiB’ or ‘4096 MiB’). Will be rounded up to the nearest MiB.

--temporary-security-credentials¶: Instead of using a consistent set of TLS credentials for all clusters, create a fresh set just for this application.

--help, -h¶: Show this help message then exit