Recipes¶

Find the thing you want to do, then follow the link. Each recipe is a code block and the handful of rules you will hit. The pages group them: Change the job settings, Control the run, Task types, Test data, Debug a failure, Compute and cost, Bundle deployments and CI and housekeeping.

I want to...¶

Goal	Recipe
Run a whole job and assert on the result	Make a copy of a job
Run one task with the upstream values I choose	Run one task on its own
Give a task the SQL rows or alert state another task would have produced	Stub a task's output
Read the task values a task set	Read task values
Re-run one task against the real values of an earlier run	Replay a task from a previous run
Run the downstream half of a job	Run the downstream half of a job, Replay a run's task values
Run only some tasks	Run only some tasks
Run the same job twice and compare	Run it
Assert on a failure, its root cause or a Spark error class	Assert on a failure
Test an if/else condition on its own	Test a condition task on its own
Run the body of a for-each task once, with one item	Run a for-each body once
Share one expensive run between several tests	Share one run between tests
Run tests in parallel	Parametrise and run in parallel

Goal	Recipe
Switch off a task that notifies people or refreshes a downstream system	Disable a task
Change the parameters of one task only	Override one task's parameters
Test what happens when a task is retried or times out	Test retries and timeouts
Run the copy as the deployed job's principal	Run as another identity
Install a package only the tests need	Add test-only dependencies
Install my own wheel's test extras	Install a local wheel with extras
Change a serverless environment's version or dependencies	Change a serverless environment
Change anything else in the job settings	Change anything else before the job is created
Set one option for every test in a directory	Set a default for every copy
Let job code detect that it runs under test	Let job code know it is under test

Goal	Recipe
Run several jobs at the same time in one test	Run jobs side by side
Stop a streaming job after a fixed time and still get its result	Stop a run that does not end
Stay under a serverless quota	Cap the number of active runs
Make a run faster	Speed up a run

Goal	Recipe
Read the rows a SQL file task returned	Read the rows of a SQL file task
Read the rows of a saved query task	Read the rows of a saved query
Check whether an alert triggered	Alert tasks
Check which branch an if/else took	Branches
Test a branch without arranging the data that selects it	Force a branch
Run a for-each task over a short list and read each iteration	For-each tasks
Test what a job does when a file arrival or table update trigger fires	Simulate a trigger
Full-refresh or select tables in a pipeline task	Choose how a pipeline task refreshes
Read the update a pipeline task started and its error events	Read the pipeline update
Run a pipeline task against a copy that writes to my test schema	Copy the pipelines of a job
Test a pipeline on its own	Copy a pipeline on its own
Test a job whose task runs another job	Jobs that run other jobs
Run a dbt selection and read `run_results.json`	dbt tasks

Goal	Recipe
Give each test its own schema	Give each test its own schema
Seed tables or files before the job runs	Seed data with a setup task
Clean up files or tables outside the test schema	Clean up with a teardown task
Assert on what the teardown removes	Tear down when the job closes
Seed with SQL or a notebook file instead of a function	Seed with SQL statements or a notebook file
Seed with Databricks Connect or pytester fixtures	Seed from the test process

Goal	Recipe
See each task's state change while the run is in flight	Watch a run live
Read what a failed notebook printed	Read what a notebook printed
Keep the job to open it in the workspace	Keep the job
Refine assertions against a kept run without running again	Read a kept run again
Step through a failed notebook task on interactive compute	Reproduce a failed notebook task
Keep every run's output as a CI artefact	Keep every run's output as an artefact

Goal	Recipe
Skip cluster start-up with a running cluster	Skip cluster start-up
Run job clusters on one node	Run job clusters on one node
Run without serverless jobs	Run the helper tasks on a job cluster
Pin the serverless environment version	Pin the serverless environment version
Charge test runs to a budget policy	Charge runs to a budget policy
Shrink clusters or queue runs	Change compute with modify_settings
Track run times over time	Record run timings

Goal	Recipe
Choose the target or pass bundle variables	Choose the target and pass bundle variables
Test against a target I deploy myself	Skip the deploy
Stop branches and pipelines from breaking each other's runs	Give each branch and identity its own deployment
Remove the test deployment at the end	Destroy the test deployment
Run the suite in GitHub Actions or Azure DevOps	Run the tests in CI
Refuse in CI what a developer's run only warns about	Be stricter in CI
Let engineers open the jobs a service principal kept	Let the team open kept jobs
Lint every job's wiring without running it	Check every job's wiring without running it
Delete what crashed sessions left behind	Clean up leftovers
Sweep the workspace from a nightly script	Sweep from a nightly script