API reference

Everything on this page is importable from pytest_databricks_lakeflow. The lakeflow fixture and the command-line flags are registered by the plugin's pytest11 entry point.

The lakeflow fixture

Session-scoped. It returns Lakeflow() unless you override it in a conftest.py. Tests that request it get the lakeflow marker, so pytest -m "not lakeflow" deselects them.

Lakeflow

Lakeflow dataclass

Lakeflow(
    *,
    bundle_root: str | Path | None = None,
    target: str = "test",
    profile: str | None = None,
    workspace_client: WorkspaceClient | None = None,
    sync: Sync = "deploy",
    variables: Mapping[str, str] | None = None,
    deploy_flags: Sequence[str] = (),
    timeout: timedelta = timedelta(minutes=30),
    poll_interval: timedelta | None = None,
    max_concurrent_runs: int | None = None,
    catalog: str | None = None,
    helpers_path: str | None = None,
    helpers_compute: str | None = None,
    helpers_environment_version: str = "4",
    disable_auto_optimization: bool = True,
    performance_optimized: bool = False,
    side_effects: SideEffects = "neutralise",
    environment_version: str | None = None,
    budget_policy_id: str | None = None,
    existing_cluster_id: str | None = None,
    single_node: bool = False,
    allow_undeclared_params: bool = False,
    test_marker: bool = False,
    notebook_export: NotebookExport = "failed",
    dbt_artifacts: bool = False,
    sweep_hours: float = 24,
    keep: bool = False,
    keep_failed: bool = True,
    strict_artifacts: bool = False,
    permissions: Sequence[
        Mapping[str, str] | JobAccessControlRequest
    ]
    | None = None,
    run_as: RunAs = None,
    queue: bool | None = None,
    disable: Sequence[str] | None = None,
    extra_dependencies: Sequence[str] | None = None,
    children: Children = "warn",
    setup: LifecycleStep | None = None,
    teardown: LifecycleStep | None = None,
    environments: Mapping[str, Mapping[str, Any]]
    | None = None,
    saved_queries: SavedQueries = "deployed",
    modify_settings: Callable[[JobSettings, Item], None]
    | None = None,
    run_finished: Callable[[JobRun, Item], None]
    | None = None,
)

Makes throwaway copies of a bundle's jobs for tests.

Tests get one from the lakeflow fixture. Override the fixture to configure it:

@pytest.fixture(scope="session")
def lakeflow() -> Lakeflow:
    return Lakeflow(target="test", catalog="dev_sandbox")

Creating one does nothing. The first test that uses it sets up the bundle, which happens once per session for each bundle target, however many Lakeflow instances use it and however many xdist workers there are.

Attributes:

Name	Type	Description
bundle_root	str | Path | None	The bundle's directory, relative to the pytest rootdir. Defaults to the nearest directory with a databricks.yml, starting at the rootdir.
target	str	The bundle target to deploy and read job templates from.
profile	str | None	The Databricks config profile for the CLI and the SDK. Defaults to their standard resolution, which reads DATABRICKS_CONFIG_PROFILE and the other DATABRICKS_* variables.
workspace_client	WorkspaceClient | None	A client to use instead of making one from profile. The CLI still uses profile, or else the client's profile.
sync	Sync	How to prepare the bundle: deploy, sync or none. Before deploy, bundle validate reads the run_as of the target and of each job, and the setup fails with LakeflowBundleError if one names another user, or a service principal on which you do not have the Service Principal User role.
variables	Mapping[str, str] | None	Bundle variables, passed as --var name=value to every bundle command, bundle destroy included. Instances with different variables set the target up separately.
deploy_flags	Sequence[str]	Extra flags for databricks bundle deploy, added at the end of the command, such as ["--fail-on-active-runs"]. Instances with different flags deploy separately. Used only when sync is deploy. -t, --target, -e, --environment, -p, --profile, --var and --auto-approve are not allowed.
timeout	timedelta	The default timeout for one job run. Each job is also created with timeout_seconds set to this plus 60 seconds, unless the template has a smaller one, so the platform stops runs the plugin cannot. A longer timeout passed to run() raises the job's timeout_seconds before that run starts.
poll_interval	timedelta | None	How long to wait between two polls of a run that has not finished. Defaults to two seconds. Zero or less raises ValueError.
max_concurrent_runs	int | None	How many runs this process may have active at once, for a workspace with a tight serverless quota. start() and run() wait for one of them to end before starting another, and raise LakeflowError when none ends within the run's timeout. Every Lakeflow in the process shares the cap of the first run started under one, and each xdist worker has its own. --lakeflow-max-concurrent-runs overrides it. Less than 1 raises ValueError. Defaults to no cap.
catalog	str | None	The catalog in which schema() creates temporary schemas.
helpers_path	str | None	The workspace folder for the helper notebooks. Defaults to /Users/<you>/.pytest-databricks-lakeflow.
helpers_compute	str | None	A job_cluster_key from the job for stub, probe, setup and teardown tasks, for workspaces without serverless.
helpers_environment_version	str	The serverless environment version for stub, probe, setup and teardown tasks.
disable_auto_optimization	bool	Stop serverless from retrying failed tasks, so that a failure fails once and fast.
performance_optimized	bool	Run every ephemeral job as PERFORMANCE_OPTIMIZED, whatever the template says.
side_effects	SideEffects	neutralise pauses dashboard subscriptions and SQL alert subscriptions in every ephemeral job, so runs do not email snapshots or notify subscribers. keep leaves them as the template has them.
environment_version	str | None	The serverless environment version of every environment copied from the template, except those whose spec sets base_environment. The helper environment keeps helpers_environment_version.
budget_policy_id	str | None	The budget policy of every ephemeral job and pipeline.
existing_cluster_id	str | None	A running cluster for every task that the template puts on a job cluster or a new cluster, stub, probe, setup and teardown tasks and tasks nested in a for_each_task included, so that tests skip cluster start-up. The job clusters are dropped. Serverless and warehouse tasks keep their compute.
single_node	bool	Make every job cluster and every task's new_cluster in each ephemeral job a single node cluster, so that runs start faster and cost one node: no workers, no autoscaling, and the single node Spark configuration and ResourceClass tag. The job fails to build if one of those clusters uses standard access mode, because single node compute needs dedicated access mode. Serverless tasks and existing_cluster_id are left alone.
allow_undeclared_params	bool	Allow parameters that the job does not declare.
test_marker	bool	Declare a __pytest_test_id job parameter on every ephemeral job, with the test node id as its default, so job code can tell that it runs under test. The node id is used even when name replaces it in the job name. Tests cannot pass the parameter themselves.
notebook_export	NotebookExport	Which notebook tasks to read with jobs.export_run after a run, into TaskRun.notebook: failed for those that ran and did not succeed, always for every one that ran, never for none.
dbt_artifacts	bool	Download the artefacts of each dbt task as soon as its run finishes, while the platform's 30 minute link is valid, and parse them into TaskRun.dbt. They go into a temporary directory per run, removed at the end of the pytest session unless keep or --lakeflow-keep is set. A tarball over 50 MB, an expired link or a tarball without run_results.json leaves TaskRun.dbt None with a warning naming the task.
sweep_hours	float	The age after which --lakeflow-sweep and sweep() delete leftover jobs.
keep	bool	Keep every ephemeral job and temporary schema. --lakeflow-keep also turns this on.
keep_failed	bool	Keep the ephemeral job when its test fails. --lakeflow-keep-failed and --no-lakeflow-keep-failed override it.
strict_artifacts	bool	Raise LakeflowError from the first template lookup when the bundle has a wheel artifact without dynamic_version: true, or, with sync="none", a local library built after the copy that the last deploy uploaded, instead of the UserWarning emitted once per session for each. Serverless is reported to cache wheels by file name (databricks/cli#1621 and #2958), so a rebuilt wheel with an unchanged version may not be picked up.
permissions	Sequence[Mapping[str, str] | JobAccessControlRequest] | None	The access_control_list applied to every ephemeral job when it is created, so that people other than its creator can open it. Each entry is a jobs.JobAccessControlRequest or a dict such as {"group_name": "data-eng", "permission_level": "CAN_VIEW"}. IS_OWNER raises ValueError, because the creator owns every ephemeral job; use CAN_MANAGE.
run_as	RunAs	The default of run_as on job() and task(). None runs each copy as you.
queue	bool | None	The default of queue on EphemeralJob.run() and start(). None leaves each run to the job's own queue setting.
disable	Sequence[str] | None	The default of disable on job(). Each copy disables the tasks of this list that its job has, and a task key that the job does not have is skipped, whereas disable on the call refuses one.
extra_dependencies	Sequence[str] | None	The default of extra_dependencies on job() and task().
children	Children	The default of children on job() and task(): "warn" or "refuse", for example "refuse" in CI. Anything else raises ValueError.
setup	LifecycleStep | None	The default of setup on job() and task(), checked here as far as it can be without a test: a string, an empty list, a statement with a stray $, a path that is not a .py file and a function the plugin cannot ship raise ValueError.
teardown	LifecycleStep | None	The default of teardown on job() and task(), checked here as setup is.
environments	Mapping[str, Mapping[str, Any]] | None	The default of environments on job() and task(), such as {"default": {"environment_version": "5"}}. Each copy takes the overrides of the environments its job has, and an environment key the job does not have is skipped, whereas environments on the call adds one. A reserved key, an entry that is not a mapping or an unknown field raises ValueError here.
saved_queries	SavedQueries	The default of saved_queries on job() and task(): "deployed" or "inline", for example "inline" to read the rows of every saved query task. Anything else raises ValueError. A per-call argument that is None, or left out, takes the field's value. Anything else replaces it whole, so disable=[] on a call disables nothing and environments={} keeps the template's environments.
modify_settings	Callable[[JobSettings, Item], None] | None	Called with each ephemeral job's settings and the test, just before the job is created. Change the settings in place.
run_finished	Callable[[JobRun, Item], None] | None	Called with the outcome of each run and the test, after the run finishes.
active_runs	set[int]	Ids of the runs started by this Lakeflow that have not finished.
active_updates	dict[str, str]	The update that EphemeralPipeline.run() started and that has not ended, by the id of the pipeline copy.

ws property

ws: WorkspaceClient

The workspace client: workspace_client, or one made from profile on first use.

run_id property

run_id: str

The id of this test session, which tags ephemeral jobs and names temporary schemas.

templates property

templates: dict[str, dict[str, Any]]

The deployed settings of each usable bundle job, by resource key, as dicts.

template_errors property

template_errors: dict[str, str]

Why each bundle job that cannot be used as a template cannot, by resource key.

access_control_list

access_control_list() -> (
    list[jobs.JobAccessControlRequest] | None
)

Get permissions as the access_control_list that jobs.create takes, or None if there are none.

template

template(key: str) -> jobs.JobSettings

Get a fresh copy of the deployed settings of a bundle job.

Parameters:

Name	Type	Description	Default
key	str	The bundle resource key of the job.	required

Returns:

Type	Description
JobSettings	The job's settings, safe to change.

Raises:

Type	Description
KeyError	If the bundle has no job key.
LakeflowBundleError	If the job cannot be used as a template, for example because it is not deployed.

check

check(
    key: str, *, rules: Collection[str] | None = None
) -> list[str]

Find the wiring mistakes in a bundle job's template, without creating or running anything.

It reports a {{job.parameters.<name>}} the job does not declare, a {{tasks.<key>.values.<name>}} whose task is not upstream of the task that refers to it, a depends_on task that is not in the job, and retry settings on a for_each_task, which belong on its nested task. bundle validate passes all of them. The rules that rules names report more: wheel a python_wheel_task entry point that the newest built wheel of its package under the bundle's dist does not have, library a library, Python file or environment dependency path that starts with / and does not exist in the workspace, dash a task reference to a task key with a dash, trigger a {{job.trigger.*}} reference, which a test's run does not resolve as its trigger would, and the opt-in equal_to an EQUAL_TO or NOT_EQUAL condition on a number or a task value, which compares text. It lints the template the session loaded: the deployed job, or the bundle validate output under sync="sync" or "none". It works outside a test, for example in a session-scoped fixture, and only reads the workspace.

Parameters:

Name	Type	Description	Default
key	str	The bundle resource key of the job.	required
rules	Collection[str] | None	The named rules to apply, from ALL_RULES. Defaults to DEFAULT_RULES, every rule but equal_to.	None

Returns:

Type	Description
list[str]	One message per problem, in the order of the job's tasks, each starting with Task '<key>':, then the
list[str]	missing dependencies of its environments, each starting with Environment '<key>':, and the trigger
list[str]	references in its job parameters, each starting with Parameter '<name>':. Empty if there are none.

Raises:

Type	Description
KeyError	If the bundle has no job key.
LakeflowBundleError	If the job cannot be used as a template, for example because it is not deployed.
ValueError	If rules names a rule that is not in ALL_RULES.

job

job(
    key: str,
    params: Mapping[str, str] | None = None,
    *,
    capture_task_values: Mapping[str, Sequence[str]]
    | None = None,
    name: str | None = None,
    run_as: RunAs = None,
    disable: Sequence[str] | None = None,
    task_params: Mapping[
        str, Mapping[str, str] | Sequence[str]
    ]
    | None = None,
    force_conditions: Mapping[str, bool] | None = None,
    children: Children | None = None,
    setup: LifecycleStep | None = None,
    teardown: LifecycleStep | None = None,
    teardown_at: TeardownAt = "run",
    extra_dependencies: Sequence[str] | None = None,
    environments: Mapping[str, Mapping[str, Any]]
    | None = None,
    task_overrides: Mapping[str, Mapping[str, object]]
    | None = None,
    upstream_task_values: Upstream | None = None,
    pipelines: Pipelines = "deployed",
    schema: str | None = None,
    for_each_inputs: Mapping[str, Sequence[object]]
    | None = None,
    trigger: Mapping[str, str] | None = None,
    saved_queries: SavedQueries | None = None,
) -> EphemeralJob

Make a throwaway copy of a bundle job.

setup and teardown run on the job's compute, as notebook tasks of the copy. The setup task runs before every other task on every run, and the teardown task after every other task, whether they passed or failed, or with teardown_at="close" alone when the job closes. Each is one of three forms:

• A plain function taking any of spark, dbutils and params, a dict of the job's parameters with their values for the run, as strings: those it declares, those modify_settings declares and those passed under allow_undeclared_params=True. It must be self-contained: it is shipped as source, so imports go inside it and it cannot use variables of an enclosing function. A value it returns becomes the task's output as JSON.
• A list of SQL statements, each rendered with string.Template from the same parameters (${name}, and $$ for a literal dollar) and run in order through spark.sql. The output is {"statements": <count>}. The list must not be empty. Each ${name} is checked here, when job() is called, so it must be a parameter the bundle job declares or a key of params; one that only run(params=...) or modify_settings would supply is refused.
• A pathlib.Path to a .py file, relative to the pytest rootdir, uploaded as the notebook. It reads the parameters from its widgets and calls dbutils.notebook.exit itself for an output.

Parameters:

Name	Type	Description	Default
key	str	The bundle resource key of the job, as databricks bundle run takes it.	required
params	Mapping[str, str] | None	Job parameters for every run of the copy.	None
capture_task_values	Mapping[str, Sequence[str]] | None	The task value keys to capture, by task key. A for-each task and the task nested in it cannot be named, because the loop's iterations cannot set task values.	None
name	str | None	Replaces the test node id in the job name.	None
run_as	RunAs	Who the copy runs as. None takes Lakeflow's run_as, which by default removes the template's run_as, so the copy runs as you. "keep" keeps the template's. A string with @ names a user, any other string names a service principal by its application id, and a jobs.JobRunAs is used as given. A service principal needs you to have the Service Principal User role on it, which is checked before the copy is created when the workspace client knows its account id.	None
disable	Sequence[str] | None	Task keys to disable in the copy. Runs skip them, and the platform disables every task downstream of them too. Both appear in JobRun.disabled_tasks, and run(only=...) refuses to name them. None takes Lakeflow's disable, and [] disables none.	None
task_params	Mapping[str, Mapping[str, str] | Sequence[str]] | None	Parameters for single tasks of the copy, by task key, in the shape the task type takes. A mapping is merged into a notebook task's base_parameters, a Python wheel task's named_parameters or a SQL task's parameters, so the template's {{tasks.<key>.values.<name>}} references survive. A list replaces a Python script, JAR or Python wheel task's parameters or a dbt task's commands, which must not be empty. A Python wheel task takes one or the other, so the one not given is cleared. Values are passed unchanged. A task nested in a for_each_task is named by its own key, and every iteration gets its parameters. A name that is also a job parameter emits a UserWarning, because the job value wins.	None
force_conditions	Mapping[str, bool] | None	The outcome each if/else condition task takes in every run of the copy, by task key: True runs the true branch, False the false branch. The task's operands are replaced by constants, so the branch no longer depends on the data upstream of it. The upstream tasks still run.	None
children	Children | None	What to do about Run Job tasks, which the copy keeps as deployed, so they run the deployed child job and not a copy. "warn" emits a UserWarning for each one that is not disabled, downstream of a disabled task nor inside a disabled for_each_task. "refuse" raises LakeflowError instead, before anything is created. None takes Lakeflow's children.	None
setup	LifecycleStep | None	A function, a list of SQL statements or a .py file to run first, as the task __pytest_setup, which every task without depends_on comes to depend on. Its outcome is JobRun.setup. None takes Lakeflow's setup.	None
teardown	LifecycleStep | None	A function, a list of SQL statements or a .py file to run last, as the task __pytest_teardown with run_if ALL_DONE, after every task that no other task depends on. Its outcome is JobRun.teardown, and a failure is logged as a warning. None takes Lakeflow's teardown.	None
teardown_at	TeardownAt	When the teardown task runs. "run" runs it at the end of every run, before the test's assertions. "close" leaves it out of every run, whose teardown is then None, and runs it alone when the job closes, before it is deleted, with the job's params, so the assertions still see what it removes. Its run is EphemeralJob.teardown_run. That costs a second run start per job. keep and --lakeflow-keep skip it, keep_failed does not.	'run'
extra_dependencies	Sequence[str] | None	Test-only dependencies for the copy, as PyPI requirements, workspace or Volumes paths of wheels, or local wheels as paths relative to the bundle root that start with ./ or ../, such as ./dist/*.whl[test]. A local path is pointed at the copy of each matching file that the last bundle deploy uploaded, its extras kept. They are appended to every environment taken from the template and, as pypi or whl libraries, to every notebook, Python script and Python wheel task on a cluster. A wheel with extras reaches only the environments, because libraries[].whl takes no extras. The helper tasks do not get them. The notebook tasks they do not reach, after environments and modify_settings, such as a serverless notebook task without an environment_key, are named in a UserWarning when the job is created. None takes Lakeflow's extra_dependencies.	None
environments	Mapping[str, Mapping[str, Any]] | None	Fields of the copy's serverless environment specs, by environment_key, such as {"default": {"environment_version": "5"}}. A field replaces the template's field, after environment_version and extra_dependencies are applied, so dependencies replaces the whole list. A key the job does not have adds an environment holding only the fields passed, which a task can use through modify_settings. None takes the keys of Lakeflow's environments that the job has, and {} keeps the template's environments.	None
task_overrides	Mapping[str, Mapping[str, object]] | None	Timeout and retry fields of the copy's tasks, by task key, such as {"load": {"timeout_seconds": 60, "max_retries": 1, "retry_on_timeout": True}}. The fields are timeout_seconds, which applies to each attempt, max_retries, min_retry_interval_millis and retry_on_timeout. They are applied before the probes, setup and teardown tasks are added, so those cannot be named.	None
upstream_task_values	Upstream | None	Tasks of the copy to replace with stubs, by task key, with the task values each stub sets, such as {"extract": {"row_count": 10}}, or a Stub with values and outputs. A stub keeps the task's key and place, so the tasks downstream of it still depend on it and their {{tasks.<key>.values.<name>}} references resolve against it, and run(only=...) adds it whenever a selected task depends on it. Their {{tasks.<key>.output.<path>}} references are replaced by the stub's outputs. It drops the task's depends_on, so the tasks upstream of it still run on a full run but no longer hold it back. It is applied after task_params, disable, force_conditions, extra_dependencies and task_overrides, so what they set on that task is lost. It appears in JobRun.injected_tasks.	None
pipelines	Pipelines	What the copy's pipeline tasks update. "deployed" leaves them on the deployed pipelines, so every run updates those pipelines' tables. "copy" makes a copy of each pipeline that an enabled pipeline task updates, as pipeline() does, publishing to schema and named after the job and the pipeline's bundle key. The copies are created just before the job, after modify_settings, which still sees the deployed pipelines' ids, and closed just after it with the same keep rules. The pipelines to copy are chosen again then, so a pipeline task that modify_settings repoints or adds is copied too. When the job is not created, the copies are deleted whatever keep_failed says.	'deployed'
schema	str | None	The full name of the schema the pipeline copies publish to, as schema() yields it. Needed with pipelines="copy", and refused without it.	None
for_each_inputs	Mapping[str, Sequence[object]] | None	The list each for-each task of the copy iterates over, by task key, such as {"loop": ["a", "b"]}, so a test of two items runs two iterations rather than the deployed list. Each list replaces the task's for_each_task.inputs as JSON, with {{ escaped so a value is not read as a reference, and may be empty. The run reports the for-each task as one task, with its iterations in TaskRun.iterations and their counts in TaskRun.for_each_stats.	None
trigger	Mapping[str, str] | None	The values the copy's {{job.trigger.<path>}} references take, by the path after job.trigger., such as {"type": "table", "table_update.updated_tables": "cat.sch.orders"}. The copy is started with run-now, which cannot say that a trigger fired, so each such reference is rewritten to {{job.parameters.__pytest_trigger_<path>}}, with each . of the path as _, and that parameter is declared with the value as its default. Each value is a string, passed as given, so a reference the platform fills with a list, such as table_update.updated_tables, gets exactly the string the test writes. A reference left out emits a UserWarning. The copy has no trigger, so jobs/create refuses its table_update and file_arrival references unless trigger sets them, and LakeflowError is raised for them when the job is created. JobRun.trigger_type stays ONE_TIME.	None
saved_queries	SavedQueries | None	What the copy's saved query tasks, SQL tasks with sql_task.query, run. "deployed" runs each saved query as deployed, and its TaskRun.sql_result is None, because the platform keeps no rows of a saved query task. "inline" reads each enabled one with queries.get and runs its text as a SQL file task, so sql_result holds its rows: the text is uploaded to helpers_path as query_<run id>_<task key>_<hex>.sql when the job is created, after USE CATALOG and USE SCHEMA lines for the catalog and schema the query sets, and deleted with the job, or kept with it. The task keeps its warehouse and parameters, and each default of a query parameter that neither the task nor a job parameter sets is added to its parameters: text, a number or a single enum value, as a string. The inlined file runs as the job's run_as, not as the query's owner, and without the SQL editor's automatic LIMIT 1000. None takes Lakeflow's saved_queries.	None

Returns:

Type	Description
EphemeralJob	The job, created when entered with with or on its first run.

Raises:

Type	Description
KeyError	If the bundle has no job key.
ValueError	If an argument is wrong on its own: children, pipelines, saved_queries or teardown_at is not one of its choices, teardown_at is "close" without teardown, pipelines is "copy" without schema or schema is passed without it, a parameter name starts with __pytest_, setup or teardown is a string, an empty list, a list with a statement that is not a string or has a stray $, a path that is missing or not a .py file, or a function that is a lambda, has a decorator, uses variables of an enclosing function, takes a parameter other than spark, dbutils and params or has no source, task_params has a value that does not fit its task's type or gives a dbt task no commands, task_overrides sets a field it does not take, force_conditions has an outcome that is not a bool, an entry of extra_dependencies is a local path that does not start with ./ or ../, environments names a key starting with __pytest_, holds an entry that is not a mapping or a field that a serverless environment does not take, upstream_task_values holds a value that is too large or not JSON-serialisable or an output that contains {{ or is not JSON-serialisable, for_each_inputs gives a value that is not a list, is not JSON-serialisable, is longer than 5,000 characters as JSON, holds None or a nested list or mixes objects with primitives, or trigger gives a value that is not a string.
LakeflowError	If called outside a test, or an argument does not fit the job: capture_task_values, disable, task_params, task_overrides, force_conditions, for_each_inputs or upstream_task_values names a task that is not in the job, capture_task_values names a for-each task or the task nested in one, task_params names a task that is not a notebook, Python script, JAR, Python wheel, SQL or dbt task, force_conditions names a task that is not a condition task or is disabled or downstream of a disabled task, for_each_inputs names a task without a for_each_task, a ${name} of setup or teardown is neither declared by the job nor in params, an entry of extra_dependencies matches no file, has no uploaded copy, is used with sync="sync" or is a wheel with extras while the copy has a Python task on a cluster, children is "refuse" and the copy has a Run Job task that is not disabled, downstream of a disabled task nor inside a disabled for_each_task, environments sets both environment_version and base_environment on one environment or brings the job past 10 environments, a task of the copy refers to an output its stub does not give, trigger names a path that no {{job.trigger.<path>}} reference of the copy uses, pipelines is "copy" and the copy has no enabled pipeline task, or, with saved_queries="inline", a saved query does not exist or has a parameter that neither the task nor a job parameter sets whose default is a date, a date range, several enum values or query-backed. When the job is created, also if a task of the copy refers to the output of a notebook task that is not a stub, or keeps a {{job.trigger.table_update.*}} or {{job.trigger.file_arrival.*}} reference that trigger does not set.
LakeflowBundleError	If pipelines is "copy" and a pipeline that a pipeline task updates does not exist.

task

task(
    ref: str,
    params: Mapping[str, str] | None = None,
    *,
    upstream_task_values: Upstream | None = None,
    capture_task_values: Sequence[str] | None = None,
    name: str | None = None,
    run_as: RunAs = None,
    task_params: Mapping[
        str, Mapping[str, str] | Sequence[str]
    ]
    | None = None,
    upstream_from: JobRun | None = None,
    children: Children | None = None,
    setup: LifecycleStep | None = None,
    teardown: LifecycleStep | None = None,
    teardown_at: TeardownAt = "run",
    extra_dependencies: Sequence[str] | None = None,
    environments: Mapping[str, Mapping[str, Any]]
    | None = None,
    task_overrides: Mapping[str, Mapping[str, object]]
    | None = None,
    pipelines: Pipelines = "deployed",
    schema: str | None = None,
    for_each_inputs: Sequence[object] | None = None,
    input: object | None = None,
    trigger: Mapping[str, str] | None = None,
    saved_queries: SavedQueries | None = None,
) -> EphemeralJob

Make a throwaway job that holds one task of a bundle job.

Each upstream task named in upstream_task_values is replaced by a stub task with the same key, so the task's {{tasks.<key>.values.<name>}} references and taskValues.get calls resolve against the stub. A stub is a notebook task, so it has no SQL, alert or other task output: the task's {{tasks.<key>.output.<path>}} references are replaced by the literals that a Stub gives in outputs.

upstream_from replays the task with the values a previous run of the job captured: each task that the task depends on is stubbed with its task_values in that run. The platform returns no task values with a run, so that run must have captured them, with capture_task_values={"<upstream key>": [...]} on job().

A three-part ref names the task nested in a for-each task, such as "fanout.loop.inner". The job holds that task in place of the for-each task, so it runs once, as an ordinary task, with input as its iteration input and the for-each task's upstream tasks as its own:

upstream = {"extract": {"rows": 3}}
with lakeflow.task("fanout.loop.inner", input={"id": 7}, upstream_task_values=upstream) as task:
    assert task.run().task("inner").parameters["item_id"] == "7.0"

Parameters:

Name	Type	Description	Default
ref	str	The job and task as "job_key.task_key", or "job_key.loop_key.nested_key" for the task nested in the for-each task loop_key.	required
params	Mapping[str, str] | None	Job parameters for every run.	None
upstream_task_values	Upstream | None	By upstream task key, a mapping of value keys to values, or a Stub with values and outputs. An entry replaces the values upstream_from holds for that task.	None
capture_task_values	Sequence[str] | None	The task value keys to capture from the task. For a three-part ref, emits a UserWarning, because the body cannot set task values when it runs in its loop.	None
name	str | None	Replaces the test node id in the job name, as for job().	None
run_as	RunAs	Who the job runs as, as for job().	None
task_params	Mapping[str, Mapping[str, str] | Sequence[str]] | None	Parameters for the task, as for job(). Only the task under test can be named.	None
upstream_from	JobRun | None	A run of the same job whose captured task values stub the task's upstream tasks.	None
children	Children | None	What to do when the task is a Run Job task, as for job().	None
setup	LifecycleStep | None	Runs first, before the stubs, as for job().	None
teardown	LifecycleStep | None	Runs last, after the task and its probe, as for job().	None
teardown_at	TeardownAt	When the teardown task runs, at the end of every run or when the job closes, as for job().	'run'
extra_dependencies	Sequence[str] | None	Test-only dependencies for the job, as for job().	None
environments	Mapping[str, Mapping[str, Any]] | None	Fields of the job's serverless environment specs, by environment_key, as for job().	None
task_overrides	Mapping[str, Mapping[str, object]] | None	Timeout and retry fields of the task, as for job(). Only the task under test can be named, as {"<task_key>": {...}}.	None
pipelines	Pipelines	"copy" makes a copy of the pipeline that the task updates, publishing to schema, as for job().	'deployed'
schema	str | None	The full name of the schema the pipeline copy publishes to, as for job().	None
for_each_inputs	Sequence[object] | None	The list the task iterates over when it is a for-each task, such as ["a"], as for job().	None
input	object | None	The iteration input of the nested task a three-part ref names: a string, number, boolean or mapping, such as {"id": 7}. Its {{input}} and {{input.<key>}} references take it through job parameters, written as an iteration writes them: a string as it is, an integer input as written, and anything else as compact JSON with every number in a mapping as a double, so {"id": 7} gives {{input}} '{"id":7.0}' and {{input.id}} "7.0". None means not given.	None
trigger	Mapping[str, str] | None	The values the task's {{job.trigger.<path>}} references take, by the path after job.trigger., as for job().	None
saved_queries	SavedQueries | None	"inline" runs the task's saved query as a SQL file task, so sql_result holds its rows, as for job().	None

Returns:

Type	Description
EphemeralJob	The job, created when entered with with or on its first run.

Raises:

Type	Description
KeyError	If the bundle has no such job.
ValueError	If ref has neither two nor three parts, input is given with a two-part ref, is not a string, number, boolean or mapping, contains {{ or is not JSON-serialisable, upstream_task_values names the task under test, upstream_from is a run of another job, task_overrides names another task, or an argument is wrong on its own, as for job().
LakeflowError	If called outside a test, the job has no such task, a three-part ref names no task nested in a for-each task, the task refers to a task that is not stubbed (its task values or any other {{tasks.<key>. reference) or to an output that its stub does not give, upstream_from lacks the captured values of an upstream task, task_params names another task, children is "refuse" and the task is a Run Job task, capture_task_values is given for a for-each task, input is missing for a nested task that refers to {{input}}, is given for one that does not, lacks a key a reference names or makes the job parameters longer than the 10,000 characters as JSON that jobs/create accepts, for_each_inputs is given and the task has no for_each_task, or an argument does not fit the job, as for job().
LakeflowBundleError	If pipelines is "copy" and the task's pipeline does not exist.

attach

attach(run_id: int) -> JobRun

Rebuild the outcome of a finished run of a kept ephemeral job, without running anything.

The run's job must still exist, as keep_failed, keep and --lakeflow-keep leave it, because its settings say which tasks were stubs and which values the probes captured. Nothing is deployed, created, cancelled or deleted, so a test that only attaches needs no bundle. The outcome is recorded in the test's report section as for a run the test started, but run_finished is not called. A child run of a Run Job task is named by its job id, not its bundle key.

run = lakeflow.attach(5437110661)
assert run.task("transform").task_values == {"row_count": 3}

Parameters:

Name	Type	Description	Default
run_id	int	The job run id, the number at the end of the Kept run: URL. Not a task run id.	required

Returns:

Type	Description
JobRun	The run's outcome, as EphemeralJob.run() returned it.

Raises:

Type	Description
LakeflowError	If called outside a test, the run does not exist, it is a task run, it has not finished, its job no longer exists, or its job was not created by this plugin.

pipeline

pipeline(
    key: str,
    schema: str,
    *,
    configuration: Mapping[str, str] | None = None,
    name: str | None = None,
) -> EphemeralPipeline

Make a throwaway copy of a bundle pipeline that publishes to schema.

The copy is made from the deployed pipeline's spec, so it runs the code that the last bundle deploy uploaded. It publishes to schema, in the publishing mode the deployed pipeline uses, runs in development mode, and leaves out the deployment, the continuous and trigger settings, the notifications and the event log. It runs as you. Only tables that the pipeline's code names without a schema go to schema; a fully qualified name still writes where it says.

with lakeflow.schema() as schema, lakeflow.pipeline("my_pipeline", schema=schema) as pipeline:
    update = pipeline.run(full_refresh=True)
    assert update.state == "COMPLETED"

The copy is deleted with its tables before the schema is dropped, unless the keep rules keep it.

Parameters:

Name	Type	Description	Default
key	str	The bundle resource key of the pipeline.	required
schema	str	The full name of the schema the copy publishes to, catalog.schema, as schema() yields it.	required
configuration	Mapping[str, str] | None	Keys to set in the copy's configuration, over the deployed pipeline's, for code that reads its catalog or schema from spark.conf.	None
name	str | None	Replaces the test node id in the copy's name.	None

Returns:

Type	Description
EphemeralPipeline	The copy, created when entered with with or on its first run.

Raises:

Type	Description
KeyError	If the bundle has no pipeline key.
ValueError	If schema is not a full name.
LakeflowBundleError	If sync is not "deploy", the pipeline is not deployed, or it no longer exists.
LakeflowError	If called outside a test, the pipeline publishes to the Hive metastore, or the copy would have more than 25 tags without the attribution tags.

schema

schema(
    catalog: str | None = None,
) -> Generator[str, None, None]

Create a temporary schema, and drop it on leaving the with block.

The schema is named <catalog>.pytest_<run id>_<random>. Pass it to jobs as a job parameter, because bundle variables cannot change per test. It is dropped even when the block fails, unless keep or --lakeflow-keep is set. It carries the pytest_expires property of --lakeflow-keep-ttl and the pytest_git_branch property, when they have a value, which the sweep reads as it reads a job's tags.

Parameters:

Name	Type	Description	Default
catalog	str | None	The catalog to create it in. Defaults to catalog.	None

Yields:

Type	Description
str	The schema's full name.

Raises:

Type	Description
ValueError	If there is no catalog.

sweep

sweep(
    *,
    scope: SweepScope = "mine",
    schemas_catalog: str | None = None,
    notebooks: bool = False,
    branch: str | None = None,
    dry_run: bool = False,
    report: str | Path | None = None,
) -> SweepReport

Delete leftover ephemeral jobs, pipeline copies, schemas and notebooks, as --lakeflow-sweep does.

It needs no pytest session, no bundle and no test, so a nightly job can call it from a script. It uses this instance's workspace client, sweep_hours and helpers_path, or your default helpers folder, and spares no run: every job older than sweep_hours goes, unless its pytest_expires time is still ahead.

Parameters:

Name	Type	Description	Default
scope	SweepScope	mine for what you created, all for what anyone created, every user's default helpers folder included.	'mine'
schemas_catalog	str | None	A catalog whose old pytest_<run id>_<hex> schemas to drop too.	None
notebooks	bool	Also delete old setup, teardown and debug notebooks and inlined saved query files that no job the sweep keeps uses.	False
branch	str | None	Delete only what carries this pytest_git_branch, and the notebooks only its jobs use.	None
dry_run	bool	Delete nothing, and report what would be deleted.	False
report	str | Path | None	A file to write the report to, as JSON.	None

Returns:

Type	Description
SweepReport	What was deleted, or with dry_run what would have been.

Raises:

Type	Description
ValueError	If scope is not mine or all.

EphemeralJob

EphemeralJob

EphemeralJob(
    lakeflow: Lakeflow,
    job_key: str,
    settings: JobSettings,
    params: Mapping[str, str] | None = None,
    capture_task_values: Mapping[str, Sequence[str]]
    | None = None,
    stubs: Collection[str] = (),
    *,
    extra_dependencies: Sequence[str] = (),
    pipeline_copies: Sequence[EphemeralPipeline] = (),
    teardown_at: TeardownAt = "run",
    query_files: Mapping[str, str] | None = None,
)

Bases: EphemeralResource

A throwaway copy of a bundle job.

The job is created on entering the with block, or by the first run() or start() when used without with. It is deleted on leaving the block, or at the end of the test that made it, unless keep is set or the test failed and keep_failed is on. Runs that are still active then are cancelled first. With teardown_at="close", the teardown task then runs alone, before the job is deleted or kept.

Attributes:

Name	Type	Description
lakeflow		The Lakeflow that made the job.
job_key		The bundle resource key of the job it copies.
settings		The settings the job is created with.
params		Job parameters for every run.
capture_task_values		The task value keys captured by probes, by task key.
stubs		Keys of the stub tasks.
extra_dependencies		The test-only dependencies passed to job() or task(), local wheels resolved to their uploaded copies.
pipeline_copies		The pipeline copies that pipelines="copy" made, created just before the job and closed just after it. They are chosen again when the job is created, after modify_settings, so a pipeline task that it repoints or adds gets a copy too.
teardown_at		When the teardown task runs. "run" runs it at the end of every run. "close" leaves it out of every run and runs it alone when the job closes, so the test's assertions still see what it removes.
job_id	int | None	The job's id, once created.
runs	list[JobRun]	The outcome of each finished run, in order. The teardown run that closing the job starts is not one.
query_files		The SQL text of each saved query that saved_queries="inline" inlined, by the workspace path of its file. They are uploaded just before the job is created and deleted with it, or kept with it.
teardown_run	JobRun | None	The run of the teardown task alone that closing the job started, with teardown_at="close", once it has finished. Its teardown is the task's outcome. None until then, and when the job never ran, keep or --lakeflow-keep kept the job, or the run did not finish.

url property

url: str | None

The job's page in the workspace, or None before it is created.

create

create() -> None

Create the job if it does not exist yet, after the pipeline copies its pipeline tasks point at.

Emits a UserWarning naming the notebook tasks that extra_dependencies does not reach, checked after modify_settings, before jobs.create. modify_settings sees the deployed pipelines' ids, which are replaced by the copies' ids after it. With pipelines="copy", the pipelines to copy are chosen after it too.

Raises:

Type

Description

LakeflowError

If the job has been closed, params names a parameter that the job does not declare after modify_settings has run, a serverless task that needs an environment has none, a task refers to the output of a notebook task, or a task or job parameter default refers to a table update or file arrival trigger that the copy does not have, which jobs.create refuses. Also if the job runs as a service principal and you do not have the Service Principal User role on it, or jobs.create refuses the job with PERMISSION_DENIED, with the platform's error as its cause. Without a service principal in run_as, errors from jobs.create are raised unchanged.

run

run(
    params: Mapping[str, str] | None = None,
    only: Sequence[str] | None = None,
    *,
    timeout: timedelta | None = None,
    stop_after: timedelta | None = None,
    performance_target: PerformanceTarget
    | str
    | None = None,
    queue: bool | None = None,
    full_refresh: bool | None = None,
    full_refresh_selection: Sequence[str] | None = None,
    refresh_selection: Sequence[str] | None = None,
    refresh_flow_selection: Sequence[str] | None = None,
    reset_checkpoint_selection: Sequence[str] | None = None,
    replay: JobRun | None = None,
    stream: bool | None = None,
    dbt_commands: Sequence[str] | None = None,
) -> JobRun

Run the job and wait for it to finish, creating it first if needed.

Every call sends a fresh idempotency token with the run request, so a request the SDK retries cannot start a second run. It is start(params, only, timeout=timeout, ...).wait(stop_after=stop_after).

Parameters:

Name	Type	Description	Default
params	Mapping[str, str] | None	Job parameters for this run only, merged over the job's params.	None
only	Sequence[str] | None	Task keys to run. +key, key+ and +key+ add upstream and downstream tasks.	None
timeout	timedelta | None	How long to wait. Defaults to Lakeflow's timeout. When this plus 60 seconds is longer than the job's timeout_seconds, which Lakeflow's timeout sets, the job's timeout_seconds is raised to it before the run starts, with a warning when the template or modify_settings set the shorter value.	None
stop_after	timedelta | None	How long the run may go on before it is cancelled, for a job that does not end on its own, such as a streaming job. The run is then polled until it has ended and returned as a JobRun whose cancelled is true, instead of raising. It counts from the start of the run and must not be longer than timeout.	None
performance_target	PerformanceTarget | str | None	The performance target of this run only, such as "PERFORMANCE_OPTIMIZED". It applies to serverless tasks and leaves the job's settings unchanged. Defaults to the job's own.	None
queue	bool | None	Whether this run may queue behind other runs. Defaults to Lakeflow's queue, and when that is None to the job's own queue setting.	None
full_refresh	bool | None	Whether the job's pipeline tasks run a full refresh in this run. Defaults to what each pipeline_task sets. The pipeline is the deployed one, unless the job was made with pipelines="copy".	None
full_refresh_selection	Sequence[str] | None	Tables that the job's pipeline tasks fully refresh in this run. Needs databricks-sdk 0.110.0 or later.	None
refresh_selection	Sequence[str] | None	Tables that the job's pipeline tasks refresh in this run. It applies to every pipeline task in the run. Needs databricks-sdk 0.110.0 or later.	None
refresh_flow_selection	Sequence[str] | None	Flows that the job's pipeline tasks refresh in this run, by flow name. Needs databricks-sdk 0.110.0 or later.	None
reset_checkpoint_selection	Sequence[str] | None	Streaming flows whose checkpoints the job's pipeline tasks reset in this run, so that they process their source again from the start. Needs databricks-sdk 0.110.0 or later.	None
replay	JobRun | None	A run of the same job whose captured task values stub the tasks that only leaves out and a selected task depends on, as upstream_task_values on job() does. The platform returns no task values with a run, so that run must have captured them, with capture_task_values={"<upstream key>": [...]} on job(). The stubs are part of the job's settings, so replay needs only and a job that has not been created yet: pass it to the first run of a new lakeflow.job(...).	None
stream	bool | None	Whether to print the start of the run, each task state change and the end of the run to the terminal while waiting, as lakeflow: <line>, whatever the output capture and --log-cli-level. Defaults to whether --lakeflow-live is set. An xdist worker cannot write to the terminal, so there the lines are only logged.	None
dbt_commands	Sequence[str] | None	The commands of every dbt task in this run, such as ["dbt deps", "dbt build --select tag:ci"], in place of each task's own commands, which the job's settings keep. TaskRun.parameters of each dbt task shows them. The Jobs API refuses them on a job with job parameters, including the ones the plugin declares for test_marker, trigger= and input=. To change the commands of one dbt task in the copy, pass task_params to job() or task() instead.	None

Returns:

Type	Description
JobRun	The run's outcome.

Raises:

Type	Description
ValueError	If performance_target is not a known performance target, stop_after is longer than timeout, dbt_commands is empty or a string, or replay is passed without only or is a run of another job.
LakeflowError	If the job has been closed, a parameter is not declared, only names an unknown task or a task that is disabled in the copy or downstream of a disabled task, which the platform would run, a selected task refers to the task values or the output of a task that only leaves out and no stub replaces, a pipeline refresh option is set and the job has no pipeline task, a selection is set and the installed databricks-sdk cannot send it, replay is passed to a job that has been created or lacks the captured values of a task it would stub, or dbt_commands is passed to a job with job parameters. Each is raised before the run starts. Also if no run ends within timeout while the process has as many runs active as max_concurrent_runs allows, as for start().
LakeflowTimeout	If the run did not finish in time. The run is cancelled first.

start

start(
    params: Mapping[str, str] | None = None,
    only: Sequence[str] | None = None,
    *,
    timeout: timedelta | None = None,
    performance_target: PerformanceTarget
    | str
    | None = None,
    queue: bool | None = None,
    full_refresh: bool | None = None,
    full_refresh_selection: Sequence[str] | None = None,
    refresh_selection: Sequence[str] | None = None,
    refresh_flow_selection: Sequence[str] | None = None,
    reset_checkpoint_selection: Sequence[str] | None = None,
    dbt_commands: Sequence[str] | None = None,
) -> RunHandle

Start a run of the job without waiting for it, creating the job first if needed.

The run is cancelled when the job closes, unless RunHandle.wait() has returned by then. Its page is recorded in the test's report straight away, so a run that is never waited for still appears there, and with --lakeflow-report-dir the cancelled run is listed in report.json. The job allows one run at a time, so start runs of different jobs to run them side by side.

Under --lakeflow-max-concurrent-runs or Lakeflow's max_concurrent_runs, when the process already has that many runs active, it waits for one of them to end before starting the run, for at most timeout. While it waits it polls those runs, so a run that has ended frees its slot even if nobody has waited for it.

Parameters:

Name	Type	Description	Default
params	Mapping[str, str] | None	Job parameters for this run only, as for run().	None
only	Sequence[str] | None	Task keys to run, as for run().	None
timeout	timedelta | None	How long RunHandle.wait() waits when it is given no timeout of its own. Defaults to Lakeflow's timeout. As for run(), when this plus 60 seconds is longer than the job's timeout_seconds, the job's timeout_seconds is raised to it before the run starts.	None
performance_target	PerformanceTarget | str | None	The performance target of this run only, as for run().	None
queue	bool | None	Whether this run may queue behind other runs, as for run().	None
full_refresh	bool | None	Whether the job's pipeline tasks run a full refresh in this run, as for run().	None
full_refresh_selection	Sequence[str] | None	Tables that the job's pipeline tasks fully refresh in this run, as for run().	None
refresh_selection	Sequence[str] | None	Tables that the job's pipeline tasks refresh in this run, as for run().	None
refresh_flow_selection	Sequence[str] | None	Flows that the job's pipeline tasks refresh in this run, as for run().	None
reset_checkpoint_selection	Sequence[str] | None	Streaming flows whose checkpoints the job's pipeline tasks reset in this run, as for run().	None
dbt_commands	Sequence[str] | None	The commands of every dbt task in this run, as for run().	None

Returns:

Type	Description
RunHandle	A handle to wait for the run or cancel it.

Raises:

Type	Description
ValueError	If performance_target is not a known performance target, or dbt_commands is empty or a string.
LakeflowError	If the job has been closed, a parameter is not declared, only names an unknown task or a task that is disabled in the copy or downstream of a disabled task, which the platform would run, a selected task refers to the task values or the output of a task that only leaves out and no stub replaces, a pipeline refresh option is set and the job has no pipeline task, a selection is set and the installed databricks-sdk cannot send it, or dbt_commands is passed to a job with job parameters. Also if no run ends within timeout while the process has as many runs active as max_concurrent_runs allows.

debug_notebook

debug_notebook(
    task_key: str, run: JobRun | None = None
) -> DebugNotebook

Write a notebook that reproduces a notebook task of a run, to debug it interactively.

The notebook sets the task's resolved parameters as widgets and runs the task's notebook with %run. It is written to <helpers_path>/debug/<job_key>__<task_key>__<run_id>, replacing any notebook of the same run, and its URL is recorded in the test's report as lakeflow_debug_notebook. It replays the parameters only: a notebook that reads upstream task values needs debugValue= in dbutils.jobs.taskValues.get().

Parameters:

Name	Type	Description	Default
task_key	str	The notebook task to reproduce.	required
run	JobRun | None	The run whose parameters to replay. Defaults to the job's last run.	None

Returns:

Type	Description
DebugNotebook	The notebook written.

Raises:

Type	Description
LakeflowError	If the job has no task task_key, the task is not a notebook task or runs its notebook from Git, the job has not run, or the task has no resolved parameters in the run, as when it did not run.
KeyError	If the task was not part of the run, as JobRun.task() raises.

close

close(failed: bool = False) -> None

Cancel active runs, then delete the job unless the keep rules say to keep it, then its pipeline copies.

With teardown_at="close", the teardown task runs alone between the two, with the job's params and no run-level ones, and is waited for for Lakeflow's timeout, unless the job never ran or keep or --lakeflow-keep keeps it. keep_failed does not skip it: a job kept because its test failed is cleaned up too, and its teardown run is among the kept runs. A teardown run that fails or times out only logs a warning with its page, and the job is deleted or kept as usual.

With --lakeflow-report-dir, each cancelled run is polled until it has ended, for at most 60 seconds, and listed in the report.json of the test that is running, whose report is still to be written: for a job that a module- or session-scoped fixture closes, the test being torn down. Outside a test the run is not polled. The job is deleted even when the polling is interrupted. The pipeline copies go after the job, with the same keep rules, so a kept job never points at a deleted pipeline, and they go even when deleting the job fails. When the job was never created, no job points at them, so keep_failed does not keep them. The SQL files of query_files stay with a kept job and are deleted otherwise. Unlike the pipeline copies, they are deleted when the job was never created even when keep or --lakeflow-keep is on. A failure to delete one only logs a warning.

Parameters:

Name	Type	Description	Default
failed	bool	Whether the test failed, which keeps the job when keep_failed is on.	False

EphemeralPipeline

EphemeralPipeline

EphemeralPipeline(
    lakeflow: Lakeflow,
    pipeline_key: str,
    source_id: str,
    create_kwargs: Mapping[str, Any],
    schema: str,
)

Bases: EphemeralResource

A throwaway copy of a deployed pipeline, publishing to a schema that the test chose.

The copy is created on entering the with block, or by the first run() when used without with. It is deleted with its tables on leaving the block, or at the end of the test that made it, unless keep is set or the test failed and keep_failed is on. An update that is still running then is stopped first. The copies that pipelines="copy" makes for a job are created and closed by that job instead.

Attributes:

Name	Type	Description
lakeflow		The Lakeflow that made the copy.
pipeline_key		The bundle resource key of the pipeline it copies, or that pipeline's id when the bundle does not define it.
source_id		The id of the pipeline it copies.
create_kwargs		The arguments the copy is created with, as pipelines.create takes them.
schema		The full name of the schema the copy publishes to.
pipeline_id	str | None	The copy's id, once created.
updates	list[PipelineUpdate]	The outcome of each update that run() waited for, in order.

url property

url: str | None

The copy's page in the workspace, or None before it is created.

create

create() -> None

Create the copy if it does not exist yet.

Raises:

Type	Description
LakeflowError	If the copy has been closed. Errors from pipelines.create are raised unchanged.

run

run(
    *,
    full_refresh: bool | None = None,
    full_refresh_selection: Sequence[str] | None = None,
    refresh_selection: Sequence[str] | None = None,
    timeout: timedelta | None = None,
) -> PipelineUpdate

Run an update of the copy and wait for it to end, creating the copy first if needed.

A failed update is returned, not raised: check its state, and read what went wrong with events(level="ERROR"). Each change of the update's state is logged and shown in the lakeflow report section of the test.

Parameters:

Name	Type	Description	Default
full_refresh	bool | None	Whether the update refreshes every table from scratch.	None
full_refresh_selection	Sequence[str] | None	Tables that the update refreshes from scratch.	None
refresh_selection	Sequence[str] | None	Tables that the update refreshes.	None
timeout	timedelta | None	How long to wait. Defaults to Lakeflow's timeout.	None

Returns:

Type	Description
PipelineUpdate	The update, which is also appended to updates.

Raises:

Type	Description
ValueError	If a selection is a string rather than a list of names.
LakeflowError	If the copy has been closed.
LakeflowTimeout	If the update did not end in time. It is stopped first.

close

close(failed: bool = False) -> None

Stop an update that is still running, then delete the copy with its tables unless the keep rules say not to.

Parameters:

Name	Type	Description	Default
failed	bool	Whether the test failed, which keeps the copy when keep_failed is on.	False

RunHandle

RunHandle

RunHandle(
    job: EphemeralJob,
    run_id: int,
    item: Item,
    first: Run,
    timeout: timedelta | None = None,
    *,
    stream: bool | None = None,
    performance_target: PerformanceTarget | None = None,
)

A run started by EphemeralJob.start(), to wait for or cancel later.

Attributes:

Name	Type	Description
job		The job the run belongs to.
run_id		The run's id.

url property

url: str | None

The run's page in the workspace, or None if the workspace has not reported it yet.

wait

wait(
    timeout: timedelta | None = None,
    *,
    stop_after: timedelta | None = None,
) -> JobRun

Wait for the run to finish.

Waiting again returns the same JobRun, and run_finished is called only the first time.

Parameters:

Name	Type	Description	Default
timeout	timedelta | None	How long to wait from this call. Defaults to the timeout passed to start(), else Lakeflow's timeout. It must not be longer than the job's timeout_seconds, which Lakeflow's timeout sets. The run has already started, so pass a longer timeout to start() or run(), which raise timeout_seconds first.	None
stop_after	timedelta | None	How long the run may go on before it is cancelled, counted from start(), for a job that does not end on its own. The run is then polled until it has ended and returned as a JobRun whose cancelled is true, instead of raising. It must not be longer than timeout.	None

Returns:

Type	Description
JobRun	The run's outcome, which is also appended to the job's runs.

Raises:

Type	Description
ValueError	If stop_after is longer than timeout. The run goes on.
LakeflowError	If the job closed before the run finished, which cancelled the run, or timeout is longer than the job's timeout_seconds, in which case the run goes on.
LakeflowTimeout	If the run did not finish in time. The run is cancelled first.

cancel

cancel() -> None

Cancel the run. It ends CANCELED shortly after, which a later wait() returns.

JobRun

JobRun dataclass

JobRun(
    job_key: str,
    job_id: int,
    run_id: int,
    run_url: str,
    result_state: str | None,
    termination_code: str | None,
    termination_message: str | None,
    duration: timedelta,
    tasks: dict[str, TaskRun],
    injected_tasks: dict[str, TaskRun],
    skipped_tasks: dict[str, TaskRun] = dict(),
    disabled_tasks: dict[str, TaskRun] = dict(),
    parameters: dict[str, str] = dict(),
    timing: Timing = Timing(),
    effective_performance_target: str | None = None,
    trigger_type: str | None = None,
)

The outcome of one run of an ephemeral job, or of the child run that a run-job task triggered.

Attributes:

Name	Type	Description
job_key	str	The bundle resource key of the job. For a child run, the child job's key when the job is in the bundle, otherwise its job id as a string.
job_id	int	The ephemeral job's id. For a child run, the id of the deployed child job.
run_id	int	The run's id.
run_url	str	The run's page in the workspace.
result_state	str | None	The parent run's result state. Not a verdict: a run whose probe ran after a failed task ends SUCCESS_WITH_FAILURES. Use succeeded instead.
termination_code	str | None	The parent run's termination code.
termination_message	str | None	The parent run's termination message.
duration	timedelta	The time from start to end of the parent run.
tasks	dict[str, TaskRun]	The user's tasks that were part of the run, by task key.
injected_tasks	dict[str, TaskRun]	The stub, probe, setup and teardown tasks added by the plugin, by task key.
skipped_tasks	dict[str, TaskRun]	Tasks left out by only, by task key.
disabled_tasks	dict[str, TaskRun]	Tasks disabled in the job, in the bundle or with disable=, and the tasks the platform disabled because they are downstream of one, by task key. When every task is disabled, the platform skips the run: its tasks have no result state and are all here.
parameters	dict[str, str]	The run's job parameters by name: the value passed for the run, or the default when none was. Parameters the plugin injects are left out.
timing	Timing	The queue, setup, execution and cleanup durations of the parent run. Zero where the platform reported none; a run with several tasks reports its time on the tasks instead.
effective_performance_target	str | None	The serverless performance mode the run used, STANDARD or PERFORMANCE_OPTIMIZED, or None when the platform did not say.
trigger_type	str | None	What started the run, as the Jobs API reports it: ONE_TIME for a run of an ephemeral job, which is started with run-now. A trigger simulated with trigger= on Lakeflow.job or Lakeflow.task changes only what the task code sees, not this. None when the platform did not say.

succeeded property

succeeded: bool

Whether every task in tasks ended SUCCESS.

Injected and skipped tasks do not count, nor do tasks EXCLUDED because their run_if or if/else condition was not met.

failed property

failed: bool

Whether the run did not succeed.

timed_out property

timed_out: bool

Whether the parent run ended TIMEDOUT or any task in tasks timed out.

cancelled property

cancelled: bool

Whether the parent run ended CANCELED, as a run stopped by stop_after does. Such a run also failed.

executed_tasks property

executed_tasks: list[str]

The sorted keys of the tasks in tasks whose ran is true.

setup property

setup: TaskRun | None

The task that ran the job's setup= function, or None when the job has none.

When it fails, the user's tasks end UPSTREAM_FAILED, so succeeded is false, and repr() names the setup failure first.

teardown property

teardown: TaskRun | None

The task that ran the job's teardown= function, or None when the job has none or the run left it out.

A run leaves it out when the job was made with teardown_at="close": the task then ends DISABLED and runs alone when the job closes, in EphemeralJob.teardown_run. A failed teardown does not change succeeded; the plugin logs a warning instead. Assert run.teardown.succeeded to fail the test on it.

task

task(key: str) -> TaskRun

Get one of the user's tasks.

Parameters:

Name	Type	Description	Default
key	str	The task key.	required

Returns:

Type	Description
TaskRun	The task's outcome.

Raises:

Type	Description
KeyError	If the task was not part of the run. The message lists the available keys and says whether the task was disabled in the job or skipped by only.

branch_taken

branch_taken(key: str) -> bool

Get the outcome of an if/else condition task.

Parameters:

Name	Type	Description	Default
key	str	The condition task's key.	required

Returns:

Type	Description
bool	The task's condition_outcome: True when the condition held, False when it did not.

Raises:

Type	Description
KeyError	If the task was not part of the run.
LakeflowError	If the task is not a condition task, or the platform reported no outcome for it.

raise_for_failure

raise_for_failure() -> JobRun

Raise if the run did not succeed.

Returns:

Type	Description
JobRun	The run itself, so run = job.run().raise_for_failure() works.

Raises:

Type	Description
LakeflowRunFailed	If succeeded is false. The message is repr(run), with the run's job parameters on a params: line of their own when it has any, followed by one line per failing task with its error, the error's root cause when there is one, and the task's run page. A failing task whose resolved parameters are known gets an indented params: line after its own.

as_dict

as_dict() -> dict[str, Any]

Get the run as a dict that json.dumps accepts, as --lakeflow-report-dir writes it to report.json.

Every field of the run and of each TaskRun is kept, including attempts and a nested child_run, with these changes: duration becomes duration_seconds and each timing a dict of queue_seconds, setup_seconds, execution_seconds and cleanup_seconds; error is a dict of type, message, trace, root_cause and code, or None; task_values is a plain dict and missing_task_value_reasons holds the reason for each of missing_task_values; the outputs of SQL, alert, dbt, run-job, dashboard and clean room tasks are the dicts of the Jobs API, except that a dbt output leaves out artifacts_headers; dbt is a dict of artifacts_dir, results (each a dict of the DbtResult fields, by unique id), failed_models and all_passed, leaving out run_results and manifest, or None; pipeline_update is a dict of pipeline_id, update_id, state, cause and full_refresh, or None; iterations is a list of tasks serialised the same way, and for_each_stats a dict of its fields, with errors a list of dicts, or None. The run also gets succeeded.

logs and the exported notebook are left out, because logs can reach 5 MB per task. Instead logs_file and notebook_file name where --lakeflow-report-dir saves them, relative to the test's report directory: <run_id>/<task_key>.log and <run_id>/<task_key>.html, with .attempt-<n> before the suffix for an attempt that was retried and .iteration-<n>, counting from 0, for an iteration of a for-each task. Each is None when the task has no logs or no exported notebook.

Returns:

Type	Description
dict[str, Any]	The run as a JSON-serialisable dict.

TaskRun

TaskRun dataclass

TaskRun(
    task_key: str,
    run_id: int,
    result_state: str | None,
    termination_code: str | None,
    termination_message: str | None,
    run_url: str | None,
    duration: timedelta,
    output: str | None = None,
    output_truncated: bool = False,
    logs: str = "",
    logs_truncated: bool = False,
    error: TaskError | None = None,
    task_values: TaskValues = TaskValues(),
    missing_task_values: list[str] = list(),
    attempt: int = 0,
    parameters: dict[str, str] | list[str] | None = None,
    attempts: list[TaskRun] = list(),
    timing: Timing = Timing(),
    effective_performance_target: str | None = None,
    condition_outcome: bool | None = None,
    sql_output: SqlOutput | None = None,
    alert_output: AlertTaskOutput | None = None,
    dbt_output: DbtOutput | None = None,
    dbt: DbtArtifacts | None = None,
    run_job_output: RunJobOutput | None = None,
    child_run_id: int | None = None,
    child_run: JobRun | None = None,
    dashboard_output: DashboardTaskOutput | None = None,
    clean_rooms_notebook_output: CleanRoomsNotebookTaskCleanRoomsNotebookTaskOutput
    | None = None,
    notebook: NotebookView | None = None,
    sql_result: SqlResult | None = None,
    pipeline_update: PipelineUpdate | None = None,
    iterations: list[TaskRun] = list(),
    for_each_stats: ForEachStats | None = None,
)

The outcome of one task in a job run.

When the platform retried the task, this describes the last attempt and attempts holds every attempt.

Attributes:

Name	Type	Description
task_key	str	The task's key.
run_id	int	The task run's id.
result_state	str | None	The task's result state, such as SUCCESS, FAILED or UPSTREAM_FAILED.
termination_code	str | None	The termination code, such as RUN_EXECUTION_ERROR.
termination_message	str | None	The termination message.
run_url	str | None	The task run's page in the workspace.
duration	timedelta	The time from start to end.
output	str | None	The value the notebook passed to dbutils.notebook.exit(). For a clean room notebook task, the value from clean_rooms_notebook_output when the top-level notebook output is missing or has no result.
output_truncated	bool	Whether output was cut at 5 MB.
logs	str	stdout and stderr of a Python script, wheel or JAR task. Empty for notebook tasks, whose printed text is in notebook.
logs_truncated	bool	Whether logs holds only the last 5 MB.
error	TaskError | None	The task's error, or None if it succeeded or did not run. For a failed alert task, which the Jobs API reports no error for, the task's termination message with type None.
task_values	TaskValues	Task values captured by a probe.
missing_task_values	list[str]	Keys that were requested from the probe but not captured.
attempt	int	The attempt number, 0 for the first attempt.
parameters	dict[str, str] | list[str] | None	The task's parameters after job parameters and {{...}} references were resolved. A dict for notebook (base_parameters), SQL, run-job (job_parameters) and Python wheel tasks with named parameters; a list for Python wheel tasks with positional parameters, Python script, JAR and spark-submit tasks, and dbt tasks (commands); None for other task types, for a task that did not run, and when the task run could not be read.
attempts	list[TaskRun]	One TaskRun per attempt, in attempt order, each with empty attempts. The last equals this task apart from attempts.
timing	Timing	The queue, setup, execution and cleanup durations of the task run. Zero where the platform reported none.
effective_performance_target	str | None	The serverless performance mode the task ran with, STANDARD or PERFORMANCE_OPTIMIZED, or None when the platform did not say.
condition_outcome	bool | None	The outcome of an if/else condition task: True when the condition held, False when it did not, None for any other task or when the platform reported no outcome.
sql_output	SqlOutput | None	The output of a SQL task as the platform reported it: query_output (with warehouse_id, output_link, query_text and sql_statements), alert_output or dashboard_output. Holds no query rows: a SQL file task's rows are in sql_result. None for any other task or when the platform reported none.
alert_output	AlertTaskOutput | None	The output of an alert task, with its alert_state. None for any other task or when the platform reported none.
dbt_output	DbtOutput | None	The output of a dbt task, with artifacts_link and artifacts_headers, or None. The link is a pre-signed URL to a tarball of the dbt artefacts that the platform signs for 30 minutes; fetch it with the headers before then. report.json keeps the link and leaves out the headers.
dbt	DbtArtifacts | None	The artefacts of a dbt task, downloaded from dbt_output.artifacts_link and parsed once the run finished, with Lakeflow(dbt_artifacts=True). None by default, for any other task, when the platform reported no link, and when the artefacts could not be read, such as a tarball over 50 MB, which logs a warning.
run_job_output	RunJobOutput | None	The output of a run-job task, with the triggered run's run_id, or None.
child_run_id	int | None	The id of the job run that a run-job task triggered, from run_job_output.run_id. The run is of the deployed child job, not a copy, so the plugin does not cancel or delete it. None for any other task or when the platform reported no run id.
child_run	JobRun | None	The run that a run-job task triggered, as a JobRun with the child's own tasks, read with get_run once the parent run finished. Its job_key is the child job's bundle resource key, or its job id as a string when the job is not in the bundle. A child with its own run-job tasks has nested child_runs, and a task disabled in the child job is in its disabled_tasks. Each of attempts carries the child run that attempt triggered. None for any other task, when the task did not run, when the platform reported no run id, or when the child run could not be read, which logs a warning.
dashboard_output	DashboardTaskOutput | None	The output of a dashboard task, with its page_snapshots, or None.
clean_rooms_notebook_output	CleanRoomsNotebookTaskCleanRoomsNotebookTaskOutput | None	The output of a clean room notebook task, with notebook_output and clean_room_job_run_state, or None.
notebook	NotebookView | None	The rendered notebook of a notebook task, with each cell's code, output and error, read with jobs.export_run once the run finished. Exported for the tasks that Lakeflow(notebook_export=...) selects: by default, a notebook task that ran and did not succeed. None for any other task, and when export_run failed, which logs a warning.
sql_result	SqlResult | None	The columns and rows a SQL file task returned, read from runs/get-output once the run finished. None for any other task, including a saved query task, for a task that did not run, and when the output could not be read, which logs a warning.
pipeline_update	PipelineUpdate | None	The pipeline update that a pipeline task started, found once the run finished among the pipeline's updates by its creation time, which lies between the task's start and end. None for any other task, for a task that did not run, and when no update was created in that time or the updates could not be read, which logs a warning.
iterations	list[TaskRun]	The iterations of a for-each task, one TaskRun per input in iteration order, read once the run finished from the for-each task run fetched by its own id. Each is built like a task, with one get_run_output and one get_run for every attempt of it: task_key is the nested task's key, output, error and the rest are its own, parameters are the nested task's resolved parameters, where the iteration's input appears wherever the nested task passes {{input}}, and attempts holds every attempt when the platform retried it. Iterations have no task_values, because the platform refuses taskValues.set in an iteration, and no notebook. Empty for any other task, for a for-each task that did not start, such as one that ended UPSTREAM_FAILED, and when the iterations could not be read, which logs a warning.
for_each_stats	ForEachStats | None	The iteration counts of a for-each task, as the platform reports them, or None for any other task and for a for-each task run that reports no stats.

succeeded property

succeeded: bool

Whether the task ended SUCCESS.

timed_out property

timed_out: bool

Whether the task ended TIMEDOUT, stopped by the platform at its timeout_seconds.

ran property

ran: bool

Whether the task ran. False if it was skipped, excluded, disabled or cancelled, or has no result state.

The platform reports no result state for the tasks of a run it skipped because every task is disabled.

excluded property

excluded: bool

Whether the task ended EXCLUDED because its run_if or if/else condition was not met.

upstream_failed property

upstream_failed: bool

Whether the task ended UPSTREAM_FAILED because a task it depends on failed.

disabled property

disabled: bool

Whether the task ended DISABLED: disabled in the job, downstream of a disabled task, or left out by only.

JobRun.disabled_tasks and JobRun.skipped_tasks tell them apart.

cancelled property

cancelled: bool

Whether the task ended CANCELED.

alert_state property

alert_state: str | None

The state an alert evaluated to: TRIGGERED, OK, ERROR or UNKNOWN.

Read from alert_output for an alert task, else from sql_output.alert_output for a legacy SQL alert task. None for any other task or when the platform reported no state. An alert task succeeds whether or not the alert triggered, so assert on this rather than on succeeded. ERROR means the alert could not be evaluated, for example because its query failed; the task then fails too.

output_json property

output_json: object

output parsed as JSON.

Raises:

Type	Description
ValueError	If the notebook did not call dbutils.notebook.exit(), if output was cut at 5 MB, or if output is not JSON. The message names the task.

log_lines property

log_lines: list[str]

The non-blank lines of logs.

raised

raised(exc_type: type[BaseException]) -> bool

Check whether the task failed with an exception of the given class.

Parameters:

Name	Type	Description	Default
exc_type	type[BaseException]	The exception class. Only its name is compared, because the task ran in another process.	required

Returns:

Type	Description
bool	True if the task's error has the class name of exc_type.

DbtArtifacts

DbtArtifacts dataclass

DbtArtifacts(
    artifacts_dir: Path,
    results: dict[str, DbtResult],
    run_results: dict[str, Any],
    manifest: dict[str, Any] | None,
)

The artefacts a dbt task wrote, downloaded and extracted once its run finished.

The platform runs each of the task's commands in turn, so run_results.json describes the last command that wrote one, such as dbt build after dbt deps.

Attributes:

Name	Type	Description
artifacts_dir	Path	Where the tarball was extracted, in a temporary directory removed at the end of the pytest session, unless keep or --lakeflow-keep is set.
results	dict[str, DbtResult]	One DbtResult per node in run_results.json, by unique id, in the order dbt reported them.
run_results	dict[str, Any]	run_results.json as parsed JSON.
manifest	dict[str, Any] | None	manifest.json from the same folder as parsed JSON, or None when the tarball has none.

failed_models property

failed_models: list[str]

The unique ids of the nodes that ended error or fail, in the order dbt reported them.

all_passed property

all_passed: bool

Whether no node ended error or fail. A node skipped because an upstream node failed is not counted.

DbtResult

DbtResult dataclass

DbtResult(
    unique_id: str,
    status: str,
    message: str | None = None,
    failures: int | None = None,
    execution_time: float = 0.0,
)

The outcome of one dbt node, a model, test, seed or snapshot, as run_results.json reports it.

Attributes:

Name	Type	Description
unique_id	str	The node's unique id, such as model.my_project.orders.
status	str	What dbt reported: success, error or skipped for a model, seed or snapshot, and pass, fail, warn, error or skipped for a test.
message	str | None	dbt's message for the node, such as the database error, or None.
failures	int | None	The number of failing rows of a test, or None for any other node.
execution_time	float	How long the node took, in seconds.

ForEachStats

ForEachStats dataclass

ForEachStats(
    total_iterations: int,
    succeeded_iterations: int,
    failed_iterations: int,
    errors: list[ForEachError] = list(),
)

The iteration counts of a for-each task, as the platform reports them on the task run.

Each iteration counts once, however often it was retried.

Attributes:

Name	Type	Description
total_iterations	int	How many iterations the task had, one per input.
succeeded_iterations	int	How many of them succeeded.
failed_iterations	int	How many of them failed.
errors	list[ForEachError]	The errors they failed with, each with the number of iterations that failed with it.

from_stats classmethod

from_stats(stats: ForEachStats) -> ForEachStats

Read the counts of a RunForEachTask.stats.

Parameters:

Name	Type	Description	Default
stats	ForEachStats	The stats from get_run.	required

Returns:

Type	Description
ForEachStats	The counts, with 0 for each one the platform does not report.

ForEachError

ForEachError dataclass

ForEachError(
    message: str,
    count: int,
    termination_category: str | None = None,
)

One error that iterations of a for-each task failed with, as the platform counts it.

Attributes:

Name	Type	Description
message	str	The error message. The platform reports the same generic message, such as Workload failed, see run output for details, for every iteration whose code raised; each iteration's own error is in TaskRun.iterations.
count	int	How many iterations failed with it.
termination_category	str | None	The platform's category of the error, such as RunExecutionError, or None.

ALL_RULES and DEFAULT_RULES

The names of the rules that Lakeflow.check(key, rules=...) applies on top of the four it always applies, as frozensets: ALL_RULES is wheel, library, dash, trigger and equal_to, and DEFAULT_RULES, what check applies when rules is None, is every rule but equal_to. See Check every job's wiring.

Stub

Stub dataclass

Stub(
    values: Mapping[str, object] = dict(),
    outputs: Mapping[str, object] = dict(),
)

What a stub task stands in for: the task values it sets and the output references it answers.

A plain mapping in upstream_task_values is the same as Stub(values=mapping).

Attributes:

Name	Type	Description
values	Mapping[str, object]	The task values the stub sets, by key, as a plain mapping gives them.
outputs	Mapping[str, object]	Literals for the {{tasks.<key>.output.<path>}} references to the stubbed task, by the path after output. exactly as the reference spells it, such as "first_row.n", "alert_state" or "rows". A string is substituted verbatim and any other value as compact JSON, so {"rows": [{"n": "3"}]} gives [{"n":"3"}], the platform's own spelling.

TaskValues

TaskValues

TaskValues(
    values: Mapping[str, object] | None = None,
    reasons: Mapping[str, str] | None = None,
)

Bases: dict[str, object]

Task values captured by a probe task.

A dict whose missing keys raise a KeyError that says why the value is missing: the task never set it, the task did not run, or the probe failed.

Attributes:

Name	Type	Description
reasons		The explanation for each missing key.

TaskError

TaskError dataclass

TaskError(type: str | None, message: str, trace: str)

An error reported by a task run.

Attributes:

Name	Type	Description
type	str | None	The exception class name as Python printed it, possibly dotted, such as json.decoder.JSONDecodeError. None when the error is not an exception, for example when the platform skipped the task.
message	str	The exception message.
trace	str	The error trace with ANSI colour codes removed.

root_cause property

root_cause: str | None

The first thing that went wrong, read from trace.

A Python chained traceback, from raise ... from e or an exception raised while handling another, prints the innermost exception first, so only the part of trace before the first chain marker is read. In that part, the text after the last Caused by: line wins, as in a Py4J or Spark trace. Otherwise, when the trace is chained, it is the innermost exception as Name: message, such as ZeroDivisionError: division by zero, while type and message still describe the outer exception. None when the trace has neither a Caused by: line nor a chain.

code property

code: str | None

The bracketed error class, such as TABLE_OR_VIEW_NOT_FOUND, at the start of message or root_cause.

None when neither starts with one. Lowercase tags such as [my_tag] are not error classes.

from_output classmethod

from_output(
    error: str | None, trace: str | None
) -> TaskError | None

Parse the error fields of a task's get_run_output response.

The error field is preferred, because error_trace is ANSI-coloured IPython output. The last Name: message line of the trace is the fallback.

Parameters:

Name	Type	Description	Default
error	str | None	The error field.	required
trace	str | None	The error_trace field.	required

Returns:

Type	Description
TaskError | None	The parsed error, or None when both fields are empty.

Timing

Timing dataclass

Timing(
    queue: timedelta = timedelta(0),
    setup: timedelta = timedelta(0),
    execution: timedelta = timedelta(0),
    cleanup: timedelta = timedelta(0),
)

Where the time of a run or task run went, as runs/get reports it.

Attributes:

Name	Type	Description
queue	timedelta	Time spent waiting in the queue.
setup	timedelta	Time spent setting up compute and the environment.
execution	timedelta	Time spent running the code.
cleanup	timedelta	Time spent cleaning up afterwards.

total property

total: timedelta

The sum of queue, setup, execution and cleanup.

It differs from the duration of JobRun and TaskRun, which is wall clock from start to end, for example when a run waited in the queue.

from_run classmethod

from_run(run: Run | RunTask) -> Timing

Read the timing of a run or task run.

Parameters:

Name	Type	Description	Default
run	Run | RunTask	The run or task run from get_run. Its durations are in milliseconds.	required

Returns:

Type	Description
Timing	The timing, with timedelta(0) for every duration the run does not report.

SqlResult

SqlResult dataclass

SqlResult(
    task_key: str,
    columns: list[str],
    types: list[str],
    rows: list[dict[str, str | None]],
    truncated: bool = False,
    warehouse_id: str | None = None,
    query_text: str | None = None,
)

The rows a SQL file task returned, as runs/get-output reports them under sql_output.file_output.

The Jobs API reference does not document file_output and databricks-sdk does not model it, so the plugin reads the raw response. The platform keeps at most 1,000 rows, for 7 days.

Attributes:

Name	Type	Description
task_key	str	The task's key.
columns	list[str]	The column names in order, such as ["n"].
types	list[str]	The platform's type name of each column, such as ["int"].
rows	list[dict[str, str | None]]	One dict per row, by column name. Values are strings as the platform reports them, such as "1", "1.5", "true" or "2026-01-01", and a NULL is None. When two columns share a name, the later one wins.
truncated	bool	Whether the platform cut the rows, which it does after 1,000.
warehouse_id	str | None	The SQL warehouse the task ran on.
query_text	str | None	The SQL file as the task ran it. Named parameter markers such as :n are kept; the values bound to them are in TaskRun.parameters.

first_row property

first_row: dict[str, str | None]

The first of rows.

Raises:

Type	Description
LakeflowError	If the query returned no rows.

from_output classmethod

from_output(
    task_key: str, output: Mapping[str, Any]
) -> SqlResult | None

Read the query result of a SQL file task from its raw runs/get-output response.

Parameters:

Name	Type	Description	Default
task_key	str	The task's key.	required
output	Mapping[str, Any]	The response as JSON.	required

Returns:

Type	Description
SqlResult | None	The result, or None when the response has no sql_output.file_output.query_result, as for a saved
SqlResult | None	query, alert or dashboard task.

PipelineUpdate

PipelineUpdate dataclass

PipelineUpdate(
    pipeline_id: str,
    update_id: str,
    state: str | None,
    cause: str | None,
    full_refresh: bool,
    _ws: WorkspaceClient,
    _since: int,
)

An update of a pipeline: the one a pipeline task started, or one that EphemeralPipeline.run() started.

The Jobs API does not say which update a pipeline task started, so the plugin lists the pipeline's updates once the run finished and takes the one created while the task ran, preferring one whose cause is JOB_TASK. The pipeline is the one the task points at: the deployed one, or a copy with pipelines="copy".

Attributes:

Name	Type	Description
pipeline_id	str	The pipeline's id.
update_id	str	The update's id.
state	str | None	The update's state, such as COMPLETED, FAILED or CANCELED.
cause	str | None	What started the update: JOB_TASK for an update that a pipeline task started, API_CALL for one that EphemeralPipeline.run() started.
full_refresh	bool	Whether the update was a full refresh of every table.

from_info classmethod

from_info(
    ws: WorkspaceClient,
    info: UpdateInfo,
    *,
    since: int | None = None,
) -> PipelineUpdate

Build the update from what pipelines.get_update or pipelines.list_updates reports.

Parameters:

Name	Type	Description	Default
ws	WorkspaceClient	The workspace client, which events() reads the events with.	required
info	UpdateInfo	The update as the Pipelines API reports it.	required
since	int | None	The time in epoch milliseconds before which the update logged no event, so events() stops paging there. Defaults to a minute before the update was created.	None

Returns:

Type	Description
PipelineUpdate	The update.

events

events(
    level: str | None = None,
) -> list[pipelines.PipelineEvent]

Read the events the update logged, newest first, with pipelines.list_pipeline_events.

The platform lists the events of every update of the pipeline, newest first, so the events of other updates are dropped and the listing stops at the first event logged before the task started.

Parameters:

Name	Type	Description	Default
level	str | None	Only events of this level, such as ERROR, WARN or INFO. Every level when None.	None

Returns:

Type	Description
list[PipelineEvent]	The events, each a databricks.sdk.service.pipelines.PipelineEvent with level, event_type,
list[PipelineEvent]	message and, for an error, error.

NotebookView

NotebookView dataclass

NotebookView(html: str, cells: list[NotebookCell] = list())

A notebook task run as jobs.export_run renders it, with every cell's code and output.

The output comes from the rendered notebook, so it is complete only once the task has terminated.

Attributes:

Name	Type	Description
html	str	The exported HTML, unchanged.
cells	list[NotebookCell]	The notebook's cells in order. Empty when the export could not be read.

stdout property

stdout: str

What every cell wrote to standard output, in order, which is what print() produced.

failed_cell property

failed_cell: NotebookCell | None

The first cell that raised an exception, or None.

save

save(path: str | Path) -> Path

Write the exported HTML to a file, which opens in a browser like the run page.

Parameters:

Name	Type	Description	Default
path	str | Path	The file to write. Missing parent directories are created.	required

Returns:

Type	Description
Path	The path written.

NotebookCell

NotebookCell dataclass

NotebookCell(
    index: int,
    code: str,
    output: str = "",
    output_html: str = "",
    error: str | None = None,
    stdout: str = "",
)

One cell of a notebook exported from a task run.

Attributes:

Name	Type	Description
index	int	The cell's position in NotebookView.cells, counting from 0.
code	str	The cell's source.
output	str	The cell's text output, such as what it printed, one result per line, with HTML tags and ANSI colour codes removed and HTML entities unescaped. Empty when the cell had no output.
output_html	str	The cell's results as exported, unchanged: HTML for an HTML result, otherwise the text the notebook wrote.
error	str | None	The exception the cell raised, as Name: message, or None when it raised nothing. A cell skipped because an earlier cell failed has no error.
stdout	str	What the cell wrote to standard output, such as what it printed, with ANSI colour codes removed. Unlike output, it leaves out standard error, displayed results and the dbutils.notebook.exit() value.

DebugNotebook

DebugNotebook dataclass

DebugNotebook(
    task_key: str, run_id: int, path: str, url: str
)

A notebook in the workspace that reproduces a notebook task of a run, written by EphemeralJob.debug_notebook().

It sets the task's resolved parameters as widgets and runs the task's notebook with %run.

Attributes:

Name	Type	Description
task_key	str	The task's key.
run_id	int	The id of the job run whose parameters it replays.
path	str	The notebook's workspace path.
url	str	The notebook's page in the workspace.

SweepReport

SweepReport dataclass

SweepReport(
    dry_run: bool,
    jobs: tuple[int, ...] = (),
    pipelines: tuple[str, ...] = (),
    schemas: tuple[str, ...] = (),
    notebooks: tuple[str, ...] = (),
    errors: tuple[str, ...] = (),
)

What a sweep deleted, or with dry_run would have deleted, as Lakeflow.sweep() returns it.

Attributes:

Name	Type	Description
dry_run	bool	Whether the sweep only listed what it would delete.
jobs	tuple[int, ...]	The ids of the jobs.
pipelines	tuple[str, ...]	The ids of the pipelines.
schemas	tuple[str, ...]	The full names of the schemas.
notebooks	tuple[str, ...]	The workspace paths of the notebooks and of the inlined saved query files.
errors	tuple[str, ...]	What could not be listed or deleted.

as_dict

as_dict() -> dict[str, Any]

Get the report as --lakeflow-sweep-report writes it, with a version key.

Errors

Two kinds of error leave the plugin. A ValueError means an argument is wrong on its own: a choice outside the ones listed, a setup= that is neither a function, a list nor a .py path, a stub value that is not JSON-serialisable, a stop_after longer than its timeout. A LakeflowError, or one of its subclasses, means the bundle, the job or the workspace did not fit: a task key the job does not have, a reference the copy cannot resolve, a run that did not finish, a deploy that failed. Catch LakeflowError for the second kind; the first kind is a mistake in the test to fix.

LakeflowError

Bases: Exception

Base class for errors raised by pytest-databricks-lakeflow.

LakeflowBundleError

Bases: LakeflowError

A Databricks CLI bundle command failed, or the bundle cannot supply a job template.

LakeflowTimeout

Bases: LakeflowError

A job run or pipeline update did not finish within its timeout. The run was cancelled, the update stopped.

LakeflowRunFailed

LakeflowRunFailed(message: str, run: JobRun)

Bases: LakeflowError

A job run did not succeed. Raised by JobRun.raise_for_failure().

Attributes:

Name	Type	Description
run		The run that failed.