Define a Hera Pipeline
This section describes how to define a Hera pipeline function. A pipeline function is a Python function that returns a Hera Workflow object. The function must not accept arguments; define workflow parameters inside the returned Hera Workflow object instead.
Pipeline Function Anatomy
Define a pipeline by creating a Python function that returns a Hera Workflow object:
from hera.workflows import Workflow, DAG, Parameter
from digitalhub_runtime_hera.dsl import step
def pipeline():
# Create a new Workflow with an entrypoint DAG and parameters
with Workflow(entrypoint="dag", arguments=Parameter(name="url")) as w:
with DAG(name="dag"):
# Define workflow steps here
...
return w
Typical Pipeline Structure
- Create a
WorkflowHera object and set the entrypoint (usually a DAG). - Use a
DAGorStepscontext to define the workflow structure. - Add steps via
step(...), providing templates, function names, inputs/outputs and parameters. - Chain steps using Hera operators to define dependencies.
- Return the
WorkflowHera object.
DSL Components
The runtime provides DSL helpers in digitalhub_runtime_hera.dsl:
step function
step(**step_kwargs) creates a workflow step inside a DAG or Steps context. It returns a Hera Task inside a DAG and a Hera Step inside Steps. Main arguments:
| Parameter | Type | Example | Description |
|---|---|---|---|
| template | dict | {"action": "job"} | Parameters template to pass to the DigitalHub function run. The action key is required. To pass inputs from other steps use the {{inputs.parameters.parameter_name}} template syntax. |
| function | str | "download-data" | Name of the DigitalHub function to execute. |
| function_id | str | "abc123" | Function ID (optional). |
| name | str | "step1" | Step name. |
| inputs | dict | {"some-input": ANOTHER_STEP.get_parameter("some-output")} | Step inputs. Keys become Hera Parameters; values can reference other steps' outputs. |
| outputs | list | ["output1"] | Step outputs. These become Hera Outputs and Artifacts. |
Other keyword arguments are forwarded to the underlying container template. step must be called inside a DAG or Steps context. The function is resolved when the workflow is built, and DHCORE_WORKFLOW_IMAGE must be set to create the container template.
Step Inputs and Outputs
Step inputs are defined via the inputs argument to step(...). This is a dictionary where keys are input names and values can reference outputs from other steps using the get_parameter(...) method. For example:
# Here the step A (let say a python function) produces an output named "data"
A = step(template={"action":"job"}, function="step-a", outputs=["data"])
# Step B consumes the output "data" from step A as its input. Because "data" is an output of step A, we use A.get_parameter("data") to reference it. Note that the input name "input_data" in step B can be different from the output name "data" in step A.
# In the template of step B, we use the template syntax {{inputs.parameters.data-from-b}} to refer to the input parameter.
# The "inputs" provided in template of step B belongs to the spec of the function "step-b", in this case a python function.
B = step(template={"action":"job", "inputs": {"parameter-name": "{{inputs.parameters.data-from-b}}"}}, function="step-b", inputs={"data-from-b": A.get_parameter("data")})
container_template function
container_template(...) builds a Hera container template for a workflow step. It returns a Hera Container object and requires template and function. Unlike step, its inputs argument is a list of input names. Use it directly for advanced scenarios or custom templates. DHCORE_WORKFLOW_IMAGE must be set; the value is used as the stepper image.
Pipeline Definition Example
from hera.workflows import Workflow, DAG, Parameter
from digitalhub_runtime_hera.dsl import step
def pipeline():
# Create a new Workflow with an entrypoint DAG and a parameter
with Workflow(entrypoint="dag", arguments=Parameter(name="url")) as w:
with DAG(name="dag"):
# First step: takes the workflow parameter and outputs a dataset
A = step(template={"action":"job", "inputs": {"url": "{{workflow.parameters.url}}"}},
function="download-data",
outputs=["dataset"])
# Subsequent steps consume A's output
B = step(template={"action":"job", "inputs": {"di": "{{inputs.parameters.di}}"}},
function="process-spire",
inputs={"di": A.get_parameter("dataset")})
C = step(template={"action":"job", "inputs": {"di": "{{inputs.parameters.di}}"}},
function="process-measures",
inputs={"di": A.get_parameter("dataset")})
# Chain the steps
A >> [B, C]
return w