Runtimes
Functions are the logical description of something that the platform may execute and track for you. A function may represent code to run as a job, an ML model inference to be used as batch procedure or as a service, a data validation, etc.
In the platform we perform actions over functions (also referred to as "tasks"), such as job execution, deploy, container image build. A single action execution is called run, and the platform keeps track of these runs, with metadata about function version, operation parameters, and runtime parameters for a single execution.
They are associated with a given runtime, which implements the actual execution and determines which actions are available. Examples are DBT, Container, Python, etc. Runtimes are highly specialized components that translate the representation of a given execution (as expressed in the run) into an actual operation performed via libraries, code, or external tools.
Runtimes define the key point of extension of the platform: new runtimes may be added to implement the low-level logic of "translating" the high-level operation definition into an executable run. For example, the DBT runtime allows defining a transformation as a task that, given an input table reference, produces a dataset applying the SQL-defined function. In this case the runtime converts the specification and references into a Kubernetes Job that runs the DBT transformation and stores the resulting dataset.
Supported runtimes
-
Run Python functions as jobs or services.
-
Run arbitrary container images as jobs or services.
-
Run model serving and inference workloads.
-
Build Hera pipelines for DAG and step orchestration.
-
Run DBT transformations for data modeling.
-
Build federated learning workloads with the Flower framework.
-
Run Guardrails AI validation functions as jobs or services.
-
Run OpenInference instrumentation functions as jobs or services.
-
Run service pipeline orchestration services.
-
Build, compile, and serve models with Apache TVM.
-
Run Hydra Python functions as jobs.
-
Run Ray Python functions as jobs.