Skip to content

Build, compile and serve the model

In this part we upload the model, create a TVM function and run its three actions. Wait for build and compile Runs to become COMPLETED before the next step. A serve Run remains RUNNING while the service is available.

With the CLI, each run reads its fields from a small YAML file; the commands expect the files in the current folder.

In the console, the Task step contains common settings such as resources and the run profile. The Run step contains the options for that execution: conversion flags, compile target/optimization/tuning, or serve model/image/workers. TVM options are no longer task settings; values stored in old tasks are not inherited by new runs.

1. Upload the model

Open Models and click CREATE. Choose the kind onnx, enter the name yolo and upload yolov8n.onnx at the bottom of the form.

dhcli upload model -p tvm-test -n yolo -f yolov8n.onnx
dhcli list models -p tvm-test

The runtime recognizes the format by itself: from the kind onnx of the Model, or from the .onnx extension of the file uploaded with the CLI (a Model of the generic kind model).

2. Create the function

Open Functions, click CREATE, choose the kind tvm and enter the name yolo-function. In the specification:

Field Value
model the key of the yolo Model, shown in its detail page, e.g. store://tvm-test/model/onnx/yolo:<id>
format auto

Create yolo-function.yaml:

kind: tvm
name: yolo-function
spec:
  model: store://tvm-test/model/model/yolo
  format: auto

A key without :<id> refers to the latest version of the Model. dhcli upload creates a Model of kind model, hence model/model in the key.

dhcli create function -p tvm-test -f yolo-function.yaml

3. Build: ONNX to Relax IR

build converts the model into Relax IR. YOLOv8n needs no conversion option: we only give the Job enough resources. If the imported outputs lack shapes, the build automatically retries with onnxsim; see Outputs without a shape.

Open the function, select the build tab and click CREATE. In the first step:

Field Value
resources CPU 4, memory 8Gi

Leave the other fields empty and complete the form.

Create build.yaml:

spec:
  resources:
    cpu: "4"
    mem: 8Gi
dhcli run tvm+build -p tvm-test -n yolo-function -f build.yaml
dhcli list runs -p tvm-test
dhcli log <run-id> -p tvm-test

When the run is COMPLETED, the Model yolo-function-ir, of kind tvm-ir, contains the IR, and the function field ir_model points to it.

Leave target_opset empty

Set target_opset only if the model really needs an opset conversion: ONNX cannot convert every operator to every version, and the build fails with No Adapter From Version .... All the build options are described in the runtime reference.

4. Compile for the cluster (AMD64)

compile turns the IR into model.so for a target architecture. First we compile for AMD64 (Intel/AMD 64-bit), to serve the model on the cluster.

The model destination is not the compiler node: the toolkit can compile this AMD64 model on AMD64 or ARM64. Only amd64_native requires an AMD64 compile node. The profile menu comes from CORE's runtime-tvm.yaml; administrators can add profiles without changing the SDK. Advanced runs can instead set a standard LLVM target object and compiler options, as described in the target guide.

Select the compile tab of the function and click CREATE. Set resources in the Task step, then target_architecture, tag and Advanced compilation in the Run step:

Field Value Why
target_architecture amd64 requires x86-64-v2 (amd64_v3 requires x86-64-v3, including AVX2 and FMA)
tag amd64 names the Model yolo-function-amd64
optimization.opt_level 3 TVM pass optimization level; separate from MetaSchedule tuning
resources CPU 4, memory 8Gi compiling needs memory

Create compile-amd64.yaml:

spec:
  target_architecture: amd64
  tag: amd64
  optimization:
    opt_level: 3
  resources:
    cpu: "4"
    mem: 8Gi
dhcli run tvm+compile -p tvm-test -n yolo-function -f compile-amd64.yaml
dhcli list runs -p tvm-test

When the run is COMPLETED, the Model yolo-function-amd64, of kind tvm-so, contains model.so and metadata.json, and the function field so_model points to it. If the compile node can execute the target, metadata.json contains its synthetic CPU benchmark. Otherwise benchmark.skipped explains why it was not run. These are compile-node measurements, not timings on an ARM device or over HTTP.

5. Serve

serve deploys the model in so_model as an Open Inference v2 service, on port 8080 (REST) and 9000 (gRPC).

Select the serve tab of the function and click CREATE:

Field Value Why
resources CPU 4, memory 2Gi starting point; one worker uses the same four threads as compilation

Create serve.yaml:

spec:
  resources:
    cpu: "4"
    mem: 2Gi
dhcli run tvm+serve -p tvm-test -n yolo-function -f serve.yaml
dhcli services -p tvm-test

When the service is RUNNING, dhcli services shows it:

NAME          ID                                 FUNCTION        KIND            SERVICE                                                         STATE
cloudy-lynx   7398740f73b14fdd957f10c2e54c8ed2   yolo-function   tvm+serve:run   s-tvmserve-7398740f73b14fdd957f10c2e54c8ed2.dev-platform:8080   RUNNING

The model is served with the name of the function, yolo-function. The platform starts the service on a node with the architecture of the model, here an AMD64 node. The next part shows how to test it.

6. Compile for ARM devices (optional)

The same IR can be compiled for ARM devices, such as a Raspberry Pi. The Job runs on a toolkit-capable AMD64 or ARM64 node, selected through profile. The target architecture chooses the generated code; it does not move the Job to that device. Check the device first:

uname -m          # kernel: aarch64 or armv7l
getconf LONG_BIT  # userspace bitness; ARM32 also needs ARMv7, hard-float and NEON

Console: in the compile tab click CREATE and enter:

Field Value
target_architecture arm64 (or arm64_pi5 for a Raspberry Pi 5)
tag arm64
optimization.cross_cc leave empty: the right compiler is chosen automatically
optimization.opt_level 3
resources CPU 4, memory 8Gi

CLI: create compile-arm64.yaml and start the run:

spec:
  target_architecture: arm64
  tag: arm64
  optimization:
    opt_level: 3
  resources:
    cpu: "4"
    mem: 8Gi
dhcli run tvm+compile -p tvm-test -n yolo-function -f compile-arm64.yaml

Result: the Model yolo-function-arm64.

Console: in the compile tab click CREATE and enter:

Field Value
target_architecture arm32
tag arm32
optimization.cross_cc leave empty: the right compiler is chosen automatically
optimization.opt_level 3
resources CPU 4, memory 8Gi

CLI: create compile-arm32.yaml and start the run:

spec:
  target_architecture: arm32
  tag: arm32
  optimization:
    opt_level: 3
  resources:
    cpu: "4"
    mem: 8Gi
dhcli run tvm+compile -p tvm-test -n yolo-function -f compile-arm32.yaml

Result: the Model yolo-function-arm32.

This model runs with the ARMv7 Go or Rust runtime image on a device with a compatible 32-bit OS, hard-float ABI and NEON. Serving through CORE on an ARM32 cluster node also requires an ARMv7-compatible init image; the default init image is AMD64/ARM64 only.

Serving after an ARM compile

Each compile writes its Model into so_model, and serve deploys so_model by default. After an ARM compile, so_model points to the ARM library, which the platform serves only on ARM nodes: without ARM nodes in the cluster the service stays PENDING. To serve the AMD64 build, set the serve field model_path to store://tvm-test/model/tvm-so/yolo-function-amd64:<id>. A service that is already running is not affected.

The ARM64 and ARM32 Models can run on compatible devices, as shown in Test the model. They can also be served by the platform on matching cluster nodes, provided the runtime and init images both support the node architecture.