Skip to content

Gateway Extension

Gateway Extension allows for attaching a given service to an instance of Envoy Gateway and specify some extra properties that control the way the service is exposed as well as some extra functionality. More specifically, the platform currently allows for distinguishing:

  • GenAI services and expose them with Envoy AI Gateway. In this way, some extra functionality may be exploited, including dedicated observability metrics collection and tracing, data monitoring, guardrails, etc.
  • Generic services and expose them with Envoy Gateway with the additional capabilities of generic observability metrics collection and tracing, data monitoring, guardrails, etc.

In the current version, support for Gateway functionality is limited and only includes the possibility to attach the guardrails to the service instance.

Guardrails

Guardrails allow for controlling and adjusting interaction with a service, for example blocking or changing the input if inappropriate or incomplete, cleaning the output, or responding with some error or default message if necessary. Typically, guardrails are attached to GenAI services, but the pattern may be applied to other services as well.

In the current implementation, guardrails should be implemented as ExtProc Envoy extension services, using e.g. Guardrail runtime.

To enable guardrails for the service, it is necessary to add to the service run a list of guardrail endpoints (one for each guardrail), defined as host:port in the same platform context. It is expected that the ExtProc-compatible gRPC port is exposed at the specified address.