Skip to content
Anvil Anvil

Testbed Runtime API

The github.com/TDB-Group/anvil/testbed package contains the protocol-neutral runtime model for generated testbeds.

The compiler creates a suite. Anvil can run HTTP and GraphQL cases against an already-running server. Driver packages can plug in WebSocket, gRPC, queue, or broker-specific runners without importing the compiler.

Suite And Report Schemas

Section titled “Suite And Report Schemas”
const SchemaV1 = "anvil.testbed.v1"
const ReportSchemaV1 = "anvil.testbed.report.v1"

Suite is the generated plan:

type Suite struct {
  Schema  string
  HTTP    HTTPSuite
  GRPC    GRPCSuite
  GraphQL GraphQLSuite
  Queue   QueueSuite
}

Report is the execution result:

type Report struct {
  Schema  string
  Summary ReportSummary
  HTTP    []HTTPRunResult
  GRPC    []GRPCRunResult
  GraphQL []GraphQLRunResult
  Queue   []QueueRunResult
}

MarshalSuiteJSON and MarshalReportJSON write indented JSON. MergeReports combines reports from several protocol runners and adds their summaries.

HTTP And GraphQL Runner

Section titled “HTTP And GraphQL Runner”
type RunOptions struct {
  BaseURL          string
  Client           *http.Client
  IncludeGenerated bool
  MaxResponseBytes int64
  WebSocketRunner  WebSocketRunner
}


func Run(ctx context.Context, suite Suite, opts RunOptions) (Report, error)
func RunHTTP(ctx context.Context, suite Suite, opts RunOptions) (Report, error)

Run executes HTTP route cases first, then GraphQL endpoint cases, against RunOptions.BaseURL. The base URL must include a scheme and host. If the base URL contains a path, that path is preserved as a prefix for every generated case URL.

When Client is nil, Anvil uses an http.Client with a 10s timeout.

Generated cases and script-loaded cases normalize empty bodies to nil. A nil body produces an empty request body and Anvil does not add a Content-Type header.

When the runtime receives a non-nil body map, it sets Content-Type: application/json. Empty maps send an empty body. Non-empty maps are JSON-encoded. Header values in the case are converted to strings and then written onto the request, so explicit script headers can override the generated content type.

Response bodies are bounded:

  • 0: default limit, 1 MiB.
  • Negative value: configuration error.
  • Above 1 GiB: configuration error.

IncludeGenerated controls which HTTP route cases run:

  • false: run scripted cases when a route has any; otherwise run generated cases.
  • true: run every case on the route.

RunHTTP clears GraphQL endpoints before calling Run. It still sees HTTP routes marked as WebSocket routes; those cases are delegated to WebSocketRunner when one is configured, or reported as skipped when no runner is configured.

For HTTP expectations, BodyContains and StreamContains are both checked against the captured response body. Streaming cases therefore work for finite-response assertions; long-lived streams need a driver-owned runner.

For GraphQL expectations, the built-in runner checks expected status and bodyContains. Normal GraphQL operations are sent as HTTP POST JSON. Subscription cases are sent as HTTP GET with query, optional operationName, optional JSON-encoded variables, and Accept: text/event-stream.

The built-in runner captures response bodies as strings and performs substring assertions. It does not decode JSON or GraphQL error arrays for assertions.

The built-in runner performs requests sequentially. Use a driver-owned runner or an external load tool when a test needs parallel execution, connection churn, or protocol-specific streaming assertions.

WebSocket Runner Extension

Section titled “WebSocket Runner Extension”
type WebSocketRunner interface {
  RunWebSocket(
    ctx context.Context,
    route HTTPRouteSuite,
    item HTTPCase,
    result HTTPRunResult,
  ) HTTPRunResult
}

Anvil does not import a WebSocket client. When an HTTP route is marked as a WebSocket and no WebSocketRunner is provided, the case is reported as skipped with the error websocket cases require a websocket runner.

The HTTP driver packages provide runner integrations for their socket implementations.

Protocol Runner Composition

Section titled “Protocol Runner Composition”
type Runner interface {
  Name() string
  RunTestbed(context.Context, Suite) (Report, error)
}


func RunnerFunc(
  name string,
  run func(context.Context, Suite) (Report, error),
) (Runner, error)


type Simulator struct {
  // unexported runner list
}


func NewSimulator(runners ...Runner) Simulator
func HTTPRunner(options RunOptions) (Runner, error)
func (s Simulator) Run(ctx context.Context, suite Suite) (Report, error)

Simulator runs every registered runner and merges their reports. Nil runners and runners with empty names are ignored. Running a simulator with no runners returns an error.

RunnerFunc rejects empty names and nil functions.

Simulator execution is sequential. It does not start protocol runners in parallel and it returns the merged report together with any joined runner errors.

Queue Broker Runner

Section titled “Queue Broker Runner”

Broker-backed queue testbeds use a small adapter:

type QueueBroker interface {
  Publish(ctx context.Context, job QueueJobSuite, item QueueCase) error
  Await(ctx context.Context, job QueueJobSuite, item QueueCase) (QueueBrokerResult, error)
}


type QueueBrokerOptions struct {
  Timeout time.Duration
}


func RunQueueBroker(
  ctx context.Context,
  suite Suite,
  broker QueueBroker,
  options QueueBrokerOptions,
) (Report, error)

For each queue case, RunQueueBroker calls Publish, then Await, then compares the observed handler error with QueueExpectation.Error.

When Timeout is positive, each queue case gets its own timeout. When it is zero or negative, Anvil uses a cancellable context without a deadline.

RunQueueBroker rejects a nil broker.

Case Shapes

Section titled “Case Shapes”

HTTP route cases use:

type HTTPCase struct {
  Name   string
  Input  HTTPInput
  Expect HTTPExpectation
  Script string
  Source *ScriptSource
}

GraphQL cases use GraphQLInput with query, operationName, variables, and subscription.

gRPC cases are JSON-shaped:

type GRPCInput struct {
  Message  map[string]any
  Messages []map[string]any
}

Queue cases use string bodies and string headers so broker runners can map them to bytes and metadata.

Script Loading

Section titled “Script Loading”
func LoadScript(file string) (ScriptFile, error)
func LoadScripts(files []string) ([]ScriptFile, error)
func ApplyScripts(suite Suite, scripts []ScriptFile) (Suite, error)

Scripts are YAML files. ApplyScripts validates each referenced case before it is appended to the suite:

  • HTTP and WebSocket cases reference route operation.
  • GraphQL cases reference endpoint name or path.
  • Cases for gRPC reference service and method.
  • Queue cases reference queue and job.

Invalid references are returned as errors before any network execution starts.

Additional validation:

  • HTTP script params must name params that exist on the route.
  • HTTP script query keys, request header keys, locals, and body keys are not rejected by the script validator.
  • HTTP expected status, when set, must be 100..599.
  • HTTP expected header names cannot be empty.
  • bodyContains and streamContains entries cannot be empty strings.
  • WebSocket message types must be text, binary, close, ping, or pong.
  • WebSocket expected close code defaults to 1000.
  • GraphQL scripts require query; expected status defaults to 200.
  • Expected gRPC code defaults to OK.
  • Expected gRPC code cannot contain leading or trailing whitespace.
  • Queue scripts must include both queue and job.

Those WebSocket message names are the protocol-neutral script vocabulary. Driver-owned runners still define the exact frame behavior they support. The official HTTP runners send text, binary, ping, and close input messages, assert text and binary output messages, and compare the close code when it is non-zero in the suite.