Test a server with cases

A test case is one call to one tool and what its result must be. Cases are files in the server's repository, so they review like code, and a pull request that breaks one shows it on its own check before it merges.

Write a case

Each file in tests/ is one case, named for its file: tests/open-tasks-are-a-list.json is the case open-tasks-are-a-list.

{
  "$schema": "https://sokie.sh/schema/test.v1.json",
  "description": "Open tasks come back as a list, quickly.",
  "tool": "listTasks",
  "input": { "status": "open" },
  "expect": { "ok": true, "items": { "atMost": 50 }, "under": "1s" }
}

• tool is the tool to call, by its name.
• input is its arguments. sokie check holds them to the tool's input schema, so a case can't fail on a typo.
• expect is what the result must be. Leave it out, and the call only has to succeed.
• on says where the case runs. See where cases run.

What a case can expect

• ok: true when the tool must answer without an error, false when it must answer with one.
• match: part of the JSON result. Every key you give must be there with the same value; the result can have more. An array you give must have each of its items somewhere in the result's array.
• items: on a list, { "atMost": n }, { "atLeast": n }, or both. The list is the result itself, or the first array its object holds, like data in { "data": [...] }.
• bytes: { "atMost": n }, the size of what the agent reads.
• under: how long the call may take, like "500ms", "1s" or "2.5s".

Where cases run

on is any of dev, preview and production. Without it, a case for a tool that only reads runs everywhere, and a case for anything above read runs on sokie dev and previews only, so nothing writes to production unless its file says so.

Cases call the real API, the same one your server calls. A case above read needs a test account there.

Run them from your terminal

sokie test runs every case on the sokie dev running for this checkout. With --preview <n> it runs them on a pull request's preview, and with --production on the live server, as you, with a token that lasts five minutes.

$ sokie test --preview 44
2 cases against pr-44--majestic-lobster-123.sokie.cloud

  ✓ create-a-task          createTask    188 ms
  ✕ open-tasks-are-a-list  listTasks       6 ms
      -32602 Unknown tool: listTasks

1 passed, 1 failed. Exit code 1.

It exits with 1 when any case fails, so a script or CI job can stop on it. --json prints every case with what it expected and what it got.

After every deploy

Once a preview goes live, Sokie runs its cases as the person who linked the repository, and reports them on the pull request's Sokie check. A failing case fails the check, so branch protection can require it. A production deploy runs the cases that may run there and shows how they went on the server's Overview; a failing case never takes a deploy back.

On a server whose API takes each caller's own key or sign-in, a case runs only when that person has set theirs. Otherwise it's skipped, and says why.

The server's Test tab has a Cases view: each case with its latest preview and production result, what a failing one expected beside what it got, and recent runs. Run again runs the cases now, as you.