TypeScript tests

A *.test.ts file holds one or more tests. Each test is an async function: plain-English steps run through ai, and anything else is ordinary TypeScript: Playwright calls on page, expect assertions, API requests, test data from any library, loops, and helper functions.

// tests/checkout.test.ts
import { faker } from "@faker-js/faker";
import { test, expect, secret } from "sedum-cli";

test(
  "a new customer checks out",
  { url: "/", tags: ["checkout"] },
  async ({ page, ai, env }) => {
    await ai.group(
      "Log in",
      [
        "type {{user}} in the username field",
        "type {{password}} in the password field",
        "click the login button",
      ],
      { user: "standard_user", password: secret(env.SAUCE_PASSWORD!) },
    );

    await ai("click the Add to cart button for {{product}}", {
      product: "Sauce Labs Backpack",
    });
    await expect(page.locator(".shopping_cart_badge")).toHaveText("1");

    await ai("click the shopping cart link");
    await ai("click the Checkout button");
    await ai("type {{first}} in the First Name field", {
      first: faker.person.firstName(),
    });
    await ai("type {{last}} in the Last Name field", {
      last: faker.person.lastName(),
    });
    await ai("type {{zip}} in the Zip/Postal Code field", {
      zip: faker.location.zipCode(),
    });
    await ai("click the Continue button");

    const total = await ai.extract("the order total");
    expect(total).toContain("$");
    await ai("click the Finish button");
    await ai("verify the order was placed and a confirmation message is shown");
  },
);

npx sedum run                                  # every test under tests/
npx sedum run tests/checkout.test.ts           # every test in one file
npx sedum run --id "tests/checkout.test.ts#a new customer checks out"   # one test

Install sedum-cli in the project (npm install -D sedum-cli) so your editor has its types. Sedum compiles the file itself with jiti; you need no build step or tsconfig.json to run tests. Types are not checked at run time; run tsc --noEmit for that.

Declaring tests

test(title, body);
test(title, options, body);

Option	Meaning
`url`	Entry URL, absolute or relative to `baseUrl`. Omitted: `baseUrl`, or a blank page without one.
`id`	Stable identity. Default: `<file>#<title>`, such as `tests/checkout.test.ts#a new customer checks out`.
`tags`	Labels for `sedum run --labels`.

Titles must be unique within a file and ids unique in the project. Each test gets a fresh browser context and runs on its own, in parallel lanes and with --retries like a YAML test unless it enters a goal (see below). Top-level code in the file runs once when Sedum imports it, so keep per-test setup inside the body. The body receives:

Name	What it is
`ai`	Runs plain-English steps; see below.
`page`	The Playwright `Page` the steps run on.
`context`	Its `BrowserContext`, for cookies, storage, and routes.
`env`	Configured `variables`, the project `.env`, and the process environment, as `sedum run` resolves them.
`testInfo`	`{ id, title, file, tags, attempt }`.

expect is Playwright’s, with its auto-retrying matchers such as toHaveText and toHaveURL, and the usual toBe and toEqual.

Steps with `ai`

await ai("click the login button");
await ai("type {{email}} into the Email field", { email });
await ai(["click the Cart link", "click the Checkout button"]);

A sentence is one step, written exactly as in a YAML test and classified and resolved the same way: click, type, press, goto, scroll, wait, verify, measure, and remember (see step classification). A list runs its sentences in order, sharing one values object.

Goals and separate verification

ai.goal(goal, values?, options?) asks the planner to complete a bounded task on the current page:

await ai.goal("Create a new account and complete its profile");
await ai('verify the page displays the confirmation message "Profile saved"');
await expect(page.getByRole("status")).toHaveText("Profile saved");

The promise resolves when the planner reports DONE and that decision passes the normal confidence and freshness checks. This means planner-reported completion, not independent verification: ai.goal does not call the Judge or add a hidden assertion. A goal-only test can pass and is reported as completed, not verified. Add a separate ai("verify ...") step for a semantic check, a Playwright expect for an exact check, or both. A later check proves only what it asserts; it does not retroactively verify every part of the goal. There is no ai.act, ai.assert, or ai.verify API. Goals, authored ai steps, Playwright calls, and assertions can be interleaved on the same page. Each goal starts with fresh planning history, counters, references, and generated-value memory, while observing the page left by the preceding code:

const customer = {
  email: faker.internet.exampleEmail(),
  firstName: faker.person.firstName(),
};

await ai.goal("Create an account for {{email}}", customer);
await expect(page).toHaveURL(/\/profile/);

await page.getByLabel("Marketing emails").uncheck();
await ai.goal(
  "Complete the profile for {{email}} using {{firstName}}",
  customer,
);
await ai('verify the page displays the confirmation message "Profile saved"');
await expect(page.getByRole("status")).toHaveText("Profile saved");

The optional values argument follows the same rules as ai(sentence, values). Every explicit {{placeholder}} must have a supplied binding; a missing one is an authoring error. With the TypeSafe provider, a goal may also choose a local, allowlisted Faker generator for a fill that has no applicable binding. Faker runs locally for each invocation. Automatically generated values can be reused within that goal (for example, in a confirmation field), but are not shared with another goal. To reuse an identity, generate it in TypeScript as above and pass it explicitly to every goal; wrap sensitive values with secret(). Automatic generation defaults to enabled. Set generateData: false in the third argument to restrict fills to supplied or remembered values:

await ai.goal("Complete the profile for {{email}}", customer, {
  generateData: false,
});
// With no values, allow goals that do not need generated input.
await ai.goal("Open billing settings", undefined, { generateData: false });

This setting applies only to that invocation. It disables automatic Faker generation, not Faker calls you make in TypeScript. Missing data cannot be generated; a blocked goal rejects. Completion still depends on the planner, so keep independent checks for the outcome you need. A goal uses the unchanged defaults of 24 requests, 18 dispatched actions, and 120 seconds. Value-selection requests count toward the same request budget. BLOCKED, abstention, no progress, budget exhaustion, cancellation, and action errors reject the goal. Once a valid goal enters planning, Sedum does not automatically retry the whole test, even if the goal succeeds and a later expect, verify step, or cleanup fails. This prevents replaying side effects; manual reruns are still possible. Tests that never enter a goal retain normal --retries behavior. Goal actions are limited to the operations offered by goal mode (currently click and type); use deterministic Playwright or authored ai steps for other work. Goals are still exclusive, tracked ai operations: always await them, and do not overlap them with another ai call. Multiple goals are sequential, not parallel. Pass values separately, not with ${}. Write {{name}} in the sentence and give the value in the second argument. The sentence text stays the same on every run, so its classification is cached and reused, and sedum validate can check it before a run. A sentence built with ${} or + is classified again for every distinct value, and validate warns about it. Values can be strings, numbers, and booleans. Before Sedum locates a target or judges a claim, it puts plain values into the sentence, so click the Add to cart button for {{product}} is resolved as the named product. The value of a type step is typed, never shown to the model. Secrets. Wrap a sensitive value in secret(). It is typed into the page, but it stays a {{placeholder}} in model requests and is redacted from report text and result.json. Printing a secret shows [secret]. Sedum learns a secret when a step first passes it, so page text judged by an earlier step can still contain it; screenshots are never redacted. List origins that show secrets with --sensitive-origin.

await ai("type {{password}} into the Password field", {
  password: secret(env.SHOP_PASSWORD!),
});

remember <target> as {{name}} stores page text for later steps in the same test, as in YAML. Always await. Steps run one at a time on one page. A step started while another is still running stops the test with an error that names the line.

Groups

ai.group names a block of steps. Every step inside it is reported under the name, such as Checkout › click the Continue button, and a failure names the group it happened in. Groups can nest.

await ai.group(
  "Log in",
  ["type {{user}} in the username field", "click the login button"],
  { user },
);

await ai.group("Checkout", async () => {
  await ai("click the Checkout button");
  await page.getByLabel("First Name").fill(first); // code runs inside a group too
  await ai("click the Continue button");
});

Reading from the page

ai.holds(claim, values?) asks whether a claim holds on the page now, so a test can branch when a screen appears only sometimes. It returns true or false, never fails the test just because the claim is false, and is recorded without a pass/fail verdict. Unlike a verify step, it does not use the configured verify grace period.

if (await ai.holds("the passkey enrollment screen is shown")) {
  await ai("click the Skip button");
}

Pass a claim, not an action or a wait instruction. Claims are classified and judged through the same deterministic and model-backed paths as verify steps at run time. Offline validation accepts readable single-clause claim forms such as the dialog is shown, the prompt appears, or the total matches {{total}} without classification. It rejects malformed placeholders, actions, waits, ambiguous sentence shapes, and call shapes it cannot inspect. ai.extract(description) returns the text of the element the description names, found the same way as a remember target. Pass a parser, such as a zod schema, to convert it:

const text = await ai.extract("the order total"); // "Total: $32.39"
const total = await ai.extract("the order total", {
  parse: (value) => Number(String(value).replace(/[^0-9.]/g, "")),
});

Extracted text is treated as page data: it is redacted from reports, like a remembered value.

Code between steps

Anything Playwright or Node can do works between steps. Use it where a sentence is the wrong tool:

Skip slow setup. Sign in through an API, or set a session cookie with context.addCookies, then page.goto the page under test.
Seed and inspect state. Call your backend, or read localStorage with page.evaluate, and check it with expect.
Exact checks. A count, a URL, or an element that must be absent is often clearer as a locator assertion than as a claim, for example await expect(page.locator(".cart_item")).toHaveCount(0).
Generate data. Use any library, such as @faker-js/faker, and pass the values to ai.
Share steps. A module is a function: export async function login(ai, user) { ... }. A helper can also wrap test() itself, for example to add shared tags; the test belongs to the file that calls the helper.

The page settles before each ai step, so a step after code sees the page that code left.

Results

A failed step stops the test. A failed verify, or a target that cannot be found, is recorded as a failed step, and the rest of the body does not run. The test fails even if your code catches the error.
An exception from your code fails the test. A failed expect or any thrown error is recorded as a failed step with operation: "code", at the file and line that threw, under its group. This includes a promise the test never awaits, such as an unawaited page.click or expect, and an ai.group or ai.extract whose code throws while nobody awaits it. To let a block fail on purpose, handle it: await ai.group(...).catch(() => {}).
A promise left behind fails the run, not another test. Sedum blames a test for an unhandled rejection only when the error’s stack points inside that test’s lines while it runs. A promise that rejects after its test finished, in a helper any test could have called, in lines that several tests declared in a loop share, or in a file’s top-level code, is reported for the whole run with its file and line, and the command exits 3. If it rejects after the report is written, Sedum prints it and still exits 3.
Misusing ai is an invalid test. A missing value, an invalid values object, or a missing await stops the run as an invalid test, like a YAML file with an error, with the file, line, and a fix.

Supplied secret and generated goal values join the attempt-wide redaction history, so later goals, verification, diagnostics, reports, and cleanup do not echo known values. They do not become fill choices for later goals. Redaction cannot mask screenshots, video, arbitrary application logging, or values Sedum never learned; use disposable accounts and the existing evidence controls for sensitive pages. Reports name a TypeScript test by its file and title, and rerun commands add --id to select just that test.

Validation and listing

sedum list imports each *.test.ts file and lists one row per test(). sedum validate also finds the sentences passed to ai(...), ai([...]), ai.goal(...), and ai.group(name, [...]), in the test file and in the local modules it imports, and checks them offline, or online with --online, exactly like YAML steps. A list or sentence held in a const in the same file is read too, when the file declares that name once and never reassigns it. Validation recognizes the ai name. For ai.goal, validation checks literal and supported const/local-helper shapes, placeholder syntax, and statically readable argument shapes. Required bindings and planner availability are checked at run time. Validation does not execute the body, generate Faker data, ask a model whether the goal is feasible, or classify the goal as one authored step. Dynamic expressions and aliases it cannot resolve produce the same honest validation warnings as other ai calls; runtime validation remains authoritative. An argument validation cannot read, such as a variable, a function call, or a sentence built with ${}, is reported as a warning. So is any way of running steps that validation cannot follow: ai aliased or renamed, ai read off the context (ctx.ai, t["ai"]), or ai, { ai }, or the test’s context handed to a helper outside the project or to a helper name the file declares more than once. Pass ai to a helper in the test file or a local module as a parameter named ai, such as login(ai, user); validation follows the call and checks the helper’s steps. A list passed to ai.group must be a literal, a const in the same file, or a function body. When it warns, validate does not call the project fully valid and exits 1, as for a YAML sentence it could not check offline. Importing a file runs its top-level code. Keep network calls and other side effects inside test bodies.

Limits

validate reads your code without running it. It vouches for the sentences it can see in the shapes above and warns about the rest; a sentence it cannot see is still checked when its step runs, and an invalid one stops that test with the file and line.
There are no beforeEach or afterEach hooks yet; use a helper function and try/finally for cleanup.
import ... from "sedum-cli" always resolves to the CLI that runs the test, whichever version is installed in the project.
A *.test.ts file cannot use YAML modules, and a YAML test cannot call TypeScript.

Get started

Use Sedum

How it works

Providers

Reference

TypeScript tests

Declaring tests

Steps with `ai`

Goals and separate verification

Groups

Reading from the page

Code between steps

Results

Validation and listing

Limits

​Declaring tests

​Steps with ai

​Goals and separate verification

​Groups

​Reading from the page

​Code between steps

​Results

​Validation and listing

​Limits

Declaring tests

Steps with `ai`

Goals and separate verification

Groups

Reading from the page

Code between steps

Results

Validation and listing

Limits