Cloud result reads
auth login session, live /me memberships,
and saved organization/project choices. They never launch login or prompt.
All accept --url, --organization, --project (ID or unambiguous name), and
--json. Explicit selections do not change saved defaults. Ambiguity requires
explicit flags. Credentials refresh under the same exclusive store lock as auth;
GET requests retry once after a 401 with refreshed credentials.
runs list returns compact recent summaries. runs show takes the Cloud run
UUID, not a report’s local run ID, and requests no full report unless
--report is present. An automatic upload prints the Cloud run UUID
(Cloud upload: accepted run <cloud-run-uuid> (local run <run-id>)); in
runs list --json it is id, and the local run ID is run_id. Passing a local
run ID to runs show reports that the run was not found.
tests history --test-id matches the exact test identity across branches, not
a substring; use --branch to scope it.
List, history and stats share these filters:
--state and --verdict filter the run, not an individual test. Inspect
the test’s evidence for its outcome; a failed run may contain passing tests.
Relative --since is anchored to --until, not to the clock independently.
Windows must be positive and at most 31 days. List/history accept --cursor
and fetch exactly one page; reuse next_cursor with the same explicit window
and filters to continue. There is no automatic page draining.
Ranked test statistics
tests stats optionally accepts --test-id for an exact identity, and
--min-observations (integer 1..1000, default 1). --sort is one of:
flaky-runs(default): recovered-retry observation count.flake-rate: recovered-retry rate.failures: failed observation count.retry-overhead: total recorded nonselected-attempt elapsed time.duration-p95: selected-attempt duration p95.duration-regression: p95 increase, requiring--compare previous.
mixed_outcomes
records both passed and failed observations. p50/p95 use nearest-rank percentiles
of recorded selected-attempt durations. Retry time sums all nonselected attempts;
missing durations are not invented. --compare previous evaluates the immediately
preceding equal-length window with the same filters. Only current-window tests
are ranked. Each row includes previous (or null), p95_change_ms and
p95_change_ratio; missing durations yield null, as does a zero-baseline ratio.
Stats return complete rankings over at most 1000 runs and 5000 test identities
per window, then the top limit; a 422 means narrow time or branch, not a
partial result. Output includes the window, run/test sample counts and, when
comparing, previous-window coverage counts. JSON uses schemaVersion: 1,
origin, organization, project, plus the API result fields; report/test
strings are never rewritten as invocation hints. No auth status text accompanies
JSON. Successful response shape is validated before printing.
Read failures exit 3. A 401 suggests login; 403/404 mean the project/run is
unavailable, 400 means invalid filters, 422 asks for a narrower query, and 429
asks for backoff. Arbitrary server error bodies and credentials are never printed.
sedum auth login, status, and logout
Connect your Sedum account through a browser, including on self-hosted instances:
--no-browser over SSH, or press Ctrl-C
to cancel. No GitHub secret is handled by the CLI. Instance URLs require HTTPS
except on loopback. Choose the instance with --url, SEDUM_CLOUD_URL, or the
last selected origin, in that order.
Your sole organization/project is selected automatically; otherwise choose in an
interactive terminal or pass --organization and --project with current IDs or
unambiguous names. status checks live memberships and renews expired access.
An account with no organization/project can still log in; create them in Cloud
and rerun status. Login does not grant new roles or configure uploads.
Sessions use rotating refresh credentials, expire after at most 30 days, and are
revoked by online logout. Offline logout removes local credentials but cannot
confirm remote revocation. CI and unattended agents should continue using
separate project upload/read credentials, not your personal login.
Credentials are saved outside the checkout in $XDG_CONFIG_HOME/sedum/auth.json
or ~/.config/sedum/auth.json (%APPDATA%/sedum/auth.json on Windows), separated
by instance origin. This is a plaintext file fallback, not an OS keychain: POSIX
files/directories require owner-only permissions, and Windows relies on inherited
user-directory ACLs. Keep this directory private and out of Git. Commands lock
the store during authentication/refresh; after a crash, remove auth.lock only
after confirming no other auth command is running.
sedum init
Run sedum init in the directory you want to turn into a Sedum project. It
creates sedum.config.yaml, tests/example.test.ts, .env.example, and
.gitignore entries for .env, .sedum/runs/, .sedum/reports/, and
.sedum/explore/. The
classification cache at .sedum/classifications.json remains trackable.
The generated test signs in to the public SauceDemo sample store at
https://www.saucedemo.com/ with standard_user and the public demo password
secret_sauce. .env.example supplies SAUCE_PASSWORD; the user must supply
their own provider key. Set TYPESAFE_API_KEY to use TypeSafe, or set
TYPESAFE_BASE_URL alongside the compatible provider’s TYPESAFE_API_KEY
(details). Running the test calls
the configured provider and may incur a charge. The generated config selects
Chromium. If no matching browser is installed, init prints
sedum browsers install chromium; on Linux,
sedum browsers install chromium --with-deps also installs system libraries.
The command reports a missing key, previews the TypeScript test it creates, and
prints a headed run command. It does not contact the provider or install a
browser itself.
On a fresh project, init asks no questions. Existing scaffold files are kept
as they are, including on repeated runs. An existing .gitignore is amended
only after confirmation in an interactive terminal. In a noninteractive shell,
the command prints any missing rules to add manually. A symlink or nonregular
file where a scaffold file belongs is an error. The Sedum wordmark appears
only in an interactive color terminal; redirected and NO_COLOR output is plain.
If .env.example already exists, init keeps it and calls out the public
SAUCE_PASSWORD=secret_sauce value needed by the generated demo when that
variable is not already available.
sedum doctor
sedum doctor checks whether this project can run Sedum before starting a test. It reports each prerequisite as PASS or FAIL, with a fix for every failure: Node 20.19 or newer, valid config and .env, an installed browser, network and authenticated access to the configured model provider, and output directory write access. The authentication check sends one small request and may incur a provider charge. Doctor does not launch a browser or run tests.
--env <name> and --browser chrome|chromium check the environment and browser a run with the same options would use; the browser check says where the browser setting came from (see configuration).
sedum doctor --json writes one versioned JSON object, including when checks fail. Both output forms omit the API key and raw provider errors. The exit code is 0 when every check passes and 3 otherwise.
sedum browsers install
Sedum uses playwright-core and does not download a browser during a test run. Install the matching managed Chromium binary explicitly:
PATH is not used as a fallback.
sedum run
sedum run [paths...] runs the configured test directory, or the named test files and directories. Paths are relative to the project root. Directories are searched recursively; symlinked test files and directories are skipped or rejected. Explicit paths form the candidate set, while a run with no paths uses tests.directory, tests.include, and tests.exclude from configuration.
Filters apply after discovery: repeat --include <glob> or --exclude <glob> for project-relative paths, use --labels smoke,auth to require both tags, and repeat --name <text> to match any case-insensitive substring of a test id or description. A TypeScript test’s description is its title. Repeat --id <id> to select tests by their exact id, such as tests/checkout.test.ts#a new customer checks out; rerun commands in reports use it. Excludes win. Bad test files are named in the result and valid files still run; the command exits 3 because the suite was incomplete. Likewise, a test that cannot run, for example because one of its steps cannot be completed, is recorded as an error on that test and the remaining tests still run; the command exits 3. Only a problem no later test can avoid stops the run early: a missing browser, a rejected provider key or configuration, or sustained provider rate limiting. A selection with no valid tests also exits 3.
--retries <n> adds up to n whole-test attempts after a failed attempt. Each attempt starts a fresh browser context and repeats its before, steps, and after phases. The JSON result keeps every attempt; the terminal summary shows the outcome sequence. Model usage and cost include all attempts, while final pass/fail counts use the last attempt. A test that passes only after a failed attempt is flagged flaky: it still counts as passed, but it appears in the flags summary, and --strict exits 2 for it as for any flagged pass. --timeout-minutes <minutes> sets a run deadline; expiration records run_timeout, preserves partial results, and exits 3. It requests cancellation of the current browser or provider operation before finalizing. If an external operation does not unwind within ten seconds, the executable exits 3 and the last atomic progress.json may be the only available result.
--non-blocking-labels <tags> quarantines tests without skipping them. A test with any of the comma-separated tags (repeatable) runs as normal, with the same retries, but a failure or a test-scoped error does not change the run result: the run is passed when every blocking test passes, and the exit code follows. A problem that stops the whole run, such as a rejected provider key, still exits 3, and so does a non-blocking test that cannot start at all, such as one with an entry URL that cannot be resolved. result.json marks each such test "blocking": false, totals.failedTests counts blocking failures only, and totals.nonBlockingFailedTests counts the non-blocking tests that failed or could not run. The terminal summary adds N non-blocking failed, the Markdown and HTML reports list these tests under Non-blocking failures, and JUnit records each one as skipped with the failure detail, so a CI step that reads the JUnit file agrees with the exit code. Flags on non-blocking tests, such as flaky, do not count toward --strict. The option combines with --labels, --affected and sharding. For example, sedum run --retries 3 --non-blocking-labels quarantine with tags: ["clients", "quarantine"] in a test’s options.
Automatic Cloud upload
SetSEDUM_CLOUD_URL, SEDUM_PROJECT_ID and SEDUM_UPLOAD_TOKEN in your CI
environment or project-root .env to upload terminal reports automatically
after sedum run. No separate upload command is needed. Local live progress
reporting is unchanged, and delivery failures warn without changing the suite’s
exit code. Each warning names the setting at fault, such as a malformed
SEDUM_UPLOAD_TOKEN, a non-HTTPS or unreachable SEDUM_CLOUD_URL, or an HTTP
401 for a token that is revoked or belongs to another SEDUM_PROJECT_ID.
Use --no-cloud to opt out for an invocation. See
Cloud configuration and CI context for
branch detection, credential handling and delivery limits.
Experimental Git-diff selection
CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_AUTH_TOKEN (or its
CLOUDFLARE_API_TOKEN alias); for TypeSafe, they are the TYPESAFE_* settings.
For OpenRouter, set provider.name: openrouter, choose a compatible decision
model with provider.model, and set OPENROUTER_API_KEY (or OPEN_ROUTER_API_KEY).
Selection makes additional billable calls.
The default base is the first available ref in this order: local main,
origin/main, local master, origin/master. --base accepts another ref.
Base and HEAD are resolved once to immutable commits. The comparison is
merge-base(base, pinned HEAD)..pinned HEAD: committed branch work only, not
base..HEAD or the working tree. No fetch or merge happens automatically.
Any staged or unstaged tracked change anywhere in the repository, including
submodule dirtiness, is refused even if an affected-ignore pattern matches it.
Commit intended changes before selection. Tracked entries with assume-unchanged
or skip-worktree flags are also refused, even when clean or ignored, because
those flags can hide local edits from Git status. Clear them deliberately with
git update-index --no-assume-unchanged --no-skip-worktree -- <path>, or run
without --affected; Sedum never changes the index for you. Sparse checkouts
with skip-worktree entries are therefore not supported for affected selection.
Nonignored untracked files are also
refused; commit intended files or ignore unrelated files. Shallow clones need
enough history to find a merge base. Sources are checked against the pinned
revision, and HEAD and cleanliness are rechecked before provider dispatch to
detect ordinary concurrent edits; this is not an atomic snapshot guarantee.
affected.ignore and repeatable --affected-ignore <glob> combine additively.
The default list is empty: docs, config, and lockfiles are not automatically
excluded. These are repository-root-relative POSIX globs, including dotfiles;
** crosses directories and a trailing / matches all descendants. *.lock
matches root lockfiles; **/*.lock also matches nested ones. Empty or invalid
patterns, and --affected-ignore without --affected, are errors. Paths are
filtered before retained patches are read, not merely before scoring.
For an umbrella PR, deliberately exclude generated fixtures and documentation
only after considering their runtime dependencies:
.env or **/.env* is a deliberate exclusion, not secret scanning: other
retained patches and complete candidate sources can still contain secrets.
Each discovered test becomes one independent binary question: could the
changed code affect behavior exercised by this test? Its question contains the
complete source and all referenced module sources; the diff is shared state. Normal
path/name/tag filters apply before relevance selection.
TypeScript selection remains file-level, not individual-test selection.
Large suites are batched without truncating inputs. Forced tests receive score
1 and reason test-or-module-changed; their sources need not be sent for scoring.
--affected rejects --shard-count greater than 1 with exit 3 before any
provider call, including with --selection-only. Independent relevance decisions
in different shards could leave tests unexecuted. Remove the shard options or
run without --affected. A single shard (--shard-index 1 --shard-count 1)
and parallel execution with --parallel remain supported.
The provisional default keeps scores ≥ 0.3. This is a selection heuristic,
not an empirically calibrated cutoff; use a lower threshold to favor recall.
--threshold accepts 0–1, with 0 keeping every candidate for a nonempty diff.
A Noul near 0 means likely unrelated, near 1 means likely relevant, and near
0.5 means uncertain. Each non-forced candidate is scored against every diff
chunk, even after an earlier score crosses the cutoff. The final score is the
maximum chunk score, with the threshold applied once. This heuristic is not a
calibrated whole-PR relevance probability or a probability that the test will fail.
Higher cutoffs trade recall for fewer selected files and can skip relevant tests.
For example, 0.4 is a tuning option, not a calibrated global recommendation.
Evaluate any threshold against representative changes before relying on it;
the default is 0.3. Pass --threshold 0.1 to use the previous default.
The normal command prints each probability and RUN/SKIP decision to stderr.
--selection-only instead writes JSON to stdout, including the base, merge-base
commit, pinned head, changedFiles, ignoredFiles, threshold, per-test decisions,
score/chunk metadata, and all provider receipts with model versions,
token usage, and cost estimates. These selection costs are separate from the
runner’s execution cost totals. Save preview output outside the repository or
under an ignored directory to avoid introducing an untracked file.
No diff means no API call and no test execution. When all paths are ignored,
there is no provider call; forced tests still run, otherwise nothing runs.
An empty relevance selection
also executes nothing; both exit 0 with explicit output and create no run result.
Invalid tests/modules, Git errors, retained binary/submodule changes, oversized inputs,
or provider failures stop selection with exit 3, never silently becoming a
successful empty selection. Ignored binary/submodule changes need no patches.
The total retained UTF-8 patch ceiling is 16 MiB, a provisional local memory
guard, not a provider context limit; exceeding it stops before provider creation.
Lossless chunks prefer complete files, then hunks, then lines; oversized lines
split at Unicode code points without breaking surrogate pairs. Each piece
carries file/hunk metadata and continuation markers. Deletions and patch payload
are preserved, with only explicit context repeated. Serialized requests include
all JSON escaping, instructions, metadata, and complete test/module sources:
the ceilings are 28,000 UTF-8 bytes for a diff plus one question and 56,000
bytes per batch. Clef also limits each request to 64 questions.
All requests are preflighted before transmission. A complete test/module input
that cannot fit, or more than 256 planned provider requests, fails before
the first call. The latter is a provisional cost guard, not a spending budget.
Chunking adds calls, tokens, receipts, latency, and potential cost. A failure in
any chunk is fatal: no partial selection succeeds and no runner starts.
Run without --affected if limits are exceeded; Sedum never automatically falls
back to the full suite or silently truncates input. The run deadline starts
after selection; Ctrl-C can cancel selection itself.
Keep full-suite CI; affected selection is not a mandatory CI gate. Models can miss indirect dependencies, lack unchanged
application source, and can be influenced by adversarial source text. Before
tuning the cutoff, compare selected tests against full runs on representative
changes and measure missed failing tests as well as tests avoided. Pin a model
version when collecting comparable results.
Deterministic synthetic fixtures and recorded responses verify mechanics, not
live accuracy or calibration. Accuracy of this hardening remains unverified;
no live accuracy spending is authorized by the plan. A live paired experiment
requires separate spend approval and representative labeled, sanitized data.
Compare old/new prompts and single/chunked scoring on inputs that fit both,
recording recall, extra selected files, chunk-count sensitivity, calls, tokens,
cost, latency, and model version before treating selection as a gate.
Design references: TypeSafe’s Noul documentation,
model limits, and
known failure modes.
Detecting browser errors
--detect browser flags steps whose page reported a problem that no claim checked. After each step settles, Sedum reads what the browser reported during that step:
console_error: aconsole.errorcall.uncaught_exception: an exception no code caught.unhandled_rejection: a rejected promise with no handler. A rejection that gains a handler moments later is dropped.http_error: a 4xx or 5xx response, or a request that failed without one (status0), for every request the page makes: the document,fetch, XHR, scripts, and images. Aborted requests, such as those a navigation cuts off, are ignored.
suspected_bug flag and lists each finding in detections with its message, URL, and status. Every absolute URL, including those inside messages, keeps only its origin and path: query strings and fragments are dropped. Reporters show findings on the step that caused them. For example, a step that clicks Save while the save request returns 500 is flagged even though the page still says “Saved”. A flagged pass exits 0 by default and 2 under --strict. Detection makes no model calls. It adds about 0.3 seconds per step while the page settles, and up to 1.5 seconds while requests are still in flight.
--detect-allow <rule> ignores known noise and is repeatable. A rule is check:pattern or just pattern; the pattern is a substring of the finding’s message or URL, or re: followed by a regular expression. Missing /favicon.ico files and ResizeObserver loop console errors are always ignored. --detect off, the default, turns detection off.
Experimental: --detect semantic
--detect semantic adds page checks that need no browser signal. Each time a step leaves the page in a state the test has not seen, Sedum sends its path, title, and visible text to Jev in one System One call and asks four questions:
error_message: does the page show that the application failed, such as “Something went wrong”, a stack trace, or an HTTP error page? Feedback about the user’s own input, such as a wrong password, does not count.stuck_loading: is the main content replaced by a spinner or “Loading…”? A loading answer counts only if the page stays unchanged for 3 seconds, so slow pages that finish are not flagged.empty_content: does a heading, section, or table have nothing under it, with no “no results” message?broken_value: does a value look unfilled, such asPrice: $, a raw translation key, ornull, null?
undefined, NaN, [object Object], Invalid Date, and {{template}}, are reported with the excerpt that contains them, but only when Jev agrees a value is broken: documentation prints those words on purpose. Findings carry source (text or model) and, from Jev, probability and severity.
Answers are cached for the run by a hash of the page’s path, title, and text, so a state seen in several tests is asked about once. Each call is about 1,000 input tokens (about $0.00004 at the checked-in Jev rate) and appears in the step’s calls with purpose detector. Semantic mode needs a System One provider, TypeSafe or OpenRouter; with Clef the run stops with detect_unsupported before any test starts. Jev’s probabilities are not calibrated: the thresholds come from the labelled pages in the bug detector evaluation, which also lists its limits. Allow rules apply to page checks too, for example --detect-allow 'empty_content:Page appears empty'.
Parallel runs and sharding
--parallel <n|auto> runs selected tests in up to n lanes at once (1 to 64). auto uses half the logical CPU cores, as Playwright Test does. The default is 1, so behavior and provider cost only change when you ask. Each lane keeps one browser. Every attempt gets a fresh browser context, and a lane restarts its browser after a browser or operational error. A lane runs a test’s retries before it takes the next test. The result lists tests in selection order, whatever order they finish in, and records each attempt’s lane.
SEDUM_PARALLEL_INDEXis the lane, from 0.SEDUM_SHARD_INDEXis the shard, from 1.SEDUM_ATTEMPT_KEYis a short random value that is new for every attempt, including retries.
--provider-concurrency <n> caps model-provider requests in flight across all lanes (1 to 32). It defaults to min(4, 2 × lanes). When the provider returns HTTP 429, every lane pauses until its Retry-After, up to 60 seconds for each pause. Requests then resume one at a time, and concurrency climbs back by one for each success. Waiting for a slot or a pause does not use up the 30-second request deadline or the three-attempt retry limit. A call that is still being rate limited after five minutes stops the run with provider_rate_limited (exit 3); lower --parallel or --provider-concurrency, or retry later. The run deadline and Ctrl-C end any wait sooner.
--shard-index <i> --shard-count <n> runs one of n deterministic slices of the selection; the index starts at 1. Sharding happens after paths and filters, so every job must use the same paths, filters, and config. As in Jest and Vitest, tests are ordered by a hash of their identity (explicit id, else path) and cut into contiguous slices:
- Shard sizes differ by at most one.
- The split does not depend on discovery order.
- Adding or removing one test moves at most
n − 1others between shards.
empty_shard (exit 3), so keep --shard-count at or below the suite size. Each shard writes its own result; merging shard results is not supported yet.
list reporter prints one line per finished test with a counter ([3/20] test PASSED …). A retried test gets one line, showing the outcome that counts and its attempt count. The steps reporter prints each test’s steps as one block when that test finishes, so output from tests running at the same time never interleaves. The summary adds the lane count, the shard, time spent in provider rate-limit pauses, and any locator cache write conflicts.
--env <name> selects a configured environment. --url-override <url> replaces the origin of each test’s initial URL while keeping its resolved path, query, and fragment; a later explicit goto step is unaffected. --browser chrome|chromium, --slow <ms>, --output-dir <path>, --strict, and --costs control browser and output behavior. --headed shows the browser and marks the element about to be clicked or filled with a browser overlay that does not change page content or intercept input. Canonical progress.json, result.json, and the offline report.html are written under <outputDir>/<run-id>/.
Output and exit codes
sedum --help lists the available commands; every command has its own --help. sedum run prints one ordered result line per test and aggregate verdict counts, plus separate low_confidence and contradiction counts. Terminals receive colored result labels and transient progress. Redirected output contains no ANSI or cursor controls and does not truncate paths or diagnostics.
Run exits are 0 for a pass (including a flagged pass by default), 1 for a failed test, 2 for a flagged pass under --strict, and 3 when the command or run could not produce a trustworthy verdict. --strict never changes the canonical verdict or flags. Model tokens and costs are shown automatically in a terminal; pass --costs to include them in redirected output.
The default list terminal reporter shows test results. Select steps for each completed step, or repeat --reporter (or give a comma-separated list such as --reporter list,junit) to use several; terminal is an alias for list. Failed or flagged steps include a needs-attention block with their recorded sentence, source, reason, page context, and evidence status. Terminal output may use color in a TTY; redirected output is plain. --reporter json writes a separate final JSON result under <reporterDir>/<run-id>/, and --reporter-dir <path> selects that project-root-relative directory. You can combine JSON with terminal reporters. The JSON copy is the canonical RunResult unchanged and validates against @sedum-dev/core/run-result.schema.json.
--reporter junit writes <reporterDir>/<run-id>/junit.xml for CI test views (GitLab, Jenkins, CircleCI, Azure DevOps, or a JUnit action on GitHub); it writes no JSON copy unless you also select json or pass --reporter-dir. When the reporter and output directories are the same, it goes in the run directory. A flagged pass is a passing testcase with its flags as properties and output; --strict adds a failure to it, so the file matches the exit code. See running in CI for the full mapping and pipeline examples. When markdown is the only other reporter, stdout prints junit <path> after the markdown <path> line; the run summary lists it otherwise.
The HTML file is created for every run with an authoritative result; open it directly in a browser, without a server.
--reporter markdown also writes report.md for a coding agent to read, always in the canonical <outputDir>/<run-id>/ next to result.json; --reporter-dir does not move it. When it is the only reporter, stdout is a single line, markdown <path>; otherwise the run summary lists that line with the other artifacts. The report opens with the verdict and a flow table sorted failed, incomplete, flagged, then passed. Then, for each failed, flagged, errored or interrupted step, it gives the sentence, file:line:col, score against the fail and pass lines, contradiction against its cutoff, page URL and title, the locator’s weighed candidates including (no match), the full recorded browser error and call log, the judged page excerpt, a relative link to the step’s frame, and a rerun command that works from anywhere in the project. Passed flows collapse to the checks that held. Page text and browser errors are escaped or fenced and labeled untrusted, so page content cannot add headings, links or images to the report. The report links frames rather than embedding them; the frames can still show private page pixels.
Frames are stored per attempt as evidence/<attempt-key>/<frame>.jpg inside the run directory, and every run and attempt gets a new directory. Sedum never deletes another run’s files, so parallel workers, retries and consecutive CI runs can share one output directory. Nothing expires automatically: remove a run with rm -rf <outputDir>/<run-id> (and <reporterDir>/<run-id> if you wrote a JSON copy or junit.xml there), and keep .sedum/runs/ out of source control.
The HTML report includes light and dark themes, failure and flagged-flow filters, per-assertion scores and decision lines, all retry attempts, and a receipt from the recorded model calls. A missing cost is shown as unknown rather than zero. --replay captures step frames and embeds them in report.html; without it the report contains no frame bytes. Captured frames can contain private page pixels, so use --sensitive-origin or omit replay on sensitive pages. Frames show the visible page, while assertion grades come from the recorded judgement and judged page excerpt. The file contains its own styles, fonts, scripts, and optional frames and makes no network requests.
sedum explore (experimental)
sedum explore <url> explores a site by a seeded random walk guided by Jev.
It reports what --detect browser reports (console errors, uncaught
exceptions, unhandled rejections, failed requests), plus HTTP 5xx responses
and broken links. --detect semantic also asks the detector’s page checks in
the walk’s existing Jev call for each new page, and --detect-allow <rule>
ignores known findings.
Safety is off by default, for disposable staging; the walk only refuses to
log itself out. --safe skips what Jev judges destructive and leaves
settings forms alone; --safe strict also skips names like delete, send or
pay.
Each run writes trace.jsonl, result.json, report.md and report.html
into --out/<run-id>/. --replay adds a frame per step to the HTML report
(--sensitive-origin turns frames off for an origin), and --shrink writes
each failure as a plain-English test. --rerun <trace> runs a recorded
trace again. Use --steps, --seed and --mode guided|heuristic|random.
See Exploration mode.
sedum validate
sedum validate [paths...] checks tests and modules without opening a browser, without a model key, and without spending tokens. That makes it cheap enough for a pre-commit hook and for every CI run.
Both commands find the project the same way sedum run does; see project configuration. The project root is the folder of the nearest sedum.config.yaml, or the current directory when there is none. Test ids are therefore the same whichever folder you run from.
- With no paths, the commands check the tests that
sedum runwould run: the*.test.tsand*.test.yamlfiles undertests.directorythat match itsincludeandexcludeglobs. They also check every*.module.yamlfile under that directory. - With paths, each path is resolved against the project root, as
sedum runresolves an explicit file. A path can be a directory, a*.test.tsor*.test.yamlfile, or a*.module.yamlfile. A*.test.tsfile is imported to find its tests, and the literal sentences in itsai(...)calls are checked like YAML steps; see TypeScript tests. Directories are searched recursively.node_modulesand dot-directories such as.gitand.sedumare skipped, unless you name one directly.
- Format. Every test and module is checked against the test file format: keys, types, data, placeholders, and a
typestep that names exactly one value. - Module calls. Every
use:call is resolved: missing files, cycles, depth, and arguments. - Sentences. Every sentence in a test or a reachable module is classified into one operation (
click,type,verify, …). - Identities. Every test must have a unique identity. That identity is its explicit
id, or else its path relative to the project root. - Entry URL. The URL each test opens is resolved against the configured
baseUrl, exactly assedum runresolves it. A test with nourland nobaseUrl, or with a relativeurland nobaseUrl, is an error. - Configuration. An invalid
sedum.config.yamlis reported with its position and exits3.
path:line:col followed by a fix:
Offline by default
Validation classifies sentences using built-in patterns and the committed.sedum/classifications.json only; see step classification. It does not read provider credentials or the project .env, does not resolve $ENV values (an unset variable is fine), and does not write any file. Because it never opens .env, a malformed or unreadable .env is reported by sedum run and --online, not by offline validation. A sentence it cannot classify this way is reported as not checked offline and fails the check. Sedum cannot confirm what that step will do, so it never reports it as valid.
To resolve such a sentence, either rephrase it with an obvious supported verb, or classify it once with the model:
--online takes the key from the process environment or the project-root .env, as sedum run does. It sends only the sentences the cache cannot answer. It writes accepted answers to .sedum/classifications.json; commit that file so CI can validate offline.
For a compatible provider, also set TYPESAFE_BASE_URL and optionally
TYPESAFE_DEFAULT_MODEL; use that provider’s key as TYPESAFE_API_KEY.
If you ignore .sedum/ for run output, keep the cache tracked:
use: calls are checked when a test reaches it, and the summary says so.
Exit codes
When both a
3 problem and a finding occur, the exit is 3. Warnings, such as a data value YAML would change, never affect the exit code.
sedum list
sedum list [paths...] prints the discovered tests with their id, tags, and path. It only parses files, so it needs no classification cache or key.
sedum validate on it for the full list.
sedum list --json prints only JSON on stdout:
idSourceisexplicitfor anidkey andpathfor a path identity.descriptionisnullwhen absent.fileis relative to the project root and uses/on every OS.- Test entries never include test data, URLs,
meta, or step sentences. Diagnostics quote only what the file already contains. - Fields may be added within
schemaVersion: 1. A change that removes or renames a field, or changes what a field means, bumps the version.
list exits 0 when every discovered file was listed, including when there are none. It exits 1 when some files could not be listed, and 3 for an invalid config or a bad path.
In CI and pre-commit
Neither command needs a browser or a key. These examples assumesedum-cli is
a dev dependency of the project; see running in CI.