Create an evaluation together with its items, optionally running test criteria against them.

Accepts three request shapes: standalone (inline data), from an existing dataset (dataset_id with optional per-item references), or with a new reusable dataset created inline from data. When the evaluation includes tasks that require execution (for example an LLM judge or custom function), an async job and a Temporal workflow are started and the evaluation is returned immediately with status running; task results and error_count populate asynchronously. When it includes only contributor tasks, taxonomy-only input, or no tasks, no workflow runs and it is returned with status completed. Optional tasks, metadata, tags, and taxonomy_params are persisted alongside the evaluation and its items.

List evaluations for the account, with pagination.

Supports filtering by case-insensitive name substring and by tags; archived evaluations are excluded unless include_archived is set. Pass the tasks view to include each evaluation's task configurations in the response. Use this for simple name or tag lookups; to filter on metadata key-value pairs or status, use the filter endpoint instead.

Retrieve a single evaluation by ID.

Returns the evaluation with its datasets, async-job progress, metadata, and task-error count. Archived evaluations are excluded unless include_archived is set. Pass the tasks view to include the evaluation's task configurations in the response.

Archive (soft-delete) an evaluation.

Sets the evaluation's archived timestamp rather than permanently deleting it, and cascades the archive to the evaluation's items and dashboards while removing it from any evaluation groups. The evaluation can later be brought back with a restore request to the update endpoint.

Update an evaluation's mutable fields, or restore it from the archive.

The action is selected by the request body: a restore request un-archives the evaluation and cascades the restore to its items and dashboards, while any other body applies a partial update to fields such as name, description, tags, and metadata (metadata is applied as an RFC 7396 merge patch). Updating an already-archived evaluation is rejected — restore it first. The evaluation row is locked for the duration of the write to avoid concurrent-update races.

Describe the data schema of an evaluation's items.

Inspects the item data and task-result fields and returns each discovered field with its flattened key path, JSON type, source, and the number of items containing it, ordered alphabetically by field name. For large evaluations the schema may be inferred from a sample of items, in which case is_sampled is set and sample_size reports how many were analyzed. Set include_archived to include archived items in the analysis.

Filter evaluations by metadata, status, and tags.

Accepts up to 10 filters combined with AND logic, each comparing a key against a value with an operator (==, !=, >=, <=, IN, NOT_IN). Filter on metadata keys returned by the metadata-keys endpoint, plus the built-in status and tag keys. Archived evaluations are excluded unless include_archived is set, and the tasks view includes task configurations in each result. Use this for metadata or status filtering; for simple name or tag lookups the list endpoint is sufficient.

Get the taxonomy JSON for an evaluation's contributor question tasks.

Returns the raw taxonomy document stored for the evaluation. Responds with a not-found error if the evaluation has no taxonomy.

Add a new test criteria to an existing evaluation.

Narrowed to contributor question tasks (contributor_evaluation.question); other task types must be configured when the evaluation is first created and are rejected here. The request is also rejected if the evaluation is archived, if a test criteria with the same alias already exists, or if any contributor annotation task for the evaluation has already been claimed or completed. Because only contributor question tasks are accepted, the added criteria is applied synchronously and contributors answer it against the evaluation's existing items — no async job or Temporal workflow is started.

Replace the full configuration of a single test criteria, identified by its alias.

The alias must match an existing test criteria on the evaluation, and the replacement configuration is validated against the evaluation's current items before being applied. The request is rejected if the evaluation is archived, if no test criteria matches the alias, or if any contributor annotation task for the evaluation has already been claimed or completed — at that point labelers are in-flight and mutating the task definition would corrupt their work.