Documentation

Evaluations

EvaluationsResource

Methods

create() ->
Evaluation
post/v5/evaluations

Create Evaluation

list() -> SyncCursorPage[
Evaluation
]
get/v5/evaluations

List Evaluations

retrieve(, ) ->
Evaluation
get/v5/evaluations/{evaluation_id}

Get Evaluation

archive() ->
Evaluation
delete/v5/evaluations/{evaluation_id}

Archive Evaluation

update(, ) ->
Evaluation
patch/v5/evaluations/{evaluation_id}

Update or Restore Evaluation

retrieve_schema(, ) ->
EvaluationSchemaResponse
get/v5/evaluations/{evaluation_id}/schema

Get schema information for evaluation item data, including field names, types, and occurrence counts.

filter() -> SyncCursorPage[
Evaluation
]
post/v5/evaluations/filter

Filter evaluations using metadata and other criteria. Supports up to 10 filters with AND logic.

retrieve_taxonomy() ->
EvaluationRetrieveTaxonomyResponse
get/v5/evaluations/{evaluation_id}/taxonomy

Get taxonomy JSON for contributor evaluation question tasks.

Domain types

class AutoEvaluationAgentTaskRequestWithItemLocator: ...
class Evaluation: ...
Dict[str,
object
]
class EvaluationSchemaResponse: ...

Schema information for an evaluation's item data structure

EvaluationTask
class EvaluationTasksProgressSchema: ...
Literal["tasks"]
str
str
class PaginatedListEvaluation: ...
Evaluations

Tasks

EvaluationsResource.TasksResource

Methods

add(, ) ->
Evaluation
post/v5/evaluations/{evaluation_id}/tasks

Add a new test criteria (LLM judge, contributor question, etc.) to an existing evaluation. Gated: rejected if any contributor annotation task has been claimed or completed. Kicks off the evaluation workflow so the new task runs against existing items.

update(, ) ->
Evaluation
patch/v5/evaluations/{evaluation_id}/tasks/{alias}

Replace a single test criteria's configuration, identified by its alias. Gated: rejected if any contributor annotation task for the evaluation has been claimed or completed.