Documentation | Evaluations › Create Evaluation

Evaluations

evaluations

Methods

Create Evaluation ->

Evaluation

post/v5/evaluations

Create Evaluation

Response fields

Evaluation

Request example

curl https://api.egp.scale.com/v5/evaluations \
    -H 'Content-Type: application/json' \
    -H "x-api-key: $SGP_API_KEY" \
    -d '{
          "data": [
            {
              "foo": "bar"
            }
          ],
          "name": "name",
          "description": "description",
          "files": [
            {
              "foo": "string"
            }
          ],
          "metadata": {
            "foo": "bar"
          },
          "tags": [
            "string"
          ],
          "tasks": [
            {
              "configuration": {
                "messages": [
                  {
                    "foo": "bar"
                  }
                ],
                "model": "model",
                "audio": {
                  "foo": "bar"
                },
                "frequency_penalty": 0,
                "function_call": {
                  "foo": "bar"
                },
                "functions": [
                  {
                    "foo": "bar"
                  }
                ],
                "logit_bias": {
                  "foo": 0
                },
                "logprobs": true,
                "max_completion_tokens": 0,
                "max_tokens": 0,
                "metadata": {
                  "foo": "string"
                },
                "modalities": [
                  "string"
                ],
                "n": 0,
                "parallel_tool_calls": true,
                "prediction": {
                  "foo": "bar"
                },
                "presence_penalty": 0,
                "reasoning_effort": "reasoning_effort",
                "response_format": {
                  "foo": "bar"
                },
                "seed": 0,
                "stop": "stop",
                "store": true,
                "temperature": 0,
                "tool_choice": "tool_choice",
                "tools": [
                  {
                    "foo": "bar"
                  }
                ],
                "top_k": 0,
                "top_logprobs": 0,
                "top_p": 0
              },
              "alias": "alias",
              "task_type": "chat_completion"
            }
          ],
          "taxonomy_params": {
            "foo": "bar"
          }
        }'

200Example

{
  "id": "id",
  "created_at": "2019-12-27T18:11:19.117Z",
  "created_by": {
    "id": "id",
    "type": "user",
    "object": "identity"
  },
  "datasets": [
    {
      "id": "id",
      "created_at": "2019-12-27T18:11:19.117Z",
      "created_by": {
        "id": "id",
        "type": "user",
        "object": "identity"
      },
      "current_version_num": 0,
      "name": "name",
      "tags": [
        "string"
      ],
      "archived_at": "2019-12-27T18:11:19.117Z",
      "description": "description",
      "object": "dataset"
    }
  ],
  "name": "name",
  "status": "failed",
  "tags": [
    "string"
  ],
  "archived_at": "2019-12-27T18:11:19.117Z",
  "description": "description",
  "error_count": 0,
  "metadata": {
    "foo": "bar"
  },
  "object": "evaluation",
  "progress": {
    "items": {
      "failed": 0,
      "pending": 0,
      "successful": 0,
      "total": 0,
      "failed_items": [
        {
          "item_id": "item_id",
          "error": "error",
          "error_type": "error_type"
        }
      ]
    },
    "workflows": {
      "completed": 0,
      "failed": 0,
      "pending": 0,
      "total": 0
    }
  },
  "status_reason": "status_reason",
  "tasks": [
    {
      "configuration": {
        "messages": [
          {
            "foo": "bar"
          }
        ],
        "model": "model",
        "audio": {
          "foo": "bar"
        },
        "frequency_penalty": 0,
        "function_call": {
          "foo": "bar"
        },
        "functions": [
          {
            "foo": "bar"
          }
        ],
        "logit_bias": {
          "foo": 0
        },
        "logprobs": true,
        "max_completion_tokens": 0,
        "max_tokens": 0,
        "metadata": {
          "foo": "string"
        },
        "modalities": [
          "string"
        ],
        "n": 0,
        "parallel_tool_calls": true,
        "prediction": {
          "foo": "bar"
        },
        "presence_penalty": 0,
        "reasoning_effort": "reasoning_effort",
        "response_format": {
          "foo": "bar"
        },
        "seed": 0,
        "stop": "stop",
        "store": true,
        "temperature": 0,
        "tool_choice": "tool_choice",
        "tools": [
          {
            "foo": "bar"
          }
        ],
        "top_k": 0,
        "top_logprobs": 0,
        "top_p": 0
      },
      "alias": "alias",
      "task_type": "chat_completion"
    }
  ]
}

List Evaluations -> CursorPage<

Evaluation

get/v5/evaluations

List Evaluations

Get Evaluation ->

Evaluation

get/v5/evaluations/{evaluation_id}

Get Evaluation

Archive Evaluation ->

Evaluation

delete/v5/evaluations/{evaluation_id}

Archive Evaluation

Update Or Restore Evaluation ->

Evaluation

patch/v5/evaluations/{evaluation_id}

Update or Restore Evaluation

Get Evaluation Data Schema ->

EvaluationSchemaResponse

get/v5/evaluations/{evaluation_id}/schema

Get schema information for evaluation item data, including field names, types, and occurrence counts.

Filter Evaluations -> CursorPage<

Evaluation

post/v5/evaluations/filter

Filter evaluations using metadata and other criteria. Supports up to 10 filters with AND logic.

Get Evaluation Taxonomy -> Record<string, unknown>

get/v5/evaluations/{evaluation_id}/taxonomy

Get taxonomy JSON for contributor evaluation question tasks.

Domain types

AutoEvaluationAgentTaskRequestWithItemLocator = { definition, name, output_rules, 6 more... }

Evaluation = { id, created_at, created_by, 12 more... }

EvaluationRetrieveTaxonomyResponse = Record<string, unknown>

EvaluationSchemaResponse = { evaluation_id, fields, total_items, 3 more... }

Schema information for an evaluation's item data structure

EvaluationTask = { configuration, alias, task_type } | { configuration, alias, task_type } | { configuration, alias, task_type } | 7 more...

EvaluationTasksProgressSchema = { items, workflows }

EvaluationViews = "tasks"

ItemLocator = string

ItemLocatorTemplate = string

PaginatedListEvaluation = { has_more, items, total, 2 more... }

Evaluations

Tasks

evaluations.tasks

Methods

Add Test Criteria To Evaluation ->

Evaluation

post/v5/evaluations/{evaluation_id}/tasks

Add a new test criteria (LLM judge, contributor question, etc.) to an existing evaluation. Gated: rejected if any contributor annotation task has been claimed or completed. Kicks off the evaluation workflow so the new task runs against existing items.

Update Test Criteria Configuration ->

Evaluation

patch/v5/evaluations/{evaluation_id}/tasks/{alias}

Replace a single test criteria's configuration, identified by its alias. Gated: rejected if any contributor annotation task for the evaluation has been claimed or completed.