Documentation | Models

Models

ModelsResource

Methods

client.models.create() ->

post/v5/models

Create a custom model record in your account and begin deploying it through a supported serving vendor.

A model here is a record for a model you deploy and serve through Scale's own inference vendors: only the launch and llmengine vendors are accepted and any other vendor is rejected. This is distinct from GET /v5/chat/completions/models, which lists the models already available to call for chat completions rather than creating or managing these records. The call is asynchronous — the record is created in a deploying status, a deployment job is recorded, and a Temporal workflow is started to perform the deployment, so the model is not ready for inference when this returns. A model name must be unique per vendor within your account; if a model with the same name and vendor already exists the request fails unless on_conflict is set to update, in which case the existing model is updated instead.

client.models.list() -> SyncCursorPage[

InferenceModel

]

get/v5/models

List the custom model records registered in your account.

Returns a paginated list of the model records managed through this API — models your account deploys through the launch or llmengine serving vendors — optionally filtered by name and by model vendor, and scoped to the caller's account. This is different from GET /v5/chat/completions/models, which lists the models available to invoke for chat completions; this endpoint returns the managed records along with their deployment status, not the catalog of callable completion models.

Parameters

ending_before: Optional[str]

limit: Optional[int]

(maximum: 10000, minimum: 1, default: 100)

model_vendor: Optional[

InferenceModelVendor

]

name: Optional[str]

sort_by: Optional[str]

sort_order: Optional[

SortOrder

]

starting_after: Optional[str]

Returns

InferenceModel

id: str

The unique identifier of the entity.

created_at:

datetime

(format: date-time)

The date and time when the entity was created in ISO format.

created_by_identity_type: Literal["user", "service_account"]

The type of identity that created the entity.

created_by_user_id: str

The user who originally created the entity.

model_type:

InferenceModelType

model_vendor:

InferenceModelVendor

name: str

status: Literal["failed", "ready", "deploying", "deployment_timeout"]

model_availability: Optional[InferenceModelAvailability]

model_metadata: Optional[Dict[str,

object

]]

object: Optional[Literal["model"]]

(default: "model")

status_reason: Optional[str]

vendor_configuration: Optional[

VendorConfiguration

]

Request example

import os
from scale_gp_beta import SGPClient

client = SGPClient(
    api_key=os.environ.get("SGP_API_KEY"),  # This is the default and can be omitted
)
page = client.models.list()
page = page.items[0]
print(page.id)

200Example

{
  "has_more": true,
  "items": [
    {
      "id": "id",
      "created_at": "2019-12-27T18:11:19.117Z",
      "created_by_identity_type": "user",
      "created_by_user_id": "created_by_user_id",
      "model_type": "generic",
      "model_vendor": "openai",
      "name": "name",
      "status": "failed",
      "model_availability": "unknown",
      "model_metadata": {
        "foo": "bar"
      },
      "object": "model",
      "status_reason": "status_reason",
      "vendor_configuration": {
        "model_image": {
          "command": [
            "string"
          ],
          "registry": "registry",
          "repository": "repository",
          "tag": "tag",
          "env_vars": {
            "foo": "bar"
          },
          "healthcheck_route": "healthcheck_route",
          "predict_route": "predict_route",
          "readiness_delay": 0,
          "request_schema": {
            "foo": "bar"
          },
          "response_schema": {
            "foo": "bar"
          },
          "streaming_command": [
            "string"
          ],
          "streaming_predict_route": "streaming_predict_route"
        },
        "model_infra": {
          "cpus": "string",
          "endpoint_type": "async",
          "gpu_type": "nvidia-tesla-t4",
          "gpus": 0,
          "high_priority": true,
          "labels": {
            "foo": "string"
          },
          "max_workers": 0,
          "memory": "memory",
          "min_workers": 0,
          "per_worker": 0,
          "public_inference": true,
          "storage": "storage"
        }
      }
    }
  ],
  "total": 0,
  "limit": 0,
  "object": "list"
}

client.models.update(, ) ->

InferenceModel

patch/v5/models/{model_id}

Update a custom model record; vendor-configuration changes are applied asynchronously by redeploying the model.

This supports three kinds of update: changing model metadata only, renaming the model, and changing the vendor configuration. A vendor-configuration change is asynchronous — it puts the model back into a deploying status, records an update job, and starts a Temporal workflow to redeploy, so the new configuration is not live when this returns; metadata-only and rename changes take effect immediately. The vendor configuration supplied must match the model's own vendor (launch or llmengine), and only those two vendors are supported. A model that is currently deploying cannot be modified and the request fails until deployment finishes. When renaming with on_conflict set to swap, the name is exchanged with an existing model of the same name and vendor instead of failing on the uniqueness constraint.

client.models.delete() ->

ModelDeleteResponse

delete/v5/models/{model_id}

Permanently delete a custom model record and tear down its deployment at the serving vendor.

This is a hard delete: the model row is removed from your account entirely and cannot be restored afterward. Before the record is removed, if the model has an associated vendor deployment that deployment is torn down at its serving vendor. A model that is currently deploying cannot be deleted and the request fails until deployment finishes. This operates on the model records managed by this API, distinct from the GET /v5/chat/completions/models catalog of models callable for chat completions.

client.models.retrieve() ->

InferenceModel

get/v5/models/{model_id}

Retrieve a single custom model record by its ID.

Returns the model record — including its vendor, configuration, and current deployment status — for a model managed through this API and owned by the caller's account. This is distinct from GET /v5/chat/completions/models, which lists the models available to call for chat completions rather than returning a single managed record.

Domain types

class InferenceModel: ...

Literal["unknown", "available", "unavailable"]

Literal["generic", "completion", "chat_completion"]

class LaunchVendorConfiguration: ...

class LlmEngineVendorConfiguration: ...

class ModelDeleteResponse: ...