Model Services¶

Model services provide a solution for quickly deploying open-source or fine-tuned large language models as callable services. With one-click deployment, the complex management of models is simplified into a standardized service format. This format is compatible with mainstream model service API call capabilities, enabling immediate use.

• Model services allow users to call selected models to perform tasks such as text generation, chat processing, image generation, and more.
• These services support the online experience of models.

The details of the model service include basic information about the service, available authorization methods, and example API calls.

Basic Information¶

• Model Service Name: The name of the service, used to identify this model service.
• Access Path Name: A unique path name for each model service.
• Model Type: The type of model used by the service.
• Status: The current operational status of the service.
• Billing Method: The billing method for the service.

Authorization Methods¶

API-Key Authorization¶

• All API requests require the inclusion of an Authorization field in the HTTP header for identity verification.
• Format: Authorization: Bearer {API_KEY}
• You can obtain your API key by clicking the "View My API-Key" link on the page.

Security Advice¶

• Store the API key securely on the backend server to prevent exposure in client-side code and avoid potential security leaks.

API Call Example¶

Request URL¶

The POST request URL is:

https://sh-02.d.run/v1/chat/completions

Request Example: Calling the API with curl¶

curl 'https://sh-02.d.run/v1/chat/completions' \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <Your API Key here>" \
  -d '{
    "model": "u-8105f7322477/test",
    "messages": [{"role": "user", "content": "Say this is a test!"}],
    "temperature": 0.7
  }'

Request Parameters:

• model: The access path name of the model service (e.g., u-8105f7322477/test).

• messages: A list of chat history containing user input, for example:

  [{"role": "user", "content": "Say this is a test!"}]
  

• temperature: Controls the randomness of the generated output. A higher value produces more creative results, while a lower value yields more stable outputs.

Response Example¶

{
  "id": "cmp-1d033c426254417b7b0675303b1d300",
  "object": "chat.completion",
  "created": 1733724462,
  "model": "u-8105f7322477/test",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "I am a large language model. How can I assist you today?"
      },
      "tool_calls": []
    }
  ],
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 15,
    "total_tokens": 40
  }
}

Response Fields:

• id: A unique identifier for the generated result.
• model: The ID of the model service being called.
• choices: An array of results generated by the model.
  • message: The generated content.
  • content: The text content produced by the model.
• usage: Token usage for this call:
  • prompt_tokens: The number of tokens used for the user input.
  • completion_tokens: The number of tokens used for the generated response.
  • total_tokens: The total token usage.

Python Example¶

# Compatible with OpenAI Python library v1.0.0 and above

from openai import OpenAI

client = OpenAI(
    base_url="https://sh-02.d.run/v1/",
    api_key="<Your API Key here>"
)

messages = [
    {"role": "user", "content": "hello!"},
    {"role": "user", "content": "Say this is a test?"}
]

response = client.chat.completions.create(
    model="u-8105f7322477/test",
    messages=messages
)

content = response.choices[0].message.content

print(content)

Node.js Example¶

const OpenAI = require('openai');

const openai = new OpenAI({
  baseURL: 'https://sh-02.d.run/v1',
  apiKey: '<Your API Key here>',
});

async function getData() {
  try {
    const chatCompletion = await openai.chat.completions.create({
      model: 'u-8105f7322477/test',
      messages: [
        { role: 'user', content: 'hello!' },
        { role: 'user', content: 'how are you?' },
      ],
    });

    console.log(chatCompletion.choices[0].message.content);
  } catch (error) {
    if (error instanceof OpenAI.APIError) {
      console.error('API Error:', error.status, error.message);
      console.error('Error details:', error.error);
    } else {
      console.error('Unexpected error:', error);
    }
  }
}

getData();

Scaling Up and Down¶

If you experience resource shortages or lag while using the model, you can scale up the model.

In the model service list, click ┇ on the right side, then select Scale Up/Down from the dropdown menu.

Enter the number of instances you want to add, such as 2, and click OK .

Deletion¶

1. In the model service list, click ┇ on the right side, then select Delete from the dropdown menu.
2. Enter the name of the model to be deleted, verify the details, and click Delete.