Last updated January 29, 2025

The /v1/chat/completions endpoint generates conversational completions for a provided set of input messages. You can specify the model, adjust generation settings such as temperature , and opt to stream the responses in real-time. You can also specify tools the model can choose to call.

Request Body Parameters

Use parameters to manage how conversational completions are generated.

Required Parameters

Optional Parameters

tools Array of Objects

tools lets you provide your model with an array of tools it may choose to call (use tool_choice to specify how the model will call tools). When provided, your model may send back tool_calls in the role="assistant" generated message, asking your system to run the specified tool, and send back the result in a role="tool" message.

Note that these tools are given to the model in the form of an extended prompt and no further validation is done. Models may make up tool names that don’t exist in the tools array you have given them. To avoid this, we recommend you perform tool validation on your end when a model sends back a tool_calls assistant message.

Heroku provides a variety of custom Heroku tools that are executed automatically by Heroku, rather than returning a tool call request to your system.

function Object

Example tools Array

[
    {
      "type": "function",
      "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state, e.g. Portland, OR"
            }
          },
          "required": ["location"]
        }
      }
    }
  ]

tool_choice Object

The tool_choice object specifies how the model should use the provided tools.

It can either be a string (none, auto, or required), or a tool_choice object. none will mean the model will call no tools. auto allows the model to call zero to many of the provided tools, and required forces the model to call at least one or more tools before responding to the user.

To force the model to call a specific tool, you may simply specify a single tool in the tools object and pass "tools": "required", or you can force the tool selection by passing a tool_choice object that specifies the required function.

messages Array of Objects

The messages object is an array of message objects.

Each message must specify a role field that determines the messages’s schema (see below).

Currently, the supported types are user, assistant, system, and tool.

If the most recent message uses the assistant role, the model will continue its answer starting from the content in that most recent message.

role=user message

user messages are the primary way to send queries to your model and prompt it to respond.

role=assistant message

Typically, assistant messages are only generated by the model, however you can create your own or pre-fill a partially completed assistant response to help influence the content that the model will generate on its next turn.

role=system message

A system message is sort of a prompt ‘prefix’ that is given to the model to help influence its responses.

role=tool message

A tool message object lets you communicate a specified tool’s result (output) to the model.

Example tool_calls Object

Here’s an example of what a tool_calls object might look like, after your model has decided to call a tool you’ve given to it as an option via tools.

[
  {
    "role": "assistant",
    "tool_calls": [
      {
        "id": "toolu_02F9GXvY5MZAq8Lw3PTNQyJK",
        "type": "function",
        "function": {
            "name": "get_weather",
            "arguments": "{\"location\":\"Portland, OR\"}"
        }
      }
    ],
  }
]

Request Headers

In the following example, we assume your model resource has an alias of “INFERENCE” (the default).

All inference curl requests must include an Authorization header containing your Heroku Inference key.

For example, all /v1/chat/completions requests should follow this pattern:

# If you're developing locally, run this to set your config vars as ENV variables.
eval $(heroku config -a $APP_NAME --shell | grep '^INFERENCE_' | sed 's/^/export /' | tee >(cat >&2))

curl $INFERENCE_URL/v1/chat/completions \
 -H "Authorization: Bearer $INFERENCE_KEY" \
 -d @- <<EOF
{
  "model": "$INFERENCE_MODEL_ID",
  "messages": [{"role": "user", "content": "Hello"}]
}
EOF