Skip to content

Speaking while generating

On a voice call, a prompt with speak_streaming: true is spoken to the caller while the model generates it; speak_streaming: "say" speaks only the say field of a json_schema response. The result then also has:

  • spoken - whether the caller heard (part of) the response
  • spoken_text - what was sent to the caller
  • interrupted - whether the caller barged in and cut the response off

Say the result as usual: it is written to the conversation but not spoken again. Once spoken is true, do not retry the prompt or say a fallback, the caller would hear both. Elsewhere (chat, a translated conversation) nothing is streamed and spoken is false.

LLM.count_tokens(prompt_or_model, text)

Count the number of tokens given a prompt or model and an input text.

Returns nil in case of error

LLM.decode_tokens(prompt_or_model, tokens)

Decode the given list of (integer) tokens into a binary string.

Returns nil in case of error.

LLM.encode_tokens(prompt_or_model, text)

Encode the given text into a list of tokens.

Returns nil in case of error

LLM.prepare_agentloop_prompt(prompt, scope)

Prepare a prompt for the platform agent loop.

Renders the prompt with the current interpreter scope and returns a map containing the prompt configuration, rendered messages, and tool schemas.

LLM.prompt(orig \\ %Prompt{}, args)

Construct a new prompt to be used in the LLM completion functions.

This function can be used to override certain properties of other prompts, or create new prompts altogether. However, prefer to use the builtins Prompts YAML files to define prompts, as this is more efficient and less error prone.

To set for instance an extra parameter on a prompt you could do the following:

myprompt = LLM.prompt(@prompts.original, endpoint_params: %{temperature: 1.3})

LLM.render_prompt(prompt, scope)

Render a prompt into its final form, an array of chat messages.

This function is mostly used for debugging and should not be needed in normal use cases.