Skip to content

Large Language Model (LLM)

This module contains functions to manipulate Large Language Model (LLM) prompts and perform LLM completions against the builtin OpenAPI integration and other LLM providers. See https://developer.dialox.ai/platform/nlp/llm/ for more information on how to work with LLM completions in general.

LLM.compact()

Compact full_transcript for later LLM.complete calls.

Runs the conversation so far through the overridable compact_full_transcript platform prompt (or an explicit prompt) and stores a compaction annotation. Later full_transcript builds keep pre-compaction system messages, place the compaction message next, and drop other pre-compaction turns.

LLM.compact()
_result = LLM.complete(@prompts.agent)

Override the platform prompt by defining a prompt with the same id, or pass a prompt explicitly:

LLM.compact(@prompts.my_compact)

LLM.complete(prompt, bindings, scope)

Perform a LLM completion request

The full result of the LLM.complete call is a map array which contains the following:

  • text - The output text that LLM produced
  • json - A JSON deserialized version of the text; the runtime detects whether JSON is available in the result and, if so, parses it. The JSON message itself can be padded with arbitrary other texts.
  • usage - The total tokens that were used for this API call
  • request_time - The nr of milliseconds this request took
  • raw - The raw OpenAPI response