Poly is a versatile LLM serving back-end. What it offers:

• High-performance, efficient and reliable serving of multiple local LLM models
• Optional GPU acceleration through either CUDA or Metal
• Configurable LLM completion tasks (prompts, recall, stop tokens, etc.)
• Streaming completion responses through HTTP SSE, chat using WebSockets
• Biased sampling of completion output using JSON schema
• Memory retrieval using vector databases (either built-in file based, or external such as Qdrant)
• Accepts and automatically chunks PDF and DOCX files for storage to memory
• API secured using either static API keys or JWT tokens
• Simple, single binary + config file server deployment, horizontally scalable

Nice extras:

• A web client to easily test and fine-tune configuration
• A single-binary cross platform desktop client for locally running models

Supported models include:

• LLaMa and derivatives (Alpaca, Vicuna, Guanaco, etc.)
• LLaMA2
• RedPajamas
• MPT
• Orca-mini

Web client	Desktop app

Sample of a model+task+memory configuration:

[models.gpt2dutch]
model_path = "./data/gpt2-small-dutch-f16.bin"
architecture = "gpt2"
use_gpu = false

[memories.dutch_qdrant]
store = { qdrant = { url = "http://127.0.0.1:6333", collection = "nl" } }
dimensions = 768
embedding_model = "gpt2dutch"

[tasks.dutch_completion]
model = "gpt2dutch"
prelude = "### System:\nYou are an AI assistant that follows instruction extremely well. Help as much as you can.\n"
prefix =  "\n### User:\n"
postfix = "\n### Response:"
memorization = { memory = "dutch_qdrant", retrieve = 2 }

See config.example.toml for more example configurations.

Custom samplers can be configured using a string-based description, see here. Any biaser (i.e. JSON biaser) is injected as first sampler in the chain.

In Poly, models are LLM models that support basic text generation and embedding operations. Models can be run on the GPU and have specific context lengths, but are otherwise unconfigurable.

A task uses a model in a specific way (i.e. using specific prompts, stop tokens, sampling, et cetera. Tasks are highly configurable. A model may be shared by multiple tasks.

A memory is a database that stores chunks of text, and allows retrieval of such chunks using vector similarity (where each chunk has a vector calculated as an embedding from an LLM). Memories can be re-used between tasks.

classDiagram
    class Model {
        String name
        Boolean use_gpu
        Int context_length
    }
    class Task {
        String name
        String prompts
        Inference parameters
    }
    class Memory {
        String name
        Chunking parameters
    }
    class Chunk {
        String text
        Vector embedding
    }

    Task "*" --> "1" Model: Generation model
    Task "*" --> "1" Memory
    Memory "*"--> "1" Model: Embedding model
    Memory "1" --> "*" Chunk: Stored chunks

The API exposes models, tasks and memories by their name (which is unique within their category).

A task configures the way user input is transformed before it is fed to an LLM and the way the LLM output is transformed before it is returned to the user, in order to perform a specific task. A task can be configured to use (optional) prelude, prefix and postfix prompts. The prelude is fed once to the model for each session. The prefix and postfix are applied to each user input (i.e. each chat message):

sequenceDiagram
    actor User
    Task->>LLM: Prelude
    loop
        User->>+Task: Prompt
		alt When recall is enabled
			Task->>LLM: Recalled memory items (based on user prompt)
		end
        Task->>+LLM: Prefix
        Task->>LLM: Prompt
        Task->>LLM: Postfix

        note over LLM: Generate until stop token occurs or token/context limit reached
        LLM->>-Task: Response
        Task->>-User: Completion
    end

When biasing is enabled, an optional bias prompt can be configured. When configured the model will be asked to generate a response (following the flow as shown above). This response is however not directly returned to the user. Instead, the bias prompt is then fed, after which the biaser is enabled (and the biased response is returned to the user).

sequenceDiagram
    actor User
    Task->>LLM: Prelude
    loop
        User->>+Task: Prompt
		alt When recall is enabled
			Task->>LLM: Recalled memory items (based on user prompt)
		end
        Task->>+LLM: Prefix
        Task->>LLM: Prompt
        Task->>LLM: Postfix

        note over LLM: Generate until stop token occurs or token/context limit reached
        Task->>LLM: Bias prompt
        note over LLM: Generate with biaser enabled

        LLM->>-Task: Response (biased)
        Task->>-User: Completion
    end

Poly is divided into separate crates that can be used independently:

• poly-server: Serve LLMs through HTTP and WebSocket APIs (provides llmd)
• poly-backend: Back-end implementation of LLM tasks
• poly-extract: Crate for extracting plaintext from various document types
• poly-bias: Crate for biasing LLM output to e.g. JSON following a schema
• poly-ui: Simple desktop UI for local LLMs

Applications that want to employ Poly's functionality should use the HTTP REST API exposed by poly-server. Rust applications looking to integrate Poly's capabilities could also depend on poly-backend directly.

flowchart TD

subgraph Poly
	PW[poly web-client]
	PS[poly-server]
	PB[poly-backend]
	PE[poly-extract]
	PU[poly-ui]
	PBs[poly-bias]

	PW-->|HTTP,WS,SSE|PS
	PB-->PBs
	PS<-.->PE

	PS-->PB
	PU-->PB
end

LLM[rustformers/llm]
PBs<-.->LLM
PB<-.->Qdrant

T[tokenizers]
GGML
Metal
PB-->LLM
LLM<-.->T
LLM-->GGML
GGML-->Metal
GGML-->CUDA

• Tommy van der Vorst (vandervorst@dialogic.nl)

Poly is licensed under the Apache 2.0 license from Git revision f329d35 onwards (only). Kindly note that this license provides no entitlement to support whatsoever. If you have any specific licensing or support requirements, please contact the authors.

We will accept contributions only if accompanied with a written statement that the contributed code is also (at least) Apache 2.0 licensed.

Specific licenses apply to the following files:

• ./data/gpt2.bin: MIT, source here, license here.
• ./data/gpt2-small-dutch-f16.bin: Apache 2.0, source here, license here.