Building a knowledge base

A common pattern: embed a set of documents once, store the vectors, and at query time retrieve the closest chunks and hand them to a chat model as context. This gateway’s /v1/embeddings endpoint (served by nomic-embed-text, the local pool’s only embedding model) covers the first half; /v1/chat/completions covers the second.

1. Chunk and embed your documents

Split long documents into passages of a few hundred words before embedding. nomic-embed-text, like most embedding models, works best on focused chunks rather than whole documents, and small chunks are what get you the precise top-k results retrieval depends on.

from openai import OpenAI
client = OpenAI(base_url="https://ai.rcfi.co.ke/v1", api_key="tak_your_key_here")
chunks = [
    "The Public Finance Management Act, 2012 establishes...",
    "County governments must submit budget estimates by...",
    "The Controller of Budget approves withdrawals from...",
]
response = client.embeddings.create(model="nomic-embed-text", input=chunks)
vectors = [row.embedding for row in response.data]
# Persist (vector, chunk_text, source) triples in whatever store you use:
# a Postgres table with pgvector, SQLite + a brute-force cosine scan, or a
# dedicated vector database. The gateway's job ends at producing the vector.

input accepts a single string or an array, batch related chunks into one call rather than one request per chunk where you can.

2. Retrieve at query time

Embed the incoming question with the same model, then rank your stored vectors by cosine similarity and take the top few.

import numpy as np
def cosine(a, b):
    a, b = np.array(a), np.array(b)
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))
query = "When are county budget estimates due?"
query_vector = client.embeddings.create(model="nomic-embed-text", input=query).data[0].embedding
scored = sorted(store, key=lambda row: cosine(query_vector, row.vector), reverse=True)
top_chunks = [row.text for row in scored[:4]]

3. Answer with retrieved context

Pass the retrieved chunks as a system message ahead of the user’s question: plain prompt construction, no special retrieval API on the chat side.

context = "\n\n".join(top_chunks)
response = client.chat.completions.create(
    model="ministral-3:3b",
    messages=[
        {
            "role": "system",
            "content": f"Answer only from this context. If it doesn't cover the question, say so.\n\n{context}",
        },
        {"role": "user", "content": query},
    ],
)
print(response.choices[0].message.content)

Keeping it in one thread

If the same conversation needs to ask several follow-up questions against the same retrieved context, open a thread with the context as its system instructions and use POST /v1/threads/{id}/continue for each turn instead of re-sending the context and prior turns on every call.

Both /v1/embeddings and /v1/chat/completions are subject to the same per-key rate limits and monthly token budget. batch embedding calls count toward the same budget as chat calls.