SIGNALS Docs
Exit Reader View
100%
Theme

AI embeddings & configuration

Native vector embeddings for semantic search and similar-record panels — provider setup, scheduling, cost controls, privacy, and graceful degradation.

Overview

Signals can generate vector embeddings for core records so global search can blend semantic matches with full-text ranking, and so detail pages can surface similar rentals, accounts, and catalogue items.

Embeddings are optional and disabled by default on fresh installations. Only the account owner can enable them and grant consent for the selected configuration. Keyword search remains available while embeddings are disabled, lack consent, have no required API key, lack pgvector, or are paused.

What gets embedded

Core: Rentals, Accounts, Catalogue items

Optional types: Activities, Conversations (+ comments), Invoices, Credit notes, Purchase orders, Equipment test results, Repairs, Attachments. Core types are enabled when embeddings are enabled; each optional type requires an owner selection in Settings → Integrations → AI. The ai.additional_types setting accepts registered morph aliases and defaults to an empty array. The AI tab shows approximate record counts, vector storage and input tokens before selection. Vector storage refreshes as you select a provider/model or enter a custom endpoint's dimensions, before saving. PostgreSQL estimates use table statistics, so run ANALYZE after a large import; token estimates assume 256 tokens per record and exclude index storage.

Each model implements the Embeddable contract via toEmbeddableText(). Content is hashed so status/total-only updates do not cost tokens.

Configuration

Supported labs

Lab (SIGNALS_AI_PROVIDER) Models Default model Dimensions
openai text-embedding-3-small, text-embedding-3-large text-embedding-3-small 1536 / 3072
gemini gemini-embedding-001 gemini-embedding-001 3072
mistral mistral-embed mistral-embed 1024
openai_compatible any (you supply the id) — any (you supply the width)

Anthropic and Moonshot are not offered — those labs expose no embeddings API. If stored settings still reference a removed lab, Signals falls back to the default provider/model and logs a warning.

Bring your own endpoint (openai_compatible)

The three hosted labs above all mean your record text crosses the network to a company you do not run. openai_compatible is the way out of that: point Signals at any server that speaks the OpenAI embeddings contract — POST {base_url}/embeddings taking {model, input, dimensions} and returning {data: [{embedding: [...]}], usage: {prompt_tokens}} — and it becomes the provider.

Run that server yourself and no record text leaves your infrastructure at all. This is the same posture the MCP server documents from the other direction: core Signals ships no model and sends no telemetry, and embeddings are the one place it sends record text out (its other outbound calls are listed in the MCP page) — so this setting is what decides whether "out" means the public internet or a box on your own network.

Local examples (require EMBEDDINGS_ENDPOINT_TRUST=local on self-hosted installs):

Host Base URL Example model
Ollama http://127.0.0.1:11434/v1 nomic-embed-text, mxbai-embed-large
vLLM http://127.0.0.1:8000/v1 intfloat/e5-mistral-7b-instruct
LM Studio http://127.0.0.1:1234/v1 text-embedding-nomic-embed-text-v1.5
LocalAI http://127.0.0.1:8080/v1 bert-embeddings
HF text-embeddings-inference http://127.0.0.1:8080/v1 whatever the container serves
Any OpenAI API gateway (LiteLLM, OpenRouter, Azure-fronted proxies) your gateway URL your gateway's model id

Three things you must get right, because there is no catalogue to fall back on:

  • Base URL — the root the /embeddings path is appended to. Include the version segment (/v1) if your server uses one. Required whenever this lab is selected.
  • Model — the id exactly as your endpoint names it. Never coerced to a default.
  • Dimensions — the vector width your model actually returns. A mismatch does not error at write time; it silently poisons similarity ranking and the HNSW expression index, so check it against your model card.

The API key is optional for this lab. Most local servers ignore auth, so an empty key is treated as configured as long as the base URL is set (Signals sends a placeholder bearer token so the header stays well-formed). Set a key when your gateway needs one.

Changing the base URL or dimensions changes the consented configuration and vector space. Generation requires matching owner consent before a full re-embed can run.

Endpoint trust and refusals

embeddings.endpoint_trust defaults to public: HTTPS with verified certificates, port 443, and exclusively public DNS addresses. An operator may set EMBEDDINGS_ENDPOINT_TRUST=local for a self-hosted install. Local trust permits HTTP, loopback and private addresses on ports 443, 80, 11434, 8000, 8080, 1234 and 8081. The lists live in embeddings.endpoint_ports. This opt-in is env-only; there is no admin control, and Cloud always enforces public trust regardless of env or stored flags.

Both classes reject metadata services, link-local, multicast and reserved destinations. URLs must have an ASCII hostname or canonical IPv4 address; IPv6 literals, zone identifiers, encoded hosts, trailing dots and numeric address shortcuts are refused. DNS may resolve to permitted IPv6 addresses. An API root may have up to four short alphanumeric, hyphen or underscore path segments (128 characters total), such as /v1. Credentials in the URL, query strings, fragments, dot segments, encoded paths and double slashes are refused. Supply credentials only through the encrypted API key field.

Saving an invalid URL displays the specific refusal and leaves the saved settings unchanged. Invalid env or stored URLs disable embeddings while preserving recorded consent. Configuration and consent checks validate the URL statically without DNS. Unresolvable hosts and provider failures leave keyword search available with the paused notice; sync items retry without deleting stored vectors or cancelling the whole batch. Repairing a URL does not grant consent automatically.

Each BYO call resolves every DNS address again and pins its connection to a validated address. Redirects are errors, including redirects to metadata services; certificate errors never downgrade to HTTP. TLS verifies the original hostname. Refusals before dispatch spend no tokens, and known unbilled failures release reservations. Provider diagnostics are replaced with a safe exception and provider-failure logs contain exception types only. Refusal logs deduplicate each reason per active request and include only the reason, scheme and host, without credentials, paths, query strings, response bodies, headers or API keys.

These controls govern the destination only. Payload redaction follows the separate payload redaction policy; endpoint trust does not change what is embedded.

Environment variables

Variable Purpose
SIGNALS_AI_API_KEY Provider API key (Cloud: required; self-hosted: optional override path is settings). Optional for openai_compatible
SIGNALS_AI_PROVIDER Lab key: openai, gemini, mistral, or openai_compatible
SIGNALS_AI_EMBEDDING_MODEL Model id for that lab
SIGNALS_AI_EMBEDDING_DIMENSIONS Optional dimension override (otherwise catalogue default); effectively required for openai_compatible
SIGNALS_AI_BASE_URL Endpoint root for openai_compatible only — ignored by the hosted labs, whose URLs are fixed

Resolution order for provider, model, dimensions, and base URL: env (when set) → admin settings repository (self-hosted only) → config / catalogue default.

A fully self-contained local setup:

SIGNALS_AI_PROVIDER=openai_compatible
SIGNALS_AI_BASE_URL=http://127.0.0.1:11434/v1
EMBEDDINGS_ENDPOINT_TRUST=local
SIGNALS_AI_EMBEDDING_MODEL=nomic-embed-text
SIGNALS_AI_EMBEDDING_DIMENSIONS=768

Self-hosted

Admin → Preferences → Integrations → AI embeddings

  • Write-only API key, encrypted at rest (optional for openai_compatible); saved credentials are never returned to the form
  • Lab / provider and embedding model (table above)
  • Endpoint base URL and vector dimensions (shown only for openai_compatible)
  • Full refresh cadence (weekly or monthly)
  • Optional monthly token budget
  • Usage log retention and vector hash revalidation interval

Changing lab or model requires fresh owner consent before generation resumes. Similarity queries always filter to the active provider/model.

SIGNALS_AI_PROVIDER, SIGNALS_AI_EMBEDDING_MODEL, and SIGNALS_AI_EMBEDDING_DIMENSIONS override the admin settings when set (useful for ops / staging without changing DB settings).

Signals Cloud

Provider credentials and model selection are env-only (SIGNALS_AI_*). Owner consent is still required before embeddings run; configured credentials alone do not grant it. The provider settings form and usage totals are hidden in the admin UI.

Scheduling

Job When Purpose
Daily sync 01:30 UTC Embed new/changed records since last run
Full refresh Weekly or monthly (setting) Re-validate hashes and backfill gaps

Jobs run on the embeddings queue. The embeddings worker uses dedicated database or Redis connections with a retry window longer than the 600-second loader timeout; set EMBEDDINGS_QUEUE_CONNECTION to override that connection. Each refresh or sync batch starts with a loader that reads IDs in bounded chunks and adds work while the batch remains pending. embeddings.batch_size defaults to 100 inputs per provider request; each text respects the model's input limit using a conservative byte bound, and embeddings.batch_max_input_tokens caps each request's combined inputs (100,000 by default). Longer chunks split into multiple requests. A request reserves its combined token budget once, then writes vectors together after checking consent and each record's current privacy restrictions and text hash. Unchanged records skip generation, and a retryable record keeps the daily sync watermark unchanged.

Semantic query vectors are cached for ten minutes by tenant, provider configuration, model, dimensions and normalized redacted text. Result visibility is checked on every request. Cached vectors contain no result IDs or permission decisions.

Cost logging & budgets (self-hosted)

Every AI call through the shared client is logged to ai_usage_logs (function, provider, model, tokens). The AI tab shows totals for the last 7 days, last 30 days, this month, last month, and year-to-date.

An optional monthly token budget pauses sync and notifies admins when exceeded. Logs are pruned like action logs (signals:ai-usage:prune).

PostgreSQL / pgvector

Production similarity ranking requires PostgreSQL with the pgvector extension. The migration enables the extension when available and creates a dimensionless vector column so provider/model swaps do not need a schema change. SQLite (and pgsql without the extension) keep a text column so the schema contract is identical. Without pgvector, similarity features stay off.

The HNSW cosine index is not created by the migration (HNSW requires a fixed dimension). When an AI model with known dimensions is activated — or a full re-embed runs — Signals creates an expression index (embedding::vector(N)) and similarity queries use the same cast so the index is used.

Privacy

Embeddable text is sent to the selected lab. The account owner sees an informed-choice notice on the AI tab before granting consent, and that notice changes wording for openai_compatible because the destination is one you chose. With a self-run endpoint the text never leaves your network; with a hosted lab or a third-party gateway, that operator receives it.

Consent identifies the owner who accepted, the acceptance time, policy version and selected configuration. Consent grants and revocations create audit entries. Disabling embeddings revokes active consent and queues vector cleanup; enabling again requires fresh owner consent after cleanup. Changing provider, model, endpoint or dimensions requires consent for that configuration. A settings-management permission alone does not allow another role to enable embeddings or grant consent.

Graceful degradation

Condition Behaviour
No API key / disabled Sync no-ops (one log line); search is full-text only; similar panels hidden
No pgvector Same as above for similarity
Budget paused Sync skipped until cleared
Record not yet embedded Full-text / existing UI unchanged for that record

API

The embeddings settings group is ai: read it at GET /api/v1/settings/ai and update it with PATCH /api/v1/settings/ai (or PUT). Normal settings permissions and token abilities apply in addition to the owner requirement for enabling and consent changes. REST and MCP settings reads expose api_key_set, a boolean indicating whether a key is configured, and never include api_key, masked text or ciphertext. Supply api_key only when writing a credential; omit it to keep the configured credential. The integrations form also leaves the credential input empty after saving.

An owner update with enabled: true records consent for the saved configuration. Settings responses include read-only consented_by (the accepting user's ID) and consented_at (an ISO 8601 timestamp). Both are null when no active consent remains. An update with enabled: false revokes consent and clears these response fields; the revoked consent evidence remains available internally for accountability. The integrations page provides a separate owner consent action and displays the accepting owner and time.

Semantic search matches global search:

MethodURLSummaryAbilityWeight
GET/api/v1/searchIndex searchsearch:read2
GET/api/v1/search/semanticSemantic searchsearch:read4

Eligible embeddable list endpoints accept filter[similar_to]=… (free text, a numeric id of the same entity type, or alias:id, for example rental:12). Record references require an alias registered in the searchable record registry and a numeric id. Other colon-containing values, such as LED: 6m, are searched as free text; PHP class names are never resolved from the input.

Use the entity aliases shown in the API reference. Only searchable record types can be used as similarity sources.

Response DTOs for rentals, accounts, and catalogue items include embedded_at and embedding_model (never the raw vector by default).

Adding a new embeddable model

  1. Add the record type to the embedding contract and enable its embedding behaviour
  2. Override toEmbeddableText()
  3. Register the record type in the searchable record registry

Vector recall and hash revalidation

Similarity retrieval over-fetches up to five times the requested count per type, with a configured candidate ceiling. Searchable types join their indexed warehouse visibility keys before nearest-neighbour retrieval. Each requested type receives its own candidate allowance. Current source scopes, policies and privacy restrictions remain authoritative after retrieval. HNSW uses a transaction-local hnsw.ef_search (200 by default, raised for larger candidate requests); it does not change the database session setting.

signals:embeddings:prune removes source orphans and queues bounded hash checks for vectors beyond ai.retention_months. Valid, unchanged vectors remain in place without another provider request; successful checks advance validated_at. Changed text is regenerated through the same consent, token-budget and rate-limit checks as scheduled sync. Subject erasure and source deletion still remove vectors immediately. The command runs through the tenant scheduler on Cloud.

Fresh installations use the Core tier. Optional generation and text search stop when a type is deselected. The registry retains every supported type for source resolution, erasure and orphan cleanup.