Scheduled Automation

Scheduled Log-Anomaly Sentinel with Vector Search

A recurring job that embeds a baseline of healthy log lines into Qdrant, then on each run flags lines whose nearest-neighbor distance is far from the baseline as anomalies, summarizes the clusters with an LLM, and notifies the on-call channel before the anomaly becomes an incident.

What This Builds

Most log alerting is rule-based: you write a regex or a threshold for failures you already know about. New failure modes slip through until someone notices the dashboard. This recipe builds a sentinel that catches novel log lines — entries that do not look like anything in your healthy baseline — using vector similarity instead of hand-written rules.

The idea, following Qdrant’s anomaly-detection approach: embed a corpus of known-good log lines into a vector store, then for each new line measure how far its nearest baseline neighbor is. Lines that sit far from everything healthy are anomalies. A Trigger.dev scheduled task runs this on an interval, an LLM summarizes the flagged clusters into a readable note, and Slack gets pinged only when something genuinely unfamiliar appears.

The Stack

  • Qdrant is the vector search engine. It stores baseline log embeddings and supports similarity (and dissimilarity) queries with metadata filtering, which is what powers the “how far from normal is this?” check.
  • Embeddings (OpenAI, or any model) turn each log line into a vector. The baseline is built from a window of healthy logs.
  • Trigger.dev runs the recurring scheduled task as a durable background job, so a long embedding pass survives restarts.
  • Slack is the notification surface for confirmed anomalies.

Step-by-Step Outline

  1. Build the baseline: collect a window of logs from a period you know was healthy, embed each line, and upsert the vectors into a Qdrant collection (with service, level, and timestamp as payload metadata).
  2. Schedule the sentinel as a Trigger.dev scheduled task (for example every 15 minutes).
  3. Score new lines: embed the latest batch, query Qdrant for each line’s nearest baseline neighbor, and keep lines whose distance exceeds a threshold as candidate anomalies.
  4. Cluster and summarize: group the anomalies and ask an LLM for a short, human-readable summary of what is new and which service it touches.
  5. Notify selectively: post to Slack only when the anomaly count or severity crosses a bar, with example lines and metadata, to avoid alert fatigue.
  6. Refresh the baseline on a slower cadence so expected drift (new normal behavior) gets absorbed instead of alerting forever.

The vector store doubles as a searchable knowledge base of past log signatures, so a responder can later ask “have we seen this line before?”

Source