How We Finally Stopped Treating Trends Like Stray Events

I was staring at our trend detection system when something clicked: we’d been treating outliers like patterns. A single spike in deployment frequency, a one-off refactor, a random config change—our old pipeline grabbed these and labeled them “trends.” We weren’t detecting patterns. We were collecting noise.

The fix came during the Trend Analysis project overhaul. We needed to stop extracting trends from individual events and start identifying structural patterns from event clusters instead.

Here’s what actually happened: I sat down with the HDBSCAN clustering output and realized we had real clusters—groups of related events that actually meant something. A cluster of “config changes” across multiple services. A cluster of “security patches.” A cluster of “database optimization attempts.” These clusters deserved analysis, not the random single events we’d been fishing out before.

The new approach—ADR v5—extracts 0 to 3 structural patterns per cluster. Each pattern gets evidence: which events support it, whether the change is up or down, what type of signal it is, metrics, the key players involved. We also started assigning domain tags to events (3-5 broad categories like “infrastructure,” “performance,” “security”) without any extra LLM calls—they come free from the extraction prompt itself.

The tricky part was matching new incoming events to existing trends. We went hybrid: check embedding similarity (threshold 0.55) and look for entity/tag overlap. It’s not perfect, but it catches the real patterns and ignores the noise.

We also killed Level 1 entity-based trend extraction entirely. It was generating false positives like a broken smoke detector. Sometimes less is more.

The migration was thorough—new tables for event_domain_tags, trend_events, plus extra columns in the trends table. We had to be careful with Ollama routing: dual-port setup, mutex locks, keep-alive set to “999h” to avoid connection thrashing, chunk sizes tuned to 5.

Testing on production data gave us 14 legitimate trends extracted from 5 clusters, with 56 events linked back to those trends. Not a massive number, but every single one made sense. No ghost patterns. No random events masquerading as trends.

What do you call a group of 8 Hobbits? A Hobbyte. 😄