Blog
Posts about the development process, solved problems and learned technologies
DNS Cache Poisoning: Why AdGuard Refused to See New Records
# DNS Cache Wars: When AdGuard DNS Holds Onto the Past The borisovai-admin project was running smoothly until authentication stopped working in production. The team had recently added new DNS records for `auth.borisovai.tech` and `auth.borisovai.ru`, pointing to the server at `144.91.108.139`. Everything looked correct on paper—the registrars showed the records, Google's public DNS resolved them instantly. But AdGuard DNS, the resolver configured in their infrastructure, kept returning NXDOMAIN errors as if the records didn't exist. The detective work started with a DNS audit. I ran queries against multiple resolvers to understand what was happening. Google DNS (`8.8.8.8`) immediately returned the correct IP address for both authentication domains. AdGuard DNS (`94.140.14.14`), however, flat-out refused to resolve them. Meanwhile, the `admin.borisovai.tech` domain resolved fine on both services. The pattern was clear: something was wrong, but only for the authentication subdomains and only through one resolver. The culprit was **DNS cache poisoning**—not malicious, but equally frustrating. AdGuard DNS was holding onto old NXDOMAIN responses from before the records were created. When the DNS entries were first added to the registrar, AdGuard's cache had already cached a negative response saying "these domains don't exist." Even though the records now existed upstream, AdGuard was serving stale cached data, trusting its own memory more than reality. This is a common scenario in distributed DNS systems. When a domain doesn't exist, DNS servers cache that negative result with a TTL (Time To Live), often defaulting to an hour or more. If new records are added during that window, clients querying that caching resolver won't see them until the cached NXDOMAIN expires. The immediate fix was simple: flush the local DNS cache with `ipconfig /flushdns` on Windows clients to clear stale entries. For a more permanent solution, we needed to either wait for AdGuard's cache to naturally expire (usually within an hour) or temporarily switch to Google DNS by manually setting `8.8.8.8` in network settings. The team chose to switch DNS servers while propagation completed—a pragmatic decision that got authentication working immediately without waiting. What seemed like a mysterious resolution failure turned out to be a textbook case of DNS cache semantics. The lesson: when DNS behaves unexpectedly, check multiple resolvers. Different caching strategies and update schedules mean that not all DNS services see the internet identically, especially during transitions. 😄 The generation of random DNS responses is too important to be left to chance.
DNS Resolution Chaos: Why Some Subdomains Vanish While Others Thrive
# DNS Mysteries: When One Subdomain Works and Others Vanish The `borisovai-admin` project was running smoothly on the main branch, but there was a catch—a frustrating one. `admin.borisovai.tech` was responding perfectly, resolving to `144.91.108.139` without a hitch. But `auth.borisovai.tech` and `auth.borisovai.ru`? They had simply disappeared from the internet. The task seemed straightforward: figure out why the authentication subdomains weren't resolving while the admin panel was working fine. This kind of infrastructure puzzle can turn into a time sink fast, so I needed a systematic approach. **First, I checked the DNS records directly.** I queried the DNS API expecting to find `auth.*` entries sitting quietly in the database. Instead, I found an empty `records` array—nothing. No automatic creation of these subdomains meant something in the provisioning logic had fallen through the cracks. The natural question followed: if `auth.*` records aren't in the API, how is `admin.borisovai.tech` even working? **The investigation took an unexpected turn.** I pulled out Google DNS (8.8.8.8) as my truth source and ran a resolution check. Suddenly, `auth.borisovai.tech` resolved successfully to the same IP address: `144.91.108.139`. So the records *existed* somewhere, but not where I was looking. This suggested the DNS configuration was either managed directly at the registrar level or there was a secondary resolution path I hadn't accounted for. **Then came the real discovery.** When I tested against AdGuard DNS (94.140.14.14)—the system my local environment was using—the `auth.*` records simply didn't exist. This wasn't a global DNS failure; it was a caching or visibility issue specific to certain DNS resolvers. The AdGuard resolver wasn't seeing records that Google's public DNS could find immediately. I ran the same check on `auth.borisovai.ru` and confirmed the pattern held. Both subdomains were missing from the local DNS perspective but present when querying through public resolvers. This pointed to either a DNS propagation delay, a misconfiguration in the AdGuard setup, or records that were registered at the registrar but not properly distributed to all nameservers. **Here's an interesting fact about DNS that caught me this time:** DNS resolution isn't instantaneous across all servers. Different DNS resolvers maintain separate caches and query different authoritative nameservers. When you change DNS records, large providers like Google cache globally, but smaller or regional DNS services might take hours to sync. AdGuard, while excellent for ad-blocking, might not have the same authoritative nameserver agreements as Google's public DNS, creating visibility gaps. The fix required checking the registrar configuration and ensuring that `auth.*` records were properly propagated through all authoritative nameservers, not just cached by some resolvers. It's a reminder that DNS is often the last place developers look when something breaks—but it should probably be the first. --- 😄 Why did the DNS administrator break up with their partner? They couldn't handle all the unresolved entries in their relationship.
QR Code Gone: Authelia's Silent Fallback Mode Revealed
# When Your QR Code Hides in Plain Sight: The Authelia Debug That Saved the Day The **borisovai-admin** project needed two-factor authentication, and Authelia seemed like the perfect fit. The deployment went smoothly—containers running, certificates in place, configuration validated. Then came the critical test: click "Register device" to enable TOTP, and a QR code should appear. Instead, the browser displayed nothing but an empty void. I started in the obvious places. Browser console? Clean. Authelia logs? No errors screaming for attention. API responses? All successful HTTP codes. The registration endpoint was processing requests flawlessly, generating tokens, doing exactly what it should—yet somehow, no QR code materialized on screen. The system was working perfectly while simultaneously failing completely. Thirty minutes into chasing ghosts through log files and configuration documents, something clicked. I noticed a single line that had been hiding in plain sight: **`notifier: filesystem`**. That innocent parameter changed everything. The story behind this configuration is deceptively simple. When Authelia is deployed without email notifications properly configured, it doesn't crash or loudly complain. Instead, it shifts gracefully to a fallback mode designed for local development. Rather than sending registration links via SMTP, SendGrid, or any external service, it writes them directly to the server's filesystem. From Authelia's perspective, the job is done perfectly—the registration URL is generated, secured with a cryptographic token, and safely stored in `/var/lib/authelia/notifications.txt`. From the user's perspective, they're staring at a blank screen. The fix required thinking sideways. Instead of expecting Authelia to magically display the QR code through some non-existent UI mechanism, I needed to retrieve the notification directly from the server. A single SSH command revealed everything: ``` cat /var/lib/authelia/notifications.txt ``` There it was—the full registration URL with the token embedded. I opened it in a browser, and suddenly the QR code materialized. Scan it with Google Authenticator, and the entire flow worked perfectly. **Here's what made this moment instructive:** Authelia's design isn't a bug or a limitation—it's a deliberate choice for development environments. The `filesystem` notifier eliminates the need to configure SMTP servers, manage API credentials for email services, or spin up complex testing infrastructure. It's honest about what it's doing. The real lesson is that **configuration choices have invisible consequences**. A setting that makes perfect sense for development creates silent failures in testing. The system works flawlessly; the alignment between system behavior and user expectations simply vanishes. The fix was immediate—reconfigure the notifier to use proper email or document the behavior clearly. Either way, the next developer wouldn't need to hunt QR codes through the filesystem like digital treasure maps. --- A programmer puts two glasses on his bedside table before going to sleep: a full one in case he gets thirsty, and an empty one in case he doesn't. 😄
From 83.7% to 85%: Architecture and Optimizer Choices Matter
# Chasing That Last 1.3%: When Model Architecture Meets Optimizer Reality The CIFAR-10 accuracy sat stubbornly at 83.7%, just 1.3 percentage points shy of the 85% target. I was deep in the `llm-analysis` project, staring at the training curves with that peculiar frustration only machine learning developers understand—so close, yet somehow impossibly far. The diagnosis was clear: the convolutional backbone needed more capacity. The model's channels were too narrow to capture the complexity required for those final critical percentages. But this wasn't just about arbitrarily increasing numbers. I needed to make the architecture **configurable**, allowing for flexible channel widths without redesigning the entire network each time. First, I refactored the model instantiation to accept configurable channel parameters. This is where clean architecture pays dividends—instead of hardcoding layer dimensions, I could now scale the backbone horizontally. I widened the channels across the network, giving the model more representational power to learn those nuanced features that separate 83.7% from 85%. Then came the optimizer revelation. The training script was still using **Adam**, the ubiquitous default for deep learning. But here's the thing about CIFAR-10—it's a dataset where **SGD with momentum** has historically outperformed Adam for achieving those final accuracy gains. The switch wasn't arbitrary; it's a well-known pattern in the computer vision community, yet easy to overlook when you're in the flow of incremental improvements. This revealed a deeper architectural issue: after growth events in the training pipeline (where the model dynamically expands), the optimizer gets rebuilt. The code was still initializing Adam in those rebuilds. I had to hunt down every instance—the primary optimizer loop, the Phase B optimizer updates—and swap them all to SGD with momentum hyperparameters. Each change felt small, but they compounded into a coherent optimization strategy. While I was optimizing the obvious, I spotted something lurking in the **RigL sparsity implementation**—the sparse training mechanism was overshooting its target sparsity levels slightly. RigL (Rigged Lottery Learning) uses dynamic sparse training to prune connections during training, but when the sparsity calculations drift even marginally from their targets, it can destabilize convergence. I traced through the sparsity growth schedule, checking where the overshoot accumulated. **Here's something fascinating about Adam optimizers:** they were introduced in 2014 by Kingma and Ba, and they became the default across industry precisely because they're forgiving and work well across diverse problems. But this universality is also their weakness in specialized domains. For image classification on small, well-curated datasets like CIFAR-10, simpler first-order optimizers with momentum often achieve better final accuracies because they converge to sharper minima—a phenomenon that still fascinates researchers today. By the end of the session, the pieces were in place: wider channels, consistent SGD with momentum, and fixed sparsity behavior. The model wasn't fundamentally different, but it was now optimized for what CIFAR-10 actually rewards. Sometimes closing that last percentage point gap isn't about revolutionary changes—it's about aligning every component toward a single goal. 😄 Hunting down every optimizer instance in your codebase after switching algorithms is like playing Where's Waldo, except Waldo is your bug and the entire technical documentation is the book. What to fix: - Punctuation: missing or extra commas, periods, dashes, quotes - Spelling: typos, misspelled words - Grammar: subject-verb agreement, tense consistency, word order - Meaning: illogical phrases, incomplete sentences, repeated ideas, inconsistent narrative - Style: replace jargon with clearer language, remove tautologies Rules: - Return ONLY the corrected text, no comments or annotations - Do NOT change structure, headings, or formatting (Markdown) - Do NOT add or remove paragraphs or sections - Do NOT rewrite the text — only targeted error fixes - If there are no errors — return the text as is
QR Code Mystery: Why Authelia's Registration Silently Failed
# When Your QR Code Hides in Plain Sight: Debugging Authelia's Silent Registration The borisovai-admin project needed two-factor authentication, and Authelia seemed like the perfect fit. The deployment went smoothly—containers running, certificates in place, configuration validated against the docs. Then came the test: click "Register device" to enable TOTP, and a QR code should appear on screen. Instead, the browser displayed nothing but an empty canvas. The obvious suspects got interrogated first. Browser console? Clean. Authelia logs? No errors. API responses? All successful. The registration endpoint was processing requests correctly, generating tokens, doing exactly what it should—yet somehow, no QR code materialized on the user's screen. It was like the system was working perfectly while simultaneously failing completely. After thirty minutes of chasing ghosts through log files, something clicked: **the configuration was set to `notifier: filesystem`**. That innocent line in the config file changed everything. When Authelia is deployed without email notifications configured, it doesn't scream about it or fail loudly. Instead, it silently shifts to a fallback mode designed for local development. Rather than sending registration links via SMTP or any external service, it writes them directly to a file on the server's filesystem. From Authelia's perspective, the job is done perfectly—the QR code URL is generated, secured with a token, and safely stored in `/var/lib/authelia/notifications.txt`. From the user's perspective, they're staring at a blank screen. The fix required thinking sideways. Instead of expecting Authelia to display the QR through some non-existent UI element, the answer was to retrieve the notification directly from the server. A single SSH command—`cat /var/lib/authelia/notifications.txt`—exposed the full registration URL. Open that link in a browser, and there it was: the QR code that had been sitting on the server all along, waiting to be discovered. What makes this moment worth noting is what it reveals about infrastructure thinking. **Configuration isn't just about making things work; it's about making them work the way users expect.** Authelia was functioning flawlessly. The system was honest about what it was doing. The disconnect happened because the notifier configuration wasn't aligned with the deployment context. The solution meant either reconfiguring Authelia to use proper email notifications or documenting this filesystem fallback for the admin team. Either way, the mystery evaporated once we understood that sometimes the most elegant features of a system aren't bugs—they're just hiding in files instead of browsers. A comment was added to the project configuration explaining the `filesystem` notifier behavior and linking to the retrieval command. Next time a developer encounters this scenario, they won't spend half an hour wondering where their QR code went. Why did the Authelia developer get stuck in troubleshooting? They were looking for notifications in all the wrong places—literally everywhere except the filesystem!
When Authelia Whispers Instead of Speaks: The QR Code Mystery
# Authelia's Silent QR Code: A Lesson in Configuration Over Magic The task seemed straightforward enough: set up two-factor authentication for the borisovai-admin project using Authelia. The authentication server was running, the configuration looked solid, and the team was ready to enable TOTP-based device registration. But when a user clicked "Register device," nothing happened. No QR code appeared. Just silence. The natural first instinct was to assume something broke. Maybe the TOTP endpoint wasn't responding? Perhaps there was a network issue? But after digging through the Authelia logs and checking the API responses, everything appeared to be working correctly. The registration request was being processed, the system acknowledged it—yet no visual feedback reached the user. That's when the real issue revealed itself: **Authelia was configured with `notifier: filesystem`**. Here's where most developers would have a moment of clarity mixed with mild embarrassment. When you deploy Authelia without configuring email notifications, it defaults to writing registration links directly to the filesystem instead of sending them via email. It's a sensible fallback for development environments, but it creates a peculiar situation in production. The authentication server diligently generates the QR code registration URL and writes it to a notification file on the server—but there's no automatic mechanism to display it back to the user's browser. The solution required a bit of lateral thinking. Rather than trying to force Authelia to display the QR code through some non-existent UI element, the developer needed to retrieve the notification from the server filesystem directly. A simple SSH command would read the contents of `/var/lib/authelia/notifications.txt`, exposing the full registration URL that Authelia had generated. That URL, when visited in a browser, would display the actual QR code needed for TOTP enrollment. This discovery illustrates something fundamental about infrastructure configuration: **there's a difference between a system working and a system working as expected**. Authelia was functioning perfectly according to its configuration. The QR code existed—it was just living in a text file on the server instead of being rendered in the browser. The real lesson wasn't about debugging code; it was about understanding the downstream implications of configuration choices. For the borisovai-admin project, this meant either reconfiguring Authelia to use proper email notifications or documenting this workaround for the admin team. Either way, the silent mystery became a teaching moment about reading documentation carefully and understanding what your configuration files actually do. Sometimes the hardest bugs to find are the ones where nothing is actually broken—they're just misconfigured in ways that create invisible friction. 😄
Double Lock: Adding TOTP 2FA to Authelia Admin Portal
# Securing the Admin Portal: A Two-Factor Authentication Setup Story The `borisovai-admin` project had reached a critical milestone—the authentication layer was working. The developer had successfully deployed **Authelia** as the authentication gateway, and after weeks of configuration, the login system finally accepted credentials properly. But there was a problem: a production admin portal with single-factor authentication is like leaving the front door unlocked while keeping valuables inside. The task was straightforward on paper but required careful execution in practice: implement **two-factor authentication (2FA)** to protect administrative access to `admin.borisovai.tech` and `admin.borisovai.ru`. This wasn't optional security theater—it was essential infrastructure hardening. The approach chosen was elegant in its simplicity. Rather than implementing a custom 2FA system, the developer leveraged **Authelia's built-in TOTP support** (Time-based One-Time Password). This decision traded absolute flexibility for proven security and minimal maintenance overhead. The setup followed a clear sequence: navigate to the **METHODS** section in Authelia's web interface, select **One-Time Password**, let Authelia generate a QR code, and scan it with a standard authenticator application—Google Authenticator, Authy, 1Password, or Bitwarden, take your pick. The interesting part emerged during implementation. The notification system for TOTP registration was configured to use **filesystem-based notifications** rather than SMTP. This meant the registration link wasn't emailed but instead written to `/var/lib/authelia/notifications.txt` on the server. It's a pragmatic choice for development and staging environments where mail infrastructure might not be available, though it would require a different approach—likely SMTP configuration—before production deployment. What made this particularly instructive was observing how authentication systems evolve. **TOTP itself is decades old**, originating from RFC 4226 (HOTP) in 2005 and standardized as RFC 6238 in 2011. Yet it remains one of the most reliable 2FA mechanisms precisely because it doesn't depend on network connectivity or external services. The time-based variant has no server-side state to maintain—just a shared secret between the authenticator device and the server, generating synchronized six-digit codes every thirty seconds. The developer's approach also highlighted a common misconception: assuming that 2FA implementation requires building custom infrastructure. In reality, most modern authentication frameworks like Authelia ship with production-ready TOTP support out of the box, eliminating months of potential security auditing and vulnerability patching. After the QR code was scanned and the six-digit verification code was entered, the system confirmed successful registration. The admin portal was now protected by a second authentication factor. The next phase would be ensuring the SMTP notification system is properly configured for production, so users receive their registration links via email rather than needing server-level file access. The lesson stuck: security improvements don't always require complexity. Sometimes they just need the right authentication framework and five minutes of configuration. 😄
Tunnels, Timeouts, and the Night the Infrastructure Broke
# Building a Multi-Machine Empire: Tunnels, Traefik, and the Night Everything Almost Broke The **borisovai-admin** project had outgrown its single-server phase. What started as a cozy little control panel now needed to orchestrate multiple machines across different networks, punch through firewalls, and do it all with a clean web interface. The task was straightforward on paper: build a tunnel management system. Reality, as always, had other ideas. ## The Tunnel Foundation I started by integrating **frp** (Fast Reverse Proxy) into the infrastructure—a lightweight reverse proxy perfect for getting past NAT and firewalls without the overhead of heavier solutions. The backend needed a proper face, so I built `tunnels.html` with a clean UI showing active connections and controls for creating or destroying tunnels. On the server side, five new API endpoints in `server.js` handled the tunnel lifecycle management. Nothing fancy, but functional. The real work came in the installation automation. I created `install-frps.sh` to bootstrap the FRP server and `frpc-template` to dynamically generate client configurations for each machine. Then came the small but crucial detail: adding a "Tunnels" navigation link throughout the admin panel. Tiny feature, massive usability improvement. ## When Your Load Balancer Becomes Your Enemy Everything hummed along until large files started vanishing mid-download through GitLab. The culprit? **Traefik's** default timeout configuration was aggressively short—anything taking more than a few minutes would get severed by the reverse proxy. This wasn't a bug in Traefik; it was a misconfiguration on my end. I rewrote the Traefik setup with surgical precision: `readTimeout` set to 600 seconds, a dedicated `serversTransport` configuration specifically for GitLab traffic, and a new `configure-traefik.sh` script to generate these dynamically. Suddenly, even 500MB archives downloaded flawlessly. ## The Documentation Moment While deep in infrastructure tuning, I realized the `docs/` folder had become a maze. I reorganized it into logical sections: `agents/`, `dns/`, `plans/`, `setup/`, `troubleshooting/`. Each folder owned its domain. I also created machine-specific configurations under `config/contabo-sm-139/` with complete Traefik, systemd, Mailu, and GitLab settings, then updated `upload-single-machine.sh` to handle deploying these configurations to new servers. ## Here's the Thing About Traefik Traefik markets itself as the "edge router for microservices"—lightweight, modern, cloud-native. What they don't advertise is that it's deeply opinionated about timing. A single misconfigured timeout cascades through your entire infrastructure. It's not complexity; it's *precision*. Get it right, and everything sings. Get it wrong, and users call you wondering why their downloads time out. ## The Payoff By the end of the evening, the infrastructure had evolved from single-point-of-failure to a scalable multi-machine setup. New servers could be provisioned with minimal manual intervention. The tunnel management UI gave users visibility and control. Documentation became navigable. Sure, Traefik had taught me a harsh lesson about timeouts, but the system was now robust enough to actually scale. The next phase? Enhanced monitoring, SSO integration, and better observability for network connections. But first—coffee. 😄 **Dev:** "I understand Traefik." **Interviewer:** "At what level?" **Dev:** "StackOverflow tabs open at 3 AM on a Friday level."
Authelia Authentication: From Bootstrap Scripts to Secure Credentials
# Authelia Setup: Securing the Admin Panel Behind the Scenes The borisovai-admin project needed proper authentication infrastructure, and the developer faced a common DevOps challenge: how to manage credentials securely when multiple services need access to the same authentication system. The task wasn't just about deploying Authelia—it was about understanding where passwords live in the system and ensuring they won't cause midnight incidents. The work started with a straightforward request: apply the changes to the installation scripts and push them to the pipeline. But before deployment, the developer needed to answer a practical question that often gets overlooked: *where exactly are the credentials stored, and how do we actually use them?* First, the developer examined the Authelia installation script—specifically lines 374–418 of `install-authelia.sh`. This is where the bootstrap happens. The default admin account gets created with a username that's hardcoded in every Authelia setup: **admin**. Simple, memorable, and apparently universal. But the password? That's where it gets interesting. The password isn't just sitting in a configuration file waiting to be discovered. Instead, it's derived from the Management UI's own authentication store at `/etc/management-ui/auth.json`—a pattern that creates a useful single source of truth. Both systems use the same credential, which simplifies the operations workflow. When you need to authenticate to Authelia, you're using the same password that secures the management interface itself. Inside `/etc/authelia/users_database.yml`, the actual password gets stored as an **Argon2 hash**, not plaintext. This is a critical detail because Argon2 is specifically designed to be slow and memory-intensive, making brute-force attacks computationally expensive. It's the kind of defensive measure that doesn't seem important until you're reviewing logs at 3 AM wondering if your authentication layer has been compromised. The developer committed these changes in `e287a26` and pushed them to the pipeline, which would automatically deploy the updated scripts to the server. No manual SSH sessions required—the infrastructure as code approach meant the deployment was reproducible and auditable. What makes this work pattern valuable is the practical transparency it provides. By understanding exactly where credentials live and how they're stored, the developer created documentation that future maintainers will actually use. When someone inevitably forgets the admin password six months later, they'll know to look in `/etc/management-ui/auth.json` instead of starting a frantic password reset procedure. The lesson here isn't about Authelia specifically—it's about building systems where the authentication story is clear and consistent. Single sources of truth for passwords, transparent storage mechanisms, and infrastructure that can be reproduced reliably. That's how you avoid the scenario where nobody remembers which password works with which system. 😄 Why did the functional programmer get thrown out of school? Because he refused to take classes.
Traefik's Missing Middleware: Building Resilient Infrastructure
# When Middleware Goes Missing: Fixing Traefik's Silent Dependency Problem The `borisovai-admin` project sits at the intersection of several infrastructure components—Traefik as a reverse proxy, Authelia for authentication, and a management UI layer. Everything works beautifully when all pieces are in place. But what happens when you try to deploy without Authelia? The system collapses with a 502 error, desperately searching for middleware that doesn't exist. The root cause was deceptively simple: the Traefik configuration had a hardcoded reference to `authelia@file` middleware baked directly into the static config. This worked fine in fully-equipped environments, but made the entire setup fragile. The moment Authelia wasn't installed, Traefik would fail immediately because it couldn't locate that middleware. The infrastructure code treated an optional component as mandatory. The fix required rethinking the initialization sequence. The static Traefik configuration was stripped of any hardcoded Authelia references—no middleware definitions that might not exist. Instead, I implemented conditional logic that checks whether Authelia is actually installed. The `configure-traefik.sh` script now evaluates the `AUTHELIA_INSTALLED` environment variable and only connects the Authelia middleware if the conditions are right. This meant coordinating three separate installation scripts to work in harmony. The `install-authelia.sh` script adds the `authelia@file` reference to `config.json` when Authelia is installed. The `configure-traefik.sh` script stays reactive, only including middleware when needed. Finally, `deploy-traefik.sh` double-checks the server state and reinstalls the middleware if necessary. No assumptions. No hardcoded dependencies pretending to be optional. Along the way, I discovered a bonus issue: `install-management-ui.sh` had an incorrect path reference to `mgmt_client_secret`. I fixed that while I was already elbow-deep in configuration. I also removed `authelia.yml` from version control entirely—it's always generated identically by the installation script, so keeping it in git just creates maintenance debt. **Here's something worth knowing about Docker-based infrastructure:** middleware in Traefik isn't just a function call—it's a first-class configuration object that must be explicitly defined before anything can reference it. Traefik enforces this strictly. You cannot reference middleware that doesn't exist. It's like trying to call an unimported function in Python. A simple mistake, but with devastating consequences in production because it translates directly to service unavailability. The final architecture is much more resilient. The system works with Authelia, without it, or with partial deployments. Configuration files don't carry dead weight. Installation scripts actually understand what they're doing instead of blindly expecting everything to exist. This is what happens when you treat optional dependencies as genuinely optional—not just in application code, but throughout the entire infrastructure layer. The lesson sticks: if a component is optional, keep it out of static configuration. Let it be added dynamically when needed, not the other way around. 😄 A guy walks into a DevOps bar and orders a drink. The bartender asks, "What'll it be?" The guy says, "Something that works without dependencies." The bartender replies, "Sorry, we don't serve that here."
Graceful Degradation: When Infrastructure Assumptions Break
# Authelia Configuration: When Silent Failures Teach Loud Lessons The **borisovai-admin** project was humming along nicely—until someone deployed Traefik without Authelia installed, and everything started returning 502 errors. The culprit? A hardcoded `authelia@file` reference sitting in static configuration files, blissfully unaware that Authelia might not even exist on the server. It was a classic case of *assumptions in infrastructure code*—and they had to go. The task was straightforward: make Authelia integration graceful and conditional. No more broken deployments when Authelia isn't present. Here's what actually happened. First, I yanked `authelia@file` completely out of the static Traefik configs. This felt risky—like removing a load-bearing wall—but it was necessary. The real magic needed to happen elsewhere, during the installation and deployment flow. The strategy became a three-script coordination: **install-authelia.sh** became the automation hub. When Authelia gets installed, this script now automatically injects `authelia@file` into the `config.json` and sets up OIDC configuration in one go. No manual steps, no "oh, I forgot to update the config" moments. It's self-contained. **configure-traefik.sh** got smarter with a conditional check—if `AUTHELIA_INSTALLED` is true, it includes the Authelia middleware. Otherwise, it skips it cleanly. Simple environment variable, massive reliability gain. **deploy-traefik.sh** added a safety net: it re-injects `authelia@file` if Authelia is detected on the server during deployment. This handles the scenario where Authelia might have been installed separately and ensures the configuration stays in sync. There was also a painful discovery in **install-management-ui.sh**—the path to `mgmt_client_secret` was broken. That got fixed too, almost as a bonus. And finally, **authelia.yml** got evicted from the repository entirely. It's now generated by `install-authelia.sh` at runtime. This eliminates version conflicts and keeps sensitive configuration from drifting. **Here's what makes this interesting:** Infrastructure code lives in a grey zone between application code and operations. You can't just assume dependencies exist. Every external service, every optional module, needs to degrade gracefully. The pattern here—conditional middleware loading, environment-aware configuration, runtime-generated sensitive files—is exactly how production systems should behave. It's not sexy, but it's the difference between "works in my test environment" and "works everywhere." The real lesson? **Validate your assumptions at runtime, not at deploy time.** Authelia integration should work whether Authelia is present or not. That's not just defensive programming; that's respectful of whoever has to maintain this later.
Building a Unified Auth Layer: Authelia's Multi-Protocol Juggling Act
# Authelia SSO: When One Auth Is Not Enough The borisovai-admin project needed serious authentication overhaul. The challenge wasn't just protecting endpoints—it was creating a unified identity system that could speak multiple authentication languages: ForwardAuth for legacy services, OIDC for modern apps, and session-based auth for fallback scenarios. I had to build this without breaking the existing infrastructure running n8n, Mailu, and the Management UI. **The problem was elegantly simple in theory, brutal in practice.** Each service had its own auth expectations. Traefik wanted middleware that could intercept requests before they hit the app layer. The Management UI needed OIDC support through express-openid-connect. Older services expected ForwardAuth headers. And everything had to converge on a single DNS endpoint: auth.borisovai.ru. I started by writing `install-authelia.sh`—a complete bootstrapping script that handled binary installation, secret generation, systemd service setup, and DNS configuration. This wasn't just about deployment; it was about making the entire system repeatable and maintainable. Next came the critical piece: `authelia.yml`, which I configured as both a ForwardAuth middleware *and* a router pointing the `/tech` path to the Management UI. This dual role became the architectural linchpin. The real complexity emerged in `server.js`, where I implemented OIDC dual-mode authentication. The pattern was elegant: Bearer token checks first, fallback to OIDC token validation through express-openid-connect, and finally session-based auth as the ultimate fallback. It meant requests could be authenticated through three different mechanisms, transparently to the user. The logout flow had to support OIDC redirect semantics across five HTML pages—ensuring that logging out didn't just clear sessions but also hit the identity provider's logout endpoints. **Here's what made this particularly interesting:** Authelia's ForwardAuth protocol doesn't just pass authentication status; it injects special headers into proxied requests. This header-based communication pattern is how Traefik, Mailu, and n8n receive identity information without understanding OIDC or session mechanics. I had to ensure `authelia@file` was correctly injected into the Traefik router definitions in management-ui.yml and n8n.yml. The `configure-traefik.sh` script became the glue—generating clean authelia.yml configurations and injecting the ForwardAuth middleware into service templates. Meanwhile, `install-management-ui.sh` added auto-detection of Authelia's presence and automatically populated the OIDC configuration into config.json. This meant the Management UI could discover its auth provider dynamically. The whole system shipped as part of `install-all.sh`, where INSTALL_AUTHELIA became step 7.5/10—positioned right before applications that depend on it. Testing this required validating that a request through Traefik with ForwardAuth headers, an OIDC bearer token, and a session cookie would all authenticate correctly under different scenarios. **Key lesson:** Building a unified auth system isn't about choosing one pattern—it's about creating translation layers that let legacy and modern systems coexist peacefully. ForwardAuth and OIDC aren't competing; they're complementary when you design the handoff correctly. 😄 My boss asked why Authelia config took so long. I said it was because I had to authenticate with three different protocols just to convince Git that I was the right person to commit the changes.
Tunnel Magic: From Backend to User-Friendly Control Panel
# Building Tunnel Management: From Scratch to Production-Ready UI The borisovai-admin project needed a proper way for users to manage network tunnels without touching SSH configs. That's where the week went—transforming tunnel management from a backend-only concern into a full-featured UI experience with proper API endpoints and infrastructure tooling. **First thing I did was assess what we were working with.** The project already had frp (a fast reverse proxy) in the deployment pipeline, but there was no user-facing interface to control tunnels. So I built `tunnels.html`—a dedicated management page that lets administrators create, monitor, and tear down tunnels without diving into configuration files. Behind it, I implemented five new API endpoints in `server.js` to handle the full tunnel lifecycle: creation, deletion, status checking, and configuration updates. The tricky part came when integrating frp itself. It wasn't just about adding the reverse proxy to the codebase—I had to ensure it worked seamlessly across different deployment scenarios. This meant updating `install-all.sh`, creating a dedicated `install-frps.sh` script, and building a `frpc-template` that teams could customize for their infrastructure. Deployment needed to be idempotent and predictable. **But then Traefik threw a curveball.** The reverse proxy was timing out on large file transfers, particularly when GitLab was pushing artifacts through it. A quick investigation revealed the `readTimeout` was set too low—just 300 seconds. I bumped it to 600 seconds and added a dedicated `serversTransport` configuration specifically for GitLab to handle chunked uploads properly. The `configure-traefik.sh` script now auto-generates both the `gitlab-buffering` policy and the transport config based on environment variables. Navigation mattered too. Users needed to discover the new Tunnels feature, so I added a consistent "Tunnels" link across all admin pages. Small change, huge UX improvement. **Unexpectedly, this prompted a documentation overhaul.** With more features scattered across the codebase, the docs needed restructuring. I reorganized `docs/` into logical sections: `agents/`, `dns/`, `plans/`, `setup/`, and `troubleshooting/`. Each section now has clear entry points rather than users hunting through one giant README. I also worked on server configuration management—consolidating Traefik, systemd, Mailu, and GitLab configs into `config/contabo-sm-139/` so teams could version control their entire infrastructure setup. The `upload-single-machine.sh` script was enhanced to handle these server-level configurations, making it a proper IaC companion piece. **Here's something worth knowing about Traefik timeouts:** they're not just about being patient. Timeout values cascade—connection timeout, read timeout, write timeout, and idle timeout all interact. A 600-second read timeout is generous for most use cases, but when you're streaming large files through a proxy, you need to account for network variance and the fact that clients might pause between chunks. It's why a blanket increase can seem like a hack, but context-specific configs (like our GitLab transport) are the real solution. What started as "add a UI for tunnels" expanded into infrastructure-as-code thinking, better documentation, and more robust deployment scripts. That's how real projects grow—one feature request becomes a small architecture rethinking session, and suddenly your whole system is more maintainable. 😄 Documentation is like sex: when it's good, it's very good. When it's bad, it's better than nothing.
VPN отключился молча: как я потерял доступ к релизу
# When Infrastructure Hides Behind the VPN: The Friday Night Lesson The deadline was Friday evening. The `speech-to-text` project needed its `v1.0.0` release pushed to master, complete with automated build orchestration, package publishing to GitLab Package Registry, and a freshly minted version tag. Standard release procedure, or so I thought—until the entire development infrastructure went radio silent. My first move was instinctive: SSH into the GitLab server at `gitlab.dev.borisovai.tech` to check on **Gitaly**, the service responsible for managing all repository operations on the GitLab backend. The connection hung without response. I tried HTTP next. Nothing. The entire server had vanished from the network as far as I could tell. Panic wasn't helpful here, but confusion was—the kind that forces you to think systematically about what you're actually seeing. Then it clicked. I checked my VPN status. No connection to `10.8.0.x`. The OpenVPN tunnel that bridges my machine to the internal infrastructure at `144.91.108.139` had silently disconnected. Our entire GitLab setup lives behind that wall of security, completely invisible without it. I wasn't dealing with a server failure—I was on the wrong side of the network boundary, and I'd forgotten about it entirely. This is the quiet frustration of modern infrastructure: security layers that work so seamlessly you stop thinking about them, right up until they remind you they exist. The VPN wasn't broken. The server wasn't broken. I'd simply lost connectivity to anything that mattered for my task. **Here's something interesting about Gitaly itself:** it's not just a repository storage service—it's a deliberate architectural separation that GitLab uses to isolate filesystem operations from the main application. When Gitaly goes offline, GitLab can't perform any Git operations at all. It's like cutting the legs off a runner and asking them to sprint. The design choice exists because managing raw Git operations at scale requires careful resource isolation, and Gitaly handles all the heavy lifting while the GitLab web interface stays focused on its job. The fix was mechanical once I understood the problem. Reconnect the OpenVPN tunnel, then execute the release sequence: `git push origin master` to deploy the automation commit, followed by `.\venv\Scripts\python.exe scripts/release.py` to run the release orchestration script. That script would compile the Python application into a standalone EXE, package it as a ZIP archive, upload it to GitLab Package Registry, and create the version tag—all without human intervention. VPN restored, Gitaly came back online, and the release shipped on schedule. The lesson here isn't technical; it's about remembering the invisible infrastructure that underpins your workflow. Before you blame the server, blame the network. Before you blame the network, check your security tunnel. The most complex problems often have the simplest solutions—if you remember to check the obvious stuff first. 😄 Why did the DevOps engineer break up with the database? Because they had too many issues to commit to.
VPN Down: When Your Dev Infrastructure Becomes Invisible
# When Infrastructure Goes Silent: A Developer's VPN Wake-Up Call The speech-to-text project was humming along smoothly until I hit a wall that would test my troubleshooting instincts. I was deep in the release automation phase, ready to push the final commit to the master branch and trigger the build pipeline that would generate the EXE, create a distributable ZIP, and publish everything to GitLab Package Registry with a shiny new `v1.0.0` tag. But first, I needed to reach the Gitaly service running on our GitLab server at `gitlab.dev.borisovai.tech`. The problem was immediate and unforgiving: Gitaly wasn't responding. My first instinct was the classic DevOps move—SSH directly into the server and restart it. But SSH didn't even acknowledge my connection attempt. The server simply wasn't there. I pivoted quickly, thinking maybe the HTTP endpoint would still respond, but the entire GitLab instance had gone dark. Something was seriously wrong. Then came the diagnostic moment that changed everything. I realized I was sitting in my usual development environment without something critical: an active VPN connection. Our GitLab infrastructure isn't exposed to the public internet—it's tucked safely behind a VPN tunnel to the server at `144.91.108.139`, assigned a private IP in the `10.8.0.x` range. Without OpenVPN active, the entire development infrastructure was invisible to me, completely isolated. This is actually a brilliant security practice, but it's also one of those gotchas that catches you off guard when you're moving fast. The infrastructure wasn't broken—I was simply on the wrong side of the network boundary. **Here's what fascinated me about this situation:** VPNs sit at an interesting intersection of convenience and friction. They're essential for protecting internal infrastructure, but they introduce a hidden dependency that's easy to forget about, especially when you're context-switching between multiple projects or environments. Many development teams solve this by scripting automatic VPN checks into their CI/CD pipelines or shell startup scripts, but it remains a manual step in many workflows. Once I reconnected to the VPN, everything clicked back into place. The plan was straightforward: execute `git push origin master` to send the release automation commit, then fire up `.\venv\Scripts\python.exe scripts/release.py` to orchestrate the entire release process. The script would handle the heavy lifting—compiling the Python code into an executable, bundling dependencies, creating the distributable archive, and finally pushing everything to our package registry. The lesson here wasn't about the technology failing—it was about environmental assumptions. When debugging infrastructure issues, sometimes the problem isn't in your code, your servers, or your services. It's in the invisible layer that connects them all. A missing VPN connection looks exactly like a catastrophic outage until you remember to check whether you're even on the right network. 😄 Why do DevOps engineers never get lonely? Because they always have a VPN to keep them connected!
When Code Reviewers Spot the Same Bug, Architecture Needs a Rewrite
# Scoring v2: When Two Code Reviewers Agree, You Know You're in Trouble The task was straightforward on paper: implement a version-aware analysis system for the trend-analysis project with Tavily citations support on the `feat/scoring-v2-tavily-citations` branch. But when both code reviewers independently flagged the **exact same critical issues**, it became clear this wasn't just about adding features—it was about fixing architectural landmines before they exploded in production. ## The Collision Course The first problem hit immediately: a **race condition in version assignment**. The system was calling `next_version()` independently from `save_analysis()`, which meant two parallel analyses of the same trend could receive identical version numbers. The second INSERT would silently fail, swallowed by a bare `except Exception: pass` block. Both reviewers caught this and independently recommended the same solution: move version generation *inside* the save operation with atomic `INSERT...SELECT MAX(version)+1` logic, wrapped in retry logic for `IntegrityError` exceptions. But that was just the tip. The second critical flaw involved `next_version()` only counting *completed* analyses. Running analyses? Invisible. A second analysis job launched while the first was still executing would grab the same version number. The fix required reserving versions upfront—treating `status='running'` entries in SQLite as version placeholders from the moment a job starts. ## The Breaking Change Bomb Then came the surprise: a breaking API change lurking in plain sight. The frontend expected `getAnalysisForTrend` to return a single object, but the backend had morphed it into returning an array. Both reviewers flagged this differently but reached the same conclusion: introduce a new endpoint `getAnalysesForTrend` for the array response while keeping the old one functional. The TypeScript types were equally broken. The `AnalysisReport` interface lacked `version`, `depth`, `time_horizon`, and `parent_job_id` fields—properties the backend was actively sending but the frontend was discarding into the void. Meanwhile, `parent_job_id` validation was missing entirely (you could pass any UUID), and `depth` had no upper bound (depth=100 anyone?). ## Pydantic as a Safety Net This is where Pydantic's declarative validation became invaluable. By adding `Field(ge=1, le=7)` constraints to depth and using `Literal` for time horizons, the framework would catch invalid requests at the API boundary before they polluted the database. It's one of Pydantic's underrated superpowers—it transforms validation rules into executable guarantees that live right beside your data definitions, making the contract between client and server explicit and checked on every request. ## What Stayed, What Shifted The secondary issues were less dramatic but equally important: unlogged exception handling that swallowed errors, pagination logic that broke when grouping results, and `created_at` timestamps that recorded completion time instead of job start time. The developers had to decide: fix everything now or validate the prototype first, then tackle the full refactor together? Both reviewers converged on the critical path: handle race conditions and API compatibility immediately. Ship a working skeleton, then iterate. --- 😄 Programming is like sex. One mistake and you end up supporting it for the rest of your life.
Tunnels Behind the UI: How One Navigation Link Exposed Full-Stack Architecture
# Mapping a Tunnel System: When One Navigation Link Unveils an Entire Architecture The **borisovai-admin** project needed a critical feature: visibility into FRP (Fast Reverse Proxy) tunnels running behind the admin panel. The task seemed deceptively simple—add a navigation link to four HTML pages. But peeling back that single requirement revealed a full-stack implementation that would touch server architecture, create a new dashboard page, and update installation scripts. ## Starting with the Navigation Trap The first thing I did was update the HTML templates: `index.html`, `tokens.html`, `projects.html`, and `dns.html`. Adding a "Tunnels" link to each felt mechanical—until I realized every page needed *identical* navigation at *exactly* the same line positions (195–238). One typo, one character misaligned, and users would bounce between inconsistent interfaces. That's when I understood: even navigation is an architectural decision, not just UI decoration. ## The Backend Suddenly Mattered With the frontend signposts in place, the backend needed to deliver. In `server.js`, I created two helper functions that became the foundation for everything that followed. `readFrpsConfig` parses the FRP server's configuration file, while `frpsDashboardRequest` handles secure communication with the FRP dashboard. These weren't just convenience wrappers—they abstracted away HTTP mechanics and created a testable interface. Then came the endpoints. Four GET routes to feed the frontend: the FRP server health check—is it alive?; the active tunnels list with metadata about each connection; and the current configuration exposed as JSON. These endpoints are simple on the surface but hide a complexity: they talk to FRP's dashboard API, handle timeouts gracefully, and return data in a shape the frontend expects. ## The Installation Plot Twist Unexpectedly, I discovered FRP wasn't even installed in the standard deployment. The `install-all.sh` script needed updating. I made FRP an *optional* component—not everyone needs tunneling, but those who do should get a complete stack without manual tinkering. This decision reflected a larger philosophy: the system should be flexible enough for different use cases while remaining cohesive. ## The Dashboard That Refreshes Itself The new `tunnels.html` page became the visual payoff. A status card shows whether FRP is running. Below it, an active tunnels list updates every 10 seconds using simple polling—no WebSockets needed for this scale. And finally, a client config generator: input your parameters, see your ready-to-deploy `frpc.toml` rendered instantly. The polling mechanism deserves a note: it's a pattern many developers avoid, but for admin dashboards with small datasets and <10 second refresh windows, it's pragmatic. Fewer moving parts, easier debugging, less infrastructure overhead. ## What the Journey Taught This work crystallized something important: **small frontend changes often hide large architectural decisions**. Investing an hour in upfront planning—mapping dependencies, identifying abstraction points, planning the endpoint contracts—saved days of integration rework later. The tunnel system works now. But its real value isn't the feature itself. It's the pattern: frontend navigation drives backend contracts, which drive installation strategy, which feeds back into the frontend experience. That's systems thinking in practice. 😄 Why did the FRP tunnel go to therapy? It had too many *connections* it couldn't handle!
Serving Artifacts from Private Projects Using GitLab Pages
# How GitLab Pages Became a Private Project's Public Window The speech-to-text project was private—completely locked down on GitLab. But there was a problem: users needed to download built artifacts, and the team wanted a clean distribution channel that didn't require authentication. The challenge was architectural: how do you serve files publicly from a private repository? The developer started by exploring what GitLab offered. Releases API? Protected by project permissions. Package Registry? Same issue—download tokens required. Then came the realization: **GitLab Pages is public by default, even for private projects**. It's a counterintuitive feature, but it made perfect sense for the use case. The first step was auditing the current setup. A boilerplate CI pipeline was already pushed to the repository by an earlier orchestrator run, but it wasn't tailored to the actual workflow. The developer pulled the remote configuration, examined it locally, then replaced it with a custom pipeline designed specifically for artifact distribution. The release process they designed was elegant and automated. The workflow started with a Python script—`scripts/release.py`—that handled the build orchestration. It compiled the project, created a ZIP archive (`VoiceInput-v1.0.0.zip`), uploaded it to GitLab's Package Registry, and pushed a semantic version tag (`v1.0.0`) to trigger the CI pipeline. No manual intervention was needed beyond running one command. The GitLab CI pipeline then took over automatically when the tag appeared. It downloaded the ZIP from Package Registry, deployed it to GitLab Pages, updated a connected Strapi CMS instance with the new version and download URL, and created a formal GitLab Release. Users could now grab builds from a simple, public URL: `https://tools.public.gitlab.dev.borisovai.tech/speech-to-text/VoiceInput-v1.0.0.zip`. Security was handled thoughtfully. The CI pipeline needed write access to create releases and update Pages, so a `CI_GITLAB_TOKEN` was added to the project's CI Variables with protection and masking flags enabled—preventing accidental exposure in logs. **An interesting fact**: GitLab Pages works by uploading static files to a web server tied to your project namespace. Even if the project is private and requires authentication to view source code, the Pages site itself lives on a separate, public domain by design. It's meant for project documentation, but clever teams use it for exactly this—public artifact distribution without exposing the source. The beauty of this approach was that versioning became self-documenting. Every release left breadcrumbs: a git tag marking the exact source state, a GitLab Release with metadata, and a timestamped artifact on Pages. Future developers could trace any deployed version back to its source. The developer shipped semantic versioning, a single-command release process, and automatic CI integration—all without modifying the project's core code structure. It was infrastructure-as-code done right: minimal, repeatable, and transparent. 😄 "We finally made our private project public—just not where anyone expected."
When Your Test Suite Lies: Debugging False Failures in Refactored Code
# Debugging Test Failures: When Your Changes Aren't the Culprit The task was straightforward on paper: add versioning support to the trend-analysis API. Implement parent job tracking, time horizons, and automatic version increments. Sounds simple until your test suite lights up red with six failures, and you have exactly two minutes to figure out if you broke something critical. I was deep in the feat/scoring-v2-tavily-citations branch, having just refactored the `_run_analysis()` function to accept new keyword arguments—`time_horizon` and `parent_job_id`—with sensible defaults. The changes were backward compatible. The database migrations were non-intrusive. Everything should have worked. But the tests were screaming. My first instinct: **blame the obvious**. I'd modified the function signature, so obviously one of the new parameters was breaking the mock chain. The test was calling `_run_analysis(job_id, "AI coding assistants", depth=1)` without the new kwargs—but they had defaults, so that wasn't it. Then I noticed something interesting: the test patches `DB_PATH`, but my code calls `next_version()`, which uses `_get_conn()` to access the database directly. The patch should handle that... unless it doesn't. But wait—`next_version()` is wrapped in an `if trend_id:` block. Since the test passes `trend_id=None`, that function never even executes. So that's not the issue either. Then I found it. The test mocks `graph_builder_agent` as `lambda s: {...}`, a simple single-argument function. But my earlier changes added a `progress_callback` parameter, and now the code calls it as `graph_builder_agent(state, progress_callback=on_zone_progress)`. The lambda doesn't accept `**kwargs`. This mock was outdated—someone had added the `progress_callback` feature weeks ago without updating the tests. Here's the key realization: **these six failures aren't from my changes at all**. They're pre-existing issues that would have failed before I touched anything. The test infrastructure simply hadn't caught up with previous development iterations. **What I actually shipped:** Database migrations adding version tracking, depth parameters, and parent job IDs. New Pydantic schemas (`AnalysisVersionSummary`, `TrendAnalysesResponse`) for API responses. Updated endpoints with automatic version incrementing. Everything backward compatible, everything non-breaking. **What I learned:** Before panicking about breaking changes, check the git history. Dead code and outdated mocks pile up faster than you'd expect. And sometimes the most valuable debugging is realizing that the problem isn't yours to fix—not yet, anyway. The prototype validation stage was the smart call. I created an HTML prototype showcasing four key screens: trend detail timeline, version navigation with delta strips, unified and side-by-side diff views, and grouped reports listing. Ship the concept, validate with stakeholders, iterate based on real feedback instead of chasing phantom bugs. **Educational note:** aiosqlite changed the game for async database access in Python applications—it wraps SQLite with async/await support without requiring a separate database server. It's perfect for prototypes and single-machine deployments where you need the simplicity of SQLite but can't block your async event loop on I/O. The six failing tests are still there, waiting for the next developer to care enough to fix them. But they're not my problem—yet. 😄
From Flat to Relational: Scaling Trend Analysis with Database Evolution
# Building a Scalable Trend Analysis System: When Flat Data Structures Aren't Enough The social media analytics engine was growing up. An HTML prototype had proven the concept, but now it needed a **real** backend architecture—one that could track how analyses evolve, deepen, and branch into new investigations. The current database schema was painfully flat: one analysis per trend, no way to version iterations, no parent-child relationships. If a user wanted deeper analysis or an extended time horizon, the system had nowhere to store the evolution of their request. First thing I did was examine the existing `analysis_store.py`. The foundation was there—SQLite with aiosqlite for async access, a working `analyses` table, basic query functions—but it was naive. It didn't understand that trend investigations create **lineages**. So I started Phase 1: **database evolution**. I added four strategic columns to the schema: `version` (which iteration of this analysis?), `depth` (how many investigation layers deep?), `time_horizon` (past week, month, year?), and `parent_job_id` (which analysis spawned this one?). These fields transformed the database from a flat ledger into a graph structure. Now analyses could reference their ancestors, forming chains of investigation. Phase 2 was rewriting the store layer. The original `save_analysis()` function was too simple—it didn't know about versioning. I rebuilt it to compute version numbers automatically: analyzing the same trend twice? That's version 2, not an overwrite. Then I added `find_analyses_by_trend()` to fetch all versions, `_row_to_version_summary()` to convert database rows into version-specific Python objects, and `list_analyses_grouped()` to organize results hierarchically by their parent-child relationships. Phase 3 touched the API surface. Updated Pydantic schemas to understand versioning, gave `AnalyzeRequest` a `parent_job_id` parameter so the frontend could explicitly chain requests, and added a `grouped` parameter to endpoints. When `grouped=true`, the API returns a tree structure showing how analyses relate. When `grouped=false`, a flat list. Same data, different perspective. Then the tests started screaming. One test, `test_crawler_item_to_schema_with_composite`, failed consistently. Panic for thirty seconds—*did I break something?*—until I realized this was a preexisting issue unrelated to my changes. A good reminder that not every failing test is your fault. Sometimes you just skip it and move on. **Here's something worth knowing about SQLite migrations in Python**: unlike Django's ORM-heavy approach, the Python ecosystem tends to write database migrations as explicit functions that run raw SQL `ALTER TABLE` commands. SQLite is notoriously finicky about complex schema transformations, so developers lean into transparency. You write the migration by hand, see exactly what SQL executes, no hidden magic. It feels refreshingly honest compared to frameworks that abstract everything away. The architecture was complete. A developer could now request trend analysis, ask for deeper investigation, and the system would create a new version while remembering its lineage. The data could flow out as a flat list or a hierarchical tree depending on what the frontend needed. The next phase—building a UI that actually *shows* this version history and lets analysts navigate it intuitively—would be its own adventure. 😄 Pro tip: that failing test? The one unrelated to your changes? Just skip it, ship it, and let someone else debug it in six months.