---
title: "Postmortem: knowledge base segregation regression"
description: "Resolved · affects Questionnaires, Agent · Sep 24, 2025"
base_url: "https://status.wolfia.com"
canonical: "https://status.wolfia.com/events/report/650"
---

# Postmortem: knowledge base segregation regression

[Status](/.md) › [Events](/events.md) › Postmortem: knowledge base segregation regression

Sep 24, 2025 · 11 months ago · affects: Questionnaires, Agent · 1 minute

## Updates

### + Resolved — Sep 24, 8:00 PM

**Incident window:** Sep 24, 2025, 10:00 AM–3:00 PM PT (≈5 hours)

**Maintenance window (completed):** Sep 23, 9:00 PM–Sep 24, 2:00 AM PT

**Status:** Resolved

**Surfaces:** Web agent, Slack agent, Questionnaires

**Trust centers:** Not impacted

## Summary

Following planned maintenance to implement tenant-level data segregation, the knowledge base (KB) was not bound to individual tenant indices. A validation check incorrectly passed and did not verify KB content/binding, allowing some answers to be generated without tenant facts. This led to degraded quality and hallucinations for a subset of requests.

## Impact

* Ungrounded or lower-quality answers for \~5 hours.
* No data loss and no cross-tenant access.
* All affected questionnaires were reprocessed; impacted customers were contacted directly.

## Root cause

A migration step omitted rehydrating and binding the KB to per-tenant indices. Validation asserted index availability but did not assert non-empty tenant-scoped content, resulting in a false positive.

## Detection

Detected at 10:00 AM PT via a customer report and our existing **“answers without facts”** alert, after the maintenance had concluded.

## Resolution & recovery

* Rebuilt and bound per-tenant KB indices.
* Temporarily blocked ungrounded answering until grounding metrics returned to baseline.
* Reprocessed all affected questionnaires and verified quality.

## Preventive actions

* Strengthen validation to require per-tenant **existence + non-empty content + correct binding**.
* Expand pre- and post-cutover canaries (Web/Slack) to verify grounded answers.
* Add a hard circuit breaker to prevent answering when facts are unavailable.
* Update runbooks and add a dry-run checklist for tenant migrations.

## Timeline (PT)

* **2:00 AM** – Maintenance completes successfully.
* **10:00 AM** – Customer report received; “answers without facts” alert fires; incident declared.
* **10:20 AM** – Issue traced to KB not bound to tenant indices.
* **11:00 AM** – Fix deployed; per-tenant KB rebuilt/bound; answering guarded by circuit breaker.
* **3:00 PM** – Reprocessing complete; metrics normalized; incident resolved.

**Questions?** Email [support@wolfia.com](mailto:support@wolfia.com) or see updates on status.wolfia.com.

---

_Powered by [openstatus.dev](https://openstatus.dev)_