Wolfia status page
  • Status
  • Events
  • Monitors
Wolfia
Track the health of Wolfia’s Questionnaires, Agent, Trust Center, and API. If something impacts your workflow, we’ll post details and progress here.

powered by openstatus.dev

Back
Sep 24, 2025
11 months ago
Postmortem: knowledge base segregation regression
QuestionnairesAgent
Resolved · September 24 at 8:00 PM (UTC)

Incident window: Sep 24, 2025, 10:00 AM–3:00 PM PT (≈5 hours)

Maintenance window (completed): Sep 23, 9:00 PM–Sep 24, 2:00 AM PT

Status: Resolved

Surfaces: Web agent, Slack agent, Questionnaires

Trust centers: Not impacted

Summary

Following planned maintenance to implement tenant-level data segregation, the knowledge base (KB) was not bound to individual tenant indices. A validation check incorrectly passed and did not verify KB content/binding, allowing some answers to be generated without tenant facts. This led to degraded quality and hallucinations for a subset of requests.

Impact

  • Ungrounded or lower-quality answers for ~5 hours.
  • No data loss and no cross-tenant access.
  • All affected questionnaires were reprocessed; impacted customers were contacted directly.

Root cause

A migration step omitted rehydrating and binding the KB to per-tenant indices. Validation asserted index availability but did not assert non-empty tenant-scoped content, resulting in a false positive.

Detection

Detected at 10:00 AM PT via a customer report and our existing “answers without facts” alert, after the maintenance had concluded.

Resolution & recovery

  • Rebuilt and bound per-tenant KB indices.
  • Temporarily blocked ungrounded answering until grounding metrics returned to baseline.
  • Reprocessed all affected questionnaires and verified quality.

Preventive actions

  • Strengthen validation to require per-tenant existence + non-empty content + correct binding.
  • Expand pre- and post-cutover canaries (Web/Slack) to verify grounded answers.
  • Add a hard circuit breaker to prevent answering when facts are unavailable.
  • Update runbooks and add a dry-run checklist for tenant migrations.

Timeline (PT)

  • 2:00 AM – Maintenance completes successfully.
  • 10:00 AM – Customer report received; “answers without facts” alert fires; incident declared.
  • 10:20 AM – Issue traced to KB not bound to tenant indices.
  • 11:00 AM – Fix deployed; per-tenant KB rebuilt/bound; answering guarded by circuit breaker.
  • 3:00 PM – Reprocessing complete; metrics normalized; incident resolved.

Questions? Email support@wolfia.com or see updates on status.wolfia.com.