Wolfia status page
  • Status
  • Events
  • Monitors
Wolfia
Track the health of Wolfia’s Questionnaires, Agent, Trust Center, and API. If something impacts your workflow, we’ll post details and progress here.
Dec 4, 2025
3 days ago
Questionnaire processing affected
Questionnaires
Resolved December 4 at 2:54 PM

Incident window: Dec 4, 2025, 6:43 AM–7:04 AM PT (≈21 minutes)

The third-party service has been restored. Questionnaire processing is resuming normally.

Status: Resolved

Sep 24, 2025
2 months ago
Postmortem: knowledge base segregation regression
QuestionnairesAgent
Resolved September 24 at 8:00 PM

Incident window: Sep 24, 2025, 10:00 AM–3:00 PM PT (≈5 hours)

Maintenance window (completed): Sep 23, 9:00 PM–Sep 24, 2:00 AM PT

Status: Resolved

Surfaces: Web agent, Slack agent, Questionnaires

Trust centers: Not impacted

Summary

Following planned maintenance to implement tenant-level data segregation, the knowledge base (KB) was not bound to individual tenant indices. A validation check incorrectly passed and did not verify KB content/binding, allowing some answers to be generated without tenant facts. This led to degraded quality and hallucinations for a subset of requests.

Impact

  • Ungrounded or lower-quality answers for ~5 hours.
  • No data loss and no cross-tenant access.
  • All affected questionnaires were reprocessed; impacted customers were contacted directly.

Root cause

A migration step omitted rehydrating and binding the KB to per-tenant indices. Validation asserted index availability but did not assert non-empty tenant-scoped content, resulting in a false positive.

Detection

Detected at 10:00 AM PT via a customer report and our existing “answers without facts” alert, after the maintenance had concluded.

Resolution & recovery

  • Rebuilt and bound per-tenant KB indices.
  • Temporarily blocked ungrounded answering until grounding metrics returned to baseline.
  • Reprocessed all affected questionnaires and verified quality.

Preventive actions

  • Strengthen validation to require per-tenant existence + non-empty content + correct binding.
  • Expand pre- and post-cutover canaries (Web/Slack) to verify grounded answers.
  • Add a hard circuit breaker to prevent answering when facts are unavailable.
  • Update runbooks and add a dry-run checklist for tenant migrations.

Timeline (PT)

  • 2:00 AM – Maintenance completes successfully.
  • 10:00 AM – Customer report received; “answers without facts” alert fires; incident declared.
  • 10:20 AM – Issue traced to KB not bound to tenant indices.
  • 11:00 AM – Fix deployed; per-tenant KB rebuilt/bound; answering guarded by circuit breaker.
  • 3:00 PM – Reprocessing complete; metrics normalized; incident resolved.

Questions? Email support@wolfia.com or see updates on status.wolfia.com.

powered by openstatus.dev