Wolfia status page
  • Status
  • Events
  • Monitors
Wolfia
Track the health of Wolfia’s Questionnaires, Agent, Trust Center, and API. If something impacts your workflow, we’ll post details and progress here.
Apr 16, 2026
13 days ago
Degraded AI features — upstream Azure OpenAI outage and Claude rate limiting
QuestionnairesAgent
Resolved · April 16 at 10:00 PM (in 45 minutes)

Fully resolved. Total user-facing impact ~45 minutes. Root cause: simultaneous Azure OpenAI regional outage and Claude API rate limiting on our primary provider. Mitigation: we have since expanded our Azure OpenAI footprint from a single region to 7 US regions (westus, westus3, eastus, eastus2, centralus, southcentralus, northcentralus), roughly 5x effective GPT capacity, with automatic health- and rate-limit-aware routing so a localized issue in any one region no longer impacts availability.

Monitoring · April 16 at 9:45 PM (15 minutes earlier)

Azure OpenAI traffic has been routed to a healthy region and Claude requests are succeeding again after backoff. Chat Agent and Questionnaire pipelines are recovering. Monitoring for stability.

Identified · April 16 at 9:15 PM (30 minutes earlier)

We are seeing degraded performance on Chat Agent and Questionnaire answer generation. Upstream cause: an Azure OpenAI regional outage combined with Claude API rate limiting on our primary provider. Engineering is failing traffic over to alternate regions and providers.

Dec 4, 2025
5 months ago
Questionnaire processing affected
Questionnaires
Resolved · December 4 at 2:54 PM

Incident window: Dec 4, 2025, 6:43 AM–7:04 AM PT (≈21 minutes)

The third-party service has been restored. Questionnaire processing is resuming normally.

Status: Resolved

Sep 24, 2025
7 months ago
Postmortem: knowledge base segregation regression
QuestionnairesAgent
Resolved · September 24 at 8:00 PM

Incident window: Sep 24, 2025, 10:00 AM–3:00 PM PT (≈5 hours)

Maintenance window (completed): Sep 23, 9:00 PM–Sep 24, 2:00 AM PT

Status: Resolved

Surfaces: Web agent, Slack agent, Questionnaires

Trust centers: Not impacted

Summary

Following planned maintenance to implement tenant-level data segregation, the knowledge base (KB) was not bound to individual tenant indices. A validation check incorrectly passed and did not verify KB content/binding, allowing some answers to be generated without tenant facts. This led to degraded quality and hallucinations for a subset of requests.

Impact

  • Ungrounded or lower-quality answers for ~5 hours.
  • No data loss and no cross-tenant access.
  • All affected questionnaires were reprocessed; impacted customers were contacted directly.

Root cause

A migration step omitted rehydrating and binding the KB to per-tenant indices. Validation asserted index availability but did not assert non-empty tenant-scoped content, resulting in a false positive.

Detection

Detected at 10:00 AM PT via a customer report and our existing “answers without facts” alert, after the maintenance had concluded.

Resolution & recovery

  • Rebuilt and bound per-tenant KB indices.
  • Temporarily blocked ungrounded answering until grounding metrics returned to baseline.
  • Reprocessed all affected questionnaires and verified quality.

Preventive actions

  • Strengthen validation to require per-tenant existence + non-empty content + correct binding.
  • Expand pre- and post-cutover canaries (Web/Slack) to verify grounded answers.
  • Add a hard circuit breaker to prevent answering when facts are unavailable.
  • Update runbooks and add a dry-run checklist for tenant migrations.

Timeline (PT)

  • 2:00 AM – Maintenance completes successfully.
  • 10:00 AM – Customer report received; “answers without facts” alert fires; incident declared.
  • 10:20 AM – Issue traced to KB not bound to tenant indices.
  • 11:00 AM – Fix deployed; per-tenant KB rebuilt/bound; answering guarded by circuit breaker.
  • 3:00 PM – Reprocessing complete; metrics normalized; incident resolved.

Questions? Email support@wolfia.com or see updates on status.wolfia.com.

powered by openstatus.dev