ELLA - Advisors unable to access existing workspaces – Incident details

All systems operational

Advisors unable to access existing workspaces

Resolved
Partial outage 70 %
Started 7 months agoLasted 9 days

Affected

Application

Partial outage from 5:00 AM to 3:00 AM

ELLA App Reachability

Partial outage from 5:00 AM to 3:00 AM

Updates
  • Resolved
    UTC
    Resolved

    This incident is resolved. All advisor access has been restored, and no data was lost, altered, or exposed at any point. Notably, the root cause was a permission change that over-indexed on security controls. the rules erred toward locking data down rather than opening it up, so the failure was one of over-protection, not exposure. We've since tightened our testing so access changes can't have this unintended effect again. We apologize for the disruption.

  • Identified
    UTC
    Identified

    We identified the cause. A recent update to our data-access permission rules was more restrictive than intended, causing some advisors to be blocked from viewing their own content. We began preparing and testing a corrected set of permission rules.

  • Investigating
    UTC
    Investigating

    We began investigating reports that some advisors could not see their workspaces and associated content (documents, deliverables, and fact-finding data) after signing in. We began investigating on 1/21, but the problem was released into production several days prior before we were found the issue. We've backdated this issue to the 15th to reflect that.