Hope College CIT - Server Hosting System Outage – Incident details

Experiencing partially degraded performance

Server Hosting System Outage

Identified
Degraded performance
Started 30 days ago

Affected

Hope Infrastructure

Major outage from 5:16 PM to 8:02 PM, Degraded performance from 5:16 PM to 8:02 PM, Major outage from 8:02 PM to 7:28 PM, Degraded performance from 8:02 PM to 1:31 PM, Operational from 1:31 PM to 1:31 PM, Degraded performance from 1:31 PM to 12:00 AM

On-Campus Server Infrastructure

Degraded performance from 5:16 PM to 1:31 PM, Operational from 1:31 PM to 1:31 PM, Degraded performance from 1:31 PM to 12:00 AM

Concerto (signs.hope.edu)

Major outage from 5:16 PM to 7:28 PM, Degraded performance from 7:28 PM to 1:31 PM, Operational from 1:31 PM to 12:00 AM

Updates
  • Identified
    UTC
    Identified

    We will be performing additional maintenance on the server with issues starting at 2pm today, for 24 hours. We don't anticipate any user-facing outages, but please reach out to the Help Desk if you experience any issues today or tomorrow with services that are hosted on-campus.

  • Update
    UTC
    Update

    For now, the problematic server is behaving as expected, but we are continuing to actively investigate the root cause of why it intermittently acts up.

  • Monitoring
    UTC
    Monitoring

    Most services have returned to normal and we are continuing to investigate the cause of this issue.

  • Identified
    UTC
    Identified

    We are continuing to work on a fix for this incident. Specific server VMs that are affected are able to be migrated to other infrastructure as needed. If a service that you rely on continues to experience issues, please reach out and we will prioritize a fix for that service.

  • Investigating
    UTC
    Investigating

    We're looking into reports that several services that run on one of our on-campus servers aren't working as expected. This includes our digital signage software, signs.hope.edu. We'll update here when we know more.