Postmortems/Spring 2017 IMAP Incident

From UGCS
Jump to: navigation, search

Description

An old certificate was left in use on the UGCS IMAP server, due to a failure in the certificate update script. This caused clients accessing the server to display SSL errors due to the certificate expiration.

Timeline

a log time ago The certificate in use was installed and the dovecot daemon restarted.

a short time ago Our newer SSL cert is updated by the certificate update script, but the dovecot daemon is not restarted. Dovecot continues to use the old script.

April 27 The certificate expires. Users accessing UGCS IMAP would be connecting with an invalid certificate.

May 29 We are notified by a user that we are using an expired certificate. We verified the issue and restarted the dovecot daemon.

Contributing Factor(s)

  • Some UGCS admins had their clients configured to whitelist SSL errors from the UGCS domain from before SSL was configured.
  • Daemons for other services are frequently restarted for a various reasons.

Stabilization Steps

The dovecot daemon was restarted.

Impact

Users saw a warning or error due to the expired certificate. Users using certain clients were unable to dismiss the error and were subsequently unable to access their email using IMAP.

Corrective Actions

  • fix the ssl script service restarter
  • Remove all SSL exceptions for UGCS from sysadmin devices.
  • Verify which daemons require restart after ssl update.