Incident Report
At 3:31PM UTC (06/01/2016) our Level 3 Data Center in Denver experienced a network outage that resulted in the full loss of service for nearly all services.
Timeline (times are UTC)
- 3:31PM Outage occurred
- 3:40PM Technician dispatched to verify local equipment
- 3:48PM Technician verified local equipment functionality
- 3:50PM Ticket opened with Level 3
- 4:05PM Ticket escalated to tier 1 support
- 4:22PM Ticket updated that Level 3 was investigating the cause of the outage
- 4:37PM Ticket updated to notify that equipment has been verified and that the fiber connections are good
- 4:42PM Ticked escalated to tier 2 support
- 5:20PM Ticket updated to notify that the connection is being reset/ rebuild
- 5:40PM Ticket updated to notify that our services should be back within 1 hour
- 6:38PM Ticket updated by ASPnix to notify that network was still offline
- 7:05PM Ticket escalated to tier 3 support
- 8:48PM Ticket updated to notify that the issue had been resolved and our services would be restored within minutes
- 8:51PM BGP connection reestablished on both ASPnix routers and service restored
Root Cause
ASPnix uses 2 private AS numbers provided by Level 3 to allow for BGP routing between our routers and the Level 3 edge routers that service our circuits. This allows for failover between routers in the unfortunate event that one fails. Due to an outstanding order that was tied to our account, their system automatically closed the order and marked our AS numbers as inactive. Within 5 – 7 minutes their routers received the updated configuration and notified that these routes are now inactive and were removed resulting in all traffic to and from our network to be unroutable.
Because of the missing AS numbers under our account, the tier 2 support technician assumed we did not use BGP and rebuilt our connection without it. Our routers are configured for BGP and advertise BGP, as a result they were unable to establish a connection and advertise our IP routes to the edge router.
Tier 3 support was able to see that our configuration was invalid and we provided our BGP information and previous AS numbers. The technician was able to restore the BGP routes and within minutes our service was back online.
Credits
We will be issuing credits to all customers in the next few days. Credits will be automatically applied to each account based on any active service(s). This excludes domains and service addons.
If you have any questions or concerns, please open a ticket to the technical support department for your service.
We’d like to thank everyone for their patience and understanding during this time and for choosing ASPnix!
