AWS power outage with data loss

AWS power outage with data loss

  • September 8, 2019
Table of Contents

AWS power outage with data loss

On August 31st, 2019, an Amazon AWS US-EAST-1 datacenter in North Virginia experienced a power failure at 4:33 AM, which led to the datacenter’s backup generators to kick on. Unfortunately, these generators started failing at approximately 6:00 AM , which led to 7.5% of the EC2 instances and EBS volumes becoming unavailable. ‘1:30 PM PDT At 4:33 AM PDT one of ten data centers in one of the six Availability Zones in the US-EAST-1 Region saw a failure of utility power.

Our backup generators came online immediately but began failing at around 6:00 AM PDT. This impacted 7.5% of EC2 instances and EBS volumes in the Availability Zone. Power was fully restored to the impacted data center at 7:45 AM PDT.

By 10:45 AM PDT, all but 1% of instances had been recovered, and by 12:30 PM PDT only 0.5% of instances remained impaired. Since the beginning of the impact, we have been working to recover the remaining instances and volumes. A small number of remaining instances and volumes are hosted on hardware which was adversely affected by the loss of power.

We continue to work to recover all affected instances and volumes and will be communicating to the remaining impacted customers via the Personal Health Dashboard. For immediate recovery, we recommend replacing any remaining affected instances or volumes if possible.’

Source: bleepingcomputer.com

Tags :
Share :
comments powered by Disqus

Related Posts

A Technical Analysis of the Capital One Hack

A Technical Analysis of the Capital One Hack

The recent disclosure of yet another cloud security misconfiguration leading to the loss of sensitive personal information made the headlines this past week. This particular incident came with a bit more information from the indictment of the accused party, allowing us to piece together the revealed data and take an educated guess as to what may have transpired leading up to the loss of over 100 million credit card applications and 100 thousand social security numbers. At the root of the hack lies a common refrain: the misconfiguration of cloud infrastructure resources allowed an unauthorized user to elevate her privileges and compromise sensitive documents.

Read More
Key Conjurer: Our Policy of Least Privilege

Key Conjurer: Our Policy of Least Privilege

Hi, my name is Reza Nikoopour and I’m a security engineer on the Security team at Riot. My team is responsible for securing Riot infrastructure wherever we’re deployed – whether that means internal or external data centers or clouds. We provide cloud security guidance to the rest of Riot, and we’re responsible for Key Conjurer, our open source AWS API programmatic access solution.

Read More
AWS and the CLOUD Act

AWS and the CLOUD Act

While news of Brexit dominates headlines in the United Kingdom, another important event took place recently in London. U.S. Deputy Assistant Attorney General Richard W. Downing addressed the myths and realities of the Clarifying Lawful Overseas Use of Data Act (“CLOUD Act”), in a speech at the Academy of European Law Conference. Following the speech, the U.S. Department of Justice (DOJ) published a whitepaper and FAQ clarifying the purpose and scope of the CLOUD Act and addressing many of the misunderstandings of this law.

Read More