AWS EBS Snapshot S3 Lazy Loading

Context

#

I was requested to restore a new AWS RDS MySQL Database from an existing AWS RDS Snapshot. After the restore process finished, the customer complained about the database performance.

Performance issues encountered:

• Migration script that mass-updates the database to a new schema
• Querying data from the database
• Inserting/updating data into the database

So I started debugging the problem by checking the database metrics in AWS CloudWatch, but there was nothing unexpected. All graphs seemed to be reasonable and within boundaries.

After checking in with AWS Support, we figured it out: lazy loading volume data from S3.

What I Learned

#

The thing is that if you create a volume (which uses EBS underneath) from a snapshot, you will encounter lazy loading. This is because each EBS Snapshot is stored on AWS S3. After you create a new volume from a snapshot, the volume is created immediately, but the data is synchronized in the background.

If you request data that hasn’t been synchronized to your volume yet, it gets requested from S3. Because of the nature of S3, it is magnitudes slower in I/O than the actual EBS volume.

After understanding the problem, I saw that the I/O latency in CloudWatch was around 35ms.

But there are solutions to prevent lazy loading data from S3 for EBS restores.

Fast Initialization of Data

#

To quickly initialize data in databases, you can count all columns in a table:

1

SELECT COUNT(*) FROM TABLE;

This directly requests the data from S3, but the first run will take some time. This is useful if some tables are accessed more frequently than others.

Fast Snapshot Restore (FSR)

#

Alternatively, you can use the FSR (Fast Snapshot Restore) feature from AWS. This prewarms the EBS volumes in your availability zones.

Why It Matters

#

Performance is critical in production environments. Restoring snapshots when something goes wrong could lead to surprisingly slow volumes. In mission-critical situations, this is a problem that should be prevented.

Key Takeaways

#

• For mission-critical databases, Fast Snapshot Restore (FSR) should be used.
• In non-production environments, read the files/tables you need first.

Links

#

• Addressing I/O latency when restoring Amazon EBS volumes from EBS Snapshots