Got this message from Amazon today:
Hello.
This is a notification that your volume vol-XXXXXXXX experienced a failure due to multiple failures of
the underlying hardware components and was unable to be recovered. We recommend recovering from your
most recent snapshot.
We regret the loss and inconvenience.
Sincerely,
XXXXXXX XXXXXXXXX
Fortunately, this was on a mysql slave and it was easy to recover from a previous snapshot. Do you plan for failure of the systems you run on Amazon EC2?
Amazon announced today the availability of Reserved Instances. Basically, you pay a one-time fee and get a significant discount on the hourly rate for that instance.
Their rates are available here.
After doing some calculations on it, I figured out the break even point between the old “On-Demand Instances” and the new “Reserved Instances.”
For the current 1 year term, the break-even point is 193 days. For the 3 year term, the break even point is 298 days. So basically, if you’re going to be running a particular instance for much more than a year, you should probably sign up for the 3 year term.
Here’s a brief comparison of the year by year costs for a small instance. I’ve only listed the small instance type since the costs for the other instance types are all proportional (e.g. large is 4x and xlarge is 8x.)
| 1st year costs | 2nd year costs | 3rd year costs | 3 year Total | |
| On Demand | $876.00 | $876.00 | $876.00 | $2,628.00 |
| 1 year term | $587.80 | $587.80 | $587.80 | $1,763.40 |
| 3 year term | $762.80 | $262.80 | $262.80 | $1,288.40 |
(I’m not sure why the spacing is so funky on this post with a table in it.)
Amazon just posted a security whitepaper which describes the security measures they have in place to protect their customers. It’s a short read (9 pages) and I strongly recommend it if you are using EC2, S3 or SimpleDB for anything. They explain their security measures and make recommendations for further protection their customers can put in place to protect their data.
One major concern I had was addressed by the whitepaper, “The AWS proprietary disk virtualization layer automatically wipes every block of storage used by the customer, and guarantees that one customer’s data is never exposed to another.” I was always curious about those disk devices on EC2 and what data might be lingering on them but never had the time to investigate.
Here’s an interesting snippet regarding their physical security:
“Amazon has many years of experience in designing, constructing, and operating largescale data centers. This experience has been applied to the AWS platform and infrastructure. AWS data centers are housed in nondescript facilities, and critical facilities have extensive setback and military grade perimeter control berms as well as other natural boundary protection. Physical access is strictly controlled both at the perimeter and at building ingress points by professional security staff utilizing video surveillance, state of the art intrusion detection systems, and other electronic means. Authorized staff must pass two-factor authentication no fewer than three times to access data center floors. All visitors and contractors are required to present identification and are signed in and continually escorted by authorized staff.”
At familylink.com, we have 4 MySQL database systems on EC2 that run that our facebook app, various other social network apps and various websites. I recently switched our disk storage for those instances from the standard EC2 instance disks to EBS (Amazon’s persistent storage for EC2) and wanted to share some brief numbers with you regarding performance.
I’m using a simple (yet quite complex) metric to measure the performance increase, load. System load is a number that show how many processes are contending for system resources (usually CPU.) For a more detailed description of load, read this article.
Enough of the talk, here’s what I saw when I switched the 4 databases over to EBS:
—Database server #1—
Purpose: 2 moderately used databases
Disk change: 2 striped local disks (raid0) to single EBS volume
Peak Load change: 2.5 to 1
Estimated disk performance increase: 5x
—Database server #2—
Purpose: 2 lightly used databases
Disk change: 2 striped local disks (raid0) to single EBS volume
Peak Load change: 1.5 to 0.5
Estimated disk performance increase: 6x
—Database server #3—
Purpose: 9 lightly used databases
Disk change: 2 striped local disks (raid0) to single EBS volume
Peak Load change: 1 to 1 (no noticeable change)
Estimated disk performance increase: 2x
—Database server #4—
Purpose: 1 heavily used database
Disk change: 4 striped local disks (raid0) to 4 striped EBS volumes (raid0)
Peak Load change: 3 to 1.5
Estimated disk performance increase: 2x
Keep in mind that in theory 2 striped disks are almost twice as fast as a single disk. That’s why I say there’s a disk performance increase of 2x on database server #3 even though there was no noticeable performance increase (we went from using 2 disks to 1 disk.)
There you go, real-world numbers from real-world sites and servers. In summary, it’s safe to say you’ll see a significant disk performance increase if you switch over to using EBS with your EC2 instances. In addition to the performance increase, it’s a no-brainer that you want persistent storage for your databases. One other huge benefit is snapshots. You can quickly and easily snapshot your database for backup purposes or for testing/reporting you may want to run against your most recent production data. See Amazon’s site for more details.
If you haven’t yet tested EBS with your systems on EC2, now is the time.
Amazon just announced two new High-CPU instance types for EC2. Here’s a quick comparison of what you can now get compared to the previous offering:
2x Small ($0.20/hr)
RAM – 3.4GB
CPU – 2 Compute Units*
DISK – 320GB
High-CPU Medium ($0.20/hr)
RAM – 1.7GB
CPU – 5 Compute Units*
DISK – 350GB
Extra Large ($0.80/hr)
RAM – 15GB
CPU – 8 Compute Units*
DISK – 1690GB
High-CPU Extra Large ($0.80/hr)
RAM – 7GB
CPU – 20 Compute Units*
DISK – 1690GB
So it really looks like a tradeoff between CPU and RAM. If you run more CPU intensive processes, then these new instance types may save you money.
*One EC2 Compute Unit (ECU) provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.
I recently had to alter(edit) a very large MySQL table on a production system to drop an unnecessary index. This particular table has over 15 million records in it and occupies over 6GB of space. Since this database runs the We’re Related facebook app, I couldn’t just bring it down for hours and make the change. This particular table stores data for a main part of the application.
Enter EC2. All of our production systems that power We’re Related are running on Amazon’s EC2 so this was an easy problem to work around. Using MySQL replication, I setup a system that would mirror this production database. The nice thing about replication is that it automatically keeps any mirrors up to date with the latest data.
Once I had the mirror up and running, I was able to run the command to alter the table on the mirror. This took over 6 hours to complete. Once done, it quickly caught up with those 6 hours worth of updates from the main system.
Tonight, I will be reconfiguring the webservers to connect to this new mirrored system instead of the original. This will result in less than five minutes of actual downtime.
End result:
downtime – < 5 minutes compared to 6+ hours
additional cost - $20 (for running an additional "beefy" server for 1 day)
That's one thing I just love about EC2, you pay for what you use. If you need a temporary server for hours or days, you just fire one up and pay minimally for it.
For us UNIX fans, Sun Microsystems is working with Amazon to offer OpenSolaris on EC2.
I’m not sure how I missed it, but they made the announcement last week.
I just signed up to be included in the beta. I’ll post an update here when I get in and start tinkering. My main focus is going to be the performance of MySQL on EC2 comparing Linux with OpenSolaris.
I’m not sure what their long term plans are, but for now they aren’t incurring any extra charges (beyond the standard EC2 charges.) I know that RedHat charges extra to run their Enterprise version on EC2.
Amazon just announced a price change for bandwidth for outbound traffic from EC2 and S3. The result is up to 26% reduction in costs for all outbound traffic depending on your volume. Here are the new prices which are effective May 1
New data transfer price (effective May 1, 2008)
$0.100 per GB – data transfer in
$0.170 per GB – first 10 TB / month data transfer out
$0.130 per GB – next 40 TB / month data transfer out
$0.110 per GB – next 100 TB / month data transfer out
$0.100 per GB – data transfer out / month over 150 TB
You can read full details here.
It looks like the easter bunny came a few days late for Amazon.
Yay! They finally offer static IP addresses for Amazon EC2. This is a huge step for anyone wanting to use EC2 to host anything on the internet. They are free of charge if you are using them with a running instance. If they are not associated with a running instance, there is a $0.01/hr charge for them.
This means that you won’t have to worry about updating DNS with a new IP address if for some reason your instance shuts down.
They currently have a limit of 5 IP addresses per account, but you can request more if you need them.
They also have a new thing called “Availability Zones.” This lets you put your instances (servers) in different locations allowing you some redundancy in case of some sort of failure of a location (like power failure or network problems.)
Facebook has announced a partnership with Amazon Web Services.
I don’t quite see what the partnership involves other than some Facebook specific documentation and extra samples of facebook apps that run using AWS.
Either way, it’s good to see some more promotion of AWS. As you know, I’m a big fan. We’ve been using AWS with our We’re Related facebook app as well as the entire familylink.com website since October and love its instant scalability and cost savings. I’ll be posting more about our Amazon EC2 experiences later.