AWS tricks #1 -- How to get rid of terminated volumes

Introduction

I spent this weekend figuring out how to install ODI 11.1.1.5 using SQL Server 2008 not Oracle 11g as the database back end and I now know two things:

1. I really ought to be doing this on Oracle instead of SQL Server as all of the blogs seem to cover the settings for Oracle.  I find it interesting that the 10.1.3.x blog posts seemed to be oriented around SQL Server and 11.1.1.x around Oracle.  However, as my environment was SQL Server, not Oracle, I was stuck with figuring it out.
2. I am not very good at installations.   This may not come as a shock to anyone, especially me.  :)

In an upcoming blog post, once I am convinced that I have everything working tickety-boo, I will document the settings and required downloads to make ODI and ODI Studio work in a 64-bit Windows server context.

Putting aside the additional grey hairs and general agita installing ODI caused me, this exercise got me back into the Amazon Web Services (AWS) cloud and that was a life-saver.  Just like a virtual machine, I could spin up as many instances of the server as I needed to blunder my way through the install and could snapshot it when I had something halfway working.  Unlike a virtual machine running on my laptop, I had 17+ gigabytes of RAM, super-fast networking (you haven’t lived till you switch from DSL to whatever AWS runs – I was getting 39 megabit per second downloads from Oracle’s website), and real server power.  

However, as I was looking at my AWS account, I saw that I had a bunch of AMIs, snapshots, and volumes that I couldn’t really account for.  Oh, I created them all right, but I hadn’t used them in ages and I was getting billed each month for them.  A mass culling ensued but it occurred to me that this is  a good time for some basic definitions.

AWS definitions

One of the (many) things that confuse me about AWS is the concept of “volumes”.  Simply put, a volume is a hard drive.  

Then there are these things called “snapshots”.  Snapshots in AWS’ Elastic Block Storage (EBS) world are just what they sound like – snapshots of volumes that you can restore back to at any time.

It is very important to note that the hard drive of an Amazon Machine Image (AMI) (a predefined server you can start) is ephemeral, i.e., your laptop’s hard drive this is not – when the instance is killed, the hard drive is gone.  It’s very easy to blow away all of your work if you don’t understand how AWS treats volumes.

This post will cover what a volume is, and how, when, and why you would get rid of a volume (you get charged for each one) as they have a way of piling up when least expected.

What’s ephemeral, and what’s not

A really good explanation of AWS’ drive ephemerality can be found here:  http://shlomoswidler.com/2009/07/ec2-instance-life-cycle.html.  If you cannot be bothered to jump to a well written blog, the gist is that EBS-backed instances have volumes that persist as long as the instance is not terminated.  

And how does that affect you?

Well, when you stop an instance (Stopping an instance is akin to powering down a server) the volume is still extant.  It makes sense (to me at least) that a volume hangs around when a server instance is stopped.

When you terminate (Terminating an instance is like you shut down the server, ripped it out of your data center, and naively took it to a recycling center which then had the hardware shipped tout de suite to Liberia for completely unsafe disposal.) an instance, there is a chance the volume will still hang around.  But why is it still there when an instance is terminated?  Doesn’t that destruction of the instance sort of imply that the hard drive volume should be trashed as well?

Remember, if you started your stopped instance back up, whatever changes you made to your boot drive are still there.  

But what about the volume that is the product of a terminated instance?  If you reattached it to a new instance, are your changes there?  Nope, all of the changes you made are gone.

An example

Let’s pretend that you fired up John Booth’s EPM 11.1.2.1 AMI (go to http://www.metavero.com), did cool stuff, and then terminated the instance.  Guess what – that C: drive is still around, and you’re getting billed for it.

Not deleted till it’s detached

Here’s what the web console to my AWS account looks like with four volumes attached to stopped instances and one 100 gigabyte volume from a terminated server instance.  
What is that volume doing there?  Did I really terminate that instance?  

As I wrote, I’m being charged (not very much, but still) for storing that 100 gigabytes.  I want to make that unloved hard drive go away.

It’s dead, Jim

That instance is deader than dead.  That drive volume is in AWS purgatory.

What’s going on?

That volume is going to hang around till you delete it.

Why?  A little searching of the tells us that the AMI must be set up to delete the volume on termination.  If it’s not, then the volume must be manually deleted.

NB – A future blog post on launching an AMI from a command prompt will explain how to launch an AMI with that parameter; if the AMI has been set up to not delete the volume on termination, you must manually delete the volume as shown below.

How to delete a volume

Luckily, this is an easy process.  Simply right click on the available volume and select “Delete Volume”.

You get one more chance to change your mind.

AWS will take a short time to delete the volume:

If you get impatient (it can take a while to delete a volume and the bigger the virtual drive, the slower the delete), you can click on the Refresh button to see the current status.

Until the volume is finally gone.

Conclusion

If the AMI is not set up to delete the volume on termination, you must do so manually, or modify the AMI via the command line interface to delete upon termination.

Set up your AMIs to delete volumes on termination or remember to check and delete available volumes – remember, you are being charged to store drives you no longer use.