Bringing the Cloud down to the Ground and no, the result is not fog, part 1

Running the metavero.com AMI on your laptop
Huh?  How can the Cloud (aka Amazon Web Services) run on the Ground (aka your laptop)?  How do you run an Amazon Machine Image outside of Amazon’s cloud?  Specifically, can the recently released 11.1.2.2 AMI be run on my laptop?  

Read on, and all will be revealed.  It’s a bit involved but oh so doable.  And most definitely one of my more satisfying hacks although to be honest, I am just combining ideas other people came up with.  So that would be in the best tradition of programmers then, e.g., I am stealing their work and applying it to my problem.  :)

One thing to note – this will be a multipart blog post as there are many steps.  I want to spread out the work give you time to digest this in bits.

A special note of thanks

Dan Pressman, Essbase wizard extraordinaire, was my QC guy on this post and contributed a lot towards figuring out how we (him, me, and you too) could get the licensing straight.  Yes, it’s a blog about Essbase Hacking, but not that kind of hacking.

In any case, this is sort of a long blog post (60 plus pages in Word which is where I initially type up all this stuff – fear not there are a ton of pictures but still there is a lot going on here)  with a lot of steps (sorry, what we’re doing ain’t all that easy).  In line with testing good practices, he who writes the solution shouldn’t test it.  Dan has it all working, and well.  I will explain the differences between his eventual solution and mine later.

What skills will you learn?

As I’ve written before, I view these posts in two ways:  a way of disseminating useful information, and a way of forcing self-training.  The first point is hopefully obvious; the second is a simple fact of my (and maybe yours) life – I am in an industry that constantly challenges me to learn, I am somewhat lazy (you should see the state of my office – the floor is almost visible through the papers), and doing unusual but job-related things makes me really learn how to do something.  

And that’s why I’m so excited about this post from a skills perspective.  Here’s a high level list of what you will understand if you follow through this whole thing:

• Amazon Web Services (AWS) Cloud architecture including Elastic Compute Cloud (EC2) and Simple Storage Service (S3)
• Interacting between the Cloud and your local machine
• VMWare conversion tools
• EPM 11.1.2.2’s resource requirements
• Windows licensing (more complicated than you might think)

Why oh why would you want to do this? 

With that introduction out of the way, here’s why you will likely want to give this a try, or at least read through the adventure that is running the Cloud on the Ground.

Cost

The initial impetus for this is mostly (at least on my part) all about cost.  I love AWS, aka the Cloud, to pieces, but I can spend $150 or more per month on it.  That’s acceptable, but I am a one-man band, and I have to try to watch every penny.

Ease of EPM installation

That is the beauty of this approach – I am not a good installer of EPM products (to put it all so mildly), but I know someone who is.  He has very generously shared his knowledge with the EPM community so we can all play with 11.1.2.2 in the Cloud.  If I take John’s work and bring it down to my laptop, aka the Ground, I have a competently installed EPM system on my laptop and I don’t have to install anything EPM-related whatsoever.

Remote access

When you use the Cloud, you must have an internet connection.  Port 3389 (that’s what Terminal Services aka Remote Desktop uses by default) blocked by your company’s firewall?  Too bad.  On a plane, or somewhere in public without a good internet connection?  Too bad, yet again.  

Running the Cloud on the Ground only requires your laptop and a battery big enough to keep it going.

Security

I am 100% uncomfortable (and you should be, too) putting real anything out in the Cloud, at least with the AMI as it is provided.  Yes, AWS EC2 is inherently secure, assuming you haven’t used a 0.0.0.0/32 rule in your security groups, but John’s AMI is really not fit for production data until it has been hardened.  Think VPNs, hardened usernames and passwords, etc., etc.  In other words, real IT (you know, what you or your clients have in house) security.  I’m not suggesting that putting this on a laptop makes it more secure (think of stolen laptops, key logging software, Trojans, etc.) but there is at least some physical security on a laptop (the Ground) that simply isn’t there when it’s on the Cloud.  I still wouldn’t put sensitive financial information on my laptop but I might not have a fit if I had less sensitive data on my local hard drive.

It’s a cool hack

This is Cameron’s blog for Essbase Hackers, right?  Isn’t this is most certainly a hack?  ‘Nuff said.

Why would you not want to do this?

Despite the cost savings, ability to connect anywhere, and overall geeky coolness of this approach, there are definitely some reasons why you would not want to do this.

Performance

I used to think the Cloud’s AWS m2.xlarge instances weren’t all that fast.  Oh, how wrong I was.  They are blazingly fast, at least compared to my poor, almost three year old Ground-based laptop.  Blindlingly fast in fact, with as much room for expansion as anyone could ever want.  Maybe if I had a bigger laptop I wouldn’t think this way, but with the old Latitude 6500 I have, trust me, the Cloud is many times faster.  Yes, this is a great reason to go buy a faster laptop, but I am trying to get as much time out of this thing as I possibly can.  On the other hand, watching Task Manager pegged at 100% a lot of the time is getting old.  

Capacity

As I wrote above, the Ground requires a big laptop, or at least one more capable than mine as I am really pushing its limits.  At the bare minimum, 64 bits (you would be surprised how many people still have 32 bit laptops – it isn’t going to be enough by a country mile), eight gigabytes (16 would be much, much better, even though the EPM instance is a compact deployment) of RAM, at least 100 gigabytes, preferably more, of free disk space, and a dual CPU.  These are the bare minimum specs – believe me when I tell you that my system is usable, but only just.  It can be very, very, very slow at times.  

Sharing

When you run the 11.1.2.2 AMI on the Ground, you are pretty much making this a box that only you can talk to.  Yes, you could stick it on a LAN and make it available that way, but even if you have a powerful laptop, it’s not going to be a substitute for a true server.

Flexibility

One of the many great things about the Cloud is that you can scale servers, clone them, and do all sorts of awesome things with them that only the Cloud or an incredibly indulgent IT department brings.  None of those possibilities are there with a laptop.  You must keep that in mind when going down this path.

Time

The converting, compressing, downloading and enabling of all of this to work on the Ground is not exactly quick.  Some of this is down to how fast your internet connection is – I have a DSL connection and fast it is not.  It literally took me two days to download the compressed VM from the web.  Did I mention I had to set up a way to host those files (I will reveal how to do that below)?.  For sure I wouldn’t attempt this on a cell phone plan.  

Cost

Huh?  Isn’t the Ground supposed to be cheaper?  Well, only if you already have a valid Windows 2008 R2 Datacenter license.  That’s right, the license that Amazon has with Microsoft is not valid when it’s transferred to your laptop.  Sooner or later (like within three days) you will start getting warnings about Windows not being activated.  You are going to need that license to do the activation.  And don’t think you can find cracked licenses (this is a hacker’s, not a pirate’s blog)

The retail cost is $2,389 and that’s without the ability to actually connect a client to it (aka CALs).  Yes, that is a little eye-watering and yet another reason why AWS is so awesome in so many ways.  However, if you’ve read this far you are determined to do it, and there are cheaper ways of getting a legitimate license.  Beyond hitting your IT department up for a license, or your consulting company management, some legal alternatives are Microsoft’s Action Pack, MSDN, TechNet, and Bizspark.  I’m sure there are other ways to do it as well but those are the ones I know of.

NB – The cheapest way to legitimately get a 2008 R2 Datacenter license is TechNet Professional.  You must qualify for the license, but once done, the current cost is $349 for the first year and $249/year renewal.  Here’s a snippet of the Technet Product List.  Note that you must either have TechNet Professional or TechNet for MS Competency Partners (that’s how Dan and I qualify) to get 2008 R2 Datacenter.

One other thing to note – TechNet grants licenses for evaluation purposes only.  Apply the key to a production environment and you are in violation of your license.  I will let you figure out how much risk you enjoy, but any prudent business avoids legal nastygrams as much as possible.

Also, don’t forget that you have to have a laptop sufficient large and powerful enough to run all of this as detailed above.

Don’t be scared off by the upfront cost as ultimately I think the Ground is the cheapest way to run the 11.1.2.2 AMI, but just know that there are many reasons to stick with the Cloud.  I have a bunch of interesting things I want to test with Essbase 11.1.2.2 that I will share in the not too distant future – for sure I’ll POC on my laptop but right after that I will be going back to AWS.

Required resources to make this all happen

With this stuff, you are on your own

• An ability to follow directions
• An Amazon Web Services account
• A desire not to incur AWS charges
• A 64 bit PC/laptop with at least eight (but very preferably) gigabytes of RAM
• A fast internet connection
• Lots of disk space (about 100 free gigabytes, but more would likely be better)
• A valid Windows 2008 R2 Datacenter license
• Patience

Here are the tools you need to grab from the web:

• VMWare Workstation (or VM Player, maybe, but I have not tested it)
• VMWare’s VCenter Converter
• John Booth’s excellent 11.1.2.2 AMI
• The 7z compression program
• Cloudberry Explorer (not absolutely necessary, but it allows you to stream multiple downloads from Amazon S3 if you go that route for downloads – tip of the hat to new Oracle ACE John Booth for suggesting that to me)

Whew, that’s an awful lot of perquisites, isn’t it?  And yet what I am going to show you is pretty awesome, if I do say so myself.  Ahem.

Converting the AMI

The trick to this is to use VMWare’s VCenter Converter software.  It is not actually meant specifically for AWS but is instead a way to virtualize any Windows (and Linux) machine.  The fact that you will virtualize a cloud machine is irrelevant to VCenter Converter, although it is geeky cool.

Give yourself some drive space

Unless you started up the 11.1.2.2 AMI with a really big drive, and even then for best performance, you will need to create a drive to do all of your converting, zipping, etc. work.  As with so many things in AWS, it’s really quite simple.  This is all assuming that the 11.1.2.2 AMI is up and running.

Create a new volume

In the EC2 interface, create a new volume by clicking the eponymous button:

Set the size

This one is set for 150 GB.

Go check to make sure it’s done

This is generally quite fast.  The State needs to be “available”.

Attach it to your running instance

This is a two step process:

Format that drive

So your Windows box now has a new hard drive.  Before Windows can address it, it needs to be initialized and formatted.

Initializing the drive

NB – You can also just go to Start->Run->diskmgmt.msc.  Below is the graphical way.

Open up Computer Management.

Then Disk Management

Set the drive online.
 

Then initialize.
 

Take the Initialize Disk defaults.
 

You now have an initialized, but unformatted drive.

Right click on that Unallocated drive and select New Simple Volume.

Follow the wizard, again taking the defaults.

Step 1 – Beginning the Simple Volume Wizard

Step 2 – Define how big it should be (go for all of the volume space)

Step 3 – Assign a drive letter.

Step 4 – Format it, using the quick format option

Step 5 – You are done!
 

Click on Finish and you should see this:

Here it is in Explorer

Before you convert, get rid of unnecessary files

There’s a c:\media directory that contains all of the install binaries.  Unless you want to keep them around, get rid of this directory as it is 24.5 gigabytes that you will be compressing and then downloading (slowly) to your laptop.  Dan caught this and his download was 15 gigabytes, compressed; mine was 29.

Converting via VCenter Converter

I’ve already given you the download link.  You will be installing this on the AWS EC2 instance.

Once installed, launch it.  This should be what you see.  Click on Convert Machine to get the process going.

Select the powered-on machine.

Select the destination type and VMWare release, along with a name and a target folder.  Yes, you can make all of this work with the free VMWare Player although I have not tried doing it with that tool.

In my case I’m running Workstation 8, so that’s the release I picked but you can run this with older releases of the tool.  Click on Next.

Get rid of the D drive – there’s no point in moving it across to your laptop as all it will contain is a copy of your VM.  Which will reside on your VM.  Which is confusing.

Set the memory size – in my case, it’s going to be quite a bit smaller than the 17 gigs that an EC2 m2.xlarge instance provides.  Also, although I didn’t do it in this screen shot you should not convert the D drive – all you want is C.

NB – I am showing this a bit out of order so the D drive is still selected.  Regardless of me not being able to demonstrate a sequential process, you need to get rid of the D drive.

Set the network to NAT.

Click on the Advanced section above, then click on “Customize guest preferences for the virtual machine”.

That will get you to the Customizations section.

I went with these settings (they seem pretty logical):

Set the Workgroup.

Review the settings and Go Man Go.

Now the conversion process begins.

You can see the process beginning by clicking on the task line.

It takes a while.

But it will finish.

And here it is in Windows Explorer.

So where does this leave you?

Aren’t cliffhangers great?  I suppose only if the reader has hung around till the end.  But you’re here, hopefully, so what do we have and where do we go from here?

At this point you have :

• Added a hard drive in AWS (and seen how trivially easy it is to do)
• Added and formatted that drive in Windows
• Downloaded and installed VMWare’s VCenter
• Converted your metavero.com 11.1.2.2 AMI to a format that VMWorkstation (or whatever) can read

I wish I could tell you that all you now need to do is download the VMWare machine to your local drive and off you go.  Alas, it’s not that easy, but the next and final post will take you through all of the steps that Dan and I painfully worked through.  It honestly isn’t that bad but there are a few twists and turns.

Hope you’ve enjoyed the hack!