- 15 Non-Certified IT Skills Growing in Demand
- How 19 Tech Titans Target Healthcare
- Twitter Suffering From Growing Pains (and Facebook Comparisons)
- Agile Comes to Data Integration
Network World - One of the promises of the public cloud is the opportunity to quickly dial up server resources in order to do a job that calls for heavy-duty batch processing. But first you need a way to manage the life cycle of that job. Fortunately, there are tools that can automate setting up and tearing down jobs in the public cloud.
In this groundbreaking test, we looked at products from RightScale, Appistry and Tap In Systems that automate and manage launching the job, scaling cloud resources, making sure you get the results you want, storing those results, then shutting down the pay-as-you-go process.
We found that each product gets to the finish line, but they each require some level of custom code and they each take different, vastly circuitous routes.
We liked RightScale's ability to both monitor and control application use, as well as its wide base of template controls and thoughtfulness for overall control. RightScale's RightGrid methodology manages the full life cycle of apps and instances, and gave us the feeling that hardware could truly be disposable in the cloud.
Yet with a bit of work, we found that both Appistry and Tap In Systems offered task automation components that could also be successful for cloud-based jobs.
In our first public cloud management test, we focused on the ability of products from RightScale, Tap In Systems and Cloudkick to simply monitor public clouds like Amazon's EC2, GoGrid and Rackspace.
This time around, our test bed was narrowed to Amazon's public cloud, and we used a variety of Amazon cloud services, including Elastic Compute Cloud (EC2) server resources, Simple Queue Service (SQS) queuing system, and Simple Storage Service (S3).
The good news for enterprises is that Amazon's pay-per-usage model can be a major cost saver. In this real-world test, we were able to complete our tasks using an extraordinarily low amount of our Amazon processing budget.
Similar batch job cost savings can be realized using Amazon competitors like GoGrid, RackSpace and others, but only if the tasks are automated using the type of cloud management tools that we tested here.
The basic procedure was similar for all three products A job that needs to be performed requires application code, data files, a place to process the data (the cloud), and a place to put the results.
There are two options: make the job into a bundle where we could define code, data, outputs and options, or do that plus have a controller get messages from the job in progress. That allowed us to take either pre-defined actions based on the messages, or allowed us to change what happens in the middle of a job.
First, we needed to create an Amazon execution image with the applications we would be using for automation. We chose ffmpeg, an application suited towards video rendering jobs for processing by arrays.
Once created, we bundled the image and uploaded it to Amazon so we would have a copy to start with. Each product then varied in terms of controlling the life cycle of the bundle. Typically, the life cycle is the sequence of events that coordinates the process of doing jobs, gathering the results, storing them, and reporting success/failure (there will inevitably be both).