Amazon CTO Vogels on competing with Microsoft in hybrid cloud, keeping partners and customers happy

Werner Vogels joined Amazon.com in 2004 and a year later became CTO. During the past decade he helped define the public infrastructure as a service cloud market. To do so, Amazon Web Services built a massive network of data centers around the world, which to this day continues to grow rapidly. This month at the AWS Summit in New York, Network World Senior Editor Brandon Butler spoke with Vogels about the state of the cloud industry, challenges facing AWS, its relationships with partners and customers, and what he’s learned from growing AWS into the business it is today.

Amazon Web Services is credited by many as the market-leader in the public IaaS market, but Microsoft is hot on your tail. One of the key differentiators for Microsoft Azure is its focus on the hybrid cloud. What are you doing to support hybrid cloud computing?

Customers have made major investments in their infrastructure, so that will not disappear overnight. Though some of our customers are going all-in with the cloud; Conde Naste, for example closed their last data center. NewsCorp has reduced from 66 data centers to six and moved everything else into AWS. Some workloads cannot move yet because they’re tied to particular hardware. So, we’ll live for a while in this world where the majority of things live in the cloud, but some things still live on premises.

Often times there are what we call IT life events that slowly start taking pieces out of the data center. So if servers go off lease, what are you going to do, are you going to buy more servers? Or would you use an IaaS? What if a software license expires, are you going to buy more software, or are you going to use a software as a service provider? Slowly but surely what is in the on-premises world will start disappearing.

Why is this happening? Because managing hardware is not a competitive differentiator for anybody – everyone has to do it. Often these on-premises environments are run by folks who have significant intellectual capabilities. You could be using them for something that really matters for your company, but instead they’re just keeping the lights on. At AWS, we’ve become really efficient at running data centers, so let us handle that.

There will still be on-premises workloads though, so we have built a whole range of tools, whether it’s Virtual Private Cloud, where customers can cordon off a piece of the cloud and then use a service like Direct Connect to provide circuits to the cloud. Or, we have a whole range of security tools that work both on premises as well as in the cloud. Identity and Access Management (IAM) has federation capabilities so you can have single sign-on on-premises and still make use of the IAM roles to exactly describe what specific individuals can do in your AWS environment. And then we’ve made it easy to migrate workloads, so if you have VMware images, you can convert them into AMIs (Amazon Machine Images) and also do it the other way back.

A number of our own management tools work on premises and in the cloud. CodeDeploy and OpsWorks were initially targeted for the cloud, but now can be deployed on premises. For backup and disaster recovery we give customers a storage gateway, a virtual machine image they can run on premises that you can attach volumes to and have those automatically backed up into Amazon. We are continuously looking at building blocks that our customers want to manage these hybrid environments.

Would Amazon ever build an on-premises private cloud management platform?

A very large part of cloud is the operational side such as reliability and security. Software takes care of large pieces of that, but those are mostly operational functions. You should expect us to continue to focus on really investing in security and operational excellence of our operational environment much more than focusing on building software. We’re not a software distribution company; we’re really an operational services company.

One concern some customers have is to avoid vendor lock-in. This seems like a concern that’s exacerbated in the cloud. How can customers ensure they’re not locked into AWS and the cloud?

Most enterprises, regardless of cloud or on-premises software, always think about their contingency plan. There are a number of architectural best practices customers can follow that make it possible to not be locked into the AWS cloud, giving them the option to move back on premises. We don’t lock you into any programming language, or any operating system, or any middleware. You can use Ruby, Java, Python or Go. Many software vendors use AWS, too, and are fully supported. This is standard, off-the-shelf software that you can find not just on AWS but on-premises as well. So we really make sure that our ecosystem is as broad in the cloud as it is in on-premises software. Meanwhile, the APIs are all built using HTTP and HTML. So you’re not locking yourself into Amazon software.

You’ve mentioned that you’re trying to build services that are compatible with existing workloads in customer data centers. But then you’re also building new services that many customers don’t have in their data centers like Lambda, and the Machine Learning service. How do you balance building tools that are specific to AWS’s cloud with ones that will work well with customers’ on-premises tools?

Well what we do is listen really closely to our customers. And our customers have no trouble giving us feedback on what they would like to see next. We’ve noticed over the years are two major pain points: Archiving and data warehousing. If you have to store more and more archiving data for regulatory requirements, it’s often just a pain. So we developed Glacier to meet that demand. We just launched great new features to Glacier that allow you to mark files as read-only, for example. You can also now attach policies to data in Glacier – like this data is never allowed to be deleted, or this data will have a lifetime of one year and then can be automatically deleted.

The other big pain point for customers has been data warehousing. By delivering RedShift, every department within an organization can have its own warehouse in a matter of hours. That’s the same reason we built WorkSpaces. A lot of organizations are supporting bring-your-own-device to work, but to do that, you need to support a well-mixed environment that your workers can use on those devices, and WorkSpaces (the virtual desktop as a service) enables that.

I’ve spoken to companies that monitor AWS use for customers who say that the vast majority of AWS use is still in EC2 and S3. To continue the company’s meteoric growth, how aggressively do you need to get customers to use services beyond just basic compute and storage?

Most of our customers are using EC2 and S3, but not only that. They’re using a lot of the other services, too. For example, RedShift is the fastest-growing service we’ve ever built. EMR (Elastic Map Reduce) is something we built on top of EC2, so it inherently uses EC2. Many of our services are integrated like that. CloudFormation and BeanStalk map down into EC2. So we have all these basic services and then more advanced services are built on top of that.

We’ve made our services flexible, too. Maybe you’re using CloudSearch from Amazon, but you think a partner’s search product matches your use case better. You can mix and match. But we’re not aggressively pushing use of any of those new services. Our approach really is to build primitive building blocks and tools, and let customers choose which ones they want to use.

One criticism I’ve heard from ISVs is they’re worried AWS will move from being a partner to a competitor by rolling out the service they specialize in. Should ISVs be concerned that if they offer a service on AWS’s Marketplace that AWS will begin offering that service too?

Most of the time when we’ve developed products that play in the same space as others in the market, those are not winner-take-all markets. There’s very healthy business to be had by all. I remember when we launched DynamoDB, quite a few people mentioned that we were encroaching on the space of Mongo or Cassandra. If you talk to the Mongo guys, they were ecstatic because it actually validated their business. So really we focus on what is the best thing we can do for our customers.

Containers are all the buzz in the technology industry right now. What do you see containers really being useful for? What will be their place in the enterprise?

+MORE AT NETWORK WORLD: 12 Hot application container startups +

More and more enterprises are taking an agile approach and are embracing a devops style to development. In that approach, it works really well to build small building blocks with each having its own scaling requirements, reliability requirements or cost parameters. Containers can become the output of the development process. Our container scheduler makes it easy to replicate containers over nodes of EC2 instances or over multiple Availability Zones; you can connect them to load balancers and disk storage.

Some AWS customers have recently called for you to be more transparent about the company’s use of green energy and how AWS services are architected under the covers. How do you decide what information you make available to the public on these topics?

In most of these cases we actually do make information available. Last year at re:Invent, James Hamilton (AWS’s Distinguished Engineer) gave a really detailed presentation about how our networks are laid out, how our data centers are laid out, what the capabilities of those data centers are, what the power infrastructures that we’ve built are, what the unique network devices that we have built ourselves are and so on.

The other topic you mentioned was green. Our goal is to have green power for all of our regions. At this moment, the Oregon, GovCloud and the Frankfurt region are 100% carbon neutral. And we’ve recently launched two programs to invest in the generation of green energy to offset the production of other sources of energy. We’re building a solar farm in Virginia and we’re building a wind farm in Indiana. Hopefully we will be able to generate enough energy in those farms to offset our usage of other types of electricity.

What have been some of the biggest lessons you’ve learned building AWS?

When we started we thought that an API and documentation was good enough. But over time, we’ve learned just how important the people side is. We know how to build really great technology that can scale and be reliable and have APIs. What we really took time to focus on is how customers are actually using it. Not everyone is a gung-ho, low-level developer who knows how to figure out APIs. We had to make sure there were the appropriate tools and functionality in all of our services.

We really had to learn how to remove all barriers. We didn’t offer support in the beginning – but that’s something customers asked for, they said ‘we need your help, and we’re willing to pay for that.’ It took a while to get solutions architects that will help review and improve customer architectures. Professional services; training and certification – all of these were things that we did not think about in the beginning, because we really were focused more on the developer ecosystem. Now, our support offering is actually a common asset to the AWS Cloud. People are picking AWS because of our support.