Cloud and Web 2.0 Insights from Structure 09 Conference

Last week I attended the GigaOM Structure 09 Conference, which is an innovation-oriented cloud computing conference. One of the interesting things about the Silicon Valley-based event: it brought together a mix of different types of companies, emerging technology products and services, and people with cloud challenges:

1. Mainstream vendors discussing their cloud computing plans, with an emphasis on enterprise offerings: their challenge is convincing potential customers that there's a pony somewhere in the cloud (please excuse the mixed metaphor.)

2. Startups, which in the main are focused on helping cloud users increase productivity by providing services that make existing cloud providers (particularly Amazon) easier to use: their challenge is the challenge of startups everywhere, which is to get people aware of their solution and interested enough to try something from an untested (and unproven) vendor.

3. Large-scale Web 2.0 companies like Facebook, LinkedIn, and the like: their challenge is two-fold:(1) managing the enormous amounts of compute, storage, and network traffic they experience; and (2) if that isn't difficult enough, having scaling those resources as more and more people pile onto the apps and the companies roll out new features as well.

The large Web 2.0 companies were represented in a panel on managing huge cloud systems, which was really fascinating for two reasons:

1. Large Web 2.0 companies are somewhat akin to scouts, in that the challenges they face will inevitably come to most compute users. One of the things we talk about with companies is that the nature of compute tasks is morphing away from transaction-focused to interaction-focused, with the latter category generating much more data from clickstreams, sensor outputs, and internal and external collaboration. Most companies will eventually face the same scale challenges explored in the panel.

2. One really got the sense from these companies that they are running as fast as they can to keep up with demand growth. In other words, they're creating solutions and fixing problems in real-time, which must make their daily lives interesting.

[For timely cloud computing news and expert analysis, see CIO.com's Cloud Computing Drilldown section. ]

Another important panel was "Better Broadband: Enabling the Cloud Era," which focused on one of the key issues confronting cloud computing-getting data to and from cloud providers.

We refer to this issue as the "skinny straw," which vividly describes the issue: the limited bandwidth available to many organizations and enterprises to communicate with cloud-based applications and storage. This issue is particularly thorny for organizations that want to take advantage of one characteristic of cloud environments: huge storage. While clouds can be great for storing and manipulating terabyte-scale data stores, if it's impossible to actually get the data into the cloud, the scalable storage available there is a tantalizing but unobtainable goal.