Containers have taken the networking world by storm, offering a lightweight, more flexible alternative to the traditional virtual machine. The major difference between a container and a VM is that a container may share common files, while VM processes are discrete and atomic, even if storage and networking is virtualized and shared. VMs are more like islands; containers can be islands or communes.
One day, we predict, most instances of operating systems atop hypervisors will be containers. In fact, OS makers are already tripping over each other to make ultra-slim, container-friendly versions of their products.
+ ALSO ON NETWORK WORLD: Best open source monitoring tools | How to tell which container is the right one for you +
So, are containers ready for prime time in your network? What are the pros and cons? To investigate those questions, we tested three container methods, Docker, Rocket/rkt and openVZ/Odin (formerly Parallels.)
Docker leads the way in terms of momentum, but there are also potential problems that need to be surmounted. Rocket/rkt addresses some of these problems, but isn’t yet production ready.
OpenVZ/Odin provides both a container methodology and a “virtual environment” reminiscent of popular hypervisor platforms, but with a few limitations.
All three are potentially very useful and also potentially very dangerous compared to traditional hypervisor and VM combinations. However, with applied care and methodical consideration, all three have great potential.
We tested these products in the context of enterprise systems infrastructure, rather than DevOps and Continuous Development. The energy behind containers, however, means that it’s very difficult to extricate systems infrastructure and its control planes versus the needs/benefits of rapid deployments, scale, and the philosophies behind continuous development practices.
Each of the products tested is in rapid transition, with the elder statesman OpenVZ/Virtuozzo playing a slightly different role, as more of a control plane product for service providers.
The problems containers solve
The evolution of containers is driven by a need for process orchestration that is lighter than VM deployment, but with some of the same systems approach to clustering, and rapid setup and teardown. All this must be done with reliability and security, which is where the heaviest criticism of each platform comes in, and it’s one of the byproducts of the evolution of cloud computing concepts.
+ ALSO: Standards are coming for containers +
In the openVZ/Virtuozzo scheme, container process and rapid roll-out are poised more towards service providers, who are offering appliance-driven infrastructure on the hoof to customers. Examples include canned website/email hosting, website merchandizing, highly language-localized appliance-like offerings, and associated services. In this scenario, instrumentation and billing for multiple baked instances of operating systems are key to a prospective client.
Odin, the renamed branch of Parallels that sponsors openVZ and offers Virtuozzo, has been doing this for a long time. It’s more predictable, more baked, and flexible enough to also be used as a hosting methodology for Docker.
Docker supplies a lightweight OS environment that’s subjugated and disciplined by controls from applied instrumentation. This instrumentation can be rudimentary or involve sophisticated management control planes.
Rocket/rkt is similar to Docker, but more primitive in some ways (and not ready for production). However, as Rocket seems to have been spawned as a response to the perceived sprawl and insecurities in Docker, we found that rkt satisfied some of our engineering instincts that we found wanting in Docker development.
Odin and its foundation of openVZ bills itself as a virtual environment, a hybrid model where a modified kernel is used to do both jobs of a traditional hypervisor, and alternately, the job of a container hosting environment. For example, you could run Docker or Rocket on openVZ.
We found that these three use very similar methodologies. Indeed cross-fertilization among these three parties, and a long list of other visible contributors, is huge. Each seems to sponsor the other in varying ways, and they’re representative of an emerging solution to lightweight instance management without the “heaviness” of traditional hypervisors.
So, do container hosting methods replace traditional Type 1 Hypervisors? Our answer today: not quite—these are different species. But we’re also noting that lightweight OS instances, such as CoreOS, skinny instances of Ubuntu Server, and even Red Hat’s upcoming Atomic distribution, are all designed to serve as the framework inside container hosting method. And we wouldn’t be surprised to see Windows 10 containers on the horizon, although MacOS containers are a stretch of the imagination.
Here are the individual reviews:
OpenVZ/Odin Virtuozzo 6.0
OpenVZ is a Linux distribution that houses hosted guest instances of VMs and/or containers. OpenVZ resources are in modern 3.X+ Linux kernels, and an OpenVZ kernel exists for downloading. Only the OpenVZ kernel is guaranteed to support all features—although a long list of partial OpenVZ services can be installed on Linux 3.X+.
+ ALSO: 12 hot application container startups +
OpenVZ guest instances can be either VMs or containers. When you add instrumentation, billing, and a management plane, you get the commercial product Odin Virtuozzo. Odin is a re-branding of this product by Parallels, a Swiss corporation.
We reviewed Virtuozzo in its Windows incarnation in 2008. Memory limitations of 32-bit servers at that time reduced the usefulness of Virtuozzo. The Windows product remains, but Virtuozzo now has the ability to use a shared kernel to do containers in the Linux realm.
We downloaded openVZ and installed it on a Lenovo ThinkServer RD640. As this scheme surrounds a Linux kernel of comparatively ancient origins, hardened as it may be, hardware compatibility needs to be ensured. We had plenty of server features, and we were told—but did not thoroughly test—that server hardware compatibility associated with Red Hat 6 will work with openVZ or Odin.
You’re tied to whichever HBA and JBOD disk combo you can muster although we verified that openVZ and Odin will work with RAID combos. IPv6 is not fully supported, although “hard” IPv6 VM/container addresses are supported and were tested.
Virtuozzo is a virtual environment, and in our testing, hosted VMs are typically Windows Servers. Linux and FreeBSD are best done as containers. Virtuozzo is controlled via a web app, and has necessary command line components that aren’t in the UI. It can be almost entirely controlled by CLI, if you desire.
Containers can be made from Docker, Rocket or LinuXContainers/LXC. There’s also a repository of OpenVZ/Virtuozzo containers available via download from Odin.
Implementation in OpenVZ is similar to Docker and Rkt. Virtualized environments are installed easily and gave the same performance as the same hardware under VMware 5.5, similarly configured. We did not use extensive benchmarks to come to this conclusion, just the SciMark browser tests we use as a loose metric for performance with Firefox 35, given similar workloads and background usage profiles.
Viewed this way, OpenVZ is much like a hypervisor, although lighter in overall processing weight than VMware, and more like Windows Hyper-V. It also doesn’t have VMware ESXi 5.X’s extensive feature list—or cost.
Testing OpenVZ and Virtuozzo
We tried OpenVZ/OVZ first. There are two branches of installation, one for RedHat-ish releases, and the other for Debian releases. Beneath each of these is a branch for pre-sysctrl and inferences for post-sysctrl. Functionality was essentially the same for all branches, as installation foundation binaries are similar.
The docs are geared for Linux-savvy installers and aren’t complete enough for novices.
OpenVZ wants bare metal although it can be run as a VM. When using bare metal, performance can be staggeringly faster. We suggest strongly not to use OpenVZ or Virtuozzo as VMs as performance cuts, especially when overallocating resources, will cause the entire host to thrash.
OpenVZ containers, or “virtual environments,” are started as a cgroup-demoted service. If the OpenVZ kernel is used instead of a standard issue Linux kernel, then we could use a filesystem called ploop to decrease the number of Linux inodes used.
After deciding the filesystem type, we installed vzctl, the main container controller, then vzquota, which controls container resource volumes/quotas. The vzctl app can then be used to deploy varying container images from the openvz.org site. The containers are very small in initial displacement, and retrieved in the form of a tarball into a cache directory, and deployed from there.
We offer a warning that not all kernels (saving OpenVZ’s kernel) used in late era Linux 3.x kernels offer full support for memory and CPU controls. We didn’t run into a problem, but we can envision scenarios where a kernel doesn’t have all of the OpenVZ toggles compiled into it, resulting in VM or container domination of the host. Ostensibly, a kernel can be recompiled with all the features turned on if desired, but we didn’t test this.
The images are sent with a GPG key so as to authenticate the image. This is important. It’s quite possible to build your own images if desired, although there are plentiful OpenVZ-sourced Linux images. Images downloaded can be deployed directly from image cache. We started many images from scripts; images deploy rapidly, but not as rapidly as Docker atop CentOS in our test environment. The difference in times are negligible, perhaps a few seconds, perhaps because of the time needed by ploop to create its filesystem.
Odin Virtuozzo 6 SP1
The commercial, instrumented side of OpenVZ is Odin’s Virtuozzo, which in our testing, arrived looking very similar to Red Hat 6. Virtuozzo took a very long time to install, and eventually was able to understand our iSCSI storage although we found iSCSI slows Virtuozzo down. NFS is supported and was only slightly faster than iSCSI in testing.
Virtuozzo offers a web page for the minutae of IP configuration (IPv4), including virtual network cards, and VLANs. This is similar to how Xen/XenServer and VMware put networks together, and is poised at multi-tenant configurations.
There are two methods of creating Odin Virtual Environments, one poised towards native hypervisor use—meaning any rational processor/platform-supported operating system—and another designed for Linux container payloads.
+ ALSO: For containers, security is problem #1 +
Those payloads can be serviced by the ploop file systems, but also pfcached/ParallelsFileCacheDaemon with performs deduplication of commonly accessed files in its own ploop filesystem cache. The pfcached uses an algorithm to choose how and when to do deduplication, and we didn’t expect to see an initial performance increase largely because the algorithm takes time to work at achieving greatest effectiveness.
OpenVZ/Virtuozzo containers are instances controlled by vzctl or prctl. One contacts them in the usual ways, ssh, RDP, or the console. In turn, containers execute in their own, normal little worlds, doing work, digesting and storing data, or perhaps offering up a web page.
Containers or VMs are isolated through virtualized Ethernet and no special inter-container or VM communications links are enabled; this must be done on your own, perhaps Puppet, Chef, or another method would be used. Virtual USB, floppy drives, DVDs are available, but only primitive video. It’s possible to build multiple hardware servers that are connected together for high availability purposes, but this wasn’t tested.
We installed four test Windows and Linux ISO binaries (our own ISO sources) as guest VMs, successfully. We did not try advanced graphics, virtualized USB ports, or sound ports. We also tested several Odin images of the same Linux distro sources and found no difference in testing between the two (ex: Ubuntu 14.04 vs Odin-sourced Ubuntu 14.04).
As with OpenVZ, images can be downloaded, ready to deploy on the host substrate. Using pre-formatted templates, user-generated or OpenVZ/Parallels/Odin images get the benefits of optimized memory utilization, although ceilings during a periodic burst activity may slow some VMs or containers, just as it would given administratively-set ceilings in other hypervisors.
CPU MHz (clock cycles), burst limits, and number of CPUs can be allocated. There is no CPU affinity, likely because this is difficult to do with in Linux kernel CPU-task affinity in general.
Using ploop makes file system snapshotting simple, and it’s necessary to do the Virtuozzo live migration of VMs or containers possible. This is aided by the fact that VM or container filesystems are contained as a large object, rather than a discrete set of files, folders, inodes, governed by miscellaneous sysctls behaviors. We did not test live migration.
In use
We could cram a stunning number of containers and VMs into Virtuozzo. Access behaved normally, and the OS found all of the 24 CPU cores in the host, and allowed us to allocate them to VMs or containers. With only a few terabytes of storage and 128GB of memory, we wondered which one would run out first—chip or storage memory. We could overallocate either disk or memory, and so we pushed it, and made some VMs misbehave.
We didn’t receive a notice that a VM or container misbehaved unless it was the VM or container that sent us the message. We were sad, but this problem isn’t unusual. One cannot set minimum CPU thresholds that would trigger an alert that the VM had crashed, only that it was pegging/peaking in terms of CPU. IPMI messaging, or a similar API set that sent host conditions to the instrumentation would be useful and it’s possible to embed nagios or other monitoring into VM/container payloads if desired. Nonetheless, we found the same with disk as a resource—you get messages when you’re full, not when there’s been no activity for 24 hours (a possible indication that the host died).
While Virtuozzo is not VMware XenServer or Hyper-V, it might suffice the needs of multi-tenant internal clouds, or small/medium service providers as a cloud platform.
Containers can also be formed and run with Docker, if Docker’s ecosystem is desired. In as much as Parallels makes their own, we find Docker is somewhat superfluous, although we tested the host platform with Docker under CentOS. Using our primitive SciMark benchmark, execution time is about 18% longer versus running the test in a native container, but only 15% longer than running an equivalent VM. Yes you can Docker, no, we don’t recommend it because it’s nesting a machine inside of a machine inside of a machine.
Docker 1.6
Docker runs as a root process on a vast list of Linux distributions, MacOS versions, and in pilot Microsoft platforms. We tested four versions of Linux hosts and one MacOS host. This is both good and bad. Docker gets much of its popularity from its simplicity, and its power to orchestrate similar and diverse containers with egalitarian controls.
Most current Linux kernels, along with MacOS, have the ability to run Docker containers. One obtains Docker for the host platform, then runs an OS instance launched “inside” Docker’s control. Docker has many conveniences for those hosting OS instances, which by another name, are simply isolated virtual machines whose resources have been orchestrated by Docker.
Part of the value of Docker is the enormous variation of instances that can be selected from Docker’s repository. The repository and container registry at Docker.com is brimming with famous app maker “official” images, some hosted at Docker’s site, others linking to the app maker’s site, still others connected to the GitHub. A stunning variety of development platforms and canned app appliances (think WordPress variations, Hadoop cluster components, etc.) are available to use.
A typical Docker instance might be an instance of Ubuntu 14.04 server, already comparatively small in terms of host resources used. Perhaps the container is running a Linux/Apache/MySQL/PHP/Perl (LAMP) instance and the generic instances are plentiful and varied. Many slimmed down images are now available from Canonical, Red Hat, and even images whose innards have been intentionally slimmed down to prevent unused processes from robbing containers of CPU cycles.
Getting things going is as simple as using the docker run command. This command instantiates an instance of the container/VM, which will use a subdirectory for storage, not unlike what Linux chroot does in terms of lowering security significance, and similar to OpenVZ/Virtuozzo.
Docker container images use the union file system/unionfs, which becomes the folder infrastructure for containers. Like OpenVZ/Virtuozzo, Docker makes layers devoted to each Docker-launched container process via unionfs. This permits similar images to share base files, requiring only one update, and conserving space for when image snapshots need to be made, or when images need updates. Update openssh, as an example, and do it for 20 containers at once, as they rely on the updated image.
The use of unionfs is convenient for Docker efficiency, but it’s also the crux of much criticism as a single bad source image craters all dependent upon it. Docker control can also be made from the user-initiator of commands using RESTful put/gets to container images. This user is as useful or as dangerous as root user, and so user controls related to access, passwords, secure repositories of SSH keys, and other standard security mechanisms are strongly recommended. It’s somewhat loose and fast, if really fun.
ISO images can be built in an automated way, if an organization wishes to maintain their own image hierarchy and control their own image security vetting.
Access to Docker internals can be performed through ssh communications, or other APIs, including communications constructs, such as puppet. Storage is demoted, and can be controlled further via user security (chroot, chmod, or other imposed file limitations/metadata controls). In some cases, you can use ploop or other filesystems to achieve some of the benefits we described with OpenVZ/Virtuozzo.
Docker as container ship
Docker is not mindfully conserving common container space in the way that OpenVZ/Virtuozzo does, in terms of de-duplication of stored common files. In theory, OpenVZ/Virtuozzo can pack containers in more tightly as a result. This said, on the same hardware platform used to test OpenVZ/Virtuozzo, we could put a huge number of Docker container instances in place, perhaps more easily.
But Docker has no control plane instrumentation that compares to Virtuozzo. Docker requires studious management of control scripts and keys, a convenience of Virtuozzo. There is a rush of third parties starting to do just this with app offerings, but those weren’t tested.
+ ALSO: Containers vs. virtual machines: How to tell which is the right choice for your enterprise +
Managing a large fleet of container hosts isn’t difficult, although it requires a slightly elevated set of skills. Docker Swarm is an API that allows a group of Docker containers to behave as an object, resulting in a cluster of containers with a single point of control. This permits rapid instance scale-out, and perhaps instance bloat.
This is a place where we had the most fun, making containers march in lockstep to our commands. This piece isn’t perfected yet, and it also is another place to warn that the users able to address Docker Swarm can also, like the Star Trek Next Generation episode, cause the Borg of containers to go to sleep—or any other command desired.
Creating a container fleet that can be managed as an object requires great responsibility. This said, there are applications like Apache Mesos that can control extremely large data sets as clustered containers. From an organizational security profile, it must be done with great care, as exposed resources are hijack-able resources.
Rocket/rkt 0.5.4
Rocket was introduced and evolves somewhat in conjunction with CoreOS, as a shaved OS designed as a low-attack surface, if high efficiency substrate, for Docker. Billing itself on GitHub as an App Container Run Time (system), it’s an application platform designed for small footprint but high stability.
Rocket lacks some key components and requires more construction savvy than either Docker or Virtuozzo. We were pleased to see that it focuses on security with source image provenance and payload control.
Based on Linux kernels, CoreOS has been evolving at a slightly slower pace than Docker, as it was an OS for clusters rather than a control plane. CoreOS is one of several sponsors of rkt, which really isn’t ready for production. Normally, it’s our policy not to include such products in review comparison without very high demand, as this industry is rife of beta expectations not meeting production reality.
As all three container methodologies use the same somewhat mature components of Linux, we include rkt more for its ideology than its practice. Rkt isn’t simple, but we feel it’s less risky because of its methodology of stricter security at many levels.
Rocket enforces a discipline that starts at container instance building management, as images must be built in a specific manner before rkt can launch them. This varies from the OpenVZ and Docker management, as both of these container controllers can use off-the-shelf ISO images, ones evolved from working machines, the container controller’s repositories/registries, or from your friend in the next cubical. Not necessarily so with Rocket.
In testing we found rkt works very similarly to Docker and OpenVZ in the basics of container runtime control: uses a daemon to control populations of containers, and controls the container instances through the life cycle. How it does this is vastly more regimented and potentially much safer than Docker.
Historically, some of the security/reliability deficiencies perceived in Docker were what inspired the development of Rocket/rkt. A manifesto was generated, designed to embody different core values around containers. App Container (appc) is the resulting specification announced to address perceptions that Docker security is weak, and that inherent systems reliability must be placed on authoritative chains, no matter how big or small the ship of containers becomes. Even rkt allows overrides, however.
To these ends, the resulting appc spec seeks to ensure that downloaded images have signatures of provenance, and integrity of assembly method. We quote the github appc spec:
The core goals of the specification include:
- Designing for fast downloads and starts of App Containers
- Ensuring images are cryptographically verifiable and highly cacheable
- Designing for composability and independent implementations
- Using common technologies for cryptography, archiving, compression and transport
- Using the DNS namespace to name and discover images
Rocket considers itself an implementation of this spec, and others have signed on to the tenets of this spec, including Red Hat, Google, VMware, and Apcera. And while it appears that CoreOS+rkt is synonymous with the appc, rkt is an implementation, as are implementations from VMware (Lightwave/Photon) and Apcera (Continuum), neither of which were available for comparison at review time.
Rkt uses the appc methodology to source its images as tar (Tape ARchive) files, rather than ISOs, so that their GPG keys are hashed with an ISO to make the container base image authoritative. This makes the source files tough to disturb (imagine file substitutions, inadvertent patch levels, malware, broken packages that can be inserted into ISOs). An ImageID results that’s unique to the image.
We created a folder to use as rkt’s rootfs or top-level filesystem. We uncompressed images, which have a pre-inserted, JSON-format image manifest of what should be inside the un-tar’d (uncompressed) image. Additional encryption steps can also be taken so that a key is required (AES-256 was our favorite) at image decompression/decryption time. We await the control panel that can do this without CLI, but it’s not tough.
Once the encryption’s done and the JSON manifest for the image is satisfied, the image can be executed. It becomes a pod, essentially a container, but the word “pod” evokes other mental images, too.
It’s a set of executables with its own ID, which becomes an object handle UUID (covered by IETF RFC 4122) to manipulate the characteristics of the pod execution. The UUID gets its own namespace, created and made subsequently manageable by rkt; this provides instance control. Rkt then takes care of, at initial execution, creating a rootfs for this UUID with a whitelisted set of folders associated with the JSON manifest. It creates the filesystem new, once, each time the container object is launched, ensuring no leftovers.
The manifest is critical, we found. Nothing works if it’s malformed. Once built, however, its reuse vets the authenticity of the source. This most recent edition will also obtain Docker images. Overall, we feel its regimentation is worth it, and instrumentation will be key to its successful evolution.
Testing Rkt
We built a testbed from a CentOS 6 host on the same Lenovo platform used for OpenVZ and Virtuozzo, launched in a minimal host environment, instead of using CoreOS, for expediency. We crafted two images, one a WordPress image from TurnKeyLinux that we’d fixed for WordPress 4.2.2, and another a generic Ubuntu 14.04 server image, that had been updated once, and built for minimal services. Later, we discovered repository images for both and used those.
Resources, such as memory allocation demanded by the pod, bandwidth, and so forth can be set with a CLI-based rkt command Here, instrumentation is missing and scripting is necessary, for now. We relearned JSON syntax.
The release we tested also permits downloading, via simple http username/password, images from the Docker (or a Docker-like) Registry. This also is the harbinger that while repositories can be used, security overrides can break the chain of authorities, thus potentially breaking audit/compliance needs unless logs are strictly monitored for this behavior and corrections to mistakes are enforced.
The scripts we made then became the basis for rapid replication into our Lenovo host server. We slipped up in our instance generation script, and when we started the machines under embedded puppet commands, we cratered the instance, leaving a smoking hole in the Lenovo as it went into instant thrash, and then locked tighter than an inflated tire bead.
We’d forgotten to set CPU shares, and starting all instances at once had created an enormous startup demand on the host cores. No server would have likely survived it. Fortunately, a reboot allowed us to correct the error of our ways (our script errors).
In practice, we obtained two more images from Docker, and after much finagling, were able to get them to execute with the rkt runtime. We scripted together a number of container launches, and were able to do a pilot scale-out of executables, but not without a lot of work. Third-party tools will help rkt, and failing that, it’ll become an unmaintained altruism.
Criticisms
All three apps run as root, as they need to pick up speed from the kernel. AppArmor and SELinux get only mild treatment in the docs for all three, but all three can use these sandboxes to isolate containers from corrupting or DoSing system resources.
At the ship/host level, processes to control the daemons that control the container’s processes must be especially protected, something that’s already a core security activity in most organizations. Then comes key control, which Virtuozzo does fairly well, but isn’t comprehensive. An architecture that correctly administers SSH keys, and controls the communications plane for the respective containers needs to evolve, and here, Virtuozzo has a start. The other software defined networking internals needed for coordination in all three products needs maturation.
But there are efficiency-through-density possibilities that have no real parallel on Type 1 hypervisors, especially where deduplication of both stored files and executed files can take place. In a way, containers are hybrid bare metal instances with skills in virtualization. Hypervisor-based VMs live in isolation, usually as discrete instances, something not necessarily true of containers. Hypervisors go to a very low level to protect instances from hijacking resources unwittingly or through malware. The walls of the sandboxes are theoretically much higher for VMs than containers, although security mistakes can easily breach policies for either type of instance.
Summary
Containers are very, very handy, and allow OS instances+executables to be exchanged in a simplified, sometimes resource-sharing (at the OS level) atmosphere. The provenance of containers, their patch/fix levels, are not entirely opaque, but unless a rigorous methodology is employed to apply audit and logging habits, they’re an explosion searching for a spot marked X.
This said, we admit the seductiveness of rapid scale-out/scale-up of instances. OpenVZ had the most mature container space in our view, and their hybrid model of onboard full-virtualization or containerizing makes sense for their target market, ISPs and Managed Services Providers.
Docker is a serious construction set with much energy behind it in terms of development. If GitHub participant counts are any indication, there’s an overwhelming amount of activity in Docker development. We noted that with the energy comes the noise of problems with Docker Registry source images, including a few we picked as samples. It makes us nervous that containers are difficult to probe, and their composite and individual components are difficult to simply view and audit for their chain-of-authorities sources—even from major organizations.
This in turn, makes us hope that the primitives associated with the appc spec, as seen in rkt, continue to evolve. There is discipline in rkt and appc compliance that merits applause, although rkt itself isn’t ready for primetime except for advanced or experimental deployments.
Tom Henderson runs ExtremeLabs, in Bloomington, Ind. He can be reached at kitchen-sink@extremelabs.com.
