5 Cool Tools for Cloud Management
HP CloudSystem Matrix
We tested HP's CloudSystem Matrix 6.3, a private-facing IaaS management tool. There's also CloudSystem Enterprise, which controls internal IaaS, PaaS and SaaS, and a Service Provider version.
Matrix is a sophisticated and complicated combination of HP blade servers and management software. Its breadth is staggering, but the system's complexity can also make it difficult to use. Matrix manages a wide variety of hardware, software and virtual machinery (chiefly VMware) in a control plane of IaaS. Its components consist of several servers, including a blade server, software controls, server storage and software. The package isn't just for HP systems, as CloudSystem Matrix can discover a long list of hardware and infrastructure by IP address range, although this wasn't tested.
Matrix, which we tested on HP blades, has a cloud-in-a-box feel. There are a number of software parts and pieces that go together and are managed through a Web-based administrative portal. The portal includes links to all the different application pieces.
The portal is quite daunting as there are so many menus, submenus and options on each screen; it begs for a huge display or two monitors. The operations of the CloudSystem seem stitched together and some parts of it seem to load another part, but we weren't sure which part was being loaded. Nonetheless, CloudSystem's breadth manages a wide variety of infrastructure.
The pivotal piece of Matrix works through an app called Insight Orchestration. Matrix has a discovery application that works on existing infrastructure, identifying assets and arranging them. These are added to a clever tool that uses icons to drag and drop a visual representation of discovered or inserted infrastructure.
Templates are then used to drag and drop objects like bare metal or virtual disks, servers, network and VLANs into a map. We could then connect the objects together, inserting details about a connection as we went through the design process.
Once the template is done, it's launched and a visual representation of deployment progress can be viewed, along with actions that might need admin approval during deployment steps. Users are then added, and we could connect to Active Directory to link users to the application.
We could also create asset pools of machines, dividing them up into objects. The more advanced versions of Matrix allow pooled/grouped assets to be branded.
Cloud apps could also be pooled in this way, so as to allow users to choose off-the-shelf configurations relating to specific or general tasks.
We had the ability to look at Cloud Maps, which were a strong visual interpretation of the cloud resources that we'd configured and deployed. We could then flip to the Capacity Advisor if we wanted to perform what-if type analysis for different scenarios. We found the user interface to be cumbersome, and procedurally not intuitive.
The user interface seems to have many options available, and is seemingly procedurally and productively simple, but we found lots of gotchas. Our mission was to deploy two ESXi servers, and during that process, our molehill turned into a mountain. After trial, error and HP support, we were able to get the VMs running.
We also had to do a lot of manual work inside of VMware to perform associations to the CloudSystem for our ESXi servers. The documents, while somewhat useful, didn't prepare us for the daunting experience that we had.
CloudSystem Matrix is complex, but it has the capacity to manage and potentially "remarket" a variety of infrastructure assets.
Puppet Labs MCollective
We first saw MCollective in our review of Ubuntu 11.04 Server and Cloud editions. What intrigued us was its ability to rapidly provision instances of operating systems, but also applications. It's poised toward developers, and is limited currently to Linux instances.
Despite the fact that the Marionette Collective/MCollective ("mc") tools are CLI, it achieves astounding speed at communicating with potentially thousands of instances as fast as the wire speed can move the messages, no matter where the instances are located. The mc tools are middleware that use a multicast-like push messaging system to controlled nodes. There is no artistic drag-and-drop rack configuration. There are no library-like user interface Web pages that one can "check out" an instance of a desired application. If CloudSystem Matrix and Abiquo 1.7 are sky-management generals-of-the-armies, MCollective is the battalion commander, bereft of the niceties, pomp and circumstance.
Inside instances that mc controls are two mc agent daemons installed from RPMs. The daemons are based on Ruby code, and can manage inter-process communications and managing packages. The client has similar components. The "collective," therefore, consists of nodes, which in turn have servers running in them -- agents that are the messengers that speak to middleware, in the client. The collective is a living and dynamic thing -- but is totally bereft of security as an object.
This means that communications must be performed over VPN links and SSH, and applications like Apache or a LAMP installation must have their own security components enabled outside of what mc manages. Fortunately, much of this can be done via mc -- but application security and link security for the collective object are two different things, we found.
The MCollective can spin up applications with frightening speed. We deployed a single instance, provisioning it with mc. We had 40 instances done in approximately 29 seconds.
We then instructed mc to install Apache into the instances, start the instances and tell us that it was done. Total time was approximately 31 seconds. Stopping all 40 Apache server instances took approximately seven seconds. Killing the instances via shutdown with verification took approximately 12 seconds.
The middleware keeps track of basic connectivity facts regarding deployed instances, but there is no database; it's a stateless, push-based messaging concept with metadata intelligence inside the messaging that makes the collective do work. The commands we used are easily scripted into scripts/batches. Had we the budget, the number of instances that we could spin up within a minute could number in the thousands. Having them do work, send messages or store the results, then shut down (and stop the cost cycles) can be stunningly quick.
The MCollective is a developer's cloud tool. It's CLI-only, but building a lightweight GUI for it shouldn't be difficult. In its adaptability, however, its security is lightweight and application security should be handled outside the mc. For the cognoscenti, it's a thunderstorm of a toolkit.
TurnKey Linux Backup and Migration Tool
Persistent cloud applications, usually SaaS-focused applications, need backup, but how that's done is often left up to SaaS site owners/admins, who often don't back up or in some cases even know what to do -- think small businesses and bloggers using WordPress.
Many don't even know where or what files to back up or how to restore a cratered site. TurnKey Linux, we found, offers fully integrated download-and-play appliances consisting of popular SaaS applications but, importantly, bundled with integral Amazon S3 cloud backup costing a few pennies at most per day.
The TurnKey Linux Backup and Migration Tools (TKLBAMs) are pre-configured customizable appliances for instances (cloud, virtualized or bare metal/physical) for popular SaaS applications.
There are more than 40 different pre-built FOSS appliances available from TurnKeyLinux.org including popular Web applications such as Drupal, WordPress, Ruby on Rails, Joomla, a basic LAMP stack and more.
All of the TurnKey Linux appliances include Web-based admin setup for each platform and other common configurations. There is a core appliance which can be used to create a customized appliance if none of the pre-built appliances are suitable. Each TurnKey-supported appliance is based on Ubuntu Linux and is automatically updated daily with Ubuntu security updates.
How it Works
Procedurally, you get an appliance and set up an account (with a link to Amazon's services if desired). The appliance is downloaded and deployed, and backups are started; a wise installer tests a restore. We obtained the WordPress appliance from TurnKeyLinux.org, and placed it on the host we use for extremelabs.com. During this process, we filled in the billing and configuration profile to be used for Amazon Web Services cloud storage charges, and for restoration purposes.
TKLBAM downloads the profile from the TurnKey Linux hub (more like an app store) for whatever appliance version is desired. This profile can be used to detect changes made after installation, such as new packages installed or files added/edited/deleted/etc. Some organizations will use a fresh appliance and populate it afterward progressively, while others will use static pages, and still others will migrate an existing equivalent running host.
After this we could use the site to restore the backup to a cloud image on EC2.
We wanted to try upgrading to the latest version using the backup that we had created, so we downloaded the WordPress appliance ISO and loaded it up in XenServer 5.6, performing the basic WordPress install.
This worked quite well; almost all of our settings, our database, our WordPress files, customizations, etc., were restored to the new instance of the blog site. The only issue we had was restoring our manual IP address settings. A reboot later, we were almost all good to go. The last thing we had to do was just an apt-get update/apt-get upgrade to make sure we were up to date.
Next we made a new backup with our upgraded appliance. By default, we had to manually backup the instance for the first time using tklbam-backup. After that, monthly full backups are enabled by default. To enable daily incremental backups all we had to do was run: chmod +x /etc/cron.daily/tklbam-backup. Our incremental backups so far have ranged from 158.5 KB up to 489.6 KB for 10 days after the original backup. The total cost so far for our site is $0.04 per month!
For TurnKey appliances using the MySQL database, database backup is also taken care of transparently. The full contents of the database are serialized and encoded in a file structure that is made specifically for optimized incremental backups. After TKLBAM calculates the delta, it uses Duplicity to encode backup contents in a chain of encrypted backup volumes, which are then uploaded to Amazon S3.
The downsides to the TurnKey Linux appliances are numerous. Zimbra, a popular email system recently purchased by VMware, cannot be backed up via TKLBAM, and appliances using Postgre database can't be backed up using the TKLBAM scheme, either. There are no current appliances using Windows applications, sadly. And there is no inherent increase in security for a given TurnKey Linux appliance, although the daily update option may help strengthen them.