Tip 3: Learn a Hypervisor
Our first two installments in the series covered soft skills, namely learning all you can about the business you’re in, and some options for project planning and estimation. I think it’s about time we heat things up with something a bit more technical.
It is my opinion that every professional developer must learn to use a hypervisor. A hypervisor is software that allows you to build and run virtual computers. A virtual computer works by slicing out a portion of your physical computer’s (usually called “bare metal”). Then within that virtual slice of resources, you can install an operating system and run it like a separate computer.
Why Would You Want to Do This?
Having the ability to run virtual computers in your development environment, is becoming more crucial each day.
In production, I have a web server, an API server, and a database server all running on separate machines. Or so it seems. In reality, my production environment runs a hypervisor on a large, very capable server. Since all four servers are virtual, I have some real benefits:
Easier IT Operations
With a hypervisor, back-ups, restores, and even moving from one data center to another is very simple.
Instead of backing up files and running database backups, I can just copy the virtual hard drive, which, again, is just a big file. Given I can buy a 2 TB spinning drive for about $80, storing several revisions of a virtual computer’s hard drive is very inexpensive. I can even transport my backup drive off-site to mitigate site-wide disasters.
Scaling out is also very easy. “Scaling out” refers to distributing the your server’s workload. You use separate machines to run databases and web servers.
Contrast that with “scaling up”. If you have one machine, you add processors, hard-disk and RAM in order to add more performance to your apps. Scaling out is generally better because you get the added benefit of removing a single point of failure as a problem for your app’s up-time and maintenance windows..
If the traffic on my app is heavy, I can copy my web servers two or three times without going through the exercise of racking a new physical computer, installing an OS, etc. It takes only a few minutes to clone a server and after that I can use a load balancer to split the workload between the copies of the original web server. If one goes down, either accidentally or as part of normal maintenance, my app stays up because there are several other virtual machines still serving the app.
Fence Me In: You Can Virtualize Each Project You Do Separately
If you freelance, it’d be nice if you could have one PC for each customer. That way their project code doesn’t mingle, and you can easily stay organized. For some clients, you might find this to be a requirement, but even if it isn’t, there are definite merits to keeping each project or client quarantined to their own development VM.
If you don’t freelance, but just naturally work on multiple projects at the same time, each project can similarly have their own VM. This is a great way to develop a microservices architecture, or work on several different products at the same time.
If you teach, like I do, you can make a separate VM for each of your classes. In my case, I have a VM for Node, MySQL, Mongo, and general programming. This can also help you if you have some software licenses that are “educational”. You’re less tempted to use them incorrectly if they are only on your teaching VM.
You can make a VM with Windows, Linux, or OS X, install any specific tools you need for the project which, since all projects are separate VM’s, will mitigate the possibility that installing something for one client will hinder your ability to work for another.
When the project is over, you can move the VM off your main development computer to cheap storage like our aforementioned 2 TB spinning disk, a big thumb-drive, or Amazon Glacier storage just in case the client comes back next year for updates.
I Can Copy Production to my Laptop.
OK for this, you would need a beefy laptop. Mine has 32 GB of RAM and about 4 TB of SSD storage, so copying several 40 GB virtual disk files is no big deal. I installed 32 GB of RAM because I usually run between 4 GB and 8 GB per virtual machine and I like to have the physical RAM to back up the workload. Strictly speaking, this isn’t a requirement. You can’t define a VM with more RAM than is physically present, but you don’t have to make sure all running VM’s don’t exceed the physical RAM. For example, I could run 7 VM’s requiring 2 GB each for a total of 14 GB, but since none of them individually requires more than 2 GB, I can run all seven on a system with 8 GB of RAM.
No Separate Database Backup
Since the entire database server exists in a file that can be copied, I don’t back up my database – I just copy the files for the virtual disk. If anything happens to the server, or the data, I can swap out the virtual drives by copying the files back to the hypervisor.1
Copying Production to Dev / Stage / Test.
Since I can copy my production VM’s down to my laptop, it stands to reason I can copy them to other places too. I typically have three of every kind of server for my app. There’s a development server where we can login and run mad science experiments or run automated testing. There’s staging, where we verify the publication process works, and allows our User Acceptance Testing (UAT) folks the chance to play with and test the newest features with live production data (as a copy from production). And finally, there’s production which the whole world can see.
When I’m creating these environments, I create production first, then do a full clone to make the staging and dev environments.
The most useful application with this practice is with my virtual database server. Every day users interact with my app, and sometimes something unexpected happens with the application data. The last thing I’d want to do is hop on to the production database and try to fix the problem there. Instead, I can copy the production database server and run it somewhere else, like on my laptop. This means I can tear the database apart without having to worry about affecting production customers. I can also install any debugging tools, which are not usually installed on the production server, to the production copy, and use my normal debugging techniques to find and fix the problem.
I Can Isolate Database and Tool Installations
Databases can be finicky beasts. Installing SQL Server, Oracle, or DB2 involves about as much as installing the operating system. These installations added quite a bit of files to the system, and not always in the predictable places on the drive.
All these databases start as a service, and there’s a lot of overhead that you might not always need. And if the database you’re using doesn’t have a really good uninstall script, you might try to remove it from your machine only to find out later that there are still artifacts from the old installation still on your system.
I keep my computer’s bare-metal OS clean by only installing Windows, VMWare and a few other essential tools and programs like MS Office to my bare metal drive. All my development computers are VM’s. This way if I need to radically change or even update an environment, it is very easy. Don’t forget, every hypervisor includes a snapshot capability, so if the upgrade somehow goes sideways, it doesn’t ruin your computer.
Always Protect the Gear!
Have you ever downloaded something off some guy’s web site or github, and briefly though to yourself, “What in the world am I doing? I’m just running this code without even looking at it first. What if there’s a virus in there? What if the code is broken and it formats my drive? ”
Of course it probably won’t, but that hunk of plastic, metal and glass sitting in front of you allows you to feed your family, seek fame and fortune, and when nobody’s watching, kill a zombie or five. Always protect the gear!. You can keep an extra virtual installation of Windows around to test software before you commit to putting anything on your bare metal.
More Capable Testing
Using a hypervisor’s snapshot capability, you can store revisions of your server installation. For example, whenever I set up a new virtual machine, I install the operating system, apply all the updates and patches. Then I take a snapshot which I usually name “Initial Install”. Later as I add more software to the server, if I don’t install something just right, I can always go back to the snapshot, and it’s like I never installed anything after the latest patches.
This can be used to test as well. You can get your app running on your virtual computer, store a snapshot at the beginning of your testing session. Then you can test your app. Go nuts! Add users, delete users, change content, and generally exercise your app. You can automate this level of user testing with application libraries like Selenium or Nightmare JS. Once your test is over, revert back to the snapshot and your virtual testing system will be in the same state it was in before you did your test, allowing you to very quickly re-test the same scenarios without creating duplicate test data in your database.
I also use this when I create training videos. I make a VM with Linux or Windows on it, and snapshot it. Then I do the “How To Install Visual Studio” video. If it turns out to be a bad take, I can revert to the snapshot and do it over.
Before you Start, Make Sure You Can
Not every computer out there can run a hypervisor. Most computers made within the last ten years can, but sometimes it requires a BIOS change. If you’re curious, boot into your BIOS and look around. You probably find some settings for virtualization, and on newer computers and motherboards, they are turned on by default. If you have an older machine, it might be off by default.
If you try to run a hypervisor on a system that doesn’t support virtualization, the hypervisor will tell you this when you try to run it.
So Where Do You Get a Hypervisor?
Like so many other tools in our toolboxes, there are free hypervisors, and some that are not free.
VMWare is the 800 pound gorilla O.G. mac daddy of all hypervisors. It comes in several flavors ranging from the low-end, VMWare Player, to the high end, VMWare Workstation. There are also pure hypervisor editions of VMWare, such as vCenter and ESX Server. With an ESX Server, you install VMWare as the operating system on the server hardware. It comes with a minimal distribution of Linux and it is used solely as a host for VMWare Virtual Machines.
That’s good to know, but we’re focused here on talking about software you can run on your development laptop. There are two versions currently that run in Windows and Linux: VMWare Workstation Pro which runs in Windows or Linux, and VMWare Player, which only runs on Linux systems. For Mac users, there is a separate product called VMWare Fusion, which for our purposes can be considered “Workstation for Mac”, which can be used in conjunction with Apple’s Bootcamp feature in OS X.
If you’re still foggy on how a VM works, check out this short demonstration:
Player vs. Workstation
The difference between Player and Workstation, aside from the differences in the host OS, is reflected in the price tag. If all you need is to create and run virtual machines on your local PC, and you’re running Linux, Player will be just fine.
If you work for an organization that uses VMWare more extensively, the Workstation variant of the tool provides the ability to upload and download VM’s, as well as browse and control any VM on your company network.
Recently, I made a change to our infrastructure that required adding a Linux server to our dev, staging, and production environments. Since I have Workstation, I can build the new VM on my laptop. Usually, I do this at night. I start the OS installer and go to bed. In the morning, I have a virtual machine ready for updates and patches.
The only trick, if you want to make a new virtual machine on your computer for deployment to a server environment like vCenter or VMWare’s ESX Server, is to make sure the version of the VM format is compatible with the ESX Server. For example, right now, the latest edition of VMWare Workstation is version 14. If you create a VM using version 14, it will not be compatible with an ESX Server that is a few years old. Since a hypervisor server is usually a major IT investment in terms of time and money, they are not often updated beyond security patches. You build an ESX server and you run it for several years before you entertain the idea of upgrading it. So there’s usually a disparity between the latest version of Workstation and VMWare’s server products.
Parallels is the de-facto standard hypervisor for those of you running a Mac. It is marketed as a way to run Windows, and therefore Windows programs, on your Mac without booting into a Boot Camp partition. You can even boot up your Boot Camp partition directly to a virtual machine so you, at least theoretically, get all the speed available on the bare metal hard drive.
Parallels doesn’t *just* support Windows as the guest OS. You can set up any operating system compatible with the x86 / AMD64 instruction sets. So you can run Linux, Unix, OS X, any version of Windows, etc. The ability to run an OSX virtual machine is useful to me because I make tutorials that often include some level of set up, like installing Node JS or perhaps Anaconda for Python work. I can use the VM to document the set up steps for a given project without affecting *my* Mac’s configuration in any way.
Oracle’s entry into the market is our first example of free software. VirtualBox runs on all three major operating systems, and in practice is so similar to VMWare that I often wonder how there isn’t a lawsuit there.
If you ran any of these tools side by side, I think you’d agree that they are very similar. VMWare feels more polished to me, but all the important features are in the others.
If you’re into Java development, Oracle has a bunch of pre-built virtual machines you can download and run saving you the time of running through Oracle’s typically non-trivial set up steps for their products.
If you’re working on Windows, there’s actually a free hypervisor built in to the operating system, called Hyper-V. You’ll need Windows Server, Professional or Enterprise (the Home edition won’t have it). By default, it is turned off. You can install it and turn it on using the control panel.
In my opinion, Hyper-V’s user interface is the least polished of all of them, but it still does what it’s supposed to do, and you can’t argue with the price or availability.
I usually mistake the KVM software acronym for “Keyboard Video Mouse”; you know, those little boxes that let you run multiple computers with one keyboard, one monitor, and one mouse?
Yeah, this isn’t that. Kernel Virtual Machine (KVM) is Red Hat’s hypervisor. Given the FOSS history at Red Hat, you can install and use KVM on any Linux machine for free. I caution you, though to read the fine print. Often in these kinds of situations, there is a separate license for development and testing versus running in production. Make sure you understand what is and is not free before committing.
Personally, I’ve never used KVM, so I can’t really comment on it’s features or ease of use, but if I were in a Linux only environment, I’d certainly look into it.
Gennymotion isn’t a stand-alone hypervisor, it is a tool that leverages Oracle’s Virtual Box to allow Android developers to run reasonably fast Android virtual devices. Instead of simulating a computer, you can simulate a phone, tablet, an IoT device, really anything running the Android operating system.
Of course, you can use the tools that come with the Android SDK. But these are written in Java, and are notoriously slow to both load and use. If you have to wait several minutes to test your code changes, that quickly becomes a pain point.
The Gennymotion tool runs the Android OS , in my opinion, significantly faster and with a more polished tool set.
Hashicorp’s Vagrant isn’t a hypervisor. It is a way to automate the creation and configuration of virtual machines using a hypervisor. The idea behind the Vagrant project is that you can write a little script, say in BASH, that can be checked into a revision control system like git. Each developer on your team installs Vagrant and your hyperfisor of choice. Then they check out the repo with the Vagrant scripts and issue a few simple commands. After the automation completes, the developer has a VM exactly to your specifications.
Vagrant scripts contain all the commands and information needed to build a VM, rather than you doing it manually. You can write a Vagrant script that spins up a VM , then installs and configures SSH . Then Vagrant’s script can install NGinx, PHP, MySQL, etc. 2 all automatically so that all developers on the project are running the exact same thing. You can even have it set up multiple VM’s, maybe a web server, and a separate database server.
For robust configurations, Vagrant integrates with configuration management tools like Chef, Ansible, and Puppet. At that point there’s no limit to what you can automate.
Vagrant uses Oracle’s Virtual Box by default (since it’s totally free). If you want to use it with another hypervisor, like VMWare, you can pay for a plugin.
And Then There’s Docker
Docker isn’t really a hypervisor either. It takes the idea of a virtual machine to the next logical step, which is to virtualize applications. I have a web app written in Flask, React, and SQL Server.
I can virtualize the specific requirements for my stack into a Docker container. That gets deployed to a server that hosts the docker containers.
The cool thing here is it gives you a really easy way to do configuration management. A good way to do this is to use a tool like Octopus Deploy, which was designed to support .NET projects. I use it, though to deploy my Python code.
Instead of deploying updated code to a server, just make a new container and swap it with the old one. If there’s something horribly wrong, you can swap the previous stable edition back in, hopefully before anyone notices.
The Big Gotcha: There Can Be Only One (On Windows)
With all these hypervisors to choose from you might be thinking, “Why not just install several and try them out?” Well you can’t. At least you can’t on Windows which is about 52% of software developers out there. If you’re using Visual Studio for mobile development, you have to use Hyper-V in order to simulate Android. If you want to work with Docker on Windows, it requires Hyper-V as well.
The “gotcha” is that you can only install one hypervisor on your Windows system at any given time. If you install Hyper-V, you can’t install VMWare unless you first uninstall and deactivate Hyper-V, and vice versa.
Microsoft MVP, Scott Hanselman, documented a work-around on this blog that involves creating a multi-boot environment where you can boot Windows with and without Hyper-V , specifically so he could run Gennymotion *and* keep his Visual Studio installation running with Hyper-V.
There are so many good use cases for adding a hypervisor to your tool kit. We’ve only touched on the low-hanging fruit here. The learning curve is pretty small. If you can install an OS, you can make a VM fairly quickly. The hypervisors I’ve mentioned here make excellent additions to a junior and mid-level developer’s resume. This is especially true if you learn the enterprise-grade VMWare or Hyper-V. I would consider it essential to a senior developer / architect’s wheelhouse.
1. In reality, I’m a “belt and suspenders kind of guy”. I *do* actually run automated backups of my production database. I have a system that mirrors my VM’s to an off-site storage array. Restoring the VM is the first line of defense in the event of catastrophic failure. If that doesn’t work, I can build a new database server and restore the traditional database backup. On my laptop, though, I just back-up my development VM’s to a big USB drive at night. I can replace an SSD, install Windows and VMWare, copy the VM’s from the drive, and be running before lunch.
2. Or insert your favorite stack here.