The following guest post was provided by Richard Rees, CISSP, a Practice Manager for EMC Consulting’s Virtualization Security and Cloud Trust group. Email him here.

The best thing about computers is they do exactly what you tell them to do, very quickly.  However, the worst thing about computers is that they only do exactly what you tell them to do.  I’m still waiting for an OS that contains DWIM code (Do What I Mean) without the evil of Clippy.  Virtualization is the same thing – except even faster over a larger scale.  Instead of computers needing DWIM capability, entire data centers need them.  This applies to security as well – the traditional problems of security in a data center exist in the virtual data center, but when mistakes are made, they’re made over a larger scale.

How to make a personnel change

Fire, by Cindy Andrie

Most security professionals have dealt with building or reviewing HR “status change” policies at one point or another.  In the case of individuals with high levels of access to systems or critical information, the typical process goes something like this:

Step 1.  Remove admin access from administrator.

Step 2.  Inform administrator that, unfortunately, the organization has eliminated his or her position

Step 3:  Give the administrator time to gather his or her things, absorb what has occurred, and leave the building, escorted if necessary

Step 4.  Do not reverse steps 1 and 2

Step 4a. Do not skip step 1.

Some organizations do this well, and some don’t.  Those that don’t may experience some interesting times – we all know the stories.  An administrator at an ISP logging into a system remotely from home after being laid off and deleting billing records.  A physical security guard getting the axe and retaliating by taking hedge clippers to the fiber interconnects in the data center.  And now, there’s a new story to add to the collection, this time with a virtual twist.

The Perils of Poor Policy

Jason Cornish was a VMWare administrator at the U.S. subsidiary of Shionogi – a pharmaceutical company.  Most pharmaceutical organizations are investing heavily in virtualization and cloud technologies with an eye towards reducing costs associated with massive number crunching activities like genetic sequencing and other research activities.  Cornish resigned in July 2010 after getting into a dispute with management.  However, Shionogi still needed his skills, and kept him on as a consultant until September.  Apparently, there was some ill-will over the reductions, as one employee refused to hand over network passwords and other tokens to management and was fired.

Ski Mask, by Dave Wasson

So, the issue should have come to a head at the end of 2010.  A new team or consultants could have come in and asserted new levels of control over the infrastructure and systems, or the individuals that replaced the laid off workers would have done so as part of the termination process.  Password reset schemes could be deployed, and in some cases, password recovery could have be done as well.  Unfortunately for Shionogi, the best practice policy for reductions in force was not followed.

Five months later, in February of 2011, Jason struck. He logged in and wiped out 15 vSphere hosts that ran 88 VMs that supported order tracking systems, financial systems, and (of course) email.  The total damage to Shionogi was estimated at US$300,000.  That value was spent on direct costs to recover from the attack, no estimates due to the fact that Shionogi was essentially closed for business for days were available in either the compliant or the material provided by the Department of Justice.

Unlike the famous ISP employee, Jason was smart enough not to log in from his house.  Instead, he chose a wi-fi hotspot at a nearby McDonalds.  Unfortunately for him, he bought his breakfast with a credit card.  His own credit card.  Five minutes before the attack.  When I was first interested in computer forensics, I took an optional course at a security conference, given by the head of fraud at Lucent.  It was a great class, where he walked through real scenarios that he had to deal with.  After the session we were talking for a bit and I asked him, “If I did *** and *** and of course ***, how would you have to change your investigation?”  He responded by saying, “We’d never find you.  You see, we catch the dumb ones.”

Lessons Learned are Old Lessons

There’s really nothing new to say here about lessons learned.  Again, the only wrinkle that virtualization truly adds to the picture is the speed and scale at which Shionogi came to a grinding halt.  See step 1.  Failing step 1, invest in intelligence and proactive monitoring.  In fact, that’s good for lots more than this one specific scenario, unlike removing your shoes at the airport here in the States.  Proactive monitoring would have stopped Jason before he even got started.  Because, you see, he logged into Shionogi’s systems over twenty times between October 2010 and January 19, 2011.  If Shionogi had invested in proactive monitoring of their firewall logs, or invested in automated tools capable of doing so, simply identifying a single rogue logon here would have saved the company over $300,000.  It also would have put Jason away much sooner.  Those twenty prior logins?  They originated from his house.

This post originally appeared on