Pets, Cattle, Insects – The Next Step in Technology History

Pets, Cattle, Insects

The Next Step in Technology History

Over the past five years, the “Pets vs. Cattle” analogy has been very popular and instructive.  It’s helped people think differently about their infrastructures, and how they design, deploy, and manage their infrastructures and applications.  It’s helped people think about creating systems that are far more tolerant of failure, and expose areas of their systems that are bottlenecks to upgrade or maintenance.

In simple terms, “Pets” are systems that cannot be replaced without significant cost.  “Pets” are usually quite expensive to keep and will be kept until they die a natural death – and the end is when they get most expensive.  The vet bills really add up at the end of a pet’s life.  Same with systems that are treated like pets.  If they’re critical to the business, they will be very expensive to replace.

“Cattle” on the other hand, are said to be raised to die. Their lifetimes are well prescribed, and they have a specific task to accomplish.  Afterwards, they are sent to market and sold for parts. If something should go wrong with that cattle, the cost is absorbed pretty easily, and it’s sent early to market to be sold off for parts. What’s made this possible?

Enabling Technologies:


  • Platform: Operating System on Metal
  • Architecture: Monolithic Apps
  • Tools: Shell Scripting


  • Platform: Operating System Virtualization
  • Architecture: Mix of Monolithic and Distributed Apps
  • Tools: Robust Config Management/Orchestration

Pets and Cattle are all about cost and replacement. “Pets” are high cost and low capacity for replacement.  “Cattle” have a lower cost and capacity for replacement. Virtualization and the new generations of configuration management and orchestration have made that possible.  But what’s the next step?

Introducing “Insects” – Your MicroServices

Consider the noble insect.  Insects’ lives are inseparable from their colony. They depend on the colony for the majority of their life sustaining resources.  Also, they are living only for their specific purpose to the colony. Consider the worker bee, the queen bee, the worker ants, and queen ants. About Ant Colonies

Enabling Technologies:


  • Platform: Containers
  • Architecture:
    • Distributed apps that can self-organize
    • Completely self-contained stand-alone apps
  • Tools:
    • Light Config Management
    • Scheduler and Deployer


What you put in a container is basically one unit of work processing.  One is free to define how large or small this unit is, but common factors apply.

In earlier blog posts, I started exploring the idea of FuncOps.  I dissected the incredibly brilliant design principles of the unbreakably robust functional programming language Erlang, and presented them in the context of Operations. Some of those principles flow through to the ideas of the Insect class that we’ll be expressing here.  But make sure to return to FuncOps to get the whole picture.

Most important to keep in mind when approaching the world of microservices is that each class of has a rigidly defined goal.

MicroServices: Role, Immutability, Evanescence, and Hive Dependence

It does one job and one job only, and it does it exceedingly well and efficiently.  It also know how to discover and be discovered by other members of the application.

Ultra-short execution cycles in the insect class.

Containers rely on the host OS’s resources, like file access, network access, etc.  It has so little of what it needs to survive on its own.  But those services are abstracted out by managing the container format – Docker is defacto, for now.

Virtual Machines do not force us to rethink the architecture of our applications.  Containers force us to reconsider.

Orchestration: Swarming

Orchestration with resilient autoremedation

Pets had a 50 year life cycle,
Cattle have a 5 year life cycle,
Insect class – seamless autoremediative orchestration and automation

Application Transformation: Population Changes

Eliminate service persistency

In-Cluster Integration: Guests in the Hive

hypervisor into the firmware and microprocessor for app startup latency

You can go too far.  Unikernels create single-purpose hardware, useful in some contexts, a waste in others.

Insect class is dependent on code can run anywhere, but always have a central service element.

What’s a Successful Hive Look Like?

Chargify and NetFlix

Chaos Monkey


MicroServices at Netflix:

MicroService and Service Oriented Architectures:

RunDeck: (Because RunDeck):




Firefox: Disable Ctrl+MouseWheel Zoom?

Oh my god, it was so annoying.  A two-fingered scroll (mousewheel equivalent) had a lot of speed to it.  If I was scrolling on a page with two fingered scroll, and then tried to switch tabs with Ctrl+Tab, the page I was on would zoom like crazy.  Firefox was picking up the Ctrl+MouseWheel instead of the Ctrl+Tab, and zooming my windows.

I figured out how to disable Ctrl+MouseWheel Zooming:


Original Setting:


Disable Setting:



“Shut up, PaaS.”

Exactly why I absolutely love/hate PaaS.

“Hey developer, hosting is broken, we have solved all the SysOp tasks for you, so that you can focus on code now!”

“Hey PaaS, thanks i am already fine with my own setup.”

“But your setup sucks! Look, ours is much smoother!”

“Shut up PaaS.”


What really still sucks – heroism

I recall my life as an ops/admin and I remember what I loved to do.  I loved to deploy good stuff for my users and then a few days later follow up with them to see what they liked/hated and needed.  I tell myself this.  But it’s not honest.  I loved being the Hero, the Firefighter, the Op of Great Sacrifice For Greater Glory.  The movie “300” always appealed to me.  But ultimately heroism sucks.

Ops Heroism Sucks

Ops heroism sucks because when fighting fires, we’re not creating anything new of value.  We’re keeping things that worked before running as before.  Commitment to mission, great sacrifice, getting off the plan to keep things afloat are all laudable in times of emergency. But the real non-hero prevents outages.

The real non-hero has automated themselves out of a job, so they can do another job – bring in NEW business opportunities and new cost efficiencies by thinking creatively about their infrastructure.

From “Shut Up” to “Thank You.”

PaaS removes a whole lot of our Heroism.  It just works.  It scales.  It has automation built in.  It manages storage and networking. It gets developers working QUICKLY.  OpenShift and Cloud Foundry are doing great things to make the SysOps and DevOps life VERY easy.  What’s left for us to do?  Am I out of a job?  Well, at least I still have the onsite gear and backups to take care of.  But the folks at RackN would also like to help you lose your job rolling out crash carts and deploying new hardware.  Check them out, and help them out at

So, what’s my new job?  Well, a lot of my old job just got more interesting.  I’m still specifying hardware.  I’m still overseeing shipments and rack and stack. I’m still working the networks hard.  I’m still watching growth and usage and supplying my users with the best system to meet their needs.  But now they’re free to hate the choice they made and change quickly.  You’re free to tell them, “it’s OK to change your mind.”  And my users say, “I’m free to choose something else?  Some other language?  Some other database?  Some other software package?  Some other app server?”  And I can say, “Yup, no big deal.”  And they say, “Thank you.” And I say, “Thank You, onsite PaaS.”

Of course I left out..

I left out Networking, Storage, and lots of other aspects of the complete IT picture.  But it’s my contention that we have the same egoistic resistance to change in bringing in PaaS as we do so many of the other tools and processes that make our businesses agile.

Don’t get stuck to the same way you’ve always done things.

Our deepest fear is not that we are inadequate. Our deepest fear is that we are powerful beyond measure.”

– Marianne Williamson A Return To Love: Reflections on the Principles of A Course in Miracles

Favorite CLI stuff

git credential helper for CLI

git config --global
git config --global
git config --global credential.helper cache 
git config --global credential.helper 'cache --timeout=3600'


    pip install powerline-status



# I like vi keybindings.  You may not.
set -o vi
# if powerline is installed, then use it
command -v powerline-daemon &>/dev/null
if [ $? -eq 0 ]; then
  powerline-daemon -q
  . /usr/share/powerline/bash/
  alias tmux="tmux -2"

Adding git branch info to bash powerline using the “default_leftonly” theme


cat /usr/share/doc/tmux-1.8/examples/vim-keys.conf >> ~/.tmux.conf


source "/usr/lib/python2.7/site-packages/powerline/bindings/tmux/powerline.conf"
set -g default-terminal "screen-256color"



set laststatus=2 " Always display the statusline in all windows
set showtabline=2 " Always display the tabline, even if there is only one tab
set noshowmode " Hide the default mode text (e.g. -- INSERT -- below the statusline)

python from powerline.vim import setup as powerline_setup
python powerline_setup()
python del powerline_setup

python from powerline.vim import setup as powerline_setup
python powerline_setup()
python del powerline_setup

golang on RHEL

yum --enablerepo=rhel-7-server-optional-rpms install -y golang uploader

[root@director ~]# export GOPATH=/root/gocode
[root@director ~]# go get
[root@director ~]# mkdir bin
[root@director ~]# cp gocode/bin/gist ~/bin

For auth, the tool looks for an environment variable called GITHUB_TOKEN You can generate one at:

export GITHUB_TOKEN="blah blah blah"



OpenShifting Around – a sensible approach for next generation

Quoth the OpenShift blog:

Cartridges play well with others – and RedHat will approve some of them:

The OpenShift v3 Cartridge format will adopt the Docker packaging model and enable users to leverage any application component packaged as a Docker image.

Orchestration will be much more robust:

Red Hat is leveraging Kubernetes and work initiated in the Geard project to bring orchestration and scheduling capabilities to OpenShift v3 and better manage large scale environments.

And they’re making all the “Enterprisey” features of JBOSS into their own containers, to make them much easier to deploy:

Lots of smart moves by RedHat, IMHO. Making life easier for systems managers like us. Only problem – gotta stay on the RHEL stack.

A DevOps fanatic, a Mainframe Sales Engineer, and an Octogenarian Professor of Creative Innovation meet at a potluck..

This weekend I had a wonderful experience living the divide that Rob Hirschfeld spells out in great detail on his blog

I was at a Pot-Luck dinner with 40 or so families, most of them well into retirement age. The only other person seemingly in their 40s I also over-heard talking about technology. I started chatting with him. It turns out that he works for IBM, and his main task, as he so animatedly pointed out to me, was to convince companies to STAY on mainframes. He did this, he excited explained, not by disproving the fact that COBOL can be run well on x86 boxes. Rather, it’s the complicated integration points of these systems with other internal and external systems that will cause the customers to incur massive business downtime and costs. He smiled broadly saying, “they’re never leaving the mainframe.” So I asked him, what if their business radically changes?

It seems a lot of folks are missing the boat on the fact that business is in a revolutionary period. Agile has spread beyond the world of softare, and Lean has made its way out of manufacturing and the shop floor, to the datacenter and beyond. GE is changing the way they do business, and they’re promoting that they can rid themselves of old rigid business processes that cause their lead times to be so vast. With competitors creating similar products at lower prices with less lead time, every business needs to re-think their business process, manage the bottlenecks, and sell the right goods to the right customers, right when they want it.

This Industrial Revolution 2.0 is nothing that the mainframe model is setup to do. And that’s where the Professor of Creative Innovation comes into the story. As the IBM guy swore to me that “it will be another 10 years and we’ll be having exactly the same conversation,” the Octogenarian Professor from RPI laughed out loud. He told us that he’s seeing the future of materials, manufacturing, and technology right before his eyes every semester. He’s challenging his excited students to solve problems by finding creative solutions. His students turn to nature, often, for inspiration. And they mimic natural, not traditionally mechanical processes, to solve the most frustrating questions of our material and mechanical world. Finally, these products are not being sold off as patents to GE or IBM. They’re building businesses right here in our area to produce them and their offshoot business.

Great innovation no longer comes from within the hallowed halls of a GE or a Bell Labs. Innovation is happening from the open collaboration of disparate experts, all with different yet overlapping incentives. Great students no longer go off for a job at GE. They start their own businesses, have the lifestyles they want, and change the world.

GE, for their part, undestands this and is doing everything possible to make its processes, systems, and factories more agile, nible, and ready to respond to quickly shifting markets, hungry for greater innovation in ecoloical benefit, lower energy use, greater or more appropriate processing, better performance, or completely novel creations. Can the very siloed world of the mainframe survive, if a mainframe user is stuck in it with only very specially trained technologists and captured by old integration points? Where is the ease and freedom and creativity there? I don’t think businesses will be able to survive with that level of rigidity. The flexible will flourish! I’m off to the Professor’s classes this week to check out what folks are building anew!

From Over-Subscription to Hyper-Subscription

I’m sick and tired of virtual machines. They’re a significant waste of my compute resources. My laptops fan always starts spinning when I start up a virtual machine. But with container based workflows, I’ve never heard my fan go on. Even better, I’ve never had to re-build kernel moduels to get the latest version to work.

When I think of the production environments that I’ve deployed, and all the wasted RAM and storage, I get kinda sad.

My goal is to automate hardware and make containerization AWESOME.

Containers will let us Hyper-Subscribe our datacenters, without some magic from VMWare to share memory segments between OSes.

Slow Droid 4 – FIXED!

EDIT:  Every time you reboot, you’ve gotta go BACK to developer options and set the ‘Background Apps’ to =< 2. 😦

Oh man, was I suffering from a slow Motorola Droid 4.  Switching between apps took MINUTES.  It was awful.  I’m running Verizon supplied Android 4.1.2.  I was blaming it on Waze or Facebook or not enough RAM. Then I did some searching and boy was I wrong.

All you gotta do is:

“Settings” -> “Developer Options” and set all the “animations” to off.  For good measure I limited background apps to 3 in number.

Now my phone is fast and wonderful.  It’s great!  If this helps you too, leave a shoutout!

All I Ever Wanted Is A Composable RunDeck

I love RunDeck. It’s just enough orchestration to get many jobs done in such an easy way. I can take whatever’s working for me, wrap it up in RunDeck and give it to someone. I wrote a long writeup on hooking it to your Active Directory, through OpenLDAP. I’m old school.  But all I’ve ever wanted is a composable RunDeck.  To bring inter-tool composition right into my face.

So then I fell in love with Chef and Puppet as they grabbed the very necessary spotlight. Suddenly, arising from the past of CFEngine, I could now code my infrastructure! Different parts of the system worked predictably, across platforms, and with such clarity and simplicity. Chef saves me thousands of lines of code, and helps me reason about my systems on a whole other level.

But these two worlds never really fit together very well. I want my “just enough orchestration,” but with the little bit of smarts necessary to interact intelligently with the abstract resources.

I was thinking, “Yeah, sure, it runs.  But how does it all fit together?  I want to compose!” OMG, then all sorts of other little tools that control other things came around, and I wanted to wrap them up in a big hug. A big smart hug, that knows just how they like to be hugged.

Crowbar 1.x originally only hugged Chef. And even only hugging Chef, it did MORE than Razor and Cobbler do (still today) combined.  And it did it all in a way that’s not “enterprisey” and is just clear and makes sense. But it didn’t get all the way to that “composable RunDeck” thing that I was looking for.

So we wrote OpenCrowbar. And now it does.