Moving from Startup to “Real Company”

I love it when I get emails from people who are reading this newsletter, particularly when they’re correcting something I did wrong. A few weeks back Mike Murphy of Proctor Free corrected a mistake in my post on OODA Loop. Thanks Mike!

I took the opportunity to ask Mike what he’d like to see covered in an upcoming post, and one of his questions was:

“Can you talk about what some of the things are to consider as you move from startup to a professional small/medium tech company? For example, when is it no longer acceptable to do “smoke test QA” and hire real QA and Dev Ops?”

For this article I’m going to address the specific question he asked, and will cover other related topics in the future.

Hiring QA and DevOps

First a quick definition on the above terms:

  • Smoke test QA – Doing a quick round of ad-hoc testing before you release your product into the wild. Smoke testing is very common in the early going, when you’re trying to see if you’ve got a product anyone will care about. You’re OK with a few bumps because the users of your product are early adopters and more forgiving. You generally test the paths that users use most and hope the rest of your app doesn’t break. There’s all sorts of issues with this approach, but the biggest one is that you generally test your product as you would use it, not necessarily as your users will use it. The second problem is that you tie up key company resources doing repetitive work.
  • QA – Short for Quality Assurance, the QA role in a software product company is responsible for developing and implementing the testing policies and procedures for your product.
  • DevOps – Short for Development Operations, the DevOps role is to create a stable environment for your product to run in. Generally they are developers themselves and work to automate as much as possible about your product’s deployment and infrastructure.

If you’ve been following this blog for any length of time, you know that I’m a big fan of automation in general, and in particular testing automation. For more background on this topic, you can read my introduction to software testing as well as my take on automated testing. Automated testing allows you to defer the QA role hire as long as possible while you’re figuring out who your customer is and trying to get to product market fit.

Hiring QA

So when do you fill the QA role? The short answer is: When quality becomes as important as speed. Generally speaking this happens when you start to see meaningful, scalable growth. How that growth is defined is different for every product, but if you’ve “crossed the chasm” from early adopters to a more mainstream audience, the importance of quality goes up significantly.

Mainstream users are far less tolerant of bugs and issues, and will churn more quickly because of quality problems. Customer churn is death in any product company. The right QA person introduced into the team as your customer mix changing can provide stability in the release process as you start to really scale.

The other thing about quality as your customer base grows is that you reduce the drag on your development team. A bad release drags the entire team down with it, forcing everyone to re-orient on fixing critical bugs. It’s hard to overstate the impact of poor quality on future customer growth as well as overall development team velocity. It’s a killer.

Hiring DevOps

The first thing I’ll say here is that every single early development hire in a company should be able to do Dev Ops work, even if they’re not the best in the world, and even if they hate it. If your development team is doing manual deployments, even in the early days, it’s a waste of time and a drag on the product overall.

It’s time to hire specialized DevOps when your environment begins to get complex. What do I mean by that?

  • You’re deploying to more than one geography
  • You need to have basically no downtime (rolling deployments)
  • You have 3-5 developers who are doing DevOps part time and are sharing the load
  • The infrastructure itself is getting more complex, to the point where it’s getting hard to keep track of

Just Remember

There’s no 100% bulletproof right time to fill either of these roles. In the early going, everyone does everything because that’s how it has to be. However, provided your cash flow can sustain it, having more specialization in these two roles at the right time can significantly increase your chances of success.

My general rule of thumb is to hire QA when quality becomes as important as speed (customer scale is starting to happen) and to hire DevOps as your environment starts to become complex enough to justify it.

One additional point on the QA role: If you are consistently putting out unstable releases and customers are complaining, you have to do something immediately. It might mean having someone do an evaluation of your development processes, or dropping in a temporary resource to address the systemic issues, but don’t let the condition persist.

If you’d like to get your specific questions answered, drop me a note here and I’ll be sure to address them in future posts.