Hiring for Strengths (Not Weaknesses) with Charity Majors – Part 1

This week on Scaling Software Teams, we present part one of our conversation with Charity Majors.

Charity is the CTO at Honeycomb, a startup that delivers next-gen application performance monitoring for distributed systems. They deliver observability that allows you to introspect and interrogate your production systems. Charity’s team is on a mission to make monitoring less cumbersome so that being on-call can suck less.

Charity’s view of hiring and success in the workplace is refreshing and rare. It’s not often that a leader has both the perspective and humility to discuss things like not being right for every role. This conversation belongs in our hall-of-fame for its candor and insight… Which reminds us, there’s some explicit language scattered throughout our talk.

In this episode, we talk about…

1. When pain and discomfort is okay in your professional life (and when it’s not)
2. Why hiring for lack of weakness is a mistake
3. Why more folks in tech should quit their job

Listen to the full episode here

Charity, thanks so much for joining us today on Scaling Software Teams.

Yay! I’m so happy to be here. Thank you.

‍

Charity what’s something about software engineering that you’ve recently changed your mind on?

Oh boy. What have I recently changed my mind on? [Laughs] I had been a Javascript hater for a very long time and I’ve recently changed my mind on that. It’s a very useful tool. It’s not Javascript’s fault that so many people do such terrible things with it.

‍

Did you have past experiences developing for Internet Explorer 6?

No I didn’t. I bypassed all of those – but I did use Javascript to develop the MongoDB Shell which was pretty much my only experience until recently.

‍

And what changed your mind? Just seeing something work?

You know, Javascript is the grep; it’s the command line tool of the programming world. It can be used for good, it can be used for harm – but it’s very powerful and it’s everywhere.

You’ve talked in the past about the competing ideas between leaning into the pain for career development – so learning – and then the real dangers of chronic pain. How do you think about the balance between those two?

Yeah that’s a very active dialogue in my head right now. I think that, of course I’m biased, but I think that learning to tolerate some discomfort is necessary if you want to learn hard things or do big things.

And yet, like there’s that fine line between, is it wearing me down or am I conquering, right? And I think that I tend to come from a place of, anything that’s hard must be good. I grew up on a farm; working hard was what we had to do to eat.

But what I’m realizing is, that shuts down your creative mind. And we all have jobs that are very creative, and when we’re in a place of pain – when we’re in that place of things being hard, we’re not creative beings, we’re slogging through. And it’s our job to steward our own career and our own life so that we have those creative open spaces to flourish in.

I think that pain should only be seen as a short term sacrifice towards a greater goal. If you’re not getting towards that goal and you don’t see the light at the end of the tunnel, and you [also] can’t see your way back to that creative place of openness – then it’s probably not worth it.

‍

So it sounds like it’s the visibility of the end?

Yeah. Because if you’re climbing a very tall, steep mountain and you believe [success is] up there, then maybe you don’t see how you’re going to get there – but you know it’s worth it.

It also has to do with other things that are going on in your life. Sometimes you have the ability to just go, “Fuck it – I’m going for broke. I’m going to take the next six months and just do whatever I can.” If you have the support of your partner, if your health is okay, if things are aligned – then I’m all for just taking those sprints.

But you can’t let that misery become your life. You have to schedule check-ins with yourself or with others. Go to therapy, or whatever it takes to like put your head up now and then and go, “Is it still worth it?” Because this is not my life, right? The state of pain is not supposed to be my life.

So thinking about that state of pain, some of us are in jobs that maybe is the right job for the mission and the company – but maybe isn’t the right job for us? And I think there’s a lot of pride wrapped up in admitting that we’re not right for the job. Can you talk to me about that? How do you navigate that?

Totally. It’s something that’s so core to the human experience: wanting to be part of something that’s bigger than us. It used to be the church, the tribe, the family. And increasingly what we have is our job; this is a major thing that we participate in that’s bigger than us. And I don’t think that’s a bad thing – as long as you know it’s an organization where the values align with yours, as long as you believe that what you’re building is a thing that should exist in the world, [and] as long as you believe in the people who you’re building it with.

I know this is hard, because of the power dynamic, but we should approach our jobs as though we are giving this organization the gift of our time, our lifeblood, and our experience – and they should be worth it. If they’re not worth it you should move on. I’m not against self-sacrifice in service of the larger good, though. Like if it’s something you believe in, that’s part of what gives our life meaning.

And lots of radicals will argue with me [about a] capitalist system, exploitation, blah blah blah… and I’m not saying they’re wrong… but this is a vehicle that a lot of us choose to participate in. So then it becomes like, okay I want this company to succeed. They need this thing for me. I can do it. It doesn’t really align with what I want for myself. Right?

And then I think that you’re just in that position of short term pain, and only you can define, how long is this sustainable for me? How long until I’m gonna at least need to know where the exit is? Maybe I can do it for six months, but then I need to know where the exit is… or maybe by then I will have learned to find the joy in it.

Sometimes it does take us a long time to find the joy in things, because that state of flow that we find so easily as engineers, I never found it as a manager never. And I’ve done it for years now, cumulatively, and I’ve never found – yet I don’t regret the time that I spent as a manager. I’m still managing people. I just learned to balance my life differently so that I can find joy somewhere, because I know it’s worth it for me to be doing the management job.

‍

So you’re saying that the mission has to matter, or the organization has to be worthy your pain.

Both. Both the mission and the organization.

‍

Right that makes sense. And then you want to lean in, lean into the pain. Because maybe it becomes a thing you like – maybe the pain goes away and then you want to check in with yourself to know if this… acute [pain] has become chronic.

Yeah totally! And sometimes there’s no shame in not choosing to do that. Saying, “This isn’t right for me right now” or “I don’t have the bandwidth for this right now. This just sounds miserable without a possible payoff of any sort for me.” That’s legit. That’s fine.

‍

Yeah there’s lots of time in our life – lots of opportunity. You don’t always need to be running up the mountain.

You can’t sprint up the mountain your entire life. You just physically can’t. My very intense work experiences have been bookended by job experiences that were super mellow, and I didn’t like it – I chafed at it. But I think I needed it.

‍

I think that pain should be only seen as a short term sacrifice towards a greater goal. If you’re not getting towards that goal and you don’t see the light at the end of the tunnel, and you can’t see your way back to that creative place of openness then it’s probably not worth it.

— Charity Majors

‍

So you’re now CTO and I’d love your thoughts on what a good CTO does for their company, and how does that compare to a VP of Engineering? Because this is a question I still don’t quite have figured out.

I don’t know that anyone knows the answer to this question.

CTO is different everywhere. VP of Engineering is more straightforward. When you’re a VP of Engineering, your job is to make sure that all of engineering executes according to the company goals, right? It’s your translation layer. And that sounds like a little, but it’s actually very hard.

Your job is to manage managers of engineers, basically, and to make sure that anything that needs to be communicated back to the exec team about, if this is going to land in time, if their projections are unrealistic, if something can’t be done, if something’s different – your job is to make sure that no one is surprised in either direction. That’s how I see it.

CTO, often your job is to be a washed-up founder who doesn’t have a place anymore. So they put you in the seat and they’re like, “Don’t cause too much trouble.” [Laughs]

‍

Probably not the idealistic vision.

Probably not. No.

I think that a CTO, ideally, you are setting the technical direction for the company. And I also think at the C-level you have a large external responsibility – it’s at least as much external as it is internal. Your VP is running things internally, ideally, and communicating with you a lot, but you have a big visible presence; you have the ability to speak for your company and your product and your technology with authority, I would hope.

‍

So the VP of Engineering is more internal facing, and the CTO is more external facing. Are you excited about being external facing?

I’m hoping to be more internal facing than I’ve been – like being CEO is so weird, and it demanded a lot of extra time. Like I was our entire marketing department for most of the last three years, which is now, thankfully, no longer true – for everyone’s sake. [Laughs]

We now have Liz Fong-Jones who’s doing the developer advocate stuff, but for a long time that was me, too. Like I was traveling around trying to do all the marketing, all the advocacy – and I’m happy to keep being that translation layer but I want to spend time with the engineers. I want to get more in the product.

When I started this company I was expecting to have three solid years of heads down technical work; just hands on, writing code. I think I got three months… when I did the Terraform blog posts, and I was having the time of my life. So I recognize that I’m not gonna get back to that state, but I need a little bit of recovery time, honestly.

‍

That makes sense.

Could you talk to us a little bit about observability, and why it matters to teams?

Yeah absolutely. Observability is something that I define using the control theory definition – it’s how much you can understand about the inner workings of your system just from the outside using a tool.

And not like hodgepodge-ing together half a dozen different tools – not like… you look at your dashboard, you see spikes, you jump into your logs, and you look for things and you jump over to another tool for tracing.

[Instead], using a single, consistent interface, how much can you introspect and understand about the inner workings of your system? And I think it’s becoming important because all of the other ways are starting to fail us; they’re just falling apart at the seams.

Most canonically, I think that there’s been, for the past 10 years or so – between the dashboards and the software engineers – you’ve had ops sitting there just explaining things to both sides – just telling software engineers what they’re looking at when they look at these dashboards…

And that doesn’t work anymore, because ops can’t fit the system into their heads and reason about it anymore…. there’s no question in my mind or anyone’s that it is better if you start from the beginning by using tooling to understand and debug your systems, and you grow up with it that way. But those tools haven’t existed – to be perfectly egotistical about it. Until honeycomb, that tool hasn’t existed outside of Google and Facebook.

And so you’ve had to use your intuition, and your scar tissue, and your remembrance of all your past outages to interpret what’s happening your systems…

So not everyone’s having to make this very large category leap all at once, and it feels kind of painful, but it’s a better world. If you give software engineers the tools to understand their systems using things that they recognize like API endpoints and lines of code and functions and queries – it’s reasonable to ask them to own their own services.

I don’t think you can ask a software engineer to own their code without good observability. You can’t ask them to do it using traditional monitoring tools, because you’re asking them to have two jobs; you’re asking them to understand and speak the entire op side of the house as well as their day job, and it just hasn’t been tractable. Now I think it is…

I think the first wave of dev ops was like, “All right ops people, learn to write code.”

And they were like, “Cool. Message received. We do.”

And now the second level of dev-ops is much more about [saying], “Okay, software engineer, to build operable systems – you need to understand enough about ops to be literate and understand what you’re building to be able to debug in production.”

And I don’t think that there’s ever been any such thing as a dev ops role – dev ops is about the philosophy, about mutual skillsets. And people have tried to paper over this divide by creating a dev ops role and it hasn’t worked. Surprise.

‍

So you still need ops [and] you still need dev, but they need to be speaking the same language?

I feel like development and operations are skillsets, and what you need are engineers who can write and ship code and own it in production. And it’s not saying that there’s no room for specialties – there is absolutely room for specialties, just less and less so.

More and more, the generalist engineer needs to have a hybrid skillset. And the reason for this is driven by the end user experience, because if what you have is developers over here writing code, never interacting with it in the real world, and over here you have operations people trying to make that shit work – the experience for the end user is shitty; it’s really bad.

What you need is for that engineer who’s writing code to follow it through to its destination, and watch it run in production – watch the users try to use it. And then you get this beautiful, wonderful feedback loop where you catch the problems early.

You’re shipping code, you have the context loaded up into your head, you watch it work, you know what you’re looking for because you just instrumented it. [If] you see it not working exactly as planned, and you course-correct before any user ever notices, right?

And even the bugs that take a while to propagate, you’re the person who’s like getting paged, you’re watching it – and it’s tight, man. This is how we keep from getting these terrible systems, where they’re paging you all night long – like those systems are the result of years, if not decades, of lack of software ownership just compounding over time until nobody knows how this shit works.

And it is hard to dig yourself out of those holes, but it has to be done and it’s worth it. And it can be done – like there’s no reason for anyone to be torturing themselves when they’re on call. The thing about on call is this is a two-sided solution, right? You have to have the commitment from management to dedicate the engineering hours and resources to cleaning it up so that on calls are not miserable – so that you can ask engineers to be on call because it’s not severely impacting their life.

‍

I saw a Twitter thread you posted talking about owning code in production and I saw a lot of some unhappy people mostly reacting to being paged.

Absolutely! Because traditionally this job that offices had is miserable. Ops has been over here just in the receiving end of the ship. They don’t have the power and the ability to clean it up, they’re just the buffer of badness and sadness and a lot of software engineers reacted as though we’re inviting them to join us there. That’s not the goal!

The goal is that everyone ships code – whether your title is ops or dev – and everyone owns it through to completion. And it’s not miserable; it’s not a bad experience. Maybe you get paged a few times during the week during business hours when you’re on call cleaning things up anyway – and maybe once a week out of hours every two or three times you’re on call. So you carry your pager, you carry your laptop, you’re available – but you’re not losing sleep over it. You’re not expecting to have to cancel your plans or travels or anything. It’s not terrible.

This is eminently achievable for everyone.

‍

So let’s get better together, rather than spread the pain around.

Yeah! That’s what dev-ops needs to be about – and this takes buy in from the entire [organization]. You can’t do it if you don’t have management support, because you’re not going to find the time to make these changes in the couch cushions between your features – it has to be an actual top level priority. And my hope is that no one in tech should be miserable – you should vote with your feet. If you can’t make the changes, go to a place where they were they actually invest in this, where they do care about it – because otherwise you’re never going to give these management teams the incentive to change unless they are losing good people to places that do make these changes.

‍

What do you think is holding back that movement from the places that don’t have their stuff together to the places that might?

I mean all of human psychology. We have loss aversion. There’s that flaming moat of shit that you have to cross to get through the interview process of all these organizations who don’t know how to interview and are gonna make you whiteboard code, which we’re getting better there, too – but it’s an unknown.

There’s fear of change. There are real attachments to the people that we’re with. Most of us come to work first and foremost for our team because we are part of something and we believe in each other and you don’t want to leave each other behind – to which I would say bring your team with you, you know? And presumably you work for a place where you do believe in the mission and you don’t want them to suffer.

And this is where I would say that you should be somewhat self-interested. You only have one life – you shouldn’t be miserable. You should be building something you believe in and not being miserable at the same time. And sometimes the only way to exert change upward is to vote with your feet and show your management chain that it is not acceptable for you to treat people this way – and show that by leaving.

We talked about hiring maybe getting a little bit better, and you mentioned whiteboarding. If you had a magic wand and you could spread or crush an idea about hiring for interview processes, what idea would you spread or crush?

The number one thing that I would say is, remember that you’re hiring for strengths not for lack of weakness.

You’re hiring a person for their very particular strengths, and you should invest in their career, and in helping them build on those strengths. And their weaknesses – just to be brought up to an acceptable level of functioning, right?

This is something that I learned at Facebook. My tenure at Facebook wasn’t amazing but I learned some great lessons there. And one of the things was know what you’re hiring for. You’re not hiring a generic engineer – presumably you need them to do a particular job and you would rather have someone who’s passionate about those things that you need, [rather than someone who’s] not terrible at anything.

When you’re investing in people, you invest in their strengths, not their weaknesses – and when you’re interviewing people make sure that you’re drawing out those strengths. We always try to ask people, “How can I see you at your best? What is your strength?”

Yes, obviously you have your process, and you should be measuring people on the same question to have consistency and all these things are good. But also, if in our process we haven’t drawn out something where you feel like you’re shown at your best – what can we do?

And that gives them back some agency, too, and some ownership. The power differentials are so extreme in a hiring situation – they think you should do everything you can to to create a level playing field and to make it clear that, you’re interviewing us just as much as we’re interviewing you.

And we also have a responsibility to see you at your best – whether or not we end up hiring you. These should be the goals. Because if there’s one thing that as a manager, or as a founder, I used to always hear as a bullshit phrase, [it’s] it was not a good fit. I always just thought not a good fit right now was this bullshit HR phrase… and sometimes it is – I’m not making apologies for that.

But more often it has been, I respect you. I hope you respect us. We have a very limited budget and this is not the best fit right now but I would love to work with you at some point in the future. This narrow tiny thing right here is what we need, and I think that this not the greatest fit right now.

And I have to kind of ruthless about that.

‍

I think that’s reasonable. My take on that is, if you can say not a fit, you have to be able describe what a fit looks like.

Yes, that is a great point. And be honest about it. Most people would rather have honest, kind feedback about why it’s not a good fit, or where they could improve, not some boilerplate bullshit.

‍

One thing we’ve fiddled with is instead of giving feedback that is negative like what someone did wrong – we just found better results when we said, “This is exactly what we were looking for.”

Totally. I would also really like to eliminate that guessing game about what people are looking for… none of us do our best work or when we’re in a state of fear. Anything we can do to make people feel comfortable and on a level playing field with us is good – but a lot of what creates that fear is uncertainty and not knowing.

So we try and tell people exactly what we’re looking for up front so that they don’t have to guess. You can’t always do a perfect job at that, but any attempt that you can make is good. And also just letting them know that questions are welcome, and if it’s not clear please ask because isn’t a game. We’re not toying with you.

Because that’s the scenario that they’re going to be in, hopefully, if they’re your co-worker; you’re not going to be playing games with them – you’re going to be be like, here’s specifically what I’m looking for. Can you help me?

 

‍