An Elegant Puzzle, Part 1 with Will Larson
SCALING SOFTWARE TEAMS – EPISODE 028
This week, we’re answering listener questions about real-life situations with Will Larson of Stripe.
Will Larson is the author of my new favorite engineering management book, An Elegant Puzzle: Systems of Engineering Management. Will has been an engineering leader and software engineer at technology companies of many shapes and sizes including Digg, Uber, and Stripe. He currently leads foundation engineering at Stripe, where he is responsible for the infrastructure and platform organization. If you haven’t heard of Stripe, they’re a huge player in the online payment space, building the economic infrastructure for the internet. They’ve raised over $700M to date and show no signs of slowing their growth any time soon.
It’s much easier for everyone involved and much more equitable for everyone involved to actually have a system that you follow, but it’s having the courage to follow the system when it’s hard is a real thing that I think is why it’s so hard in some situations to actually commit to this approach.
Listen to the Full Episode Here:
‍
Episode Highlights
• The first metric you need to improve in your hiring process
• How Stripe defines developer productivity
• Why the best policy might be to throw away your policies
‍
Full Transcript
Wes Winham:
Today we’re doing something a little different for part two of our conversation with Will Larson. If you didn’t listen to part one, I’d recommend going back and checking that out. Will Larson is the author of my new favorite engineering management book, An Elegant Puzzle, Systems of Engineering Management.
Will has been an engineering leader and software engineer at technology companies of many shapes and sizes including Dig and Uber when they were scaling from 200 to 2000 engineers. And currently at the very fast growing Stripe. He currently leads foundation engineering at Stripe where he’s responsible for the infrastructure and platform organization. If you haven’t heard of Stripe, they’re a huge player in the online payment space. They’re building the economic infrastructure for the internet.
They’ve raised over $700 million to date. And they show no signs of slowing their growth anytime soon. This week we’re going to be doing a Q&A where we dive into specific stories and problems submitted by listeners. If you want me to ask one of our guests about a situation you’re working through, send me an email at wes@woventeams.com. And now here’s part two of my conversation with Will Larson. There is just so many tools in your book that I found useful related to specific engineering problems. So I’d like to talk through some of those problems. I went out and asked some folks about problems they’re having in their team and I’d love to hear your thoughts.
Wes Winham:
I know these are always … The right answer is always going to be it depends. So I’ll try to make up some details. So feel free to ask me and we’ll make it up on the spot. So we have an organization that’s had a data science team for the first time for maybe the past 18 months. And really only one thing has made it into production. Part of this team’s mandate is shipping things of production, but there seems to be a mismatch between good ideas and stuff that ends up in the application. Where would you look to help diagnose that problem?
Will Larson:
The first place my head goes to is kind of like structure. The first thing I would check is that. Whenever you roll out a new function, you kind of have this decision about doing it vertically or kind of embedding. So you have a data science org and they’re kind of off kind of somewhere together? Or do you actually have each of the data scientists join a specific team? Like working on a given product or something like that. And I think the vertical approach is very hard to actually ship.
Wes Winham:
That’s what’s happening. This is a vertical setup.
Will Larson:
You see this same challenge when you roll out SRE or when you roll out TPM or you roll out any function where they’re just vertical. They tend to be misaligned with the actual priorities of the teams that they’re working with because they have literally … There’s literally no aligning function. They have some sort of new goal, like I was talking with one person and they were launching an experimentation engineering team. And their goal was to increase the rate of experimentation. But the teams they needed to launch the experiments weren’t goaled on experimentation. And they were goaled on kind of shipping.
Will Larson:
And it just wasn’t working, but it was because the actual incentives were misaligned across these groups. So to me, changing the model, actually going through and embed I think is quite important. Too, if you have this vertical model, you can really only consult is typically the challenge. Where you then have to show a lot of value to the teams. How do you show them that working with data science will get them better results? So something that I’ve done on the infrastructure side is we will kind of do a swarm where we will have … If you’re launching like an SRE function for example, have the entire group go work with a team that’s having the particular problem you’re most able to help. And then you’re able to build a bit of a case study around it where instead of just talking hypothetically, it’s data science can be useful. Instead you’re like, we worked with the growth team last quarter and we launched these features together.
Will Larson:
And they actually have these results. And kind of moving from this belief about data science being useful to an actual case study of it working at your company with results that got recognized in a word by leadership.
Wes Winham:
So if I’m on the application side, it sounds like if I can’t change the vertical alignment, I might be able to pull data science in for one project and then publicize that project is a success.
Will Larson:
Yeah. Typically I don’t find many companies where people feel like they have too many resources. So typically you can kind of find a way to harness this new function in a way where you get some real value out of it. But you also get to support them proving out their concept. And finding that kind of mutual kind of value for each other is I’ve found to always be possible and quite valuable.
Wes Winham:
It seems like as an engineer, is where I think we’re particularly not inclined to pull the … We should promote what’s working market what’s working. That seems like a rare tool in the bag.
Will Larson:
You can actually create so much … One of my biggest things I’ve been thinking about for the last two years is I think sometimes companies think people do the wrong things because they don’t care enough or aren’t working hard enough or aren’t passionate enough or something. But I’ve almost always found that people who are doing the wrong things are missing context. So how do we create nudges, how do we create automatic feedback loops where people get the missing context? How do we change people’s behaviors by believing in them as rational people? And that makes us believe that we haven’t given them the information they need to be successful versus framing it as a they’re not good or they don’t care. Which is these kind of weird blamey things that make it not our responsibility that they’re not doing data science. They’re not doing reliability or not doing whatever the top of mind thing is for you.
Wes Winham:
These data science folks, they just don’t really care about our customers and the product. They must be just a bunch of academics.
Will Larson:
Yeah that’s an easier story than, I’m not doing what I need to do to give them the context and alignment to be successful. So let’s another team, this team is they’re about 20 engineers, so roughly four teams split off. Half the teams are application teams and they’ve recently split into front engineer versus back engineer titles. What they found is that they’ll plan a scrum, an iteration from a customer perspective and they’re like, yeah this is great. And they’ll look around and they’re like, oh man, we don’t have any work for the front end engineers. So you kind of have to make [inaudible 00:06:48]. So it’s been really hard to keep things staffed.
Will Larson:
This is an interesting one. So I think one of the things I really like about Kanban is it forces you to reflect on your actual process of delivering work. And I worked at … I can think of one company I worked at a couple jobs ago where we had this Kanban board and it was kind of spec, design, front end, back end, production, or validate and then production or something. And every single thing was blocked on back end. And were like, well the back end engineers are just not very fast, which is one belief, but I was one of those back end engineers. So that was not my belief. But then we had to think about why have we ended up here? And I think the powerful thing about Kanban is it shows you where your systems throughput isn’t working.
And it forces you to ask this question. And I think almost certainly the thing that you have to say is that you actually haven’t … Either you haven’t hired the right people, or you aren’t picking the right work for these skills of the team. Or you should revert this kind of distinction right now. And it really depends on the skills of the folks that you’re working with. I think if everyone was joined together before, it’s almost certainly the case in my mind that they’re mostly full stack. Some of them are stronger on front end, some of them are stronger on back end. But probably all of them can be some degree of kind of both.
And I think pulling back from this idea of I’m a front end engineer in a team of the size you’re talking about so that you can actually work on the constraints, there’s this idea that you only get value from shipping. And if you are doing work, you can’t ship because you’re doing front end work because you’re blocked on back end work. That work’s never going to do something of value or at minimum it is less valuable than other work you could be working on. So I think not creating these artificial constraints by specializing too early is important. But you also have to be honest about if the skillset constraints are real, and they really are folks who can only work on front end engineering, calling them something else doesn’t fix your problem. So you just have to be honest about the difference between naming and reality.
Wes Winham:
Where in a company’s lifestyle, maybe size have you seen it be worth it? In a generic sense obviously the skills of the people matter to split between full stack software engineer and specialization?
Will Larson:
This is an interesting question. I think successful companies that are hiring well and are, the business itself is doing well, and have a good culture that people want to join, often don’t have to do the specialization for an extremely long time. At Stripe we have different interview loops. Like if you’re interviewing to be a front engineer you’re going to have a different kind of set of interviews you perform than if you’re hiring to be a kind of a infrastructure engineer. But the actual title, the [inaudible 00:09:36] et cetera are going to be identical. And I think that in many cases, not specializing is really powerful if you can.
Conversely I found some companies can’t hire if they don’t specialize. And so I think it’s necessary in some hiring conditions. But generally I think trying to do as little specialization in terms of this sort of thing is powerful because it’s not just that there’s smart engineers and there’s back end engineers. It’s there’s platform engineers, and there’s infrastructure engineers. And there’s reliability engineers and there’s the designer who’s kind of an engineer but leans a little bit more on the design side than the UX engineer side. And there’s all these gradients where if you can avoid clarifying all of them it’s really powerful.
But often the way I think you should make this decision is are there people who are doing work that’s critical to the company? But your ladder, your performance cycle can’t properly recognize and reward. And if you’re in that scenario, that’s when I would make another level, another ladder, another role. But as long as you’re able to properly reward the work, I wouldn’t create a new role for folks in general.
Wes Winham:
So if I’ve got a software engineer that is just really driving across the scope, across the organization putting front end engineering forward, but they’re not meeting some other skill ladder component, so they can’t go to the next level, that would be a sign that maybe we need to create a separate track so this person gets rewarded?
Will Larson:
I think that’s right. You often see a lot of ladders designed around infrastructure. For example, you have to be a staff engineer you have to design a distributed system supporting hundreds of thousands of requests per second or something. And then you’re a front end engineer and you’re like, well, this kind of recludes me by definition. And so making sure you don’t have anything definitionally that makes it impossible for someone who is more product engineering focused or UX focused or more infrastructure focused, you want to be able to support all those paths kind of equivalently according to your company’s needs.
Wes Winham:
So in this next scenario my organization is undergoing kind of an agile transformation moving from waterfall monthly releases, they’re really painful, everyone goes around. Brought in a new VP of engineering who’s making changes that are moving some of those numbers. The challenge is I’m an engineering manager and there’s just a lot of rumbling because the VP of engineering communicates things exactly once and they do it mostly talking out loud and then they’re kind of bored of talking about it. And I see ideas that I think are actually good but I see my team not happy with them because of that lack. What are some things that I can step in and do to help?
Will Larson:
There’s kind of three interesting things here. The first, one of the hardest parts of being a manager is helping teams emotionally process change. And so what they’re describing is definitely a real thing that managers spend a lot of time on. And it’s hard. It’s hard to be frustrated about a change yourself and then help 10 people process their frustration with you in a way where you don’t reinforce it. Which then means you have to spend even more time tomorrow helping them process it. But it’s like how do you help them actually get rid of their frustration in a constructive way?
Second I think a common mistake is that your leadership will be doing something you disagree with and you will be annoyed at them. But you won’t actually ask them to change. For example, in this case I think a lot of folks who are new kind of senior leaders aren’t familiar with the drumbeat of repetition that is kind of the core of senior leadership, which is not you say it once. It’s that you keep saying it in every form, in every discussion. You write it down, you say it in all hands. You include it in the top company priorities. You just continually drumbeat it. And I think my impression is that that new senior leader, that new VP of engineering is either a new kind of senior leader, or this is one of their particular skill gaps that they know they need to work on.
So first, I think they’d be extremely unlikely to go talk to that person that they’re shocked by this feedback. It’s probably, they probably already know and it’s important for them to get that feedback. And I think the thing I found most successful is that particularly for things like this where the person already knows this problem, probably this is just my guess, I’d say the majority of feedback I get I’m already kind of aware. But I’m kind of in a rut where I’m like, yeah I know I’m bad at that. But aw man, I’m so busy with this other stuff, I’m just going to keep doing that poorly. That if you can come with them and if you send out an email and I’m glad to draft it with you, I think that’d be really helpful. And if you talk to ATH or sorry, your all hands or whatever you call it next week, that’d be great too.
And if you talk to my team. If you actually give really specific suggestions, I think it can help people who are either overwhelmed with the area or are kind of uncomfortable with it in a way where they’re avoiding it. Versus just telling them there’s a problem. It’s rare in my experience that folks at that level of seniority are completely unaware of the things they’re doing wrong. And so giving them suggestions I think can help be … It’s not fair that you have to give them suggestions, but it’s effective. And so I think as you manage up you have to be honest about what’s effective versus what you should have to do with your manager. That’s relevant but not very effective to get caught on what your manager shouldn’t require of you versus what you can work with them on to be helpful to your team.
Wes Winham:
So my … I would go to this leader probably in person, get a feel for whether they … Give them the feedback. Get a feel for whether this is a surprise or not. And if it’s either way, I have the opportunity to suggest specific actions and maybe offer to help.
Will Larson:
Exactly. And even building on it, I think you can go in person, but there’s a way to frame … If you aren’t frustrated, there’s a way to frame any advice in a way where it’s going to come across fine. And so if I were writing this email I’d say something like, “I’m just so excited about the agile rollout. This is exactly what we need. I have heard a few folks that are surprised about the rate of change. And I think it would be really powerful to send out another email or two just emphasizing how valuable this is. And that we’re doing it. So folks know that we’re going to continue this until we’re done.” There’s a different way to write that email, which is, “Folks hate agile and this rollout is going very poorly. And I think you really botched this. And I’m frustrated that I have to deal with it because this is your initiative, not mine.”
And that’s not going to work in person or in email. But if you find the right way to say it, it’s going to … You can get it through over email. You don’t have to do it in person necessarily. Although in person is always a little bit easier for sure.
Wes Winham
I love that framework. Start with the shared thing we’re both excited about. And then give information so that you can … And then a specific suggestion. That’s awesome.
Will Larson:
Scaling software teams is brought to you by my startup Woven. Will’s book, An Elegant Puzzle has quickly become one of my favorite books on engineering management. I started Woven because I’m passionate about helping engineering leaders build better teams. And I think that Will’s book would be a huge asset for any engineering leader. That’s why we’re giving away free copies to anyone who signs up to learn more about Woven. If you go to woventeams.com/book and schedule a meeting with me to talk about your hiring plans, we’ll send you a free copy of the book. Again, that’s woventeams.com/book for your free copy of An Elegant Puzzle, Systems of Engineering Management. That’s woventeams.com/book.
Wes Winham:
So let’s … I’m at another company. I’m bigger, I’ve got several teams. And one day I look out, I’ve been working hard on [inaudible 00:17:29] inclusion on my team. I have a relatively balanced mix. But one day I look around and I realize all of my most important projects and initiatives are being … There’s no diversity there. It’s all older, white dudes in my particular story here. What are some things I can do to fix that? I’ve been thinking a lot about inclusion and kind of what’s a definition of an inclusion that is useful for managers and leaders who want to make change in their organizations?
Will Larson:
What I focused on this idea of there’s kind of two core dimensions, which is community and opportunity. So community is do I feel like there are other folks who understands do I feel involved? Do I feel comfortable? Do I feel safe? Just like what is your kind of anxiety level? Or kind of comfort level on this day to day basis. And then opportunity though is like who’s getting these most important projects? Who’s getting promoted? Who are all the senior staff engineers? Or who are all the senior leaders? These kind of outcome metrics that are not kind of debatable. These are actually the state of play.
And so one of the things that we did, and I’ve noticed many companies like this, right? Where you find it’s not even necessarily that all the people getting these certain projects are of a certain kind of characteristic, it could just be that they’re all from the same company or they’ve worked with founders before for example. I’ve seen that many companies are kind of the loyalists who might in fact be a diverse set of folks are the only ones who kind of get important projects, something like that. So we actually kind of designed this project selection process, which is little bit heavy. But we think about any project which is a scarce resource. And some projects are not scarce resources.
Wes Winham:
Like I need someone to help with our compliance research needs to be done in the next three days. This is not something where everyone’s chomping on the bit to be allowed to do it, right?
Will Larson
Yeah.
It’s not like, people aren’t desperately sad that they didn’t get to do that kind of that fire drill. But there are projects which are kind of perceived staff level projects, right? Where if you did this really well, this might be how you become one of the senior most engineers of the company. On a rewrite or a kind of rolling out a key piece of technology, integrating a key partnership, something like that. And so we actually try to follow this process now for anything that’s a scarce project where we send out an email broadly to kind of … We say all of engineering or all of the infrastructure engineering group or whatnot.
And we outline the projects. And we encourage people to apply. We give them usually like five to seven days to actually decide to apply. We want to make sure that people who might need to talk with their friends, talk with their manager, talk with their peers to kind of build kind of confidence they should apply. Have some space to do that. If you only have a really short timeframe, then you kind of get people who are willing to self-authorize, which is going to be like a subset of your overall organization. Then the next thing I do, and we had to see this fail a few times before we started doing this. Is then I go through, I and maybe some of my peers will go and find every person who we think could do this and hasn’t volunteered for something similar recently.
And we go nudge them to apply. Just be like, hey, I think you’d be an amazing person to do this. I think you should apply. And I think that helps people who are kind of on the fence lean into it and kind of decide to actually apply. Then we evaluate, we try to give written feedback to everyone who we don’t pick so they know how we did the evaluation. And that creates a sense of confidence in the program. And then we pick someone and we’re off to the races. But I’ve really found that structure is important to make sure people have the opportunity to apply. And people feel like it’s worth their time to apply. And if you don’t have that, I think it’s very hard to get out of this trap of just leaning on the same folks over and over and over.
Because it’s more efficient to do it that way, right? It’s a lot easier than running this process. But it doesn’t create the same kind of organizational bench longterm where you have many people who can run these projects over time.
‍
‍
