Optimising sociotechnical systems

Ben, welcome back to Ship It!

Great to be back again. I can’t believe it’s been a year since our first round.

Do you remember the episode, the one when we first recorded? The episode number?

Yeah, it was – well, this time last year; when we chatted again recently, I couldn’t believe it had been a year. But I check back on my calendar, sure enough, a year ago. Crazy.

So that was episode four. If it was five and this would have been 55, that would have been perfect. However, by the time you’re listening to this, it will be exactly one year, give or take a day, from when the previous one came out. That was episode four, “OODA for operational excellence.” What is new with you?

Well, still definitely going around that OODA loop. Since we chatted, I’ve been doing a lot more research and exploration of the OODA loop. I was very privileged to be part of a five-series conversation which was streamed on YouTube with some OODA experts, and we had some really, really interesting discoveries… Not really discoveries, but insights into OODA and things like that.

Over the last couple of years I’ve been trying to figure out how to build better OODA loops for businesses through technology. By the time this goes out, I feel fairly safe to say without touching wood and making a tap in the microphone that that company will be live and will be rolling that out. So… Very exciting.

So OODA still works. That’s the important one. OODA as a principle works well.

Absolutely. I do a sporadic podcast with a friend of mine and one of the things we say on there is OODA is pretty much axiomatic at this point. Whether you call it OODA or whether you call it something else, the thing that’s going on there is going on, whether you like it or not. You don’t do OODA, OODA does you. [laughs]

Wow, okay. Why do you say that? Why do you say that OODA does you? That’s a very interesting expression. I haven’t heard it before.

Yeah, it’s because people always say – you know, there’s this move within our industry… There’s this over-focus on frameworks, and we’re going to use this framework, and we’re going to do this methodology, or we’re going to… Pick your poison there. But OODA is not something you do. OODA is something that happens to you, because you are an entity moving around within your environment and it’s a biological imperative that you improve your capacity for independent action, whether you as an individual person or a business.

So OODA is happening whether you like it or not, and whether you call it OODA or not, because there’s plenty of other things that approximate Boyd’s work out there. So at this point, all of the exploration and research I’ve done, I do believe that it’s just something that’s happening, and whatever labels we use doesn’t really matter, because underneath there’s that principle of reacting to the environment that literally everybody has to participate in.

So a lot of great words were mentioned in these few minutes, and I’m going to mention another one, which I think goes really well with this… And then you have to tell me what generator do you use the generate your talk names, because I think they’re spot-on. Maybe it’s you… [laughs] You gave a talk, The Paradox of Control, and I really liked that title. I really like the way you formulated it. So what gave you the idea or how did you come up with that title? I thought it was an excellent one, by the way.

So this was a talk that I gave at the NATO Command and Control Center for Excellence last November. The C2COE is a kind of a practitioner body that is aimed at improving military command and control systems. My background is – when I was in the military, when I was in the Royal Marines, I was a signaler, so I was one of the low-level cogs in the command and control system, if you like.

So I went and looked at all of the recent research, breakthroughs, technology, approach, and I put that up against what I know of OODA and what I know of evolution and ecosystems. What I’ve found was a paradox - it’s that the command and control is over-focusing on the control part, and it’s focusing wrongly on the control part… Because the more you try and control a complex system, the less in control it is. Example - forest fires. With the huge forest fires that are happening in the States, there’s quite a strong argument there that one of the reasons they’re so huge and so out of control is they’ve been trying to control the outbreak of these fires for so long that it’s built up loads of potential in the system, and now they’re completely chaotic.

So I feel like can command and control, when put into the context of the military procurement system, you end up with these big programs worth millions or billions of pounds or dollars, very technology-first, very much designing a system that they expect people to jump through the hoops of… And I experienced this in the military, being given a piece of kit that was utter piece of rubbish, and being expected to use it, having never been – apparently no one on our level ever being consulted about the development of it.

Can you tell us what was that piece of kit?

So back when I was in the Marines - and this is going to make me sound extremely old, but it just goes to show how out of date the military system is…

Experienced. Seasoned.

Experienced, let’s go with that. That’s fine, I’ll take it. We were still using analog radios, believe it or not, around about 20 years ago, despite the fact that mobile phone networks were a thing, satellite communications were a thing, 3G was a thing, digital comms was a thing… But we were still using these God-awful radios that were developed in the ’60s and ’70s. Crazy.

Reliable, maybe? Was that the reason?

No.

No.

I’ll tell you a story in a second about probably the reason I left the Marines actually, but…

It wasn’t because of the radio, was it?

It was. It bloody was.

Really? Oh, my goodness me.

I’ll tell you in a second. Let me just finish this thought and then I’ll go back to that. So the piece of kit that we were testing was a way of securely communicating over HF radio. And HF radio, it’s a ridiculously low board limit because the frequencies are very low. You can’t physically fit much signal into it. And it was just awful. This was in the days where you used to encrypt radio signals by pulling a punched piece of paper through a physical piece of equipment. If you got caught by the enemy, the first job you had to do was burn the piece of paper so they couldn’t decrypt your signals. Seriously. 20 years ago, this is.

Wow, okay. Are you sure it wasn’t 50 years ago? Ben, just how experienced are you?

No. It was developed 50 years ago, but not updated for 30 years. So anyway, the story of the why I ultimately decided to punch out of the Marines and think that I had better things to do with my life… 2003, just after I’d been teaching myself to code, I was sat in Saudi, and this was just before we did the assault on the al-Faw peninsula as part of the Iraq invasion.

My job was to get all the comms ready. Bearing in mind, these communication systems are radios that you set up. Back in those days, you used to hand-tune the antenna lengths to make it match the frequency of what you were talking. We were trying to get comms with a ship that was maybe 50 kilometers away.

Right. Wow.

So there’s me and all of my buddies, sat in the desert for three days, twiddling antennas and moving masts around so they don’t interfere with each other. It was an absolute ball-ache. Then the Navy SEALs rocked up, because they were on the assault with us. They were doing a practice session with us. And they rocked up probably a few hours before kickoff time. Fairly nonchalant.

This signaler, my counterpart in the SEALs, pulled his truck up, turned it around, reversed it up next to our command post tent, got out of his truck, pressed the button, up went a hydraulic mast, and he picked up his handset and he had comms immediately. So what had taken me three days took him approximately three seconds. And I already knew that there was a better kit out there and…

So that was it. That was the moment that broke you.

Yeah, that was it. “The mission doesn’t make sense to me.” So that’s one of the reasons I’m now doing what I’m doing rather than being a sergeant major in the Marines or whatever. [laughs]

So compared to that, Kubernetes must be child’s play. Still, you don’t do that, because complexity - there’s better ways, and there’s a different type of complexity that you want to spend your time with. How does this compare to the technological landscape and the infrastructure landscape that we have today? Because I see a lot of similarities, but I’m wondering which ones are you seeing.

That is a very, very astute observation, because the dynamics that played out, of our kit being behind the times - you can see that playing out in reports about the Russian kit now. But the dynamic is exactly the same as what’s playing out in the technology environment. It’s obsolescence, and it’s an evolutionary struggle of ongoing improvement. Can you imagine if you had a piece of technology now that was 50 years old? We’re 20 years in front of when I just had that story.

No, I can’t… No way.

You just cannot stay competitive with these long procurement cycles. For us in technology, that means that the time horizon or the half-life of your current systems gets shorter and shorter and shorter, so you have to be continually reinventing yourself. If you launched a tech company five years ago and you became successful, and you’ve now got competitors… Say you launched five years ago and you launched on, I don’t know, AWS, maybe you were using Kubernetes at that point, maybe you’re using Docker, but probably you were using virtual machines, maybe you’ve terraformed it up, maybe… But you’ve become successful, you’ve gained a load of internal inertia, and now you’ve got a competitor snapping at your heels, and that competitor is using something like Vercel, using something like Fly.io, or railway.app for their infrastructure. They will be building - it might not be an order of magnitude faster than you, but it will be faster than you. Exactly the same dynamic of what I just shared about the radios back then is now in effect at probably two, three, ten times the rate in our daily lives as technology professionals.

Okay. I had a recent conversation about this, where we were talking about Terraform and ECS versus something more modern, like Hasura or Fly.io. People keep reverting back to what they know. So there’s an element of being comfortable, and they say, “No, I’ll be quicker, because what I know will make me quicker.” What would you say in those conversations? What would be the argument that you would use for the new wave of infrastructure, the new way of tech?

Yeah, I’ve come across this argument, and it does make sense in certain contexts, I think. Government IT is a good example. In the UK, in many departments in the government there’s this mantra of “Use boring technology.” There’s even a website for it; I can’t even remember what it is, BoringTechManifesto or something. Again, this is something that may have made sense five years ago.

If you’re five or ten years ago and you had the choice of all those sexy technology or stick with Java, then that might be defensible back then. But if you had the choice now of choose something that is a huge CI/CD pipeline, multiple days to get anything out the door… Technology moves on, and you can always make that choice of familiarity giving you that immediate getting-off-the-ground advantage, but it’s always a trade-off. You will always have to be knowingly trading off that there are other teams around that are using technology that’s more modern, and they might not be even as good as you… Good if there is some sort of objective measure of goodness within a technology team; but they will be using better tools.

Like with anything, there’s no right answer. But if you’re a company that’s maybe fallen behind the times a bit, there is a very strong argument for saying, “If you’re going to make a leap forward, make a leap forward right to the cutting edge of this evolutionary landscape, rather than making an incremental step forward.”

Okay. I’m wondering how much of this is linked to that – you know, when you create and destroy, create and destroy. Some call it disruption. I think you have another way of phrasing it. How much of this plays into that? …in that, if you’re comfortable and if you say, “No, no, let’s just do what always worked,” what got you here will not get you there, and you’re trying to get there. You’re not trying to get where you’re at. How much of that do you think translates into our technological choices? Whether it’s Kubernetes, whether it’s VMs, whatever we choose.

Yeah, that’s a great question. There’s always the pressure to get started as quick as you can. That’s the whole lean startup, all that kind of thing. So given that you start a journey with the resources at your disposal, there’s obviously a strong incentive to begin that journey with what you have right now. Take a small step forward, prove that it works.

I was in a really great conversation last night, actually, and the issue is that it’s not a turn-based game. People don’t wait nicely in line, your competitors don’t wait nicely in line, the market doesn’t wait quite nicely in line. It’s not chess where you take a move, somebody else takes a move; everyone’s moving all the time. So the problem with comfort is that you can become accustomed to being comfortable and you’re not ready to be uncomfortable. The more comfortable you stay for the longer time, the more uncomfortable you’re going to be very very quickly when the situation changes, because you’re just not used to doing new stuff.

You can see that in – government services, and again, I’ve mentioned them before… But the pressures that society faces as a result of technological change are now butting up against - across all of the world really, apart from maybe Singapore and some other forward-thinking places - are now butting up against the inability of civil service, political systems, to change. And when you’re now looking down the throat of it, economic crisis and climate crisis…

I was listening to the radio on the way back from dropping my little girl off to school and they were saying that, “We’ll see, we’re going to get planning permission in two weeks or a couple of weeks.” I’ve worked in the Planning Inspectorate a little bit… So that’s probably the end two weeks of probably several years of back and forth and utter waste of time. And now we’re going to begin building, hopefully in the next few months, and we’ll have a new power station in 10 years. Wow.

[laughs] Wow. Yeah. I think this is so relevant. The energy sector is going through some huge change. We’ve been talking about renewable energy for a long, long time, and there are certain events happening around the world which have accelerated this. But is it… I don’t want to say too late, because it’s never too late… But is it maybe five years too late? We could have done this five years earlier; why do we have to wait? But then I think there’s something in us… We are waiting until we can wait no more and then we make the change. Maybe there was a better way. Maybe, I don’t know.

Yeah, there clearly is a better way, but the incentives against following that better way are just heavily counted against doing that; political systems and incentives. Yeah, so that’s an extremely difficult nut to crack.

With our own infrastructure with Changelog, every year we try to do things differently. We try to challenge ourselves and go in a different direction. This year was no different. This year – I mean, who goes from Kubernetes to PaaS? I think very few do that. We did it, and Jerod is so happy about that move. I would have not expected that outcome, and others which are coming. It just goes to show that if you try things that you wouldn’t normally try, because - it’s not that they don’t make sense, but you go in a slightly different direction. You’re trying to look for the learnings, you’re optimizing for the learnings, not necessarily for where’s the majority going, where it’s safe to go, what has been proven.

Okay, PaaS’es have been around for a long time, but there are certain elements which are new, such as for example WireGuard, or IPv6, or other build packs. And there are other technologies like that which have matured and have been combined in a different way. I think this goes back to the paradox of control. It’s not about control, it’s about how things combine. And we are combining the same elements in a slightly different way. And they may be new elements. This combination of things which, to me, is fascinating. Can you tell us a bit more about that from your perspective?

Yeah. That’s exactly what I meant about the paradox of control. When you do an experiment, if it’s a good experiment, you probably don’t have a great deal of control over the immediate outcome. But doing many, many experiments paradoxically gives you more control over your destiny, because you are learning, you’re adapting. And that ability to learn and adapt is way more important to have as a capability than the outcome of a single experiment along the way.

It’s like the process of evolution… There’s many mutations and lots of them are unsuccessful for the individual, or in some bad cases, the species. But ultimately, the process of evolution gives you literally everything that you see as you look around outside your window, inside your window, the technology that we’re talking on. Again, OODA is always happening, so you either become good at it, or you fall behind. There is just literally no other choice, in my view.

Right, yeah; that’s a good one. So this makes me think about more specifics and the tools… Some of the tools that you’re mentioning to me, that are even new to me. You were mentioning Metabase the other day, you were mentioning N8n.io… Again, let’s make that clear, N8n.io. Notion… I think many already know Notion, but I think there’s ways of using it and ways of thinking about it that many maybe aren’t doing that. Can you tell us a bit more about these types of services? Why are you attracted to them? What is the value proposition there? Because there’s something there, for sure.

Yeah. 100%.

What do you see in this mix and in this combination?

Yeah. Up until now, we’ve been talking about developing products, I guess - developing products and services that you use to deliver to your customers. And that is an ever-growing feast of potential technology platforms and affordances that you can use to build products. The whole point of being in business is to find a problem for a section of the market and to solve it in a way that you deliver value much greater than what you charge, and it costs you to deliver that value much less than you charge for it.

So all of the product management and all of the technology talk is about that bit. That’s all external to the organization that delivers that, but the organization is the foundation. So I got very jaded… And this is probably how I ended up on this OODA loop and research and whatnot, trainers… As a on-the-tools developer, DevOps engineer, leader, with doing a good job on all the stuff that we’ve just been talking about - putting a good team together, building a good platform, and it’s just not quite landing either, because of poorly-aligned priorities, or a misunderstanding of the market, or increasingly, just utter friction internal to the company. So as I’ve been doing all this research on command and control and OODA and whatnot, I realized that we can take a bunch of this great technology that is becoming increasingly more available, and we can use it to build internal systems as well. Because if you put all of your investment into the externally facing parts of your technology and you’ve got a bunch of other interns running around doing Excel to try and figure out what the hell’s actually going on, which is increasingly what happens… Because the way I see it is that you’ve got different company departments – so a company gets to a certain size and it fractures into different functions. You get your head of X, head of Y, marketing, product, whatever. And they go off and they start building their part of the pie. And that’s the right thing to do, because you need a marketing department, you need sales, you need ops, you need customer support… The problem is, you get to a certain point where now all of those individual departments are choosing the right tools to support their journey. So they’ll pick - I don’t know, a HubSpot, or a Amplitude or whatever. There’s millions of these different services you can use to build a better department.

And what I’ve observed in many companies is that’s going really well for that department, but when it comes to taking a step back from operations and figuring out strategy, direction, priorities, and sense-making, and situational awareness, there’s a void in the middle. Because that void is filled by these same expensive people you’ve hired to build out your function, in some cases, trying to aggregate data in Excel very often. Or you’ve got a team of admin folks in the middle of this company who are usually not that well paid, and sometimes not very experienced, and they’re also trying to take all this data and aggregate it together in order to provide some kind of insight, or run payroll, or do any other number of biologically important things that keeps the company ticking.

And I realized a couple of years ago - why would you use Excel when you can connect up… All of these good frontline services have APIs; why would you have a human go into the user interface, download a CSV, manually do all that stuff the same time, at the same way, in the same error-prone fashion every month, or every week, whatever your reporting cycle is, when there’s all these affordances like N8n, like Hasura?”

So I spent the last couple of years basically building a stack of these open source tools that you can run yourself, that you can basically put into the center of your company and turn that into what I’m calling a mission control, which is an idea for the military… Right in the middle of any military unit, there’s always a usually a tent or a room somewhere. It’s got a big map of the area of operations on the wall, and that’s where all the information feeds in. It’s basically the same concept, but for a modern tech-enabled business.

Okay. How do you pick those tools? Are they the same set of tools? I don’t think that’s the case, because it’s very specific to what that company needs. And every company is different, and there are different stages. How do you know what to pick?

Same as all those other experiments. I’ve tried a bunch of stuff that hasn’t worked. So the observation here is that when you’ve got a company that might have between 50 and 100 SaaS products in use to support operations, the combinatorial nature of that means that that’s going to be different for every company. One company will choose a HubSpot, one company will choose, I don’t know, something else; one will be on Amplitude, one will be on Orbit, whatever. And the company will have a different makeup and a different social structure, and they will have evolved differently, and they’ll have different goals, and everything. There is definitely no product that you can just take and plunk into the middle of a company and say, “Here’s your business in a box.”

Again, what I was talking about in that talk, The Paradox of Control - instead of trying to build a product and you know flog a product, which is the best thing, candidly as a business – a product, I could do that for a huge margin, because it’s a software as a service thing and it would be great. But it’s not the right thing. This is something you have to build internally within a business. It’s something that all businesses are doing, but they’re doing badly, with Excel or bolting together bits and pieces.

So that combinatorial nature, all the different software in use to support operations means that you’ve got this factorial shaped problem when you add a new thing. If you’re integrating everything with everything, you’ve now got the number of previous things, of links that you have to build. So the observation of this tool is if you put something in the middle, which is this central nervous system if you like, the only thing you have to integrate to is that, and you build up this consolidated model of what’s going on in the world. That, I think, you can templatize it, you can leverage a stack of technologies that I’ve figured out with trial and error over a bunch of consulting engagements that have cost quite a lot of other people’s money… So I know that there’s a stack of five or six different tools that work well together and can build this capability very quickly. And some of the ones you mentioned, Hasura, Metabase, N8n, Notion - they’re all part of that stack.

But if I was to go into a company and they already had Asana, or they already had Monday.com, then obviously Notion would drop away and you’d use what was already there. It doesn’t have to be me doing this if a business is doing this themselves. There’s a landscape that you have to build your foundations into before you start building the internal systems.

So as good as these tools are, and as relatively easy it seems to combine them, there must be something more to this for it to be valuable. What I’m thinking - and this is an assumption and you tell me if I’m wrong - is that there are certain principles, certain approaches that are way more valuable than the tools that you choose, so that you can swap any which one out. It’s almost like our nervous system. It’s just continuously regenerating parts of it in different stages, but it still functions as a whole; it won’t shut down.

So there must be some – I don’t want to call them rules, but there must be some principles, some learning, some things which are just like truths, like axioms, if you wish, that will always be that way. Things that just happen and you don’t do as a result of those things. Can you tell us? I mean OODA, I think, is one of them. Are there other things like this that basically hold all this together? It’s almost like rules of the universe, but like rules of technology, rules of organizations. I don’t know what to call them, but I think you know what I mean.

Yeah. We’ve already touched on OODA. I think OODA is the thing that is like the box that this all fits in. Whether you use O-O-D-A to describe it, there is a process that’s happening, and that process does seem to be very well-described by Boyd’s work. Whether you take his diagram and his words for it doesn’t really matter, but that’s the principle of what’s going on.

Or you call it Agile, or CI/CD, or whatever.

You can call it whatever you like.

It’s a representation of that. But yeah, I really like that.

Obviously, if you call it Agile with a big A, you actually also have to be agile, which doesn’t always match up, as we know. So there are several things that Boyd said that I think are worth picking into here, outside of the context of OODA. One is that he said “People, ideas and technology, in that order, always.” Again, that’s the big stick that I use to try and beat this paradox of control into submission.

When you go the other way around, technology, ideas, people, it doesn’t tend to work that way. Technology always needs to be in service of people, rather than people jumping through hoops of technology. It seems like a simple thing, but you just have to look around the world and look at customer service experience in many businesses and understand that people have got that the wrong way around.

So although I am a technology person, I’m also a person, and this stuff has to be in service to the people who are doing the thing. That’s why in that Paradox of Control talk, I pick out this bottom-up approach of building technology. Instead of coming up with a top-down five-year vision of a product and then deconstructing all the bits you need to make and doing the waterfall building it, the world of five years in the future that that lands into will be completely different. As we all know, that’s the whole point of Agile. So the approach here is instead you collect capabilities together and you use those as the situation demands, reactively, to build something that is loosely aligned with a big direction. That’s one of Boyd’s other sayings.

The other one is – the thing that he talks about which is what drives the OODA loop and drives the dynamic of the OODA loop is that organisms or entities that are separate from their environment have this biological imperative to build their capacity for independent action. Now, that is quite a long sentence. Capacity for independent action essentially means that you are able to operate in the world without detriment. You have your own capacity to do your own things and your own moves forward. The issue that I’ve found that led me to this kind of technology approach was that you go into many businesses and you start talking about OODA and situational awareness and everything else, and people are like “Yeah, great story, bro, but I’m reactive 18 hours a day. That sounds really nice, but sorry, I don’t have time to implement that.”

So the other thing that you get when you start using better tools is you get capacity back. If you have, let’s take an example from our world, an engineering manager, or several, who spent half of Friday afternoon preparing reports… Very manual process - they go and download some stuff from Jira, and then go and download some stuff from GitHub, and they prepare a report that they then feed into the management meeting; that is not value-adding at all, because what you have is a human who is being a cog in a machine, because they’re doing straight-through processing of data that is available.

If you, instead, move that into a low code tool or a tool that you build yourself or whatever, you it move out of the human’s remit into a technology thing that will run reliably every time, take much less time and give that human back their capacity to be human, to make decisions, to take actions, to think. I became quite frustrated with the course that I put on and that you were one of my students on; I ran out of steam with that, because I was like, “And so what?”

Once I’ve done all of that explanation of OODA, situation, awareness, mission control, all of that good stuff in the military, it’s like, “And now how do I tell people how to implement that?” And that’s what this thing is, it’s a way to start implementing that to give people back capacity, so that they can exert the right type of control over their environment, rather than being humans acting as cogs in machines.

Yeah. I can definitely see how we are literally working our way backwards. We talked about technology, which actually should come last. So forget the tools, forget what you’re choosing… Now we are somewhat touching on - did you say ideas, or was it the process?

Ideas.

Ideas, okay. So these are the ideas that are behind that technology, and I think now we are coming to the people. So what do people do day to day to start implementing those ideas using the technology that exist today and that most don’t know about? Going back to the people - and I think that’s where maybe we should have started; that’s an interesting observation, even for me. Let me say so myself this time… [laughs] So what are the important things when it comes to people, when it comes to building healthy organizations, that build healthy products, that are just a joy to be part of and an honor to be part of? What needs to happen at a people level?

Yeah, that’s a great question, and I don’t think there’s anything like a single answer to this. But one of the books that I’ve really enjoyed reading and companies that I’ve enjoyed following is Corporate Rebels. There’s the Western kind of management, kind of doctrine, which is very much about building machines. You have a bunch of people, you build into a machine that makes money, you put this much money into the machine, and you get this much money out.

Maybe 20 years ago, before the internet came along and we came into a much more networked world, that kind of scientific management… I mean it still has its place, obviously; when you do a business model, you still need to know that it all hangs together and it makes sense. But that’s, I think, no longer the predominant paradigm of actually doing day-to-day operations and day-to-day management. What people need is… People have ideas, and they make decisions, and they take actions. The problem is that those actions and those ideas are… You only receive information about the effectiveness of those as ripples in time. So you do something now, and you find out how effective it was in the future, and somehow you need a way to close that gap.

Again, paradoxically, when you had businesses that were all collocated and there was a team of people in one place who were all doing this human stuff, you would have almost like a tribal wisdom about what’s going on. There would be a sense within there of what’s actually happening. I think that’s something that, broken down, especially when people start going remote and they synk, but it doesn’t need to be, because much of the sensory apparatus that we have as people in business are now technology-mediated. All of your customer data is flowing through a system of one sort; it’s no longer Joe in sales picking up the phone and talking to people. It’s actually probably much more algorithmic of, “This is our customers’ behavior when they use our product, when they see our website.”

So although we’ve lost a little bit of that human-to-human behavior, actually the richness of sensory data available to us has vastly improved. So the other aspect of technology here is that you can use technology to make that more accessible and understandable to humans in the way that would have been that kind of tribal knowledge beforehand. And that’s another thing, I think that traditional command and control hierarchical view… We’re in a much more networked world now. So you need to look at ideas like data mesh and things like that to understand that access to this data needs to be democratized. So technology should be there for people to use.

Like Metabase, for example, is a good idea. You mentioned it earlier, plenty of other BI tools. But with Metabase, you can give people access to raw data and they can aggregate it and slice it up without knowing SQL. They can share that without having to take a screenshot and email it to somebody. So it is like a nervous system. It’s building the nervous system of the business such that it is more useful to the brain of the business, which is the people.

Right.

So now that we are a lot more distributed, everyone’s remote, more remote than they were before, especially in the sector that we operate in, how can we make those people still feel connected and still feel like they belong to the same tribe? Because I think that is very challenging these days. It feels like it’s you in a room, best case, maybe at the kitchen table… How do you make them feel like they belong, part of a group that used to be the office, and is no longer the case?

Yeah, that’s a very good question. There’s something that I’ve been playing with here actually. I’m going to try out for the first time, live, on you. Let me know what you think of this. So another thing that came out of this research into command and control. A distributed system is necessarily an asynchronous system. There’s no single objective source of truth. There’s eventual consistency, and I think that the trap that we fall into, which is maybe a legacy of how businesses used to run, is that we over-focus on coordination.

A distributed system really wants to work in a much more eventually consistent way. So to build a distributed system, you use things like CRDTs and operational transform, and you don’t have necessarily an objective point in time truth, or not one that’s readily extractable. You can have a historical point in time truth. You can go back an hour when everything’s consistent, say “Okay, this is what we knew then.” But you can’t do that right now. And I think the mistake that we make as we get more and more distributed is that we try and brute-force coordination, when what we should actually be focusing on is alignment. As long as all that stuff is pushing in the right direction and that direction is clear, it doesn’t actually matter if we have an objective truth at any single point in time right now, because people are in different time zones and everything else.

I think the way you make people feel connected in a distributed context is you have a very clear mission, values, vision, all of that stuff that people talk about but don’t do very well, because they’re continually focusing on the day-to-day operational side of things.

So focus on increasing the capacity and the operational side of things by automating what you can, so that all of that consistency and whatnot is made part of the machine if you can, to the greatest extent possible. Use that capacity that you have to build human-to-human connections and connect those people with the alignment of the business, what the group is doing because ultimately, that’s what makes people feel part of something, is being part of something important that you’re moving towards; not “I missed a meeting, and now I don’t know what’s going on.” Does that make sense?

It does, very much so, and I know how important something like the culture is. If you don’t get that right - and this just needs to be part of the group; they just need to understand what the culture is, they need to be part of it. It’s what you do when no one’s watching, but it’s really what you do day to day. What does your day-to-day look like? Does your day-to-day or your ideal day-to-day match to the rest of your team, so that you feel that you belong? If everyone does the same thing, you start seeing events, you perceive those events, and you realize, “Oh, actually, that looks familiar, because I’m doing something similar.” That’s how you feel that you belong, or at least it’s a small part of it. An important one, but small part.

The other one is the clarity. Having the clarity in the group, within the team; what is it that you’re doing? And - this is the hard one - why? Why is it that you do what you do, and do you agree on that? Do you truly agree on that? I think those things are really hard, and they are becoming even more important these days to get right.

Yup.

Because otherwise, you won’t feel that you belong. There isn’t that office element where maybe 50% of the time you do something other than work, best case; it’s actually usually more than that. You don’t have those high social interactions, those high bandwidth social interactions which make these other things even more important. So the next thing which I’m thinking is how can you get that clarity? How can you capture the culture? How do you go about those things, which are hard, more important than ever, and I think there are much better building blocks rather than an office space, where you just hang out eight hours a day.

Yeah. That’s a difficult one. To what extent can you even capture culture? Because it’s an aggregate quality of a group of people; it’s not an objective thing. So it’s necessarily diffuse and nebulous and very difficult to grasp. You can make judgments about… You can do, I don’t know, questionnaires and tests on various different dimensions of culture.

But the other thing is to what extent do you need to have a homogeneous culture? Diversity, obviously, is very important. The thing with the military is you have diversity of decision-making, but you have homogeneity of action and standard operating procedures. That homogeneity means that people know their place and know what to do, but it also means that the culture is quite rigid. There’s a way of doing things in the military. If you don’t do it that way, you will very quickly be asked to consider your position, or you will consider your own position and you’ll move on. So we have to be really careful about over-applying stuff in the military.

Because one of the other aspects of this command and control paradox, the paradox of control is that in order to have that clarity of direction and mission, I think it’s actually necessary to give up control of immediate outcomes. In the leadership position, in order to become a distributed organization, that leader has to give up a lot of control and has to have trust in order to be able to do that in their subordinates. Because one mistake I often see is that leaders want absolute certainty about what’s going on, and that’s very uncomfortable to give up. But in order to build a distributed system, you can’t have – what is it? CAP theorem? I forget. I think the C stands for consistency?

Consistency, yes, availability and partition tolerance. Yeah, that’s it.

Yeah. So partition tolerance is what you need to very much be building for in an organization, because you’re async. You’re not online all the time, you’re not connected all the time, you’re not available all the time. So I think the biggest thing that I see businesses that seem to be doing really well, is that leadership ability to give up control and accept uncertainty. Because that rigid demand for certainty means that actually over the space or time, you’re losing control.

Yeah, I can see it. I can definitely see it. So I’m wondering, do you have an example of a business that you’ve worked with that you feel got all those three elements - people, process, ideas, technology - right, and what did that look like?

Yes, to a degree. I don’t think I’ve ever worked somewhere… I’m hoping to build a company that does this myself. But I’ve worked in places that have elements of it. As an example, one of the early places I tried this out was with a restaurant business. Obviously, during the pandemic that we’ve just had, restaurants had a bit of a nightmare. This restaurant business that I worked with was already using chatbots and quite a large network of automations for doing some of this stuff, which was working super-well for the operational side of things. People can feel connected because they were going through Facebook Workplace. They had chat bots in there that can answer questions for them.

The problem was that there was that lack of consistency of where that data sat and lack of situational awareness of what was going on. So general managers in that business, to find out how profitable they were, they would have to send an email to a central office who would go and look all the point of sale data and come back to them in two days… Which when you’re trying to make decisions on how many people to put in, how many staff to put on - you need that; you need to have that without asking a human for it. That’s an example of humans filling cogs in machines. That answer, in any business that has an operational element, the stuff that people need to know to get their job done needs to be right here, front and center, on a screen in front of them, readily available.

That’s what I started using this stack to build. We centralized all the point of sale data, we built some views over that, we joined in all the HR data about who people are, how much they get paid. And then from that, you can build a view either as an aggregate at the business level, or as an individual general manager of a restaurant to say, “Okay, how profitable was I at 2:00 on a Tuesday?” And the answer in most places is pretty much not profitable, because you’ve got one or two people walking through the door, and you’ve got five staff on, and you’ve got that minimum level.

So you get this sense of the rhythm of the business through the data that flows through the business, and that worked super-well at the operational level. It didn’t work so well because there were other aspects of the business that I didn’t get around to tackling, like finance and the logistics supply chain and things like that. I think had that engagement continued, we might have got there. That was a relatively multi – 50 different locations, different kind of socio-economic conditions, they were all around the country… A restaurant in Canary Wharf versus a restaurant in, I don’t know, West London - very different demographics. We would have got there eventually, I think. That was one example.

Okay. So, Mission Ctrl, the company that you are just about to spin up… How are you thinking of using all those three elements within Mission Ctrl to basically show what something as close to the ideal would look like? What is the combination of all three elements? How are you thinking about it, and how are you thinking especially from the tooling side? Because again, technology, I think that is the enabler. People - I think it’s just a few of you right now, and the ideas, we’ve heard them. We had 50 minutes, 60 minutes almost, of hearing all those ideas. How do you imagine yourself combining all those into Mission Ctrl?

Yep. That’s a very good question. The typical engagement that I have in my mind is that – I’ve got to the point where I’ve picked a decent set of tools and I’ve got something that I can spin up on low-cost cloud infrastructure very quickly. Which is good, because that means that the cost of that is minimal and we can spend the cost on doing the human things of uncovering where the business wants to get to, what the goals are, what the drivers are, what the constraints are. Because you have to have that kind of big-picture understanding first.

The rough view is that we would spin up the cloud infrastructure, which would be all the bits that you need; that would be owned by the business that I go work with. I’m a purely service-based offering here. I’m not making any margin on any of the tools or anything like that. It’s a purely service-based thing to build a capability that that business owns and operates and it becomes part of them. Then, there’s that discovery process of where does the business want to get to, what are the biggest kind of bottlenecks or constraints, how might we want to reconfigure parts of the business model, and the operations in light of having this new capability?

The last you want to do when you… This is the big problem with approaches like RPA. You don’t just want to go in and blindly start automating the stuff that’s there that might not necessarily want to be there. You’re just accelerating chaos if you do that. If you’ve got a process that’s not particularly well-designed in the first place, you don’t necessarily want to just blindly automate it, because then if it’s really badly designed, you actually could be driving the business into the ground if that process isn’t profitable or whatever. There has to be that big picture, first look. Then, it’s a case of building a targeting list and biting off the parts that are causing the most pain.

If you go into an engineering business, it might be that, “Oh wow, we’ve got all of these expensive engineering managers that are spending 20% of their time collating reports.” Click, that’s easy, we can sort that one out. We might have something where the breakdown of customers and orders and their behavior once they become customers is unclear, because you’ve got the customer acquisition data held in one system and the operations in another. Then what we might do is just centralize that and build some dashboards over the top so that we can see different segments of customers and how they behave, which is not impossible to do in Excel, but it absolutely sucks and it’s really painful to keep that up to date. Whereas what we can do with these capabilities that I just mentioned is you can have a real-time feed from all of those operational systems into the single source of consistent data. Backward-looking, because we don’t really care about… It’s not a transactional system, it’s a business intelligence system so it works in retrospect. Then you use Metabase, or something similar. There’s another great one called cube.dev, which is a headless BI system.

The general principle is that all of these different parts of the system, where possible, should be open source. They should run on cheap PaaS platforms that we mentioned, so that the overhead of them doesn’t require a full-time engineer just to keep it up and running, and you can still use all of the engineers to build the product. But at any point, the bus factor of me is zero because it’s all built off commodity open source software. So if you’re an engineering company, if the worst was to happen, then you would quite easily be able to manage and operate that system yourself.

Right. Okay. For Mission Ctrl, the business, the website, what do you pick? How do you run your stack? I’m curious.

Yeah. Okay. Great question. I’ve nerded out on this long enough. So I have a content management system, open source, called strapi.io. That’s where I keep the structure of the website and blog posts and things like that. That is statically rendered into a Jamstack, Next.js site that is hosted on Vercel. Strapi runs on railway.app. Vercel is where I run my front-end systems that’s a statically compiled, very fast to load, optimized HTML build. Vercel also gives you serverless functions on the back end. Just a Next.js thing, you write functions in a certain way in a certain directory in your code, and you’ve got an API endpoint.

Another massive part of this stack is Hasura. So a very important part of this stack is where you keep all of that consolidated data. Hasura is fantastic, because underneath it is a proper Postgres database. So you’ve got all of the affordances that Postgres gives you to make sure that your data is well-formed and foreign keys are pointing in the right direction, and all of that. Then Hasura gives you a free API over the top of that in the form of a GraphQL schema that’s automatically generated.

That just cuts down all of the backend programming that you have to do, because you want to build a specific user interface. You can pick a Retool or Appsmith or you can build something yourself in Next.js, and the amount of backend programming you have to do is very minimal, because you just do a GraphQL query. In fact, you can even generate code, typed code if you’re using TypeScript. I might not have mentioned this before but my background is in functional programming and I like types a lot.

So yeah, you can build this quite robust, but very easy to operate system comprised of all these different parts. N8n, you can run as a cloud service, so you can pay N8n themselves to run you that. But if you ever find that you’re putting a load of volume through and it’s costing you too much, you can always run it yourself, again on the same platform, whereas I’ve sometimes seen things like Zapier or Integromat.

In that system, as soon as something gets expensive, you now have a very expensive rebuild job to do. Either you suck it up and you pay several hundred pounds a month, or more sometimes, which is probably worth doing versus the alternative. But that’s money that straight off your bottom line. Or you stop the world and you rebuild it into something that runs on some sort of cloud infrastructure, and that’s obviously expensive in engineering time. Or you do it this other way and you say, “Okay, I can run it myself on my own infrastructure” and you’ll just pay usage on Railway or whatever. Just a Docker container.

You gave me a lot of tools that I’ll be checking out. I’ll be dropping all of those in the show notes. Some of them I’ve never heard before, so this is super exciting for me. I really like the people aspect which you brought up. I thought that was a very important element. It’s just so easy to get engrossed in all the tooling, basically start nerding out and you forget, “Hang on, there’s humans,” and it’s not just you; you have to understand this, it’s humans that you work with.

And if they think that is the wrong choice, then you know what? It doesn’t really matter whether you like it or not. It doesn’t really matter how amazing you think Kubernetes is. For Jerod, it’s really hard, and this is a personal example. A PaaS, as some would say, “A PaaS? You’re going backwards.” “Well, no, not really, because there’s a new breed of PaaS’es out there that have new capabilities.” While it may look like a PaaS, actually, it’s a very different beast these days.

There’s possibly one more, but I’m wondering, a key takeaway for me… But I’m wondering, as we prepare to wrap this up, what would you say is the key takeaway for our listeners that stuck with us to the end?

So the key takeaway and the reason that I’m doing what I’m doing here, actually, is the key takeaway from 2000 years ago. It’s a Greek philosopher - they often come up with some pithy one-liners. It’s “You don’t rise to the level of your expectations, you fall to the level of your training.” So in the modern world, building technology-based businesses - you don’t rise to the level of your expectations, you fall to the level of your systems.

So increasingly, the systems that we build is the gating factor in how well we do. Because if you want to be competitive in today’s world, you have to have the systems to do that. If you were to, I don’t know, run any business that’s still in my village that doesn’t even have a card machine, you still pay cash.

Does the thing even exist? I don’t remember the last time I use cash, that’s why I asked. Everything is online these days. When do you use cash? Seriously… On holiday.

Exactly. So these guys now are incredibly vulnerable to the next chain restaurant, or whatever that rocks up, that can deliver their services cheaper, that can actually access delivery services. So the level of systems that you have, the level of sensible systemization you put into your business is increasingly the gating factor.

Those systems could be social systems. i.e. in the military, we have very specific ways of communicating; that’s a social system. It can be the socio-technical system, which is the mix of people and technology. Or it can just be the pure automation and efficiency play that you get from that, and there’s this whole spectrum, and you need to pay attention to that. You can no longer succeed where your competitors are paying attention to this serious competitive advantage I think nowadays.

So what I’m hearing is, if you’re using Excel, that is your limit. If everything boils down to Excel or it boils down to just email reports being thrown around, that is the level that you will be at. It doesn’t really matter what you use on the frontend if what runs the business are those - I don’t want to call them outdated, but less modern systems.

Yeah. There’s better alternatives. So if you’re using Excel and email, you are like an athlete with the flu. You are not going to win. Even if you would normally win, even if you’ve got impeccable training, if your muscles and your sinews are equivalent of people in a business, they’re really fantastic and you’ve trained them very well… If you’ve trained yourself extremely well, but you have flu, or you have vertigo or something, you’re not going to perform.

It’s exactly the same with businesses. You have to take the best of what’s available out there. And you’re right, Excel is definitely… The problem with Excel, it’s easy to use, so people start using it for databases and for operations, or for managing projects. And it’s not the best tool, it’s not even in the top five of best tools for any of those things. Even for modeling, we’ve got better options like causal.app which I shared with you the other day. So for me, I could just probably return my business, Excel extermination as a service, and it wouldn’t be inaccurate.

I think it’ll be very catchy. I think many people will recognize, “Oh, Excel, I know this.” Oh, no, no, no, this does the opposite Excel of what they want. [laughs]

Yeah, he wants to exterminate Excel.

Yeah, the Excel exterminator. Okay, so I’m going to defer the title of this episode to you, because you’re very crafty with them. That’s the first thing. The last thing is, I know that riding motorbikes is a very important element to you. I know that the talk that we mentioned, The Paradox of Control, starts with that story, which is a great one. You mentioned about you starting to ride on a track. And I’m wondering, how is that going? Which track do you use and how is that going?

I don’t actually have a motorbike anymore. I got to the end of the finance period on the motorbike that I had, and I just had too much work on, so I didn’t roll over, but I will be getting another one at some point. Just checking my wife isn’t in earshot. This is the other thing, building capability. I ride with friends who have been riding for a long time. And because they’ve gone through this muscle memory, they push it hard. Some of them really hard. They crash all the time, they really push it a lot harder than I do. But they also have experience, that time under tension.

So the thing that I’m doing, riding at the speed that they are completely at ease; they’re not even slightly taxed, and I’m at max capacity, trying to stay on the bloody machine. So I think there’s a really important lesson there, is that building systems in a business context is very much like building skill in a performance context as an individual.

You have these guys that are just pootling along at the same speed as I am, but able to literally put one hand on the handlebars and look over their shoulder to check my body position, and I’m hanging off the bike. I’m probably not at 100%, because that’s not how you build skill, but the difference there, and there’s other examples in the martial arts… So what riding a bike taught me was that your performance is predicated on your previous practice, not what you think you’re going to be able to do. It’s the same with systems in business.

Oh, yes. I can see a strong correlation there. That’s why I brought it up. Okay. Well, Ben, thank you very much for joining me again. Definitely doing this next year, or in a year’s time, maybe even sooner, but definitely next year, because I’m really enjoying these conversations. Thank you very much for joining.

Yeah, definitely. We said last time afterwards that we’d have to get together for a barbecue over summer. So we definitely need to make that happen this summer.

Oh, yes. Oh yes, that’s another one. Thank you very much for that. Thank you, Ben.

Absolutely. Alright, great stuff.