Changelog Interviews – Episode #575
Shift left, seriously.
with Deepak Prabhakara & Schalk Neethling from BoxyHQ
This week we’re going deep on security and what it takes to shift left, seriously. Adam is joined by Justin Garrison (co-host of Ship It), plus two members of the BoxyHQ team — Deepak Prabhakara, Co-founder & CEO and Schalk Neethling, Community Manager and DevRel as well as fellow Changelog Slack member.
We discuss how to shift left, the role of the developer and the burden of security, the importance of tooling, the difference between authentication and authorization, and a mindset change for when security takes place — it’s a matter of “when” not “who.”
Featuring
Sponsors
Vercel – With zero configuration for over 35 frameworks, Vercel’s Frontend Cloud makes it easy for any team to deploy their apps. Today, you can get a 14-day free trial of Vercel Pro, or get a customized Enterprise demo from their team. Visit vercel.com/changelogpod to get started.
Synadia – Take NATS to the next level via a global, multi-cloud, multi-geo and extensible service, fully managed by Synadia. They take care of all the infrastructure, management, monitoring, and maintenance for you so you can focus on building exceptional distributed applications.
Read Write Own – Read, Write, Own: Building the Next Era of the Internet—a new book from entrepreneur and investor Chris Dixon—explores one possible solution to the internet’s authenticity problem: Blockchains. From AI that tracks its source material to generative programs that compensate—rather than cannibalize—creators. It’s a call to action for a more open, transparent, and democratic internet. One that opens the black box of AI, tracks the origins we see online, and much more. Order your copy of Read, Write, Own today at readwriteown.com
Fly.io – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog and check out the speedrun in their docs.
Notes & Links
- BoxyHQ.com
- SAML Jackson - Enterprise SSO made simple
- Enterprise SaaS Starter Kit
- MVSP (Minimum Viable Secure Product
- Minimum Viable Secure Product (MVSP) checklist
Chapters
Chapter Number | Chapter Start Time | Chapter Title | Chapter Duration |
1 | 00:00 | This week on The Changelog | 01:23 |
2 | 01:23 | Sponsor: Vercel | 03:28 |
3 | 04:53 | Shift left! | 02:43 |
4 | 07:35 | Shift lift, no more right? | 01:46 |
5 | 09:22 | Too much burden on devs? | 05:01 |
6 | 14:23 | There's too much tooling | 04:20 |
7 | 18:42 | So, who does security? | 02:25 |
8 | 21:08 | Secure defaults | 05:29 |
9 | 26:37 | Auth for BoxyHQ | 03:00 |
10 | 29:37 | Right vs wrong auth | 04:16 |
11 | 33:54 | Sponsor: Synadia | 04:20 |
12 | 38:13 | Security as code | 04:14 |
13 | 42:27 | Justin's first automation | 06:36 |
14 | 49:02 | Tooling to reconcile security | 05:58 |
15 | 55:01 | Sponsor: Read Write Own | 01:21 |
16 | 56:22 | It's when, not who. | 03:13 |
17 | 59:35 | Minimum Viable Security Product | 02:47 |
18 | 1:02:22 | I can point to the drive | 01:14 |
19 | 1:03:36 | Build vs buy? | 04:03 |
20 | 1:07:39 | SaaS Vs boxed software | 02:49 |
21 | 1:10:28 | Can you hire for that? | 01:52 |
22 | 1:12:20 | Adam loves Pipedrive (btw) | 01:33 |
23 | 1:13:53 | We're not doing everything | 04:36 |
24 | 1:18:29 | Why is BoxyHQ open source? | 02:31 |
25 | 1:21:00 | How to be open and open to contribs? | 01:12 |
26 | 1:22:13 | Digging into commits | 01:10 |
27 | 1:23:23 | Gaming LOCs | 00:44 |
28 | 1:24:07 | Wrapping up | 01:34 |
29 | 1:25:41 | Up next | 02:34 |
Transcript
Play the audio to listen along while you enjoy the transcript. 🎧
So we’re here with Deepak - I don’t know how to say your last name. I didn’t ask you… Can I try? Prabhakara… Is that right?
Close, close.
What’s the proper way to say it?
Prabhakara.
Prabhakara, okay. I was pretty close on that one. And Schalk Neethling… Is that right?
Yeah, that’s good. I mean, if you want to do like proper afrikaans pronunciation, then it’s Neethling… But Neethling works in a pinch.
Okay. I’m going to pinch here, so I’m going to work with that one there. And we also have Justin Garrison here, from Ship It.
How’s it going, Adam?
Nice to have you here as a co-host. It’s super-cool to get you onto the network, but then also to get you onto this show, with our friends over at BoxyHQ, talking about security-related things… I mean, I think dev sec/ops sometimes is in my backpocket of things to talk about with folks when I meet them, and shifting left is always good for devs to be more mindful, I suppose of the code we write and ship into production. And maybe this is more in the Ship It lane, Justin… Just like, how do we put the code we put out into production, out there in a more secure way? So a lot of fun topics planned for today.
I think maybe the one to open up with is maybe the easy button, the shift left. It seems to be a “common” phrase for folks that say, “I get it. How do we define shifting left?” What does that actually mean, to shift left? Does it have a different meaning depending upon who says it and where? I guess that question is for everybody?
I think in a fundamental – there’s different layers to it, obviously. So very high-level it’s really about where the responsibility - of anything, because it could be shift left testing, it could be shift left security, shift left ops - is where does that responsibility lies… And left would mean more towards the developer or the person responsible for it, in some ways; or rather responsible for shipping something, to keep it simple. And on the right is really when it goes to – it gets deployed, and gets used.
So a lot of things traditionally have happened more to the right, which is after deployment, or closer to deployment. Shift left is really about bringing that closer back to the person really building things today, or building software, of course; that’s what it’s all about. That would kind of be my fundamental description of what shift left is.
So if we shift this towards the source, whoever created the thing, do we get rid of the right? Is this like a “Hey, you no longer need those security compliance tools or teams after the thing’s running”? Or is this now we do it in two places, and we have double the work?
So it’s definitely not getting rid of the right. I mean, there’s obviously those classic questions about who does it… For example, in this case development on one side, security on the other. Can you do away without a traditional security team? And the answer is actually no, because you can’t have the developer doing everything, of course; that’s one thing. Just not enough mindspace for anyone, no matter how big the team is, because it’s really about the focus… So that is really more about how you coordinate, rather than getting rid of one side or the other.
[08:28] Yeah. I think I see that in the accessibility [unintelligible 00:08:29.22] the same way, where you do want to shift left and give some of the responsibility over to the developer and the designers… But you’re never going to be able to get rid of your accessibility specialists, who can really make sure that all the nuance is not being missed.
But I think maybe another way one can look at it is when you have a bunch of stuff running in CI; so when you open a pull request, it can take care of a bunch of things that the maintainer now doesn’t have to review, because they can kind of run the CI and it’ll throw up like “Oh, you’re formatting is not quite cool.” ESLint can scream at you because you did something, and you can take care of that. But then you get into the minute things, the really detailed things. That’s where your specialists come in and really have additional knowledge.
I suppose if we think that the shift left movement is more towards putting more responsibility on the developer, I feel like developers already have a lot of responsibility. To then also take on one additional hat, or half a hat, or a portion of a hat is obviously going to be better for the product and better for the service and the company, but I just wonder, do we put too much burden already on developers to cover so much? I mean, even with npm; if you’re a frontend developer, pulling code from npm, it’s like a wild west out there in terms of like packages these days. We just had this major everything package that was meant to be a prank, but really caused some major harm… And that’s totally possible. And so we have friends like Feross Aboukhadijeh, who runs Socket.dev - not a promotion; love him, by the way - that really is helping developers to shift left, to be able to pull packages from the “open source supply chain”, which is not something I really believe in, but it is a chain of supply… So sure, open source supply chain… In a way that doesn’t require the developer to dig into every single dependency, every single line of code… Because it’s just impossible. It’s impossible.
So when we talk about shifting left and giving more responsibility to developers, how do we do that in a way that doesn’t add that extra burden? What are the things you’re doing at BoxyHQ to do that, and what do you think about it holistically, to not overburden the developer and say “Well, you’ve got to learn it, you have to have the responsibility”? How do we do that in a way that doesn’t just add more to them?
Yeah, I mean, I’ll take the example of shift left security, but it kind of applies to shift left in general… If you think about the spectrum, in a startup it’s very different from a way larger enterprise, that has to do shift left… Mainly because in a startup everything is fused into one. You have a bunch of people who are just responsible for everything: shipping, deployment, thinking about security in the middle, usability, all of that. So in many ways, you’re already wearing multiple hats. That’s what a startup is all about. But then as you grow, and as you start to say “Okay, now actually I should have a separate function here”, then it just becomes how do you now start to bridge the two? So that spectrum is kind of fused together in smaller companies, and kind of starts to spread apart in larger ones.
In smaller ones it’s much easier, because you kind of already have a bit of the context, the knowledge. It may not be full, but it’s enough to kind of do it. Whereas as you start to get bigger, you need enough tooling/automation in the middle to kind of help with - as you said, how do you unburden folks with? Everything that they already do, to have to do one more thing.
[12:06] So it’s more about providing them the right tooling to kind of give them enough context, give them just enough to kind of deal with what they have to do at that point in time. But then have enough resources to go back later, and then - you know, if it’s on a deadline that they’re dealing with, or fighting a fire, and then you can always come back to it, and… They then have that foundation to work upon. So that’s how we think about it at BoxyHQ. It’s really about process, tooling, and people. Those three kind of come together to really help you do the shift left. And it’s not possible without all of them coming together.
And I think what you mentioned, Adam, what the folks at Socket.dev are doing - I think Jackson is doing some of that, right? Because I think one of the ways to allow a shift left is to ease the burden with tooling. And I think you’ve hinted at this, Deepak.
So when you start talking about SSO, single sign on, and then you start exploiting that, and you get into all the nitty-gritty and the nuance there, and you then start as a startup, for example, having to deal with stuff like SAML, which is an older way of doing it that’s not OAuth 2 or whatever, there’s a lot of complexity that’s being introduced there. And now you’re thinking like “Oh, my goodness, how am I going to be able to deal with it?” And that is where something like Jackson comes in and it says “Well, you just talk the modern protocol, and let us deal with talking to the more older SAML-based system.” And Deepak, you can dig into that a lot more than I can. But what I’m trying to get at is this idea that I think tooling can play a role here, and I think Boxy is building some of these building blocks, these security things in the open, using open source, to ease the burden.
What’s Jackson? I’m not familiar with the tooling there.
It needs to have the full name, honestly. You need to say the full name.
Yeah, so SAML Jackson is effectively our enterprise single sign on product…
Nice. Okay.
Say my name! [laughter]
And you know, because we don’t have a marketing department to veto things, it kind of stuck… [laughs]
It’s very good. People say naming things is hard, and then you just nailed it. Where’s the line here, where you’re talking about moving responsibilities to the developer, and adding tooling for them… But at some point, when I try to write some code, there’s too much tooling. Everything gets in the way. I’ve got ChatGPT, I’ve got my reliability tooling, I’ve got some sort of like Docker helper, I’ve got a cloud helper, and then I’ve got the security helper. At some point, as a dev, I just need to write some code, and I need to not think about all of those things for a hot minute, so I can just get the text in the editor… And then I can go back and say “Oh, was this right?” or “How does this actually apply to things?” Where does the tooling get in the way, versus help a developer?
You know, I come from the pre-cloud world, so early 2000s, when I started my career, we had to go into a data center and then spend about a week there, hanging by the window, trying to get some Linux commands sorted out. Then you’d set up a rack, and then you can be useful, right? But today, of course, you can do all of that on the click of a button on AWS, or GCP, or any cloud provider out there.
So we’ve taken that, let’s say a week as an example here, for going to a data center and setting up infrastructure; we will compress that down to a couple of minutes on the back of tooling automation. So that’s how we think about it. You’re taking a building block, packaging it up together, it has some interfaces that you deal with, but you don’t need to now know how the actual infrastructure underneath works, or is even set up.
[15:59] Except for when it breaks, right? When something’s not secure or it breaks. That week in a data center taught me so much about hey, when something broke, I was like “Oh, I bet it’s this cable. I know it’s this plug”, because that plug was loose, and now I can go back and I can find exactly where it was. Because that click of a button helps eliminate all that learning. And at some point, I just need to go deep. It’s like, let me go to a data center and find the actual physical server. So I feel like those tools are helpful to accelerate some things, but I’m curious - because it feels like we need to be able to give someone that data center experience, to understand what’s happening under the hood, and how it’s actually built.
Absolutely. I mean, at least for me, the best way to learn is like you mentioned - you’re knee-deep in there, and something’s broken and you’re going to have to fix it… It doesn’t matter where in the stack it is. But I think that’s pretty hard today, because there’s so many layers of abstraction. And I say it in a good way, because - you know, how much time can we spend in this fast-paced world today to go and say “Let’s give a data center experience to, let’s say, our SREs”? You now just say “Okay, I’ll spin up another EC2 instance”, and just move stuff over and carry on, because you can’t be bothered to go figure out what exactly happened at an infrastructure level.
So in many cases, you don’t even have access to it, because it’s impossible. We have abstracted away so much that the only interface into that is a virtual machine. And you’re not too concerned about where exactly it runs, and how it’s orchestrated, because the cloud provider is kind of doing that for you. And that’s what it is today. If you’re building software, it’s taking all those abstraction levels, building on top of each other, and then saying “Okay, my business logic is here, so I’m just gonna go focus on that.”
As you mentioned, if something breaks, now you suddenly have to say where in the layer it is. But that’s getting better, of course. We’ve got enough tooling to say “Actually, this is in my application, and not anywhere else.” That’s, I think, where we’ve evolved to, and that’s where a lot of tooling can get in the way, but at the same time it will also help you get out of it.
At a high level, you need to understand which layer you need to jump into when something’s wrong. And as you said, a lot of it is [unintelligible 00:18:07.13] with experience, but in other places it’s just about specialization. You’re saying “Okay, as an SRE, my responsibility is just looking at what’s actually happening at an infrastructure, operating system level, but anything beyond that, now I don’t understand, because it’s application level”, and then you go and call somebody else on the next level.
And pretty much all those layers - they’re all divided by who built that layer.
Exactly.
Because the SRE is like “I built that layer, so I understand how this piece works.” And the developer’s like “I built this layer, so I understand how that one works.” Where does security as a shifting left paradigm - who builds that layer? If it’s a tool to help a developer, who’s the person that goes and debugs it, and knows how it works?
So the security gets very complex by definition, because it’s extremely layered. You’ve got security at the infrastructure level, even though you may not see infrastructure. For example, what kind of network is opened up to the wide world? I mean, today, if you think about it, as full-stack engineers we don’t really need to dive into networking. But at the same time, we don’t fully understand the implications of opening up something to the outside world.
I always use this kind of security by default example. In effect, at least from a technical standpoint for a developer, things are only as secure as the default option. Because if you think about, MongoDB, when they kind of came out and were getting popular, they made it super-easy to deploy something… As they had to, right? Because from a developer experience, you wanted folks to just get up and running. And I had this case many times, where I’ve helped other companies kind of figure out what exactly went wrong… And most of the times it was because they deployed MongoDB somewhere, but the default option is that there is no authentication. And that’s how – you know, 20,000 [unintelligible 00:20:01.15] databases were out there, and then Boxy came in, scanned it, got in, because the port is well known…
[20:11] No one’s at real fault here, because by default – an even simpler example is your modem. Username is admin, password is password; you always get that – at my parents place, they’ve never changed it until I went in and figured things out, and gave them the right… You take away the default, which is insecure, and then put in something, which then causes a usability issue, but at least it’s secure. So it’s that trade-off between what’s easy to use, versus what’s secure, but a little more difficult to use.
How do you understand how it works? Because that default password on the router - it’s probably okay as long as it’s not open on the WAN. And that’s what default providers started doing. It’s like “Oh, look, all these routers are available on WAN access, with admin admin”, and guess what? People can do whatever they want. And those defaults - yeah, that matters a lot, until it’s like “Did I plug in the data center? Or did someone else? Or did I click a button?” And for BoxyHQ, what sort of defaults are you changing there? What are you actually relying on a developer to understand and know, versus just doing it for like “Hey, guess what? Mongo is insecure by default”, or at least it was. “So we’re just going to do that for you.” But you have like three more steps to set up authentication; you need an environment file, something like that.
Yeah, exactly. So that’s obviously fairly – it depends on the context as well, because for us, that secure by default, I think we’ve learned that over time. Now, of course, Mongo also patched that up, so by default, you will have some kind of reasonable layer of security, and then you can kind of build upon that.
So I think that kind of awareness has already come in. We also had cases where - you know, Internet of Things got popular maybe a decade ago. And because – I mean, from the outside, we always think an attack is quite sophisticated, mainly because we don’t really understand how that actually went about. But the reality is people get in through very simple things. They try a lot of times, and they find one or two weaknesses, and then it kind of cascades into more things internally. I think classic a case was a fish tank in I think a hedge fund - or a casino; I forget exactly - and that was IoT-enabled and connected to WiFi, and that was open… So they got into the actual internal network on the back of a fish tank that was connected.
Could you imagine…? Who didn’t VLAN that network, right? Come on… That’s not a default in most networks though. VLANs are something that you definitely – because every network is designed, right? It’s flat by default, I suppose, so not secure by default, and then you unflatten it by VLAN-ing, and you would have an IoT VLAN that has blocks, allows and disallows, of course.
But who’s responsible for that? At a startup it’s flat, and - who knows? But does BoxyHQ VLAN my network? I still need a network admin for that, right?
Yeah, exactly. And even if you did all that, and somebody installs it and you don’t really know them - because as the person [unintelligible 00:23:14.00] you may not know that there’s a fish tank that’s enabled, unless you have the right tooling to scan that and say “Okay, now we’ve got another asset that I did not know about.” So I think that’s why I was talking about the layering, because you’ve got your external security, then you’ve got kind of your internal security, and then kind of more deeper things, like infrastructure, and things like that. Because ultimately, if somebody gets into your infrastructure, you’re in big trouble anyway. They can do whatever they want to. And every layer, you’re trying to protect them from getting deeper into your stack.
[23:47] So the way we think about it very simply is building blocks; a bit like Lego, right? Who’s constructing it, and where that kind of responsibility lies. So our – and of course, it’s a very large spectrum, as mentioned, and our kind of entire focus is on not so much preventing hack, but assuming that you will get hacked as a baseline, and then you need to think about “Is my product truly secure if they get to this level? Do I have the right set of primitives that prevent that?” For example escalation of privileges… Can they get to the user management database? Can they get to our sensitive data? So that’s kind of the different building blocks we think about at a high level, and then kind of try to make that as developer-friendly, at the same time having those reasonable, secure by default. And as a power user, then you can build upon that with your own security construct.
So that’s kind of our approach… Trying to be opinionated, of course, so that you have – that’s why the minimum viable security that we spoke about broadly… And this is an initiative from Google’s SecurityScorecard and a bunch of really great security companies. They call it the MVSP; that’s minimum viable secure product. And the fundamental idea there is that you need some level of basic security in place, at a minimum, for everyone. And then you can build upon that based on your context, the vertical you’re in, the kind of requirements you play with. So that’s how we think about it, we give you those minimum viable security blocks, and you then have that foundation to build upon.
I think one of the interesting things that I learned since joining - because I’m new at Boxy; I’m here for like two or three months, maybe - is the significant difference between authentication and authorization. And where SAML Jackson plays that SSO role, so that’s your single sign on role, and where that is focused on authentication, not authorization. And I think the authentication is this first level of security almost, where this gets you into the system. But at that point, the system does not know – it knows that it’s you, but not what you’re allowed to do yet. So that was interesting, I don’t know if we want to talk a little bit more about the nuance between authentication and authorization… Because what I’ve found when I’ve spoken to people before is that oftentimes those are used interchangeably, when they’re actually nuanced and different. The one gets you into the system, the other one deals with “Now that I’m in the system, and I’ve been authenticated, that I am who I said I am, what can I do inside this system?”
Is that something that you find developers are writing often? That interchange between things - is that something that’s like a common thing for developers that are using BoxyHQ to like “Hey, I do this all the time. I need this help”? Or is that something that’s kind of specialized for “Hey, we set this up once”, and then we have a structure in place for authorization, and we can say “Hey, we trust that someone else said you are who you are”? They have a key to the door; now they’re in the car. Do they have a driver’s license? Can they actually drive the car? Those are different things.
Or the skills.
Or my kids can get in my car, but they cannot drive the car. Is that something that you’re looking at, and that’s something that’s common? Or is this more of a framework helper for people that are like “Hey, what can I do now that I have access to the application?”
Yeah, so there’s always two aspects to it. One is more infrastructural, in the sense that what is the foundation there, right? Can you give folks the building blocks to say “You can now build, for example, an authentication policy around this”? So in our case, we do that with just purely enterprise single sign on, which is a way to get into any kind of system on the back of well-known established identity providers, like Okta, Azure Active Directory… And just to even do that, you kind of have to decide for all the protocols in between, how they talk to each other. There’s massive amounts of security vulnerabilities on the back of that simply, because of the protocols involved, and not necessarily knowing how to do it right. So even within authentication, to get authentication right is a lot of work. You can make things work, but then are you doing the right things in terms of being as secure as you can? Of course, there’s nothing like 100% security, but as much as you can.
[28:24] Yeah.
Would that be a maximal security viable product? Is that like the opposite of minimal?
[laughs] I think that’s pretty hard, because if you look at the breaches, even the best companies in the world get hacked. And they’ve got all the resources, that you could think of. So I don’t think there’s a maximum level to it, it’s more your risk management around it, and how much you put into the most priority risks that you have. So it’s always a trade-off in security. So you can have minimum, but I don’t think – because the maximum is never being able to get into your system.
If I unplug it, it’s maximally secure. Right? [laughter]
Exactly. Go offline, and don’t let anyone into the room. But at that point, it’s pretty unusable. And because systems are evolving so quickly, and you kind of have to – I mean, just over the holidays, even though I didn’t take a long break, but you come out of it and the number of npm updates you have to do is like insane.
Who is shipping code?! Why?! [laughs] Dependabot, I’m on break, alright? Chill for a minute! You mentioned right; so you mentioned authentication being right. But systems always change. Is there a right and wrong? Is there a - there’s a correct way for right now, or for this level of access? Because I feel like if there was a solid definition of “This is the right way, that’s the wrong way”, the right would be default. And I would hope it would be default, but I don’t think it is. And at least from my experience, that constantly changes depending on the people that need access to the app, how the app is interfacing with other applications… So how does BoxyHQ or shifting lefts move that definition from “Oh, this was right last week, but this commit changes what the new correct way of doing it is”?
Exactly. So there I think the important bit - there is a context, because you’re building new things… Requirements now change. Obviously, security also changes with the incoming requirements. A lot of the times you’re just getting something out, but not necessarily fully thinking about what are the implications of doing that. So in a lot of ways, first order effect is very easy, because you kind of see it; it’s cause and effect. But second, third, higher order is very, very difficult, even for the people who had experience… Because a lot of the times you may not have seen it, or it manifests in a very different way, which then leads to other things. So that second, third order, higher order effects is quite difficult. Security by default is an example there, because why would you think about that when you do something like a default option?
So in many ways that evolves as well, and so does your security on the backup resources available to you. So you’re a startup now, you’ve found product-market fit and you start to scale up… Then it’s a way different way to approach security, because now you’re popular, someone will try to hack you… Denial of service attacks, DDoS attacks… Because it’s so much cheaper to carry out attacks today than actually to prevent it. There’s a huge cost differential.
That’s kind of how you start to think about this, and that’s how security evolves for you as well as a company. And as you mentioned, is there a right way? There’s always a right way for the moment, and that then quickly changes.
[31:55] So that’s why shift left as a process is also important, because you’re constantly thinking about it. Now that the vectors have changed, you now say “Okay, actually, now I should go back and rethink a lot of things that I did not right. And can I get it to a phase now where I get it right enough that I can carry on to the next thing?” So that trade-off between making something work, to making something secure is quite a broad spectrum, and it changes very quickly, on the basis of risk, and resources available… Sometimes you know it’s there, but you know that it’s not exploitable… In which case it’s fine, because you’ve kind of dropped that risk now. So that’s how everything you do kind of evolves. It’s iteration at the end of the day with security as well.
For sure.
And that’s why I think it’s important to think about different layers as well. You can’t just make your product secure without that external testing, that external approach to it. Someone who doesn’t know your system might actually have a much better understanding of how to break it, than you were in the weeds and think that you have everything covered. Way simple things, like API; if you think about API keys - you may not have thought about everything. So you let someone else come in and kind of do that. That could be tooling, that could be people, processes… That’s why data is important, and you have to do it continuously. Otherwise, it just breaks down eventually.
Always be improving. Kaizen. That’s what we do around here at least…
Absolutely.
How does security as code shift left, I suppose, the shift left? Because security as code seems very similar, where you just treat security like any other workflow when you’re a software developer, the full entire lifecycle. It seems like shift left, but how does that change the lens of how you look at security when you think about that?
Yeah, so security - I think that’s quite new as well. Security as code in itself is really about automating certain things, which fundamentally you couldn’t do earlier, because you had not built it in the right way to be automated. So an example is when you’re talking about authentication authorization. Let’s say you build an authorization engine, or you pick one off the shelf somewhere. That fundamentally means you build a policy around it. And now that policy, of course, will change over time. But then, because the policy is now in a way that actually can be embedded in code, it’s quite easy for you to go in and automate tests around that. So now it suddenly goes from just having been a policy, to you being able to tweak those things around. So that kind of is an example of security as code. Just like infrastructure as code. Earlier you built infrastructure, but didn’t have a way to replicate it. Infrastructure as code now allows you to kind of bring up the same set of instances if you lose something.
Same way with security. You codify it so that you can kind of rebuild the whole thing again, and then tweak some parameters if you wanted to. Let’s say you had to add a new role. And now you’re in a position to say “Okay, now with this new role, can I go back and test everything that I had before, and then improve upon that?” Because now I know that for example, if you add a billing role, then within your application you know what is kind of segmented off, and now all your tests can kind of utilize that security as code to kind of automate things around.
So security as code is really just giving developers the ability to kind of automate things, as opposed to kind of being a utopian way to say “We can change everything around it”, because that’s fundamentally not the case. Because every time you change something, you need to have enough context around it to do the right kind of testing, release cycle, and post deployment as well; are you testing fully end to end?
So that’s why security as code is fairly new. You hear about it in like DevSec ops. That’s kind of more on the – you know, after you’ve written code, and it’s going through your CI/CD pipeline to get deployed. And at the same time, you can have a similar thing within your product on the backup policy. And the interesting thing there is, because fundamentally your product has usage on one side, but also there’s compliance, potentially, or regulatory frameworks you deal with, most of the times you need to collect some kind of evidence around the fact that you’re doing something correctly, or at least you’re following a process around compliance. And most of the time, this is just boring evidence that you’re collecting, like for example access control log, or the fact that you have an access control policy… But who understands it? Is it just the product people, or can your compliance manager also come in and get a full understanding of what this was about?
[41:54] So in many ways, security as code helps you make that bridge, that jump from people building it and know what it is about, know the full context, to saying from an external point of view “Can I now automate that piece and let them know that an admin is responsible for X, Y, Z, and he has evidence that they’ve followed that?” So that’s where we see the real value of security as code; you’re automating things on one side, but they’re also opening up evidence gathering on the other side, which is a crucial part of compliance.
Yeah. The first automation I ever wrote was a wiki page. I wrote documentation. Documentation was the first form of automation, because I could write it down once, and someone else would do it. And that was automation to me, because I could say “Hey, I need a new server set up this way”, I would point them to the wiki page, and they would do it. And that was the best – the very first automation. And that had all the context in there. It had the screenshots, and it had the paragraph of why we checked a box versus not checking a box… And that enabled someone else to go do the work. It was automated for me, not for them; they had to manually go through it and then be like “Oh, let’s take this wiki page and make it a Bash script.” And we can take that and we can say “Hey, we can take out all of the context, and we just trust that the steps are correct at some point.” And someone that runs the Bash script, they don’t know what’s happening; they don’t know why it’s happening, they just know something happened. And in a security as code sense, I feel like we’re removing some of that context. We’re saying “Hey, you just have to trust that we got this right in a meeting that we planned three years ago, and now you’re able to just run the script and have this interface for “Don’t worry about it.” It’s a magical Bash script that does the secure thing.”
But now my question is, who’s responsible? Because you’re saying very specifically that shifting left for security, and developers writing that security as code, versus the compliance manager. And I feel like the compliance should never be a developer’s responsibility. A developer is responsible for “I’m doing the secure thing by default, that I know with my context…” But as we say, layers, right? Like, the infrastructure layer changes, they update OIDC on the layer below you, and your thing has to change, and the person above you has to change… And so the only piece I wrote was in the middle there. And so now somewhere, someone, a compliance manager or some team needs to go through it and say “Hey, does the full path through all of those layers still meet the requirements of whatever security posture we need to have, whatever compliance checks, whatever regulations we have?” Someone else needs to be able to do that full stack, because their context isn’t “What does the application do?” Their context is “Is this path, this token, or whatever, is this part secure?” Is that right?
Yeah, absolutely. So we’ll take an example. Let’s say you’re building a high-frequency trading platform.
I would never do that. That sounds difficult. [laughter]
Let’s say you are.
Let’s say your key metric is latency. You need to reduce latency as much as you can. But there’s a flipside to when things go wrong. There’s obviously a financial aspect, because you’re losing money now, because your trading system is broken… But then there’s a lot of regulatory burden on the back of that. Not just for you as a company, but externally as well. The other SEC is gonna come knocking on your door. So now you’ve kind of moved that [unintelligible 00:45:21.22] from “A system broke and wasn’t available” to “Why did it break? What went wrong? Was there a security implication of this breakdown? Did something get stolen? Was it a deliberate attempt to kind of benefit financially for someone?” So you start to get into all of this external context that your product may not have.
[45:45] So you now suddenly see that what is traditionally an availability incident in any other b2b context is no longer that, because of the regulatory environment you’re playing in. So there the burden to do this kind of automation, plus knowing the context… You mentioned documentation… They do a lot here, because error codes get documented… You kind of have to follow that path on what actually went wrong, which system spoke to the other, and that whole traceability around “Where did the request start? Where did it break down? Which system was responsible?” And on the back of that, figure out everything else. So a combination of logs, your own metrics around it… The code as well, because at that point, you need to know the exact code that did the thing. So there, you see that the burden around security as code is very different from let’s say someone building a CRM application. That kind of context, security – I mean, anything as code will help you there. But then, of course, you’ve written more code. So what happens to that as code that you’ve written, be it infrastructure as code, security as code, or ops as code?
So in many ways, that’s why compliance exists, because they’re now trying to say “Here’s your process. When things go wrong, you’re thinking about how things go wrong, and what you’ve done in the middle, to kind of prove that you’re at least doing the right things there.” So it becomes a – it’s never a technological thing at the end of the day. It’s a mix of that with your process, your people, the regulatory burden you carry… And that also changes the pace at which you can move as well. The way to deploy a trading system is very different from a traditional SaaS application, where you can have multiple deployments a day, because you can always roll back, come back.
So in many ways, as you said, you write documentation, it gets old, then there’s a disjoint now between where code lives and what the documentation says about it. So a lot of times it is about how close you can couple them, because when one changes, you’re then automatically forced to change the other.
You’re absolutely right. Documentation is a static thing, and it doesn’t match reality. But the people we hire are the reasonability of “Oh, I see where they went with this, I see how it is now.” But my infrastructure as code does the exact same thing. My TerraForm code isn’t the exact state of reality for my infrastructure, but it’s not like – some of it can be changed, but also, I’m not holistically defining every last piece of my infrastructure, because that’s too much context that I need. And with Kubernetes we do what’s called infrastructure as software. It was a term my co-author and I coined a few years ago; it’s continual reconciliation of what does the state of the world look like, and what did we want it to define as? And as soon as one of those things get out of sync, it goes in and changes it. Git Ops is the exact same thing. Git Ops does infrastructure as software, where it says “Hey, that thing that’s applied didn’t apply back here. I either need to make a commit back to Git, or I need to change the world back to what I think it should be.” Is there a tool for infrastructure as software for security that says “Hey, we’re going to continually reconcile it to keep that state as close as possible, so when the human looks at the code in Git, it is reasonably assumed that is actually what’s in the real world”?
Good question. There’s something that exists today. I think it’s a combination of things, because there is tooling today, but it’s abstracted away. You spoke about Kubernetes. There’s a lot of tooling around Kubernetes cloud native. Then you think about cloud - there’s a bunch of things around cloud security that you can plug in. But again, each layer kind of does specialize in what it does. At the end of the day, for you externally, you’re thinking about cloud security, you’re thinking about your product security, application security across your DevOps pipeline… And then post deployment, and all of those things.
[49:54] So you have today the tooling that’s fairly fragmented, and that’s why security is quite an exciting field today. It’s not yet reached a place where – like infrastructure was a decade ago, where a lot of exciting things were happening, but you didn’t have that holistic approach to it. And the question whether you can get that is also a big one, because security is very fragmented. Your responsibility in each layer is quite different.
So you get tooling in each place, and the bigger question that we are also addressing is “Can all of that be brought under – if not one single umbrella, but at least helping join those pieces together?” …which today is done quite manually on the back of compliance. You want to get a SOC 2 or an ISO 27001 - again, compliance is not really security, but for a lot of practical purposes, they kind of blend together. One doesn’t necessarily mean the other, but it’s kind of assumed to be… That’s why the larger companies will say “Do you have this?”, in which case we check all our boxes, and then we could carry on. Because today, there is no easy way for them to actually say “Does this company have true internal security?” So they kind of rely on some external signals to say “Okay, they think about it [unintelligible 00:51:08.28] and therefore we trust them with it.”
At a very high level there isn’t one tool that kind of unites everything, but you have enough tooling in each area of security you’re dealing with, which then has to be brought together. And that’s why that’s so hard for the CSO and his team today… Because they are not building things, but they’re defining security around it, and I think that that mismatch is what is causing a lot of the context gathering that is quite manual today… Which on the back of security as code should improve as you kind of go along, because as we saw, you can kind of expose a bunch of things to the other side, saying “Okay, now automated”, you can gather X, Y, Z and feed that into your compliance process.
Also maybe with the shift left thing, I think other than trying to shift things onto a different crowd, splitting it off, saying “Oh, this is now another responsibility for you”, I think all of that is maybe just changing your mindset. So you’re kind of thinking about security - you don’t necessarily need to be a security specialist, but at least it’s part of your thought process. When you do something, you stand back for a second and you’re like “What could I potentially compromise by not doing this, or by doing this thing?” It’s like if you use an LLM and you stick your ID number or whatever in there. Now you’ve exposed this [unintelligible 00:52:37.24] Before you do that, you stop and say “If I put this in, do I even know where this goes?” If you have a key, like an API key access key, you have to think about “If I put it just in Git, and it goes to GitHub, this is not a good thing. I don’t want that.” That’s why we have tools that will say “Whoa, don’t commit this. There’s something in here that –” So they get to thinking about that. But then, I think to what you were speaking, Justin, this whole thing about syncing made me think of React and all these new things, where like –
[laughs] [unintelligible 00:53:10.17] I’m sorry.
You know, this whole idea of when something changes, everything has to be synced back, so that your UI is a reflection of your data, and all that kind of stuff. I think one of the ways that this happens is like if you think about a system where you have a single sign on process, so you’ve signed on, but now it brings in a bunch of users and things can change externally… So now how do you ensure that the central system, the SaaS system, for example, is in sync with all the things that happen outside? So I think something like Directory Sync comes into play there, where it has a responsibility of picking up on webhooks that happen, that says “This event happened. Somebody joined the company, somebody left the company. Somebody was promoted. So as you are promoted, you now have more access.” And this needs to be synced through. But then at the same time, as all of this stuff happens, there needs to be some kind of auditing, and that is where something maybe like audit logs can come in, which collect all of this action that happens, and then over time, if an incident happens, now you have a source of truth, so to speak, that you can interrogate and ask like “Okay, on this day, between these hours, what happened within my system that caused this breach to potentially happen?”
So I mean, I’ve said a whole bunch of things there, but I think the shift left – security as code is different to that, but the shift left thing is changing just your mindset. So just when you’re doing stuff on a daily basis, you kind of just stop for a moment and consider whether… Like, if I go back to frontend code, if I’m going to have a button, I should just use an HTML button, right? I shouldn’t use a [unintelligible 00:54:50.18]
Don’t overcomplicate this… [laughter]
What you’re saying is resonating with me, because I kind of feel like it’s a when versus who. And we began the conversation with shift left, and the idea was - Deepak, you mentioned it’s not after deployment, it’s predeployment. And so we began with a conversation assuming “Oh, we’re putting more burden and pressure on the developer”, which - yes, kind of… Unless you do what you just said, which is it’s more of when, versus who. It’s at development, not so much by the developer. It’s meant to happen prior to going into production, so that you’re not shipping the defaults, in the old Mongo days or other scenarios, like a modem, you’ve got insecure defaults. It’s so that you can think about those things prior to shipping, and it doesn’t mean that it has to be the developer necessarily, it’s the team behind development… Which includes developers, but also may include compliance, it may include this MVSP, which is more of a kind of policy in a way of how to secure cloud applications… And as I’m reading this, there’s like security headers in there, there’s how you allow your customers to test the system, there’s all these sort of protocols that are in there, that can be tested… But this is something that the whole team says “Okay, this is what we want to do when it comes to shipping a cloud application that’s secure.” And that happens before development, it happens during development, but prior to production. So it’s more of a when versus a who; that’s kind of how I see your perspective shifting, at least the way you described it.
Absolutely.
It seems like a lot of that really depends on that minimal piece. Like, the bare minimum you need. Because the context that I need to remember as I’m writing code is very different if it’s a trading application versus my personal website.
Way different.
Yeah. That minimal piece – and it depends on expectations, right? Like, am I going to be expected that this will never get hacked, or this is only secure against this level of sophistication, or something? And even what the user is going to expect out of it. Because just the example of users - like “I have to go change my password.” And I’ve worked in enterprises for 10 years now, and every single one of them, I go change my password and the prompt says “Please wait 24 hours for your password to replicate across the entire ecosystem of all of our apps.” And that is the expectation of a user that I had, that I’m like “Oh, I may go to a site and have to use my old password for the next day.” And that is okay that that state is not immediately synced because of the amount of pressure and scalability and all these other things we need to say “Oh, if I need that replicated everywhere right now”, guess what? That’s actually like a big task, to have everyone scale up their systems to receive all those webhooks. Versus just like a little prompt on there that says, “Hey, just wait 24 hours.” And guess what? All the scaling problem goes away, all of that expectation of “This is everywhere right now, and it’s going to be the same - not security, but the same password everywhere.” It can be just different based on whatever minimal means in your context, at your application level, and whoever’s writing that code. Because I can solve those problems at different layers.
Well, this is a minimum viable secure product, so in this case the word minimum is still used there. But it’s five scrolls for me, three or five scrolls… So it’s a lot. So this is minimum in comparison to a cloud application…
Is that metric system scrolls, or is that standard system scrolls? Because I don’t know if this is a –
It’s my digits scroll… But that’s a cloud application, so I would say a minimum version of this for a personal website would be like a one-liner. Like “Don’t get hacked.” Or I hope it doesn’t get hacked, because it’s personal. Not that big of a deal, unless like your email’s attached to it, and you’ve got other things happening as a result… But anyways, I digress.
[01:00:14.10] Yeah, we’ve been pretty bad at this as a software industry. We’ll store passwords as plain text… I mean, it still happens. It’s crazy. But there’s legacy systems that can’t change… And password complexity causes more pain than actually securing it, because from a usability perspective [unintelligible 01:00:33.15] and then they just take the easier path out. GDPR, for example, the reason it had to come in was – it’s just common sense; secure the data that you collect, and don’t use it in bad ways.
You would think that’s common sense, but I don’t know… [laughs]
But that’s the reason, right? The reason that came about is because companies were misusing it. They were not doing the right thing. They were just throwing it out everywhere… Could they even use it the way they used it? I mean, we don’t even know today where our data is sent and how it’s being used. So GDPR was kind of that attempt to say “Okay, now just hold on; think about it - you collected a lot of data, but…” And it’s not even just about sensitive data. If you hold 6,000 email addresses, that’s a big problem, because that leak is effectively – and it’s also the quantity you’re collecting, not just specific things.
So I think that’s why all of these compliance regulations exist, because we just don’t do it the right way, until we are compelled by an external event to kind of make that happen. So security kind of has to be forced upon; otherwise you kind of happily say “If I don’t have to do it, I won’t” or “I’ll do it later”, and that data kind of comes probably never. In many ways, from an external point of view, it feels like it’s slowing things down, but it’s more about saying “Think about it.” Because if we don’t force you to think about it, you’re never going to think about it. So I think that’s where we’ve been pretty bad, and that kind of mindset that we were speaking about has to kind of come back… Because we have a lot of responsibility as data collectors. That’s the crown jewel that we’re collecting, and we’re gonna make it harder for anyone to steal that away.
And that’s changed a lot, right? Like you mentioned in the beginning, from a data center, where if I was collecting email addresses, I could go point to the hard drive that had the email addresses on them. That’s where it is. It’s not leaving here, but now I can’t do that, right? I have key-value stores in Cloudflare. I’m like “Where does that exist?” I don’t know… What is my reasonable amount of expectation to secure that data? It’s like “Well I have an access key, and that’s how I get to it, and how I can write to it, so no one else should be able to read it…” But I have to trust that my provider is doing the right thing beyond the layers that I have control over. I have the access key, and they do everything else. But the dynamic is very different from a globally distributed key-value store, versus a hard drive in my closet. Those are very different things around how I think about security, and what my responsibilities are for that security. So I guess it really depends on, again, those layers; I have a key to the door to the data center, and no one else has card access to it. Versus I have a key to the API to give me access to the data, versus – and those are just bits. If someone stole those bits, I would never actually know. But if I lose my keys, I know immediately; my physical keys [unintelligible 01:03:27.22] It’s very different how we go through those mindsets.
It’s similar too also to the build versus buy aspect, right? Because in the aspect of the data center, and pointing to the hard drive, you’ve likely built the model. You’ve built the servers, you’ve racked and stacked, you’ve provisioned, all that good stuff… Whereas from the API perspective, you have the key, and what’s behind the API is a black box, and you’re focusing on trust. So you’ve in that scenario chosen the buy versus the build. And I think security is one of those things where - wow, you really have to have a lot of awareness and knowledge to do the build part. And not a ton aside from judgment, and maybe peer pressure to do the buy part. To buy a decent solution is like “Okay, what does Google tell me when I say “WorkOS versus”? Is BoxyHQ in that list?”
[01:04:18.29] I know Michael Grinich from WorkOS, big fan of them, and y’all are an open source solution to a similar problem, where they’re proprietary. And then it’s like “Well, do I trust open source more? Do I trust a proprietary less? Do I build? Do I buy? Do I want to even learn SAML? Do I wanna learn Scheme? Do I wanna learn all these protocols? Do I want my team focusing on that versus the product we should be building?” And that’s where you really sort of almost are forced by market nature to buy, versus build. Even if you’re not that knowledgeable to even know what to buy, you have to buy the most trusted default or trusted solution that comes up. And yes, I still use Google. ChatGPT isn’t for everything. So when I am learning something, like “Hey, what’s well-known-brand versus?” And I usually choose from that, because that’s what the wisdom of the crowd has said. But let’s open up that, the buy versus build, because I feel like there’s two different paradigms there. You could be knowledgeable enough to build, but does the market allow you to even do that? Maybe you have to be at a certain scale to say “We should build this, versus buy this is.”
The first stepping stone there was really infrastructure. The shift from data center to clicking a button on AWS…
Right.
…if you didn’t do that - let’s say this was like the early 2000s, when AWS came out. If you chose to go to a data center, that means you were like at least a week or two behind, right? …just at the get go, from your competitor who chose to build on AWS. At the same time, as you kind of had to maintain it ,you will probably lose more time as you kind of evolve. So it kind of just immediately becomes a competitive advantage of speed. And we know speed for a product is super-important, so you don’t get left behind. If you’re not at the right place, at the right time, with the right product that you have, that fundamental – like, Stripe changed that fundamentally as well. We were building payment systems which took forever, and didn’t quite work as well… To now just integrating Stripe and carrying on. So what is code to you as a business, versus everything else? Twilio is a great example as well. You wouldn’t build audio/video today.
Not if you’re smart.
Just pick up Twilio. Exactly. So I think that is then that whole question of “What am I building upon?” I know there’s other questions that come up; you’re dependent on a platform, and all that… But as you start off, you’re kind of then saying “If I had to build everything, like Twilio, and then do my value-add on top, it’s gonna take me forever.” So that has changed a lot now. You can roll things out over a weekend with almost no cost. That I think has become a competitive advantage question now. What do you build? What is most essential to your core? [unintelligible 01:07:10.12] that’s undifferentiated and commoditized? So that, I think, is that buy versus build question that comes up. That’s why SaaS has kind of taken off in the last decade. I think the average company uses about 80 SaaS vendors on average, which is quite a lot.
That’s a lot. Yeah.
We see that as well. We’re such a small company; we have about 30 I can think that we use internally.
How is that different from what it was in the ’90s? Because I would go buy box software. And that was the ‘90s version of a SaaS, was like I went to Circuit City or whatever and I bought a box off the shelf, and that was the software that I was buying, and I would run that forever, as like “This is the software I didn’t have to write.” And I feel like the SaaS term is thrown around like it’s something new, but really it’s been around for a long time, that model of “I don’t have to build this part of it. I don’t need to build my own Excel. I just go buy Microsoft Office in a box. And I paid for it once.”
[01:08:11.00] And I know plenty of people that used Office 2003 for two decades. And it was the thing that ran for them. Versus a SaaS product, where they have to keep updating, and retraining, and doing things… Because there is an amount of “I can buy this once and own the software, and move faster long-term, because I don’t have to retrain people off of –” I remember when Office 2007 came out with the ribbon. It was a huge thing. I was working at a Help Desk, and I wrote I don’t know how many pages of documentation about “Hey, guess what? The ribbon is here. And you’re going to have to change how you use Microsoft Word, because everything is different now.” And nobody wanted to move to it. So it’s like, that slowed us down as a company, because we had to stay with what the SaaS was providing. We had to upgrade, or whatever.
And I feel like – what you mentioned before, like if you go to a data center versus AWS, you’re gonna be a week behind… Except for the fact that like if I go to AWS, I have to learn an organization, and IM, and security groups, and I have so many more layers to learn, versus I plugged in a machine and installed Linux. And when I was working in Disney Animation, we had a render farm, and we could rebuild the entire render farm, like 4,000 or 5,000 machines, in a day. And I went to Disney Plus, and we were in AWS, and if I wanted to replace 4,000 or 5,000 machines, that was like six days. And the difference between like how fast – like, to reimage and to refresh the system, there was a speed difference there. And owning the hardware and owning software is a very different mindset than what a SaaS… And someone else can say what you can or cannot do, and how fast you can do it. And how does that apply to security? Because I see on your site you have a self-hosted option. I can go download the software, it’s open source, I can run it in my data center, on my machines, and I can run it for - I don’t know, even if it’s not supported by you, I can run it forever, as long as the hardware is there… Versus the SaaS, I can get started now, and see how this works, and you’re going to take care of the layer below me. Maybe I don’t need that. Maybe the actual long-term effects of running this myself, and actually just having a little bit of expertise on how that runs in troubleshooting it is valuable to my company, to be able to say “Oh, I don’t need to open a support ticket to fix this. I can go find the open source and have my developers do it right now.”
Yeah, you bring up a very interesting point. I think, to your point, one thing we always overlook in the build versus buy is “Can you hire the right set of people to do that?” Internally. So your choice of, for example, today React versus something else. Or even like Golang versus Java, right? It’s fundamentally about who can you bring on board to kind of support it; even if you build something, it needs to be supported in the long run.
So I think a lot of the build versus buy choices also happen – it may not be the best one that you pick up, but it could be the one that has the most talent pool out there. Or you think that you will reach a point where this thing is gonna get big. A lot of people bet on React when they saw who was behind it, saying that “This will become big, so we will have enough people who picked up the skills [unintelligible 01:11:16.16]
So I think, to your point about that on-prem versus SaaS - we cannot talk about Salesforce as an example, right? They were the first ones to kind of – well, they did a great thing with that whole “No software in the cloud, you don’t have to worry about it.” And today, if you think about it, it helps you move fast, right? You’ve got not just a CRM, but a whole platform around. You can now do pricing quotes on the back of that, you can align your account management team around that, customer support can happen there… So you’ve suddenly gone from just a CRM to a whole range of things within that same platform, so to speak.
Do you find that distracting for companies?
[01:12:01.17] I mean, most companies have to do it. So I’ve seen this evolve. Your [unintelligible 01:12:04.21] you hired a sales team, now your Salesforce is where they live and breathe… Nobody likes Salesforce, but everybody has to be there from a sales, marketing and customer support.
Can I plug something real quick? Because I do not like Salesforce, and I do our sales for us… And I started to use a thing called Pipedrive.
Pipedrive is awesome. Yeah, exactly.
I love it. I’m just so in love with this product. Like, I just want to mention it right now, because they’re not even paying me to do it. It’s phenomenal. And I could use it as a one-person sales show. Now, if I had to scale and hire somebody next year to help you with sales - which we’ve considered - I would bring them into Pipedrive, and I could take a step back, but still be involved. Anyways, I’m not gonna plug them too far. They’re just amazing. And Salesforce has never – it’s just been too big. Just too big, you know? Anyways.
And that’s the distraction I’m talking about. At what point do you need Salesforce, all of the features? Because I can’t tell you how many people for the last four years asked me “How do I learn AWS?” I’m like “There are over 200 services. You do not learn AWS. You learn one piece that you want to do, and everything else is a distraction.”
Right.
And I feel like Salesforce is the same thing, where it’s like “Hey, you wanted to do CRM, and that’s it? Go to Pipedrive.” Don’t do all of the things that Salesforce does. For a large enterprise, for someone that knows they need all the feature sets, that makes sense. But for the smaller people, for the people getting started, for the people who are like “Hey, I want to solve this problem…”
Or indies, man. Yeah.
Yeah. Maybe Excel is too difficult. If I go look at Excel and I’m like “Actually, there’s too many functions here. I don’t need all of them”, I can hide most of them. But at some point, I just need like maybe a calculator. Because actually, what I needed – I didn’t need Excel to do that thing.
Yeah.
And I think that’s why we’re so focused in. We’re not trying to do everything. Explicitly, we’re not trying to do everything. That’s why we have these couple of verticals that we’re focusing on. Enterprises are so. We’re not trying to solve the entire authentication authorization problem. We’re trying to solve a very specific area in that. And I think that the reason, Adam, why you previously said [unintelligible 01:14:09.29] I think the reason why that is so long is because that is for when you want to sell to enterprise. Because now you get all kinds of –
Right. And I expect it to be long, too. That was not a knock – I expect it to be. I mean, if you’re going to secure a cloud application, it should be pretty holistic. So the many scrolls was expected.
Yeah, yeah. But I think one of the reasons why it is as complex and as vast as it is, is because who you’re building this cloud application for is most likely a company that has all kinds of compliance rules around all of this… And to even satisfy the minimum is a big list.
Exactly. And it’s a trade-off. Like you mentioned, Salesforce vs. Pipedrive. You can move fast with Pipedrive, but there might come a time when you need something that it doesn’t have. Because there’s integrations and things like that, and it’s possible, but it kind of becomes a trade-off and then you say “Okay, now maybe I have to migrate to Salesforce.” I’ve seen this happen as well. As sales grew, they kind of were almost forced back into going to Salesforce. Or you’re doing it in an Excel sheet for a while, and that’s fine, and then you’ve got to move somewhere.
So we see this in security as well. It’s a trade-off. We turn away a lot of prospects because we tell them that we can’t do everything. And if you’re thinking about doing everything, we’re not the right product at this stage. And that’s a trade-off as well in the build versus buy. It’s “Okay, do I pick this up from BoxyHQ? Do I go and build it? Or do I find a vendor who does everything?” So these are all the questions that constantly come up.
We obviously will evolve over time as well. As I mentioned, you go from single product, to multi-product, to kind of a platform… So that’s kind of the bigger vision around solving this holistically. But that’s kind of a 10-year timeframe, not two, three years. And what Salesforce did to the on-prem CRM, Pipedrive is kind of doing it to them, right? It’s the evolution. And there’ll be a next generation…
[01:16:11.17] You might think so, that’s for sure. I know I’ve ran CRMs on a desktop, in a networked environment, when I had no idea what a network was… And I was just – just the worst. Anyways… And running local software… So thank God that Salesforce came along and used the cloud to make the software run in the cloud, networked, versus on-prem, the LAN network. It revolutionized things. But I agree with Justin, it just does so much. You don’t learn Salesforce, you learn “Can I just have the CRM?” And it’s just too complicated, because it thinks you want everything, and they have this massive scale enterprise large sales team mindset, which I suppose is more of a juxtaposition of how we think about security. Is it developers only? Or does that shift-left include other folks who can help developers in the development process to get to where they need to go?
The one thing I think that is really interesting, at least with y’all’s choice, is the open source nature. You can go and self-host BoxyHQ yourself, you can do I think a premium version of it too, which - I’m not sure how you licensed that… But there’s two different flavors, so you can self-host, and I’m assuming both built on open source… But then you can also go and do the SaaS version of it. But I think what I’m trying to promote more there is less about your tears and how you’re selling it, but more so that there needs to be back to that default, that secure by default scenario where “Can I choose a viable open source version that is free to me?” Free to contribute, free to use, free as in everybody else can use it as well… And it becomes this model that seems to be a version of yes for you all, but at the same time, does that then make it one single point of failure? Like, if everybody starts going the BoxyHQ way to do SSO and secure their SaaS products for enterprise, does that mean that you become more of a centerpiece for an attack? Maybe yes. That comes back to the build versus buy, make the right choice… And like you said before, Deepak, it’s just constantly iterative when it comes to security. Iteration, always improving, being aware, and fine-tuning and making changes as that changes. But I’m also very happy that you’ve chosen open source. Can you talk about the choice to – I mean, because you can kind of cannibalize yourself, by yourself, by being open source.
Absolutely.
Right? But how did you choose that? Why was open source super-important to you all?
Yeah, so there’s two fundamental points for us when we were thinking about go to market. We are serving the developers. And as a developer myself, and having seen my team multiple times before, it’s the best way to go and evaluate something. You’re either thinking about “Do I build this? Or do I pick up something that gives me some kind of foundation to build upon?” But then you can also evaluate it without having to talk to anyone, no sales, marketing language… You can truly see it functional, you can see the value… And that’s how you – it’s like the freemium version for SaaS, effectively. Some [unintelligible 01:19:11.22] we don’t even know who they are, they’re trialing it, and then they might come back around for more questions, or they might stumble into an issue and they’re on our community.
So that was kind of one major aspect, to think about how we’re discovered. But at the same time, we also realized that we’re quite close to infrastructure, and commoditized. We’re building on top of protocols. We’re not trying to reinvent the thing… Fairly interoperable protocols; we wanted to keep it that way. And that’s the best way to kind of bring in contributions from the community. Integrations [unintelligible 01:19:46.22] providers. Some providers we can’t get access to. We have a lot of help from the community who are using that, and are trying us out, and ran into trouble, and could help us troubleshoot, and submit changes to make it work.
[01:20:03.06] Those two were kind of the main reasons why we chose to go – and our license is Apache 2, so no restrictions. Fully free to use it. And what we have done is in an open core model there are certain premium features we identified, that only the larger companies need, and those are kind of locked up as requiring an enterprise edition. But that doesn’t prevent anyone else from not being able to use the full functionality of everything else.
So that was our thinking here, post infrastructure, how developers discover and evaluate us. But in the process, what we really found insightful was we get used in the tech stack in ways that we didn’t anticipate before. Because they can lift and shift us, right? So they’ve used us in places that we did not even think was the use case before. So that was quite insightful, to see the new use cases kind of emerging.
How do you deal with that case where someone uses you in a way you didn’t think, but then maybe they want to contribute? What would you consider an enterprise feature? How do you deal with open, but open to contributions?
We obviously tie that in with our roadmap that we think about, at a very high level, on where we want to kind of evolve as a company, both from a product and kind of who we serve perspective. And all the contributions, ideas coming in is kind of evaluated and prioritized based on that. Because there’s many directions we can go. Like Schalk mentioned, we could do all of authentication, but that’s a very different problem to solve, versus going deeper into developer security. That’s the path we’ve picked. So we’re quite strong about saying no to things that we think is not kind of core to what we’re doing.
But at the same time, we have the APIs to extend it and use it for [unintelligible 01:21:51.21] And where we see repeated use cases, we kind of then go out and see a – there’s a lot of verticals we don’t have experience in, but it’s kind of [unintelligible 01:22:01.14] And then we try and sit down and see if that’s a vertical we could just go after from a pure go to market perspective.
Very cool.
I was just clicking around on the open source, because I love seeing commits. When something is open source, I love just digging into GitHub and looking at the commits…
Right. I didn’t do that. I didn’t dig into the actual source. I looked at it, but I didn’t dig in.
Deepak, I wanted to commend you. This is actually amazing, because you have – not the most commits. Dependabot obviously has the most commits, because Dependabot is a 100x developer out there. But Deepak, of your commits, you have 516,000 lines added, lines of code. But even better than that, you have 470 removed. And that’s amazing. That is a great stat right there. You were only 46,000 away from like breaking even on like – you could just have zero code left… [laughter]
I mean, obviously, it’s a bit skewed by the package lock-in stuff, but I love removing code. Sometimes the metric is more [unintelligible 01:23:00.29] it’s better.
I always strive to be neutral in my code. Like, by the time I leave a company, I want to have as many lines written as removed. That is how I know that I made an impact of getting rid of stuff, and adding – like, whatever it is, but break those even, and get that in a neutral spot is a goal of mine.
Interesting.
There’s a funny anecdote – I mean, I won’t name the company I was at, but wherever I was, they came up with this crazy metric to just reward people… And more recognize, not even monetary reward. And they said “The developer with the most number of lines of code to commit” - and this was back in the days of CVS, before Git… And people game the hell out of it. You know, change the whole formatting to leave like two lines between every line of code… And then it became a game of who can game it better to get to the top. [laughter]
Semicolons go on their own line… And then we have two lines at the end of the file… Yeah, you can totally – if this isn’t my promo doc, I am getting more lines, that’s for sure.
Well, let’s leave it there then. I think we covered most of it. I definitely love hearing about the journey, and the security journey, I suppose, for developers, now that I think about it… I mean, I think one takeaway for me was really just the shift left of my shift left, which is - that is when, not who. I think that was really an eye opener. Because every time I hear the term shift left, I always think, in almost a defensive way, like “Gosh, can developers really just do more? Can we just keep piling on more?” But I think now that I think about it, like Schalk said, just from a different perspective, that it’s really about when - which was my words, but really synthesized from his meaning - was when versus who. It’s predeployment that it’s everyone’s job to think about it. But when we’re actually building the application, not once it’s built and shipped do we then begin to think about security. Because we’re gonna build better companies, we’re gonna build better products, we’re gonna build better teams… Which might even mean keeping people around longer, paying them more, profiting more… All the mores that you can put into that stack seem to be better if you can shift that left to the when versus just the who. So that’s the takeaway for me.
That’s super-insightful. I didn’t think about it that way at all.
Well, that’s what we had you on here, so we can give you some feedback. That usually happens, to some degree… Alright, well let’s leave it there. Thanks for coming on. Thanks for sharing this security journey, and just thanks for joining me, man. It’s been fun gabbing with you a bit, man. It’s our first time together. Maybe we’ll do it again some time. Who knows?
Yes, this was great.
Awesome. Yeah, I mean, honored to be here. It was an absolute pleasure.
Very cool. Thank you.
Thank you so much.
Thanks so much.
Our transcripts are open source on GitHub. Improvements are welcome. 💚