Critical Point War Stories with Kevin Riggle

(Melanie) I think something that we really benefited from the most at Uber was the fact that our core incident response team engaged every single day.

It -- one of the challenges that I see with companies who come to us for incident response, when they haven't done adequate preparation is that, you know, individual teams have their own IR plan, they don't work together, and they don't use it on a regular basis.

And so they're constantly, like, brushing the dust off [laughs] and I wanna have incident response plans to mirror every day operations as much as possible so that people can do it in their sleep. I don't want it to be this, like, strange, crazy thing that they're trying to figure out under pressure.

(Kevin) Because I got into security by running the incident response program, or helping to run the incident response program at Akamai, I have a very weird perspective on this, 'cause Akamai's incident response program for everything --

(Melanie) A resilient organization should be able to eb and flow, and like, put together the right people at the right time depending on what is the urgent thing that's happening at the moment.

And if you think that your company is not having an incident everyday, I -- I'm worried for you.

(Kevin) Howdy, folks.

We're back again on the War Stories Podcast, talking about another incident.

We are here today with Melanie Ensign, a friend of mine from Boston days, but also formerly of Uber, who has a story to tell us about a time that -- this time someone else broke production.

We're getting our first security incident on the podcast, so I'm super excited to chat with her.

We are --

(Melanie) But it's not the Uber incident that everyone is expecting, I'm sure.

(Kevin) Okay. Great. Yes. Yes. I don't want to say that there are a few that I can think of which might qualify for that --

(Melanie) It's not any of those. It's not even any of those --

(Kevin) Yes. Yes.

Uber had a bad run for a while, and we'll bring people something they haven't heard about. And first, we'll roll the titles.

[smooth intro music plays]

And we're back!

Once again, the War Stories Podcast on Critical Point, here with Melanie Ensign. I'm Kevin Riggle.

Uh, Melanie! Uh, can you tell us a little bit about yourself and how you found yourself in a place, not to break production, but to help some folks whose production had been broken?

(Melanie) Sure. So, first of all, thank you so much for having me today. It's real exciting to -- to be talking to you again after -- I think we've both been so busy, I think, running our own businesses the last couple of years.

Um, I am the founder and CEO of a company called Discernable.

We are a dedicated and specialized communications firm, specifically for security and privacy organizations. So our clients are chief security officers, chief privacy officers, and their teams.

And our mission is to help them become more effective communicators in order to earn more influence across the business so that they are not treated as the clean-up crew to everyone else's decisions.

(Kevin) So when production breaks, when there is a security incident, you should call Melanie -- or ideally, you should call Melanie before there is an -- production security incident and -- I've just been really enjoying your newsletter.

Like, I've been getting a ton of value out of that, yeah, as I try to figure out how to communicate about my work.

And --

(Melanie) It turns out that communications is, in fact, a century's old scholarship of its own, not something that engineers invented. [laughs]

(Kevin) Also, not something most of us got taught in school, like... I had a, like, half semester communications course, and it was incredibly valuable, that, like, people were kind of, complained about it, but it still wasn't enough. I need more. [laughs]

(Melanie) Yeah. While they -- while engineers were learning computers and technical things, like, we were studying people and how they communicate.

We -- I've even done a good deal of studying engineers, specifically, and how they communicate. Right?

(Kevin) Yes. Which is a little different than anyone else. Yeah. Oh, that would be a -- I would really love to read about that.

(Melanie) [laughs]

The -- the special snowflake communicator. It's -- I mean -- I just -- because of course, there -- I -- don't want to imply that this is a monoculture, you know.

Not everybody's the same, but there are a lot of shared, I think, challenges because it's not a skill that is inherently and proactively developed with technical teams, right? That there seems to be an incorrect assumption this is just a skill you develop

5:00
via osmosis, because you showed up to a team, and so you should just magically learn how to communicate effectively with all different types of stakeholders, right?

And we have found that even engineers that are good at communicating to say, perhaps, their board or business leaders, it doesn't necessarily mean that they're good at communicating with other engineers.

So when you start working on things like shared KPIs and road maps, you know, we're constantly working with security teams to help them negotiate with infrastructure teams and product teams, and you know, SRE teams.

How can we get security outcomes through the planning process and not just checking it off at the end?

(Kevin) Oh, nice. So you're helping teams communicate... Like, technical teams communicate internally and with other technical teams, as well as outwardly focused? I think I have had you as being primarily outwardly focused.

(Melanie) Most of our work is internal because that's where the work gets done. Um, we -- the external work is kind of a reflection of what's been done internally right? I don't have an external story to tell if we haven't done the work internally first.

(Kevin) And the external story will reflect a lot of whatever is going on internally, like, for better or for worse.

I -- there are some incidents that I -- not my story to tell, but that I think spring vividly to mind, where, yeah, the ways that we exposed or didn't expose the internal implementation details matter a lot to how the incident got resolved.

(Melanie) Yeah, exactly. And even when it comes to incident response a lot of teams individually, different departments in different functions have their own plan for how they deal with their special flavor of incidents and what they consider to be an incident, right?

Um, but prior to an incident, very few companies are thinking about how do all of those plans work together, [laughs] and how do we make sure that those plans are as close as possible to our everyday work so that we are practicing them over and over and over again.

We see this a lot, particularly with security incidents where, you know, cross-functional partners who don't work on security every day... If we are giving them a brand-new playbook when there's a breach, I can promise you, they're not going to execute that well because you know, they're rusty every single time they have to use it, right?

So a security incident playbook needs to mirror what people would normally do for other types of incidents, right? It needs to make the decision-making a little bit more automatic, and taking that responsibility off the engineer.

It shouldn't be up to engineers to decide whether or not legal gets looped in. Legal needs to make that decision. Right? Which means we need to have ways to surface the information to legal at the start of the incident so that, you know, these non-technical worlds are not playing catch-up every time they get looped into an incident.

(Kevin) So, people should come talk to you about communication internally and externally around security and privacy because... Well, and then that is, like, such an overwhelming amount of the work, like, would be implemented.

GDPR at Stripe and then CCPA at Lyft. Like, every engineering team at the company had to do at least a sprint's worth of work, so we were talking to all of them. That was -- you know, we spent 15 months. I talked with, you know, practically everybody at the company.

Still, people will come up to me, be like, oh, yeah, we worked together on CCPA at Lyft. I'm like, I'm really sorry I don't remember you. Because, like, I talked with literally everyone.

You were lovely. I'm really sorry for making you do all this work. I'm glad that we completed it successfully, but yeah...

(Melanie) Yeah. A lot of engagements, and you know, one of the things that we encounter a lot is when we first start working with a client on incident preparedness, and I'd say, if this incident or type of incident were to happen tomorrow, what would you like to be able to say to all of your stakeholders? And then we'd spend the next X number of months or weeks, depending on the scope, to make that true before you need to say it, right?

Um, and so there's, you know, a lot of, I think, just, like, very basic preparedness where you're not gonna be able to say something unless you make it true first, right? That is shockingly not a common train of thought. [laughs]

Um, and, you know, the other thing is that they're not thinking about, kind of, um, what I -- what I call the hiccups, the fuck-ups, and the give-ups of incidents, meaning that it's not just breaches. It's not just, like, a technical compromise that is going to be

10:00
an incident.

You know, something like when people discovered that Facebook was using phone numbers, not for -- just for the 2FA feature that we all expected, but also for targeted advertising.

Like, that was a massive incident for the company in terms of public perception, regulatory investigations. It's -- it's a huge distraction for a lot of teams at an organization when you have something like this, right?

So I've talked about this before. I've written blog posts about it, that you have to really expand your definition of what's an incident because it's not just an outage, it's not just a bug, and in fact, the incident that I'm gonna tell you about today, there was no actual technical compromise. There was no vulnerability. Um, but it created an incident for us on Christmas, of all days.

Um, but I do have -- I mean, I do have to say that, you know, one of the things that Uber was really good at while I was there was we had our core incident response team engaged on a daily basis because we worked together on everything from bug bounty reports to product security to actual, like, breaches or regulatory investigations.

Like, all of these things, the same core team, from legal, comms, security, public policy, like, we were lockstep. So we had our process very well practiced. And we knew all the people, our playbooks, you know, worked well together, we had good internal communications on most of these things, and you know, so when this thing popped up on Christmas, in a matter of hours, me and two engineers had put it to bed, right?

Because it was, like -- it was more of a public perception problem, and a rumored incident rather than an actual compromise of our infrastructure or corporate network. Um, it, nonetheless, required our core incident response team to come together very quickly on a holiday to make sure that what I was telling to the journalists was accurate, factual, and, you know, not going to cause additional issues for us.

And I will just -- I will tease the ending by saying... [laughs] The reason it ended as well as it did, was because of that process we had, where, you know, this all starts with a bug bounty report, and I knew about the bug bounty report from day one. Right?

So, as the communications advisor, I knew the whole story, I had all of the context. When I'm contacted by a member of the press, I know exactly what's happening.

(Kevin) So that's -- so -- we've gone -- these always -- every time I get to the edit, I'm like, this has already gone six different ways, and there's so many different, cool threads to pull on.

So that's a little bit about what Melanie is doing right now at Discernable, is helping teams build a process so that on Christmas morning, when, you know, somebody raises the flag and is like, we've gotta deal with this, you've got a process which, within a couple of hours, can get everybody on the same page, going in the same direction, knowing what the next steps are, and put something to bed so you can go do Christmas things, if your family -- the --

(Melanie) Or not do Christmas things, if that's how you choose to spend the day. [laughs]

(Kevin) Right, yeah. If you don't do Christmas, then do something else. Yes. Uh...

But literally anything but deal with an incident on Christmas. Um... 'Cause there are only some of us who would consider that a lovely present to find wrapped under the tree.

All right. So with this as context. You're at Uber. It is Christmas Day, 2017. What was the first thing that you noticed?

(Melanie) Sure. So on that particular day, I was contacted by the weekend editor at a tech publication.

I'm not going to name them because I'm not trying to embarrass anybody with this story.

(Kevin) And what's a weekend editor?

(Melanie) So, a weekend editor -- sure, that's a great question.

A weekend editor is essentially somebody who -- they cover the weekend, so they're filling in for other members of staff who work Monday through Friday, you know, unless there's, like, breaking news, right? This wasn't a breach or something like that, so it didn't warrant bringing in, like, the full-time security reporter to cover it.

But because it was a holiday, most of the full-time staff for that publication were on PTO. So this is essentially somebody who covers weekends, covers holidays, things like that.

(Kevin) And so they kind of wear all of hats, they cover all of the bases, because -- for the publication.

(Melanie) Yeah. I mean, you could -- I think you can kind of think of it as, like, on-call when other people are on PTO, or you know, on leave or something like that

(Kevin) For a news organization. Okay. That's great. I love that.

(Melanie) [laughs] So this was a person who, despite working at a, you know, a tech publication, their typical beat was not cybersecurity, right, so I don't fault them for not understanding the technical details of -- of what was in this particular bug bounty report, which was the topic of discussion.

(Kevin) Okay. And so what had they seen?

(Melanie) So what had happened -- right, so what had happened was, there was a security researcher who had submitted about a half a dozen reports to Uber's bug bounty program. They had been labeled duplicates by the team and closed without payment.

And so this particular individual understandably was frustrated and upset with how much time they had spent submitting these reports and to get no financial payout for them.

(Kevin) Because they believed this is a real issue.

(Melanie) That is correct.

(Kevin) Not just, like, not outside the scope of Uber's bug bounty program, but they're, like, there's a real thing here that you all aren't seeing.

(Melanie) Yes, yes.

And as a result of that, they then decided they were going to escalate and publish their own blog post about how they had been screwed by Uber's bug bounty program. And kind of, you know, denied what they felt they deserved. Which is not an uncommon situation for bug bounty programs.

(Kevin) And this kinda thing gets picked up by, like, Hacker News or...

(Melanie) I mean, it definitely was because it was a slow news day. I mean, it was Christmas, [laughs] right? So it's -- it was a story because it was a slow news day. And it became an incident for us because it was happening on a holiday when a lot of people were not working, and certainly, when I did not really want to be on the phone with reporters, right?

But it's the nature of -- of the job, right?

So... What happened was -- so this... The weekend editor reached out to me for a comment on, you know, this researcher's blog post. And because of the way that the -- Uber's bug bounty program manager ran the program -- and I will say, she is one of the smartest, most astute peons I have ever met in bug bounty.

(Kevin) Nice. Which is a hard field.

(Melanie) Yes.

Because of the way that she ran the program, I already knew what was happening, before this editor reached out to me.

So, the moment he reached out to me, I not only gave him a statement, I gave him links to a whole bunch of comments on social media posts that had been shared by other members of the community when they saw this individual's blog post.

And these comments validated Uber's position, validated the position of the bug bounty team members, and called out specific and abusive behavior that had been demonstrated by the researcher.

You know, again, we don't always know who's on the other end of these reports, right? I try really hard not to make assumptions about what's going on in other people's lives that, perhaps, is motivating some of this behavior, right?

Um, that being said, when we work with the bug bounty teams -- and Uber's team was really good at this at the time -- you know, we're focused on what is the long-term, desired outcome of each of these relationships, right, whether it's a researcher who you have had a long history with, who, perhaps, consistently submits valuable reports to your team, or in this case, is a first-time reporter to your program.

And -- and even if you disagree with their technical analysis, being professional, being respectful, is still paramount in the way that you speak to them, right?

And because of that, I mean, it -- it's interesting, and somewhat amusing to me, in hindsight, that the security program at Uber, in this moment, was --

20:00
it was -- benefiting from a more positive public perception than Uber's parent brand, at the moment.

(Kevin) Right.

Which is not a thing the security program is usually doing, and often, we try not to get noticed, and this was, like, was what this... Delete Uber, was that 2016, or...? Was that?

(Melanie) So, it was the be -- that was the beginning of 2017. And so this is almost 12 months later. This was December of 2017.

(Kevin) 'Cause that was when Susan Joy Fowler had published her post about, like, sexual harassment at Uber, and there had been, yeah, a lot of negative press.

(Melanie) 2017 was a difficult year for Uber.

(Kevin) So that maybe, also, piqued the weekend editor's attention. They're, like, we know that, you know, people will click on a news article about bad things happening at Uber because that's just been part of the narrative for, you know, a year at this point, then yeah.

(Melanie) Yeah. I was playing into people's existing -- I don't wanna call it bias because I think some of it was well-deserved -- but their preexisting opinion, right?

Um, and so -- yeah, it was -- and so it was interesting when I responded to this editor and was able to say exactly, you know, I knew exactly what he was talking about so we could respond right away, right?

It didn't take hours for me to track down people with knowledge of this of this situation. I knew exactly what was happening because I had sat next to the bug bounty team as they responded to this researcher's correspondence because, again, I -- the PM on this team was so smart, and she immediately recognized when emotions were escalating, right, and when things started to kind of go off the rails.

And, you know, we came from a position of disclosure by default, at least in the way that we communicated with people. Because we did not know if we would always have control over what got disclosed publicly or not. Right?

And so we had to hold ourselves to a higher standard and take the high road, no matter what. That was just how we operated.

Um, and so, you know, I had been looped in with the bug bounty team from the beginning on this particular report, and as a result of the way that the program manager responded and handled this situation, you know, members of the security community who otherwise did not have a favorable perception of Uber as a company, were publicly coming to defend -- the defense of this PM and the bug bounty program, to say not only is Uber correct in a technical assessment, but they behaved professionally. They were actually the adult in the room here.

And this other individual, like, you know, exhibited some very problematic behavior in some of the, you know, quite frankly, sexist language that he was using with the program manager.

And, you know, it's one of those things where, you know, individual engineers aren't necessarily always thinking that every email or every bug bounty correspondence is going to end up on the Internet, but the reality is is you don't know.

And the interesting thing is that, through all of my time with this bug bounty team -- I worked at Uber for 4 years -- one of the members of their team, because we were constantly going through this process of focusing on your long-term outcomes, right? We're not trying to win an argument, we're trying to, you know, get to a specific result so that we can -- we're in a better position tomorrow.

This particular individual said -- told me that the communications skills that they had learned working with me improved their communication in their marriage. Because all of a sudden, they stopped trying to win arguments.

They were like, the most important thing to me is the longevity of the relationship. And it wasn't just about, you know, like, self-censoring, or not mentioning those problems, but about making sure that, above all else, your partner understood that the relationship was more important to you than the argument.

(Kevin) Yeah. Ooh, okay. Yeah. That's -- I mean, I have this problem as an engineer more than I -- you know, more than I like, and sometimes it really is important that there are specific technical details that really need to be right, but that -- I -- that that framing -- that -- that framing is actually kind of blowing my mind a little bit, or it's never been you know, put so crisply to me.

(Melanie) And to be honest, sometimes you need to reassure the other person that the relationship

25:00
is the priority in order for them to accept when you are right.

So, you know, again, it's about being honest with yourself, and being able to articulate what is it that I actually want this relationship to look like a week from now, two weeks from now, a year from now. Sometimes, the answer is I would rather not be in this relationship a week from now, right? And sometimes, the answer is I need more from this relationship. I need something to change. And other times, the answer is I just need it to stay the way it is, and I need to articulate that this is going well. Right?

So, you know, and so thinking about communication in terms of outcomes, rather than what can I say that sounds, like, really smart because you make different decisions about your word choice, your tone, whether communication should be esynchronous [sic] or not.

I mean, bug bounty is a difficult area for communications because, you know, a lot of times, there can be a language barrier. There can be -- you know, even just the cultural difference of working inside of a company versus being a external researcher, and the different levels of visibility, and of course, the bug bounty platforms themselves are primarily asynchronous and text-based and not everybody is very clear in written communication.

(Kevin) Well, and one of the things that we found -- I found when I've been on the receiving end of bounty reports is, um... that, you know, a researcher has some model of our systems from an external perspective, and I have a model of the systems from the internal perspective, sometimes the folks doing triage at the, you know, bug bounty platforms don't have a model, you know, of either, and so that's where sometimes stuff gets stuck.

And some poor researchers can go months and months and months trying to push something through the front-line triage folks until somebody who understands the system internally comes and looks at the report and is like, oh, no, actually, this is real, and we need to prioritize it., where the triage folks have been struggling with it.

And so then, like, the poor researcher is, you know, coming in with three months of frustration carried on their back, to an interaction that got --

(Melanie) Well, and in fact, one of the things that was really important to us to factor in in the way we communicated in, in this particular situation, was that this was somebody who, you know, just based on our cursory, like, you know, OSINT review, was fairly new to bug bounty, coming from a background in consulting, where you do get time you'd -- er, I'm sorry -- you get paid for the time that you spend looking for these things. And so this person felt validated demanding payments for the work that they had done unsolicited. [laughs]

Um, and from that perspective, is -- like, I understand why this is their expectation. It's inaccurate, but I understand where it's coming from. This person is not -- they did not start out being unreasonable. Um, there was some misunderstanding, there were some inaccurate... expectation. But the communication did not start unreasonable. There was a logical thought process behind what they were asking for.

(Kevin) And -- I -- recognizing on your side that, like -- I don't know -- so -- so -- if folks are, you know, have not been on the receiving end of bug bounty reports, like, it's -- you're effectively publishing an email address on the Internet that that anyone on the Internet can send things to, so you get everything from, like, super sophisticated people who have found something real. Like, if Florian Weimer sends you a GPG encrypted email, you had better drop everything and decrypt that email because that might be the next Heartbleed. Ask me how I know. To, like, you get people who will run, you know, commercial, off-the-shelf, vulnerability scanners against your system, and sometimes they find real things, but mostly, you've already run those, and so...

There's a lot of, like, low-effort stuff that you have to filter out, and then you get people who are just, like, legitimately -- not to put too fine a point on it -- crazy, who are having some kind of psychological issue, and.

(Melanie) Yeah. I mean -- and the reality is, like, all of those things can be true on both sides of this equation. And so, you know, I'm -- when I'm working with bug bounty teams, I'm thinking just as much about the mental state of my engineers as I am about the people that we're communicating with on the outside. And one of the reasons why this work

30:00
has become so important with the companies that we work with, is that the burnout and the mental health of, you know, especially triage engineers, has just become such a huge priority for these teams, in some part because it does have a very real impact in productivity, but also because, as engineering managers mature as humans, they realize that treating people like shit is not great management. [laughs]

(Kevin) Good. Yeah.

(Melanie) And so -- you know, thinking about... And so, you know, we're thinking about how the emotional drain that comes from these types of, like, escalation incidents -- and I call them incidents even though there's no breach because it's a very real moment in time where there's, like, a black hole that's just sucking resources from the team. Right?

And just, like, that emotional labor is often neglected and overlooked if -- you know, 'cause a lot of times, the -- they just -- the technical teams aren't thinking that way at first, right? And so, you know, like, it's not uncommon for... security engineers to not want to touch bug bounty rotations because they just know it's gonna drain them.

It's a very different skill set, communicating with people that you don't know versus your team members that you work with on a daily basis, and -- and then constantly having that fear of is this thing that I'm about to say going to come back to me as an explosion of emotion? Right?

And so, you know, it's -- we do a lot of work with these teams, like I said, to just help folks regulate their emotions, be very clear in their communication -- of course, with an eye to, again, what is the long-term outcome that we're striving for? Right?

It is -- you know, it -- conversations are not battles that we're trying to win. It's an exchange of information. We're trying to get from point A to point B, um...

And hopefully, if it's a valuable relationship, we're able to continue it and to strengthen it, which means that we're gonna be deliberate in what we say and how we say it.

(Kevin) Theoretically, even though, like, you know, the internal team is internal to the company, and the external reporter is external to the company, we're both on the same team here of wanting these systems to improved be more secure, to be more safe... or there's at least the possibility of that, in any interaction.

And so when we find that value's alignment, then, yeah, that makes a difference in understanding that that's a thing that we're looking for, understanding that it's an iterated game, and...

That -- that was a thing that I don't think that I had quite clocked until I did this for a while, was that it's not just, you know, a new, random person every time. Sometimes, people will come back, submit dozens of reports, they'll all be super valuable.

You know, I have some friends who I, you know, met initially because they submitted us a couple good reports, and we started, interacting more and more, and that was just, like, the genesis of the relationship, carried on, even after both of us left, you know, the work that we were doing and the respective companies that we were at.

So... That there is a possibility of, yeah, somebody coming back for years and years and years, being a valuable, you know, collaborator, if we do this right.

(Melanie) And not only that, but I have seen a number of times where that researcher actually gets hired by the company to -- doing their security team. Right? And -- I mean, you know how hard it is to hire really good security engineers.

(Kevin) We're not on the market, ever.

(Melanie) [laughs] I mean, and not only in terms of being able to find them and attract them, but you have somebody who's coming in with already baseline knowledge of your system. Right? Not a hundred percent complete, but not zero, right? And that's incredibly valuable.

And so you also wanna think about what is the culture that we're demonstrating to this individual? Right? Like, is this someone that we would potentially want to invite to join our team at some point? And have we communicated that we're the type of organization that they would be interested in?

I think something that we really benefited from the most at Uber was the fact that our core incident response team engaged every single day. Right? Whether it was a bug bounty report or an actual security investigation or just some random, like, Twitter thread, that we were running down to see if it was true, right? Um... We --

35:00
we interacted on a daily basis, which meant that we had the relationships, we had the influence, and we could make decisions very quickly, which also made actual incident response in, you know, during, like, severe incidents so much easier. Right?

Everybody knew their roles and responsibility. We could, you know, one of the challenges that I see with companies who come to us for incident response when they haven't done adequate preparation is that, you know, individual teams have their own IR plan, they don't work together, and they don't use it on a regular basis, and so they're constantly, like, brushing the dust off, [laughs] and I wanna have incident response plans to mirror everyday operations as much as possible so that people can do it in their sleep. I don't want it to be, this, like, strange, crazy thing that they're trying to figure out under pressure.

(Kevin) Because I got into security by running the incident response program, or helping to run the incident response program at Akamai, I have a very weird perspective on this, 'cause Akamai's incident response program for everything from, you know, one of the links at one of our data centers is down to Heartbleed, and so, I just think of incident response as, like, you know, well, running an incident response program is the security team's job.

And it was actually kind of a weird thing that Akamai did, and that meant that when I got to other companies, and I was, you know, wound up updating the security incident response program documentation, and, like, that was the first I had seen it after having been there six months because we never used it. Was a eye-opening experience for me.

(Melanie) And then the reality is it's not too different from product sprints. Right? So a resilient organization should be able to eb and flow and, like, put together the right people at the right time, depending on what is the urgent thing that's happening at the moment.

And if you think that your company is not having an incident every day, I'm worried for you. [laughs] Because it means that you're -- you're probably missing something. You're not aware of something that's happening, right?

And so across an organization, there is a thing happening all the time, and it should be seamless for us to use the same process to bring in the right people at the right time without being, like, oh, my god, we have to declare this a crisis, and okay, well, you know who I don't want making decisions? People freaking out about a crisis, right? That is not how the human brain is designed, right?

Um, you know, when -- when you're in fight-or-flight mode, you are not usually executing sound judgment, right? And so the more that we make security incidents this, like, anomalous thing that requires, you know, some special, like, binder, you know, of information, the weirder it is for people.

And I just -- I want people across the company to go, yeah, I know how to do this, it's just like all the other incidents that exist in my world. The difference is, security's in charge of this one, right? I'm taking the lead from security, but I understand the process, the workflow, the approval process, all of it. Like, that doesn't -- that shouldn't be different, right?

(Kevin) Because that -- that's the same one we used last week.

Yeah, I think there is so much value in those crisis situations in being able to fall back on your process, where you're like, you know, I'm freaking out about a thing, so my, you know, my proper response, the thing that will cause me to stop freaking out about the thing is by falling into the response pattern that I know, which is the existing incident response program, which we used every week for the site is down or whatever.

(Melanie) It was interesting -- I was having a conversation with a friend earlier this week who also works in incident response, and he reminded me of this, like, old adage that I think originated in the military. I'm not a hundred percent sure. My military history is, like, zilch an inch, right?

But it was essentially, like, you never rise to the occasion, you fall back to your training. And so, you know, very rarely are people actually exceeding expectations when it -- when they're under pressure. Right?

And even -- so outside of security, I'm a certified rescue scuba diver. And we talk about the same concept all the time. We refer to it as task loading, right?

If you're expected to do a task under pressure in an urgent situation, and it's unfamiliar to you, it requires a lot more cognitive resources and attention to complete that task, and you might not do it well, right? So I want

40:00
the response in an emergency to be as close as possible to what you do all of the time so that it doesn't require your entire focus and attention to just do this one thing.

(Kevin) All of the rituals that we build around an incident should be a comfort, should be a -- a, like, thing to fall back on, and not a thing where you, like, you, say, have to pull out the binder for the first time and start leafing through it because you're already existing in a regime of not abnormal operation, and you don't want to add additional abnormality on top of that.

That, just, like you say, causes people's heads to explode.

(Melanie) Yeah, I mean, it -- of course, I hope that every single day, your process isn't going up to the most severe, like, escalation possible.

(Kevin) You don't want a Sev 1 every day. Yeah?

(Melanie) Right? Exactly. Exactly.

But you should be able to manage, like, some type of Sev every day.

(Kevin) And that's not also a failure of your company, a failure of your process. Incidents are part of normal operation.

(Melanie) No. It's a failure of the Internet.

(Kevin) Well, it's -- like, reality, kind of, will throw curve balls at us, no matter --

(Melanie) Yeah. Exactly. It's just -- I just don't know what else you're doing if not this every day. Right?

(Kevin) Well, and so you are doing this, on some level, and with some process, however formally or informally specified, so if you have a lightweight enough incident process that you can use it every day, like, this, that's a real strength, yeah.

(Melanie) Yeah. And, of course, you don't want the same people on your team constantly thrown into the fire. They are going to burn out. But it -- again, it's -- as an organization, as a living organism, we need to be able to eb and flow and have that resilience so that people can step into those roles for each other so that people get a break. Right?

Which means this can't be something that only lives in your head, that is, like, such engrained institutional knowledge that another person can't step into that role, right?

(Kevin) I need to go on vacation ever, and yeah. And it was -- being, you know, sometimes I was the, you know, subject-matter expert, and I was coming into somebody else's incident, and sometimes I was running the incident. Or, on the same incident, sometimes I would, you know, step back and forth in those roles as the needs arose, and so the fact that we all at Akamai got really practiced with that was a real strength of the organization, I felt.

(Melanie) I remember the exact moment in my career when I realized that this was something I had to focus on for my own role.

Um, I was actually -- so it was while I was at Uber, and it's crazy that it took me to my role in Uber to actually have this epiphany, but I think the reality is is earlier in my career when I was young and less wise, I made myself available for everything. Right? And it wasn't until later in life that I was, like, you know, it doesn't always need to be me. [laughs]

And so this moment I had this epiphany, I remember, I was actually diving a shipwreck in the Bahamas, and we're about a hundred feet below the surface, and I just remember -- and I don't know why this thought came to me -- but I very clearly had this voice in my head that was like, if Uber has a breach right now, there's nothing you can do about it.

Like, even -- 'cause even if I wanted to end my dive, and go to the surface, like, even if there's someway for me to find out, a hundred feet below the surface, that Uber was having an incident, I can't shoot to the surface. I have to do my safety stops so I don't get decompression sickness. Like, there is just -- I cannot control how quickly I get to the top and retrieve my phone from my dry bag, right?

Um, and in that moment, it was both, one, motivating that I needed to get my shit together and make sure that my team could fill in for me, and number two, it solidified that scuba diving was forever going to be the thing that balanced my work with my life because when I'm under water, there's nothing I can do about it.

And I found that to be the most freeing experience of my entire career. To realize that, if I have my ducks in a row, and I have prepared my team with the plan that gets them 99.9% of the way, as if I'd done it myself, I can go diving and not worry about what happens because I know my team is going to do a stellar job.

(Kevin) What goes into that plan?

(Melanie) Hopes and dreams, Kevin. Hopes and dreams.

(Kevin) Oh, okay. Yeah.

(Melanie) No, I'm kidding. I'm kidding.

So a lot of it has to do with, like, building a process with your cross-functional partners so that my team is not responsible for everything that happens obviously, right? Um, we have very clear --

(Kevin) Especially the security. Oh, my god, we touch everything

(Melanie) Yeah.

(Kevin) Like, yeah.

45:00
(Melanie) Yeah.

So having very clear roles and responsibility in terms of who owns what, who has to sign off on what, who gets looped in, who actually gets to have an opinion, right.

Like, I think one of the biggest problems, especially for an inexperienced person who's thrown into an incident like this, is you don't know whose opinion you're supposed to listen to, right?

My team has very clear instructions on here's who you actually, here's who has to say yes. If anyone else has an opinion, you can consider it, but they're -- they're not an approver. So --

(Kevin) They don't block anything. Yeah.

(Melanie) Exactly. Exactly.

So if they have a great idea, good suggestion, great, let's incorporate it, but if they're just hung up on something and all the other approvers don't care, move forward. Right?

Um, and so you really have to be able to, like, empower people to move quickly, which means that it needs to be very clear who makes what decisions, what the SLA is for people who need to contribute to that decision, right? There's gonna be some things in an incident that only your legal team can decide.

Well, my legal team can't take four days on some of these things, right, so we determine in advance what is your role, where are you the gatekeeper, and in the places where you could potentially be the bottleneck, how quickly are you willing to respond?

Because if you can't respond as quickly as we need you to, then we need to have a conversation about whether or not the organizations are stacked appropriately, in order to enable that velocity, right?

I will be the first one to go to bat for a headcount with a cross-functional partner who's like, I'm the only attorney at this whole company. I can't turn this around in less than two hours. Welp, let me go to bat for you and figure out how to fix that because you have to respond in under two hours. Otherwise, it's gonna be bad for you and for me, right?

(Kevin) So, yeah, I mean, I guess it's building the incident response plan, and then, you know, using that as part of the day-to-day.

(Melanie) And this is why preparation happens so far in advance of needing it. Right? Because you can't, in the middle of an incident -- like, I hate the saying of -- it's, like, never waste a crisis. I hate that.

No one -- no one is building long-term infrastructure during a crisis. They're just trying to put the fire out. Right? I need you to be thinking about, again, what are the long-term outcomes that you want to see from this program? Let's build to that now so that we can quickly make decisions in the moment and be proud of where we end up.

(Kevin) It's not just financial risk. It's not even in reputational risk the way risk management people understand it. It's like, relational risk. On a very human level.

(Melanie) Yeah. Well -- and even, like, the -- the risk of a misunderstanding can be incredibly costly. Right? Even -- even if you are really trying to do something good, right, maybe it's not anything malicious, but if you didn't communicate it clearly, now there's a huge misunderstanding and a blow-up on the Internet that you have to, you know, defuse.

I don't know, could we have spent, like, four more minutes carefully crafting that message so that it was clearer? [laughs]

(Kevin) Right. Yeah.

So it sounds like some of the work was, like, having you embedded fairly closely with these engineering teams who were doing bug bounty triage to help them, sort of craft, the message, and craft the relationship, and with that, you know, keep that north star in mind of, like, the relationship that you wanted to have with the researchers on an ongoing basis. And -- just a little support.

(Melanie) Yeah. Well, I think you can really divide the value of my role, sitting so closely with those folks, it being, one, I was a constant reminder of thinking about those -- those long-term outcomes, right?

But also, I could just be a sounding board for them to go, does this say what I think it says? [laughs]

You know? Like, has -- getting a second opinion of you -- because, again, if we're thinking about should this become public, it's gonna be read by more than just somebody with like, technical background. Right? How will this read to a journalist? How will this read to a regulator? How will this read to a customer?

(Kevin) I -- we got in the habit of the doing that when I was at Akamai, even if just on an informal basis. You know, pass the laptop around. I'm gonna send this email, does this look kinky to you? And it made a big difference.

(Melanie) I do that. I'm a communications professional, and I have people in my life and on my team where I'm saying, can I get a gut check before I hit send on this? Because I need to make sure that it says what I intend it to.

(Kevin) And then building processes which support that in the organization, and the organization giving people the time to take that extra four minutes, rather than...

(Melanie) I -- I mean, that -- that's a crazy measurement, just

50:00
in general, because... Fewer tickets isn't necessarily a good thing. [laughs]

(Kevin) No. Right. Yes. Yeah.

In the same way that fewer incidents is not necessarily a good thing like, if it means that you're missing -- (Melanie) Fewer declared incident, or --

(Kevin) Exactly.

(Melanie) Fewer discovered -- incidents -- (Kevin) Discovered incidents, yes. Yes, yes. That's the real trade-off we're making here, is just not acknowledging when things go wrong. And -- or might be going wrong.

(Melanie) Yeah. And -- and I think a lot of times people are hesitant to, like, make some -- to call something an incident 'cause they're like, ah, that means it has to become this, like, thing, and I was like, if you're really honest about the fact that sometimes we have to put more attention on this and more attention on this, and that becomes the role of leadership, to give people permission to focus on this instead of this. We're gonna slip on this because this is more important.

Like, a healthy organization can move like that. Right? And when we turn any type of incident, whether it's an outage or a security incident, or you know, a DeleteUber situation. Like, when that becomes, like, a special scenario, where people are freaking out, it's a million times worse. Right?

I mean, I think one of the things that that I took away -- I will just say from working with the bug bounty team at Uber in general, is just how important it was to have a variety of backgrounds and experience and disciplines represented in that room, you know, myself and legal were often attendees at the payout meetings because we were thinking about expectations we were setting, things we may inadvertently be communicating by the payout amounts or perceived inconsistency and things like that.

And you know, I want technical teams to know that they're not alone. There are cross-functional partners who want to help and support you. But often, they do not feel comfortable walking into that room without an invitation from you.

So, if -- if you would like some cross-functional help, reach out to those partners and start inviting them to your discussions.

(Kevin) Yeah. Yeah.

And then, start collaboratively figuring out how to communicate across those divides. A lot of my work, over the last few years, has been, like, translating from lawyer to engineer and vice versa 'cause speak two very different languages, and yeah, that was -- that's... And eventually, you know, we each learned to speak the other's language well enough to communicate in those rooms, but it took a while and a lot of patience.

But, yeah. Yeah, having those disciplines represented, especially in something like bug bounty...

(Melanie) I think assuming good intent until proven otherwise goes a long way. Um, I -- I worked with the Duo Security team after the acquisition to Cisco, and one of the -- that team from Duo brought their values from Duo to the security organization at Cisco, and one of the ones that really stood out to me was be kinder than necessary.

It changed the way people spoke to each other. It changed the assumptions people made about each other. And, quite frankly, it weeded out people who didn't want to be kind to each other.

Like, who wants to work with those people, right? And so, like, that, to me, was just really profound. I mean, it was something that you know, the Duo founders implemented when they started the company. And, you know, I'm not surprised that it lasted as -- that this value continued even after the acquisition because it -- people clung to it because it just had such a profound effect on the way that they worked together, and it's something that I think more of us could adopt, more of us could benefit from being kinder than necessary.

(Kevin) Yeah. Especially in security.

(Melanie) Which is also really great for bug bounty communications.

(Kevin) Yes. Yes. Yes.

Yeah. And still managing to surface problems when they needed to be surfaced without, like, being jerks about it.

(Melanie) You know, being kind doesn't mean that we're not truthful. It doesn't mean that we don't raise issues. It means that we do it in a way that is respectful of each other. Right? I'm not going to hide a problem that I think is going to bite us later, but I'm going to be very respectful and deliberate in the way that I communicate that to my peers, to leadership, you know, even to our clients.

There are times where we have to give them bad news, or let them know that something terrible is coming around the corner. And doing that in a way that is kind actually helps people focus on the real issue, rather than the emotion.

(Kevin) Yeah. That's a good point.

That -- yeah, we get -- deeper into the problem, we're able to gauge more fully and completely with the problem as the problem rather than... It's not obscuring the problem, yeah. That's a really good point.

(Melanie) Right. Or -- or distracting people from, you know, like, if I have hurt your feelings that's what you're gonna think about for a long time rather than the answer to the question. Right?

(Kevin) Uh, Melanie, where can people find you online?

(Melanie) Great. So the easiest place to find me online is LinkedIn. And I'm also on Mastodon. My handle is Wednesday@DefCon.Social.

(Kevin) And your company is Discernable, Inc. Is that the...?

(Melanie) That is correct.

(Kevin) Link in the description below.

If you want to hire Melanie to help you and your security and privacy teams communicate with each other and with internal stakeholders and external stakeholders...

Yeah, Melanie, this has been really lovely.

(Melanie) Yeah. I really enjoyed it. Thank you so much for having me.

(Kevin) Good. Yeah.

This has been the War Stories Podcast on Critical Point.

I'm Kevin Riggle, here with Melanie Ensign, formerly of Uber, now doing her own thing, and 'til next time, folks.

[smooth, rhythmic outro music plays]