The Reporter Called Her Christmas Day - Melanie Ensign - Bug Bounty & Incident Management
Download MP3I think the thing that we really benefited from the most at Uber was the fact that our core incident response team engaged every single day. One of the challenges that I see with, with companies who come to us for instant response when they haven't done adequate preparation is that, you know, individual teams have their own IR plan. They don't work together, and they don't use it on a regular basis, and so they're constantly, like, brushing the dust off. And I I wanna have I wanna have incident response plans to mirror everyday operations as much as possible so that people can do it in their sleep. I don't want it to be this, like, strange crazy thing that they're trying to figure out under pressure.
Kevin:Because I got into security by running the incident response program, helping to run the incident response program at Akamai. I have a very weird perspective on this because Akamai used the incident response program for everything.
Melanie:A resilient organization should be able to ebb and flow and, like, put together the right people at the right time depending on what is the urgent thing that's happening at the moment. And if you think that your company is not having an incident every day, I'm worried for you.
Kevin:Howdy, folks. We're back again on the War Stories podcast talking about another incident. We are here today with Melanie Ensign, friend of mine from Boston days, but also formerly of Uber, who has a story to tell us about a time that this time someone else broke production. We're getting our first security incident on the podcast. So I'm super excited to chat with her.
Kevin:We are
Melanie:But it's not the Uber incident that everyone is expecting. I'm sure.
Kevin:Okay. Great. Yes. Yes. I don't want to say that there are a few that I can think of which might, qualify for that.
Melanie:It's not any of those. Not any of them. Yes. Any of those.
Kevin:Yes. Yes. Uber had a bad run for a while, and, we'll we'll bring people something they haven't heard about. And first, we'll roll the titles. And we're back.
Kevin:Once again, the War Stories podcast on Critical Point here here with Melanie Anson. I'm Kevin Riggle. Melanie, can you tell us a little bit about yourself and how you found yourself in 8 plays, not to break production, but to help some folks, whose production had been broken.
Melanie:Sure. So first of all, thank you so much for having me today. It's really exciting, to to be talking to you again after, I think we've both just been so busy, I think, running our own businesses the last couple of years. I am the founder and CEO of a company called discernible. We are a dedicated and specialized communications firm specifically for security and privacy organizations.
Melanie:So our clients are chief security officers, chief privacy officers, and their teams. And our mission is to help them become more effective communicators in order to earn more influence across the business so that they are not treated as the cleanup crew to everyone else's decision.
Kevin:So when, production breaks or when there is a security incident, you should call Melody. Or, ideally, you should call Melody before there is a
Melanie:Yes.
Kevin:Production Yes. Security incident. And, oh, I've just I've been really enjoying your newsletter. Like, I've been getting a ton of value out of that, yeah, as I try to figure out how to communicate about my work and
Melanie:It turns out that communications is in fact a centuries old, scholarship of its own, not something that engineers invented.
Kevin:Also not something most of us got taught in school. Like, I had a, like, half half semester communications course. And it was incredibly valuable that, like, people were kinda complained about it. But it still wasn't enough. I need more.
Melanie:Yeah. I mean, like, they while engineers were learning computers and technical things, like, we were studying people and how they communicate. We I've even done a good deal of studying engineers specifically and how they communicate. Right?
Kevin:Which is a little different than anyone else. Yeah. It's oh, that would be a a I would really love to read about that.
Melanie:The the special snowflake communicators. It's I mean, I I just, because, of course, they're I don't want to imply that this is a monoculture. You know, not everybody is the same, but there are a lot of shared, I think, challenges because it's not a skill that is inherently and proactively, developed with technical teams. Right? That there seems to be an incorrect assumption that this is just magically learn how to communicate effectively with all different types of stakeholders.
Melanie:Right? And we have found that even engineers that are good at communicating to, say, perhaps, their board or business leaders, it doesn't necessarily mean that they're good at communicating with other engineers. So when you start working on things like shared KPIs and road maps, you know, we're constantly working with security teams to help them negotiate with infrastructure teams and product teams and, you know, SRE teams. How can we get security outcomes through the planning process and not just checking it on yet.
Kevin:Oh, nice. Oh, so you're helping teams communicate, like, technical teams communicate internally and with other technical teams as well as outwardly focused. I think I have had you as being primarily outwardly focused.
Melanie:Most of our work is internal because that's where the work gets done. We the external work is kind of a reflection of what's been done internally. Right? I don't have an external story to tell if we haven't done the work internally first.
Kevin:And the external story will reflect a lot of whatever is going on internally, like, for better or for worse. I there are some incidents that I not my story to tell, but that I think, spring vividly to mine where yeah. The the ways that we exposed or didn't expose the internal, matter a lot to how the incident got resolved.
Melanie:Yeah. Exactly. And and even, you know, when it comes to incident response, a lot of teams individually, different departments, and different functions have their own plan for how they deal with their special flavor of incidents and what they consider to be an incident. Right? But prior to an incident, very few companies are thinking about how do all of those plans work together, and how do we make sure that those plans are as close as possible to our everyday work so that we are practicing them over and over and over again.
Melanie:We see this a lot particularly with security incidents where, you know, cross functional partners who don't work on security every day. If we are giving them a brand new playbook when there's a breach, I can promise you they're not going to execute that well because, you know, they're rusty every single time they have to use it. Right? So a security incident playbook needs to mirror what people would normally do for other types of incidents. Right?
Melanie:It needs to make the decision making a little bit more automatic and and taking that responsibility off the engineer. It shouldn't be up to engineers to decide whether or not legal gets looped in. Legal needs to make that decision, Right? Which means we need to have ways to surface the information to legal, at the start of the incident so that, you know, these nontechnical worlds are not playing catch up every time they get looped into an incident.
Kevin:So people should come talk to you about, communication internally and externally around security, and privacy because well, and then that is like such an overwhelming amount of the work. Like when we implemented GDPR at Stripe and then CCPA at Lyft, like every engineering team at the company had to do at least a sprint's worth of work. So we were talking to all of them. That was, you know, we spent 15 months. I talked with, you know, practically everybody at the company.
Kevin:Still people will come up to me, be like, oh, yeah. We worked together on CCPA at Lyft. I'm like, I'm really sorry I don't remember you, because, like, I talk with literally everyone. You were lovely. I'm really sorry for making you do all this work.
Kevin:I'm glad that we completed it successfully, but yeah.
Melanie:Yeah. A lot of engagements. And, you know, one of the things that we encounter a lot is, you know, when we first start working with a client on incident preparedness, and I'd say, you know, if this incident or type of incident were to happen tomorrow, depending on, depending on, the scope to make that true before you need to say it. Right? And so there's, you know, a lot of, I think, just just, like, very basic preparedness where you're not gonna be able to say something unless you make a true first.
Melanie:Right? That is shockingly not a common train of thought. And, you know, the other thing is that they're not thinking about kind of, what I what I call the the hiccups, the the fuck ups, and the give ups of incidents, meaning that it's not just breaches. It's not just, like, a technical compromise that is going to be an incident. You know, something like when people discovered that Facebook was using phone numbers not for just for the 2 f a feature that we all expected, but also for targeted advertising.
Melanie:Like, that was a massive incident for the company in terms of public perception, regulatory investigations. It's it's a huge distraction for a lot of teams, at an organization when you when you have something like this. Right? So I've talked about this before. I've written blog post about it that you have to really expand your definition of what's an incident, because it's it's not just an outage.
Melanie:It's not just a bug. And in fact, the incident that I'm gonna tell you about today, there was no actual technical compromise. There was no vulnerability, but it created an incident for us on Christmas of all days. But but I do I do have to I mean, I do have to say that, you know, one of the things that Uber was really good at while I was there was we had our core incident response team engaged on a daily basis because we work together on everything from bug bounty reports to, product security to, you know, actual, like, breaches or regulatory investigations. Like, all of these things with the same core team from legal, comms, security, public policy.
Melanie:Like, we were lockstep. So we had our process very well practiced, and we knew we knew all the people. Our playbooks, you know, worked well together. We had good internal communications, on most of these things. And, you know, so when this thing popped up on Christmas, in a matter of hours, you know, me and 2 engineers had put it to bed.
Melanie:Right? Because they it was, like, it was it was more of a public perception problem and a rumored incident rather than an actual compromise of our infrastructure or corporate network. It nonetheless required our core incident response team to come together very quickly on holiday, to make sure that what I was telling to the journalist was accurate, factual, and, you know, not going to cause additional issues for us. And and I will just I will tease the ending by saying the reason it ended as well as it did was because of that process we had where, you know, the this all starts with a bug bounty report, and I knew about the bug bounty report from day 1. Right?
Melanie:So as the communications adviser, I knew the whole story. It had all of the context. When I'm contacted by a member of the press, I know exactly what's happening.
Kevin:So that's so we've we've we've we've gone these always every time I get to the edit, I'm like, this is already gone 6 different ways. And there's so many different cool threads to pull on. So that's a little bit about what Melanie is doing right now at Certible. It's helping teams build a process so that on Christmas morning, when, you know, somebody raises the flag and is like, we've got to deal with this. You've got a process, which within a couple of hours can get everybody, on the same page, pointed the same direction, knowing what the next steps are and put something to bed so you can go, you know, do Christmas things, with your family.
Melanie:Or not do Christmas things. Or not
Kevin:do Christmas things.
Melanie:That's how you choose to spend the day. Right.
Kevin:Yeah. If you don't do Christmas, then do do something else. Yes. But literally anything but deal with an incident on Christmas, because there are only some of us who would consider that a lovely present to find wrapped under the tree. Alright.
Kevin:So with this as context, you're at Uber. It is Christmas day 2017. What was the first thing that you noticed?
Melanie:Sure. So on that particular day, I was contacted by the weekend editor at a, tech publication. I'm not going to name them because I'm I'm not trying to embarrass anybody with the story.
Kevin:And what's a weekend editor?
Melanie:So weekend editor sure. It's a great question. A weekend editor editor is essentially somebody who they cover the weekend. So they're they're filling in for, other members of staff who work Monday through Friday, you know, unless there was, like, breaking news. Right?
Melanie:This this wasn't a breach or something like that. So it didn't warrant bringing in, like, the full time security reporter to cover it. But because it was a holiday, most of the full time staff for that publication were on PTO. So this is essentially somebody who covers weekends, covers holidays, things like that.
Kevin:And so they kind of wear all the hats. They cover all of the bases be for the publication.
Melanie:Yeah. I mean, you could I think you can I think you can kind of think of it as, like, on call when other people are on PTO or, you know, on leave or something like that?
Kevin:For a news organization. Okay. That's great. I love that. Yes.
Melanie:So so this was a person who, despite working at a, you know, a tech publication, their typical beat was not cybersecurity. Right? So I don't fault them for not understanding the technical details of of what was in, this particular bug battle report, which was the, topic of discussion.
Kevin:Okay. So what had happened?
Melanie:Right. So what had happened was there was a, security researcher who had submitted, about half a dozen reports to Uber's bug bounty program. They had been labeled duplicates by the team and closed without payment. And so this particular individual, understandably, was frustrated and upset with how much time they had spent, submitting these reports and to get no financial payout for them.
Kevin:Because they believe this is a real issue.
Melanie:That is correct.
Kevin:Not just, like, not outside the scope of Hooper's Bug Bounty program, but they're like, there's a real thing here that you all aren't seeing.
Melanie:Yes. Yes. And as a result of that, they then decided they were going to escalate and publish their own blog post about how they had been screwed by Uber's Bug Bounty program, and kind of, you know, denied what they felt they deserved, which is not an uncommon situation for bug bounty programs.
Kevin:And this kind of thing gets picked up by, like, Hacker News or I mean, it
Melanie:it definitely was because it was a slow news day. I mean, it was Christmas.
Kevin:It's Christmas. Okay. Yeah.
Melanie:Right? So it's it was a story because it was a slow news day, and it became an incident for us because it was happening on a holiday when a lot of people were not working and certainly when I did not really want to be on the phone with with reporters. Right? But it's the nature of of the job. Right?
Melanie:So what happened was so this, The Weeknd editor reached out to me for a comment on, you know, this, researcher's blog post. And because of the way that the Uber's, Bug Bounty program manager ran the program and I will say she is one of the smartest, most astute PMs I have ever met in bug bounty. Right.
Kevin:Which is also a hard field.
Melanie:Yes. Because of the way that she ran the program, I already knew what was happening before this editor reached out to me. So the moment he reached out to me, I not only gave him a statement, I gave him links to a whole bunch of comments and social media posts that had been shared by other members of the community when they saw this individual's blog post. And these comments validated Uber's position, validated the position of the bug bounty team members, and called out specific and abusive behavior that had been demonstrated by the researcher. You know?
Melanie:Again, we don't we don't always know who's on the other end of these reports. Right? I try really hard not to make assumptions about, what's going on in other people's lives that perhaps is motivating some of this behavior. Right? That being said, when when we work with the bug bounty teams, and Uber's team was really good at this at the time, you know, we're focused on what is the long term desired outcome of each of these relationships.
Melanie:Right? Whether it's a researcher who you have had a long history with, who perhaps consistently submits valuable reports to your team, or in this case, is a first time, reporter to your program. And and even if you disagree with their technical analysis, being professional, being respectful is still paramount in the way that you speak to them. Right? And because of that, I mean, it it's interesting and somewhat amusing to me in hindsight that the security program at Uber in this moment was benefiting from a more positive public perception than Uber's parent brand at the moment.
Kevin:Right. Which is not a thing is that security program is usually doing, and often we try not to get noticed. And this was, like, what was this? Delete Uber was was that 2016 or was that
Melanie:So it was the be that was the beginning of 2017. And so this is almost 12 months later. This is the summer of 2017.
Kevin:Okay. Because that was when, Susan Joy Fowler had published her post about, like, sexual harassment at Uber, and there had been, yeah, a lot of negative press.
Melanie:2017 was a difficult year for Uber.
Kevin:So that that maybe also piqued the weekend editors attention. They're like, we know that, you know, people will click on a news article about bad things happening at Uber because that's just been part of the narrative for, you know, McGarrett at this point then. Yeah.
Melanie:Yeah. I was playing into people's existing I don't wanna call it bias because I think some of it was well deserved, but they're preexisting opinion. Right? And so, yeah, it was and and so it was interesting when when I responded to this editor and was able to say exactly you know, I I knew exactly what he was talking about so we could respond right away. Right?
Melanie:It didn't take hours for me to track down people with knowledge of this of this situation. I knew exactly what was happening because I sat next to you, the bug bounty team, as they responded to this researcher's, correspondence because, again, I the PM on this team was so smart, and she immediately recognized when emotions were escalating. Right? And when things started to kind of go off the rails. And, you know, we we came from a position of disclosure by default, at least in the way that we communicated with people, because we did not know if we would always have control over what got disclosed publicly or not.
Melanie:Right? And so we had to hold ourselves to a higher standard and take the high road no matter what. That was that was just how we operated. And so, you know, I had been looped in with the Baghdadi team from the beginning on this particular report. And as a result of the way that the program manager responded and handled this situation, You know, members of the security community who otherwise did not have a favorable perception of Uber as a company were publicly coming to defend the defense of this PM and the bug bounty program to say, not only is Uber correct in a technical assessment, but they behaved professionally.
Melanie:They were actually the adult in the room here. And this other individual, like, you know, exhibited some very problematic behavior in some of the, you know, quite frankly, sexist language that he was using, with with the program manager. And, you know, it's one of those things where, you know, individual engineers aren't necessarily always thinking that every email or every bug bounty correspondence is going to end up on the Internet, but the reality is is you don't know. And the interesting thing is that through all of my time with this Bugbani team, I worked at Uber for 4 years. 1 of the members of their team, because we were constantly going through this process of focusing on your long term outcomes.
Melanie:Right? We're not trying to win an argument. We're trying to, you know, get to a specific result, so that we can we're in a better position tomorrow. This particular individual set told me that the communication skills that they had learned working with me improved their communication in their marriage because all of a sudden, they stopped trying to win arguments. They were like, the most important thing to me is the longevity of the relationship.
Melanie:And and it wasn't just about, you know, like, self censoring or not mentioning those problems, but about making sure that above all else, your partner understood that the relationship was more important to you than
Kevin:more than I like. And, you know, more than I like. And sometimes it really is important that, you know, there are specific technical details that really need to be right. But that that that that framing all that that framing is actually kind of blowing my mind a little bit, or it's never been, you know, put so crispy to me.
Melanie:And to be honest, sometimes you need to reassure the other person that the relationship is the priority in order for them to accept when you are right. So, you know, again, it's it's about being honest with yourself and being able to articulate what is it that I actually want this relationship to look like a week from now, 2 weeks from now, a year from now? Sometimes the answer is I would rather not be in this relationship a week from now. Right? And sometimes the answer is I need more from this relationship.
Melanie:I need something to change. And other times the answer is I just need it to stay the way it is, and I need to articulate that this is going well. Right? So, you know, in in so it's like thinking about communication in terms of outcomes rather than what can I say that sounds, like, really smart, because you make different decisions about your word choice, your tone, whether communication should be asynchronous or not? I mean, bug bounty is a difficult area for communications because, you know, a lot of times, there can be a language barrier.
Melanie:There can be you know, even just the cultural difference of working inside of a company versus being an external researcher and the different levels of visibility. And, of course, the bug bounty platforms themselves are primarily asynchronous and text based, and not everybody is very clear in written communication.
Kevin:Well and one of the things that we found I found when I've been on the receiving end of bug bounty reports is, that, you know, a researcher has some model of our systems from an external perspective. And I have a model of the systems from the internal perspective. Sometimes the folks doing triage at the, you know, bug bounty platforms don't have a model, you know, of either. And so that's where it sometimes stuff gets stuck. And some for researchers can go months months months trying to push something through the frontline triage folks until somebody who understands the system internally comes and looks at the report and is like, oh no.
Kevin:Actually, this is real and we need to prioritize it. Where the, triage folks have been, struggling with it. And so then, like, the poor researcher is, you know, coming in, you know, with 3 months of frustration, you you know, carried on their back to an interaction that yeah.
Melanie:Well and and in fact, one of the things that was really important to us to factor in in the way we communicated in this particular situation was that this was somebody who, you know, just based on our cursory, like, you know, OSINT review, was fairly new to bug bounty coming from a background in consulting where you do get time you'd or I'm sorry. You get paid for the time that you spend looking for these things. And so this person felt validated demanding payment for the work that they had done unsolicited. And and from that perspective, it's like, I I understand why this is their expectation. It's inaccurate, but I understand where it's coming from.
Melanie:This person is not they did not start out being unreasonable. There was some misunderstanding. There was some inaccurate expectation, but the communication did not start unreasonable. There was a logical thought process behind what they were asking for.
Kevin:And, like, recognizing on your side that, like I don't know. So if folks are, you know, have not been on the receiving end of bug bounty reports, like, it's you're effectively publishing an email address on the Internet, that anyone on the Internet can send the things to. So you get everything from, like, super sophisticated people who have found something real. Like if Florian Weimer, sends you a GPG encrypted email, you would better drop everything and decrypt that email because that might be the next Heartbleep. Ask me how I know.
Kevin:2, like, you get people who will run, you know, commercial off the shelf vulnerability scanners against your system. And sometimes they find real things, but mostly you've already run those. And so, there's a lot of like low effort stuff, that you have to filter out. And then you get people who are just like legitimately not to put too fine a point on it crazy, who are having some kind of psychological issue. And
Melanie:Yeah. I mean, and and the reality is, like, all of those things can be true on both sides of this equation. And so, you know, I'm when I'm working with bug bounty teams, I'm thinking just as much about the mental state of my engineers as I am about the people that we're communicating with on the outside. And one of the reasons why this work has become so important with the companies that we work with is that the burnout and the mental health of of, you know, especially triage engineers has just become such a huge priority for these teams, in in some part because it does have a very real impact in productivity, but also because as engineering managers mature as humans, they realize that treating people like shit is not great management. Right.
Melanie:And so,
Kevin:you know eventually.
Melanie:Thinking about and so, you know, we're thinking about how the the emotional drain that comes from these types of, like, escalation incidents, And I call them incidents even though there's no breach because it it's a very real moment in time where there's, like, a black hole that's just sucking resources from the team. Right? And and and just like that emotional labor is often neglected and overlooked if, you know, because a lot of times, they they just the technical teams aren't thinking that way at first. Right? And so, you know, like, it's not uncommon for security engineers to not want to touch bug bounty rotations because they just know it's gonna drain them.
Melanie:It's a it's a very different skill set communicating with people that don't know versus your team members that you work with on a daily basis. And and then constantly having that fear of, is this thing that I'm about to say going to come back to me as an explosion of emotion? Right? And so, you know, it's we do a lot of work with these teams, like I said, to just help folks regulate their emotions, be very clear in their communication, of course, with an eye to, again, what is the long term outcome that we're striving for. Right?
Melanie:It is you know, conversations are not battles that we're trying to win. It's an exchange of information. We're trying to get from point a to point b. And, hopefully, if it's a valuable relationship, we're able to continue it and to strengthen it, which means, like, we're gonna be deliberate in what we say and how we say it.
Kevin:Theoretically, even though, like, you know, the internal team is internal to the company, the external reporter is external to the company, we're both on the same team here of wanting these systems to improve, to be more secure, to be more safe. Or there's at least the possibility of that in any interaction. And so when we find that values alignment, then, yeah. That makes a difference in understanding that's the thing that we're looking for. Understanding that it's an iterated game.
Kevin:And that was the thing that I don't think I quite caught until I did this for a while. Was that it's not just, you know, a new random person every time. You know, sometimes people will come back, you know, submit dozens of reports. They'll all be super valuable. You know, I have some friends who I, you know, met initially because, you know, they submitted us a couple of good reports and we started, you know, interacting more and more.
Kevin:And that was just like, you know, the genesis of the relationship, that carried on even after both of us left, you know, the, you know, work that we were doing in the respective companies that we were at. So, that there is a possibility of, yeah, somebody coming back for years years years, being a valuable, you know, collaborator if we do this right.
Melanie:And and not only that, but I have seen a number of times where that researcher actually gets hired by the company to join their security team. Right. Right? And, I mean, you know how hard it is to hire really good security engineers.
Kevin:We're not on the market ever.
Melanie:Yeah. I mean, and not only in terms of being able to find them and attract them, but you have somebody who's coming in with already baseline knowledge of your system. Right? Not a 100% complete, but not 0. Right?
Melanie:And that's incredibly valuable. And so you also wanna think about what what is the culture that we're demonstrating to this individual. Right? Like, is this someone that we would potentially want to invite to join our team at some point? And have we communicated that we're the type of organization that they would be interested in?
Melanie:I think the thing that we really benefited from the most at Uber was the fact that our core incident response team engaged every single day. Right? Whether it was a bug bounty report or an actual security investigation or just some random, like, Twitter thread that we were writing down to see if it was true. Right? We we interacted on a daily basis, which meant that we had the relationships, we had the influence, and we could make decisions very quickly, which also made actual incident response in, you know, during, like, severe incidents so much easier.
Melanie:Right? Everybody knew their roles and responsibility. We could, you know, one of the challenges that I see with, with companies who come to us for instant response when they haven't done adequate operation is that, you know, individual teams have their own IR plan. They don't work together, and they don't use it on a regular basis. And so they're constantly, like, brushing the dust off.
Melanie:And I I wanna have I wanna have incident response plans to mirror everyday operations as much as possible so that people can do it in their sleep. I don't want it to be this, like, strange crazy thing that they're trying to figure out under pressure.
Kevin:Because I got into security by running the incident response program, helping to run the incident response program at Akamai, I have a very weird perspective on this, because Akamai used the incident response program for everything from, you know, one of the links of at one of our data centers is down to Heartbleed. And so I just think of incident response as, like, you know, well, running an incident response program is a security team's job. And that was it was actually kind of a weird thing that Akamai did. And that meant that when I got to other companies and I was, you know, wound up updating the security And then the reality is it's it's not too different.
Melanie:And then the reality is it's it's not too different from product sprints. Right? So, a resilient organization should be able to ebb and flow and, like, put together the right people at the right time depending on what is the urgent thing that's happening at the moment. And if you think that your company is not having an incident every day, I'm worried for you because it means that you're you're probably missing something. You're not aware of something that that's happening.
Melanie:Right? And so across an organization, there is a thing happening all the time, and it should be seamless for us to use the same process to bring in the right people at the right time without being like, oh my god. We have to declare this a crisis and it Okay. Well, you know who I don't want making decisions? People freaking out about a crisis.
Melanie:Right? That that is not how the human brain is is designed. Right? You know, when when you're in fight or flight mode, you are not usually executing sound judgment. Right?
Melanie:And so the more that we make security incidents this, like, anomalous thing that requires, you know, some special, like, binder, you know, of information, the the weirder it is for people. And I just I want people across the company to go, yeah. I know how to do this. It's just like all the other incidents that exist in my world. The difference is is security is in charge of this one.
Melanie:Right? I'm taking the lead from security, but I understand the process, the workflow, the approval process, all of like, that doesn't that shouldn't be different. Right?
Kevin:Because that that's the same one we used last week. Yeah. I I think there is so much value in those crisis situations in being able to fall back on your process where you're like, you know, I'm freaking out about a thing. So my, you know, my proper response you know, the thing that will cause me to stop freaking out about the thing is by falling into the response pattern that I know, which is the existing incident response, you know, program which we use, you know, every week for, you know, the site is down or whatever.
Melanie:It was interesting. I was having a conversation I was having a conversation with a friend earlier this week who also works in incident bonds, and he reminded me of this, like, old adage that I think I think originated in the military. I'm not a 100 percent sure. My military history is, like, Right? But it was essentially like you never rise to the occasion.
Melanie:You fall back to your training. And so, you know, very rarely are people actually exceeding expectations when it when they're under pressure. Right? And even so outside of security, I'm a certified rescue scuba diver, and we talk about the same the same concept all the time we refer to as task loading. Right?
Melanie:If you if you're expected to do a task under pressure in an urgent situation and it's unfamiliar to you, it requires a lot more cognitive resources and attention to complete that task, and you might not do it well. Right? So I want the response in an emergency to be as close as possible to what you do all the time so that it doesn't require your entire focus and attention to just do this one thing.
Kevin:All of the rituals that we build around an incident should be a comfort, should be a a, like, thing to fall back on and not a thing that where you, like, you say have to pull out the binder for the first time and start leafing through it because you're already existing in a regime of not abnormal operation. And you don't wanna add additional abnormality on top of that. That just, like you say, positive people's heads to explode.
Melanie:Yeah. I mean, of course, I hope I hope that every single day, your process isn't going up to the most severe
Kevin:Right. Like, recession. Every day.
Melanie:Yeah. Exactly. Exactly. But you should be able to manage, like, some type of sev every day.
Kevin:And that's not also a failure of your company, a failure of your your process. Incidents are part of the law.
Melanie:Failure of the Internet.
Kevin:Well, it's it's a, like, reality kind of will throw curveballs at us no matter I
Melanie:just don't know I just don't know what else you're doing if not this every day. Right. It
Kevin:Well and and so you are doing this on some level and with some process, however Yeah. And
Melanie:Yeah. And and, of course, you don't want the same people on your team constantly thrown into the fire. They are going to burn out peep but but, again, it's as an organization, as a living organism. We need to be able to ebb and flow and have that resilience so that people can step into those roles for each other so that people get a break. Right?
Melanie:Which means this can't be something that only lives in your head that is, like, such ingrained institutional knowledge that another person can't step into their role. Right?
Kevin:I need to go on vacation ever and yeah. And it was like being, you know, sometimes I was the, you know, subject matter expert. I was coming into somebody else's incident. And sometimes I was running the incident. Or on the same incident, sometimes I would, you know, step back and forth in those roles as the needs arose.
Kevin:And so the fact that we all at Akamai got really practiced with that was a real strength of the organization, I felt.
Melanie:I I remember the exact moment in my career when I realized that this was something I had to focus on for my own role. I was actually so it was while I was at Uber, and it's crazy that it took me to my role in Uber to actually have this epiphany. But I think the reality is is earlier in my career when I was young and less wise, I made myself available for everything. Right? And it wasn't until later in life that I was like, you know, doesn't always need to be me.
Melanie:And so this this moment I had this opinion, I remember I was actually diving a shipwreck, in the Bahamas, and we're about a 100 feet below the surface. And I just remember and I don't know why this thought pained me, but I very clearly had this voice in my head that was like, if Uber has a breach right now, there's nothing you can do about it. Like, eve because even if I wanted to end my dive and go to the surf, Like, even if there's some way for me to find out a 100 feet below the surface that Uber was having an incident, I can't shoot to the surface. I have to do my safety stop so I don't get decompression sickness. Like, there is just I cannot control how quickly I get to the top and retrieve my phone from my dry bag.
Melanie:Right? And in that moment, it was both, one, motivating that I needed to get my shit together and make sure that my team could fill in for me. And number 2, it solidified that scuba diving was forever going to be the thing that balanced my work with my life because when I'm underwater, there's nothing I can do about it. And I found that to be the most freeing experience of my entire career to realize that if I have my ducks in a row and I have prepared my team with the the plan that gets them 99.9% of the way is if I'd done it myself, I can go diving and not worry about what happens because I know my team is going to do a stellar job.
Kevin:What goes into that plan?
Melanie:Hopes and dreams, Catherine. Hopes and dreams.
Kevin:Okay. Yeah.
Melanie:No. I'm kidding. I'm kidding. So a lot of it has to do with, like, building a process with your cross functional partners, so that my team is not responsible for everything that happens, obviously. Right?
Melanie:We have very clear
Kevin:security. Oh my god. We touch everything.
Melanie:Yeah.
Kevin:Like yeah.
Melanie:Yeah. So having very clear roles and responsibility in terms of who owns what, who has to sign off on what, who gets looped in, who actually gets to have an opinion. Right? Like, I think one of the biggest problems, especially for an inexperienced person who's thrown into an incident like this, is you don't know whose opinion you're supposed to listen to. Right?
Melanie:My team has very clear instructions on here's who you actually here's who has to say yes. If anyone else has an opinion, you can consider it, but they're they're not an approver. So
Kevin:They don't block anything.
Melanie:Yeah. Exactly. Exactly. So if if they have a great idea, good suggestion, great. Let's incorporate it.
Melanie:But if And so you really have to be able to, like, empower people to move quickly, which means that it needs to be very clear who makes what decisions, what the SLA is for people who need to contribute to that decision. Right? There's gonna be some things in an incident that only your legal team can decide. Well, my legal team can't take 4 days on some of these things. Right?
Melanie:So we determine in advance what is your role, where are you the gatekeeper, and in the places where you could potentially be the bottleneck, how quickly are you willing to respond? Because if you can't respond as quickly as we need you to, then we need to have a conversation about whether or not the organizations are staffed appropriately in order to enable that velocity. Right? I will be the first one to go to bat for headcount with a cross functional partner who's like, I'm the only attorney at this whole company. I can't turn this around in less than 2 hours.
Melanie:Well, let me go to bed for you and figure out how to fix that because you have to respond in under 2 hours. Otherwise, it's gonna be bad for you and for me. Right?
Kevin:So yeah. I mean, I guess, building the the incident response plan and then, you know, using that as part of the day to day.
Melanie:And this is why preparation happens so far in advance of needing it. Right? Because you can't in the middle of an incident like, I hate the saying that's, like, never waste a crisis. I hate that. I'm I'm no one no one is building long term infrastructure during a crisis.
Melanie:They're just trying to put the fire out. Right? I need you to be thinking about, again, what are the long term outcomes that you want to see from this program? Let's build that now so that we can quickly make decisions in the moment and be proud of where we end up.
Kevin:It's not just financial risk. It's not even reputational risk the way risk management people understand it. It's, like, relational risk on a very human level.
Melanie:Yeah. Well and even, like, the the risk of a misunderstanding can be incredibly costly. Right? Eve even if you are really trying to do something good. Right?
Melanie:Maybe maybe it's not anything malicious. But if you didn't communicate it clearly, now there's a huge misunderstanding and a blow up on the Internet that you have to, you know, diffuse. I don't know. Could we have spent, like, 4 more minutes carefully crafting that message so that it was clearer? Right.
Kevin:Yeah. So it sounds like some of the work was, like, having you embedded fairly closely with these engineering teams who were doing bug bounty triage to help them sort of craft the message and, craft the relationship. And with that, you know, keep that North Star in mind of, like, the relationship that you wanted to have with the researchers on an ongoing basis. And, just a little support.
Melanie:Yeah. Well, and I I think you can really divide the value of of my role sitting so closely with those folks. It being 1, I was a constant reminder of thinking about those those long term outcomes. Right? But, also, I could just be a sounding board for them to go, does this say what I think it says?
Melanie:You know? Like, how's getting a second opinion of you because because, again, if we're thinking about, should this become public? It's gonna be read by more than just somebody with, like, technical background. Right? How will this read to a journalist?
Melanie:How will this read to our bike later? How would this read to a customer?
Kevin:I we are in the habit of doing that when I was at Akamai even just on an informal basis, you know, past the laptop around. I'm gonna send this email. Does this look kinky to you? And it made a big difference.
Melanie:I do that. I'm a communications professional, and I have people in my life and on my team where I'm saying, can I get a gut check before I hit send on this? Because I I need to make sure that it says what I intended to.
Kevin:And then building processes which support that in the organization and the organization giving people the time to take that extra 4 minutes rather than
Melanie:I mean, that that's a crazy measurement just in general because fewer tickets isn't necessarily a good thing. Yeah.
Kevin:Right. Yeah. Yeah. In the same way that fewer incidents is not necessarily a good thing. Like, if
Melanie:it Fewer declared incident or fewer Exactly. Fewer discovered incident.
Kevin:Fewer discovered incident. Yes. Yes. Yes. That's the real trade off we're making here is just not acknowledging when things go wrong and or might be going wrong.
Kevin:Yeah.
Melanie:And and I think a lot of times peep people are hesitant to, like, make some to call something an incident because they're like, oh, that means it has to become just like the and I was like, if you're really honest about the fact that sometimes we we have to put more attention on this and more attention on this, and that that becomes the world leadership to give people permission to focus on this instead of this. We're gonna slip on this because this is more important. Like, a healthy organization can move like that. Right? And when we turn any type of incident, whether it's an outage or security incident or, you know, a delete Uber situation, like, when that becomes, like, a special scenario where people are freaking out, it's a 1000000 times worse.
Melanie:Right? I mean, I I think one of the things that that I took away, I I will just say from from working with the bug bounty team at Uber in general, is just how important it was to have a variety of backgrounds and experience and disciplines represented in that realm. You know, myself and legal were often attendees at the payout meetings because we were thinking about expectations we were setting, things we may inadvertently be communicated by the payout amount or perceived inconsistency and things like that. And, you know, I want, you know, technical teams to know that they're not alone. There are cross functional partners who want to help and support you, but often they do not feel comfortable walking into that room without an invitation from you.
Melanie:So if if you would like some cross functional help, reach out to those partners and start inviting them to your discussion.
Kevin:Yeah. Yeah. And then start collaboratively figuring out how to communicate across those divides. A lot of my work over the last few years has been, like, translating from lawyer to engineer and vice versa because speak 2 very different languages. And, Yeah.
Kevin:That was that's, and eventually, you know, we each learned to speak the other's language well enough to communicate in those rooms, but it took a while and a lot of patience. But yeah. Yeah. Having Having those disciplines represented, especially in something like bug bounty.
Melanie:I think assuming good intent until proven otherwise goes a long way. I I worked with the the Duo security team after the acquisition to Cisco, and one of the the that team from Duo brought their values from Duo to the security organization in Cisco. And one of the ones that really stood out to me was be kinder than necessary. It changed the way people spoke to each other. It changed the assumptions people made about each other.
Melanie:And quite frankly, it weeded out people who didn't wanna be kind to each other. Like, who wants to work with those people? Right? And so, like, then to me was just really profound. I mean, it was something that, you know, the the duo founders, implemented when they started the company.
Melanie:And, you know, I'm not surprised that it lasted as that this value continued even after the acquisition because it it people clung to it because it just had such a profound effect on the way that they work together. And it's something that I think more of us could adopt, more of us could benefit from being kinder than necessary.
Kevin:Yeah. Especially in security.
Melanie:Which is also also really great for bug bounty communication.
Kevin:Yes. Yes. Yes. Yeah. And still managing to surface problems when they needed to be surfaced without, like, being jerks about it.
Melanie:Yeah. I mean, being being kind doesn't mean that we're not truthful. It doesn't mean that we don't raise issues. It means that we do it in a way that is respectful of each other. Level.
Melanie:Right? I'm not going to hide a problem that I think is going to bite us later, but I'm going to be very respectful and deliberate in the way that that I communicate that to my peers, to leadership, you know, even to our clients. There are times where we have to give them bad news or let them know that something terrible is coming around the corner. And doing that in a way that is kind actually helps people focus on the real issue rather than the emotion.
Kevin:Mhmm. Yeah. That's a good point that, yeah, We get deeper into the problem. We're able to engage more fully and completely with the problem as the problem rather than, it's not it's not obscuring the problem. Yeah.
Kevin:That's really the point.
Melanie:Right. Or or distracting people from, you know, like, if if I have hurt your feelings, that's what you're gonna think about for a long time rather than the answer to the question. Right?
Kevin:Monie, where can people find you online?
Melanie:Great. So the easiest place to find me online is LinkedIn, and I'm also on Mastodon. My handle is Wednesday at defcon.social.
Kevin:And your company is Discernible Inc. That's it. Link in the description below. If you want to hire Melanie to help you, and your security and privacy teams communicate with each other and with internal stakeholders and external stakeholders. Yeah.
Kevin:Melanie, this has been really lovely.
Melanie:Yeah. I've really enjoyed it. Thank you so much for having me.
Kevin:Yeah. This has been the War Stories podcast on Critical Point. I'm Kevin Riggle here with Melody Ensign, formerly of Uber. Now doing her own thing. And till next time, folks.
Creators and Guests
