Critical Point War Stories with Kevin Riggle: Software Engineers Talk Breaking Production

[music]

Kevin: Hi there folks. My name is Kevin Riggle. I run a little cybersecurity consultancy called
Complex Systems Group. This is the "War Stories" Podcast on Critical Point.

We are here with Willie Williams to talk about that time he broke production because that's what we do here, telling incident stories in public. Willie, do you want to introduce yourself a little bit and then tell us what it was that got you into a place where you could break something?

Willie: Sure. I'll go in reverse order, which is right now I'm the CEO of Mori, which is a genealogy startup. Prior to that, I've spent a lot of time in Silicon Valley and San Francisco doing startups. This story actually takes place way back when I was at MIT. I was actually a summer intern. [laughs]

Kevin: Summer interns, yes. Those are dangerous. [laughs] Where were you interning at?

Willie: Akamai, at the time.

Kevin: I know what Akamai is because I worked there. We might have even overlapped. For our viewers who've never heard of it, which is a surprising fraction of people, what is Akamai?

Willie: Well, that's a great question now. Back in the day, because I'm sure they've evolved, back in the day, this is, let's say, circa 2004, coming out of the dot-com bust, Akamai was an awesome company that rose in the late 90s or early 2000s.

Primarily, what they did was, the way I used to describe it to folks, is, if you were going to run an ad [during] the Super Bowl, and in the ad, you were going to link to a website, you probably wanted the website to run on Akamai's servers instead of your own.

They had all this technology that allowed this large burst of traffic. They can handle it. Your service wouldn't go offline at the most critical moment. These days, Cloudflare is a great example. These edge network setups, CDNs, and whatnot. 20 years ago, that was...

Kevin: Content distribution network is the [buzzword]...Akamai was the first or one of the first, if I recall, and one of the few Web 1.0 companies to survive,although that was a near thing for many years.

Willie: They were MIT as well.

Kevin: The current CEO is still an MIT professor. When I would work career fairs, I would get people coming up being like, "Oh, you're Akamai. I took Professor Leighton's class," because he taught algorithms every other spring semester. People were like, "Ah!" [jazz hands], which was always fun.

What were you doing at Akamai? What team were you on there?

Willie: It was me and another intern. We were working on Akamai's geolocation services at the time. Again, transport back to 20 years where we aren't carrying...

Kevin: The Internet of...

Willie: In the Internet of, we don't know exactly where you are. We don't carry these little GPS chips in our 
phones or all the devices type of thing. Most people are on desktops even, rather than just laptops.

Kevin: Probably on dial-up still in 2004.

Willie: Yeah.

Kevin: We had DSL, but a lot of people still on dial-up.

Willie: It was very special [laughs] to have university speeds, I remember that. I have another great story from a friend. Maybe I'll tell you afterwards.

Akamai very smartly figured out that if you could determine where a computer was sending the request geographically, they could route the request to the nearest server they had. Allowed you to drop latency and also give information on maybe where they should put more servers in certain data centers.

Kevin: That's how you handle those Super Bowl spikes, is by rather than sending all of the traffic across the country to your server in, without loss of generality, Cupertino, you send it to a server that is closer to the user, and so you save the backbone for only the important stuff.

Willie: As Google was actually proving out right about that time. Latency has a qualitative effect. Its distribution. If one tail of distribution goes above a certain level, it's not that those users just wait around for that response to come back. They actively go and do something else. The faster you can make the websites, the more users engage and work with them, and the better your Super Bowl ad will perform.

The system that was built for doing this was [laughs] Akamai just had logs, and logs, and logs of ping traces, traceroute data, that they had from everywhere. It was just strings.

Kevin: From here to here on the Internet, it takes 50 milliseconds to get to this hop, to the next hop, and 50 milliseconds...Then five hops later, you're at the...This was just traceroute data between basically all the servers that Akamai ran?

Willie: Exactly. What it also would do is, within there you had names. You would see like, if I do this from my dorm, you would see I have my IP, but then you could see like mit.edu. Then it would go up to the Cox network and then over and then back down into the local ISP.

It turns out that those names provided signal information. The project was to map these hostnames to locations. We were one part of an effort, which was like, figure out a general area, and then they could use, they actually did a very cool thing where they would triangulate where the final location was because 
they could send like three pings or a series of three pings and between three geographically distributed data centers. Given the response time, you can draw those three lines and see, "This person is probably in Monrovia, Iowa," or something like that.

The interns... They had [already] set most of this up. We knew what was going on, but it's an intern project. They were like, "We need you to improve on the system. It's really great. We happen..."

Kevin: They'd already figured out, that like, an mit.edu IP was probably localized to the Cambridge area, Cambridge, Massachusetts, but the question was now, how do we do one better than that?

Willie: Yeah. They realized this technique had produced some fruit. I might say it this way. They were like, "Well, we'd love it to produce more fruit." Day one was literally we sat down and they were like, "Here's what we got." We opened up the file and it was a 10,000 line, 20,000-line Perl script.

Kevin: Love it.

Willie: [laughs] I don't remember who wrote it, we were having to...

Kevin: It was 2004 and it was Akamai. That was how we did everything back in those days.

Willie: That's how we did everything. As a side story, I was in the MIT Media Lab and 
we once lost a server, for anyone who's never lost a server where you're like, "We know it's here. We just don't know where it is." [laughs]

Kevin: It pings on the network. We just don't know physically where the box is.

Willie: These are the days they were.  We looked at this script and we're like, "We should do something better." Like all computer science students do at least once in their lives, we were like, "We're going to design a language to do this, a domain-specific language to allow us some primitives, I know, [laughs] to allow us some primitives to stuff."

Kevin: It was in the water, like the Ruby on Rails folks were just getting going. It was like, yeah.

Willie: We kind had progressed through the summer. We took this approach. We progressed through the summer. Actually, it worked pretty well.  One day, I'll say it was probably about two-thirds of the way through the summer, we walk in, and, the Akamai office was not a high-activity office. They used to have the NOCC, which was their Network Operations Command Center. That looked cool. That looks like something that was sold to every single college student because it had all these screens. It looked like something out of a Bond movie, but...

Kevin: It was Mission Control.

Willie: Yeah. Our area was not. Our area was, we had a lot of fun. Everyone was great. It was quiet. It was a bunch of engineers working.

Kevin: Cube farm.

Willie: I'll say it's a Tuesday. We walk in on this Tuesday and it's probably late July and our office is on fire. People are running back and forth and up and down and [I'm] thinking. I turn and I'm just, "Yeah." [laughs] I'm like, "Folks are doing stuff."

Kevin: Something is happening.

Willie: We had our own little project, our own little office. They stuck the two interns in the little closet. We worked in there and drank soda.

Kevin: There we go. [inaudible]

Willie: I got in and spent about like an hour working and just outside the door, we could see folks running back and forth. Running back and forth. No idea [what's going on]. [mimes looking back and forth]

Kevin: [laughter]

Willie: After about an hour and a half, our boss, who's a great fellow [laughs] and he's, I wish I could 
hold it. He's holding a phone. One of the old wireless phones, and peeks his head into our office.

He's like, "Hey guys, how are you doing?" I'm like, "We're good." He's like, "Did you guys happen to push a change last night?" We're like, "Yeah, probably." [laughs]

Kevin: Pushing them regularly.

Willie: "Yeah. It's like right before we went to bed." He's like, "Oh, interesting."

"Did it have to do with like the domain name project, like, the hostname project with the pings and all the stuff?" It's like, "Well, yeah. That's the only thing we're working on."

He's like, "Huh, when did you push that change?" We like, go and look at the log. And he's like, "Stay right here for a second." His office was right next ours. He walked back to his office, hung out, came back five minutes later.

He's like, "Do you know...? The best part of that story is if I had this guy's name, but I still cannot remember it. He's like, "Do you know who so and so is?"

We're like, "No." He's like "You should look him up. Google him." Google is still a new word, great. He was the CTO of Yahoo. He's like, "How much is he worth?" He was like $2.1 billion or something at that time.

Kevin: Back when Yahoo was still a big deal.

Willie: Yeah.

Kevin: Sorry Yahoo.

Willie: We're like, "Cool. We still have no idea." Then he reaches up and unmutes his phone and all you hear is just this stream of yelling, cursing, like flowing out of this phone. You can hear voices and it's just punctuated by more yelling someplace. [laughs] Then he's like "Mute."

He's like, "So that's going to go away  in a second, because we figured out what the problem was. The problem turned out to be that the change you pushed last night filtered through our system, but the end result was we ended up...our systems believed all the traffic coming from Yahoo was coming from Yahoo Japan and so we routed...

We were routing everything back over the Pacific and then back over to the US, [laughs] which is, at the time, burning them, I'm sure, millions and millions and millions of dollars. [laughs]

Kevin: Those exchange costs are not cheap even today.

Willie: Yeah. The Internet of 20 years ago was not as...didn't have as many people on it. The advertising dollars were percentage points of what they are now and it was still a lot of money.

He was like, "Don't do that again." [laughs] I think to Akamai's credit, we never heard another thing about that ever again. We finished our project. [laughs]. Went on with our lives.

Kevin: If you hadn't done it, somebody else would have. That's the nature of that kind of work. Can you say a little bit more about what the change was? You had pushed out a new geo-mapping database or...?

Willie: Yeah. Roughly, we would make upgrades to our system for roughly mapping text to location, so we would look at this string. We had a set of levels of location, country, city, town levels. Later on, I would call them geocodes or whatnot.

Then we just had this traceroute data, these traceroute lines. We would, to the best of our knowledge, try and get as close as possible.

Kevin: You're actually doing this heuristically. You've got what was a line in the Perl script that said, "If this traceroute line matches .mit.edu, then return Cambridge Massachusetts"?

Willie: Return Cambridge, Massachusetts, or Zip Code or something like that.

Kevin: You've replaced this Perl script, this god-awful monstrosity of a Perl script with this DSL for describing these rules, and so you ship out a new version of this. This isn't running on every request.

Willie: Well, so, our little part is not, right. What they would do is they would batch process it and then store it in a key-value store. Then when they would see this data come back, they could very quickly do these lookups.

Kevin: When the DNS request comes in, there's some like, which IP address should we return logic, which is trying to figure out which IP is going to be closest to the end user?

Willie: I should also mention that our work was...There were two facets. One was converting this Perl script over into a domain-specific language, but the other one was actually doing the work of building heuristics for adding new mappings.

I think this is ultimately where we actually had screwed up, where we were looking and doing something and it's... [laughs] Actually, it was vaguely reminiscent of...Did you ever play SimEarth?

Kevin: Yeah, a little bit. A long time ago.

Willie: Way back when. It was kind of like the second or third game in the series. The best part about SimEarth is, if you had forgotten your key or your password, they would give you a trivia question. It was like, "How many moons does Saturn have? or something like that.

You could give a response and then you could play the game. This is back in '93, '94. There's no Internet to  look stuff up at and you don't know what the latest technology is. I remember being a kid and being like, "I want to play this game. I need to..."  It was like I had to ask for permission to get the password, but if I could answer the question, I was going, I was good.

Kevin: That's... [laughter] diabolical.

Willie: Yeah, great fodder for a certain type of kid.

I remember, and so you'd go and you would try and find this information out, looking at books, whatnot. Hit rate of like...because the game's view of the world sometimes got out-dated.

I was playing a year or two afterwards, sometimes they discover a new moon of Saturn, but if you look it up, it's like, "This number is 14," when it's 15. Anyway, the Akamai work was a little bit like that, where you'd see a host name and you'd be like, "What do we know about this in the world?"

I learned a great many things with this lesson around like how you should write tests, what deployment processes look like. I read a lot of the things that really helped my future self. I guess that's what college and college internships are for.

Kevin: Yeah, the good ones, which was always a strength of Akamai.

Willie: Yeah. [laughs]

Kevin: One of the things that Akamai I thought always did real well, and that we tried to do when I had interns, was to give them real work and to sometimes put them in places where they could break Yahoo, because it meant that they were working on things that mattered to the people who worked at Yahoo.

Willie: [laughs]

Kevin: Something about the understanding of the world had gotten out of date relative to the script or the...

Willie: The reality is, we had just made a mistake. We had somehow been...Somewhere in there, we were like, "This hostname looks a lot like it should be in California, when it's really in Japan," something like that.

Kevin: Interesting. As an example of that, we talked about MIT, back when you and I were at college, if your IP address started with 18., so 18.1.2.3, or 18.27.., or...I'm trying to remember the IP address of my server in college, but whatever it was. 18.243.0.151.

Willie: [laughs]

Kevin: Anything 18. or after meant that it was an MIT computer, and so you could infer things about it.

Willie: We had that rule.

Kevin: Yeah, exactly. Within the last 5 or 10 years, a bunch of that IP space has gotten sold off to AWS. Now you can no longer make that assumption, right?

Willie: [laughs]

Kevin: We keep spinning up EC2 boxes with some of my client work and I see that it gets an 18. IP address and it makes me feel things.

Willie: [laughs]

Kevin: Because of how IPv4 worked and how IPv6 adoption never took off, those IP addresses got increasingly valuable. Akamai owned a lot of them and AWS owned a lot of them for the work they did. MIT owned a lot of them and didn't need nearly as many of them as we had.

As an example, I know that there are still servers around there that do...because I was responsible for one until a couple of years ago, which did a certain amount of, "If you have an 18. IP address, that means you're probably an MIT student or affiliate, and therefore allowed to access these things." I'm not sure that that got updated when those IPs got reassigned.

Willie: [laughs]

Kevin: I'm lucky that that did not result in any traffic getting routed to Japan accidentally, but exactly the same kind of change.

When you were deploying things, what did that look like?

Willie: That looked like some cowboy stuff. [laughs] There were a couple of different failures of process here. We used version control, but it was CVS.

Kevin: Interesting.

Willie: Just a sign of the times.

Kevin: We were on Perforce in my era. That's also the best version control system that 2004 can provide. [laughs]

Willie: Again, to Akamai's credit, they gave us a lot of leeway on this. We didn't have a release checklist, we didn't have any kind of process where we were going to sign off on this. The thing that the team learned was that, they thought we were somewhat sandboxed. I don't think they were thinking that we were going to really...Their assumption was, if these interns make a mistake, [laughs] the worst thing it's 
going to do is drive up a little bit of latency in one particular spot, and worth it for the recruiting.

Kevin: If you'd been working on core mapper code, you might've been using more of the serious tooling, because I know that we had release tooling and version control stuff built out for that kind of stuff, which was substantially more sophisticated. You just threw the code over the wall and something picked it up 
that you didn't have to know a whole lot about.

Willie: The whole thing was very loose, which was great. We learned and we got a lot of room to try stuff. I know the fix was fairly easy, because [laughs] we could just go back and look at the heuristic mapping changes we had made, and roughly hit undo and roll that out. The biggest weight was, again, the stuff 
that was outside of our fog of war, which was [laughs] all of the real deployment processes that they had for doing these kind of batch operations.

Kevin: Yeah, because we would never have rolled that out to the whole Internet at once. It was physically impossible to talk to that many servers at once, so there were staged rollouts and a certain amount of alerting, although...yeah.

I find it a little bit interesting that they didn't give you too much of an explainer on what 
the backend was and how this was going to fit in. I'm interested that they were just like, "Well, it's this little piece..."

Willie: It helped us focus down. We had this very specific problem. It was like, think about this one. 
Also, I would say, with a grain of salt, this is a story from 20 years ago, they might have actually done it.

Kevin: Fair point.

Willie: [laughs] It was summer, and this is two months in and is the bright...not the bright spot, but the burning part of the summer or of that internship. I'm sure there were other parts where they may have walked us through some of them. It's like most things, like when you onboard into a system, you don't...Without a lot of the context, some of the onboarding isn't as effective as it could be.

Kevin: Necessarily. Some of the reason, some of the way that you learn things as you spend more time working in a system is when things break. You're like, "Oh, that's how that works." [laughs] "Whoops!"

Willie: Exactly. [laughs] "Those two things are linked? I didn't know that."

Kevin: That is, as we get into the system safety stuff that I'm always on about, that is the whole work of system safety, is discovering what things are potentially interacting that you don't think are potentially interacting, and then figuring out how to constrain that.

I assume one of the constraints was that they maybe put some...Did they put some controls in on the rollout of these things or on what kinds of changes were being made, or was there an evolution of the process after that?

Willie: Not visible to us, by my memory. [laughs] I think that the contract and the abstraction of the internship stayed pretty much the same. "You work on this, we map more of these unknown hostnames to geolocations, and focus your energy on figuring out how to do that more efficiently or faster or whatnot, and then we'll make sure the interns don't footgun themselves." [laughs]

Kevin: Footgun the entire company. It cannot possibly have been the first time that we pushed out a bad mapper change, and it was not certainly the last, because the... [laughs] Stories that are not mine to tell.

Like, "If a couple interns who are working on a fairly constrained project can do something with this kind of impact, then that means that we need to think harder about the systems that we're building. We have the 
best interns available," [laughs] "so replacing them with new interns is just going to..." You never made that same mistake again, did you?

Willie: No. It's also funny because later on, as I had interns, I always thought of this story in terms of where responsibility lies and what you want in terms of...I liked the leeway that Akamai gave us. It felt very responsible and adultish, like, "Hey, we're gonna give you a real problem that we work on and that matters."

I also remember the fact that sometimes interns just do things. [laughs]

Kevin: You were what, maybe a rising senior then, so you were at most a year or two out from working on this kind of thing full-time? There's just not that much difference from being a rising senior and a new grad hire. They could have given you exactly the same project as a new grad and had exactly this.

That was also, I don't know, for me...I deleted the entire source code repository of my first internship, right before the boss was due to get on a plane to come back to the States from England.

Willie: [laughs]

Kevin: Similarly, I was working...They put me on real stuff, I was shipping code to customers from day one. He was obviously not happy, but he also walked me through the process of recovering it [the repository] 
and emailing out to the project list, being like, "The repository is gonna be down for a day while we do this."

He didn't throw me under the bus. He was like, "We need you to be part of the recovery process, but this is..." When I had interns, that was something that I kept in mind, of what should I think about my role and responsibility towards these people and their work, is very similar.

Willie: It was also very good for me, personally, that particular internship, because that was this transition 
where...because I spent a lot of time, for example, at the MIT Media Lab. I love the Media Lab, and the Media Lab's great, but the Media Lab version of shipping to production was a demo.

Kevin: Demo day.

Willie: Demos have this whole other thing with them, about how they fail and whatnot. It was like a point-in-time success. If it worked when you demoed, it worked.

Kevin: Day of.

Willie: Day of. Coming up with it, I'm like, "OK, this..."

Kevin: Sponsor Day, that was it, yes.

Willie: Yeah, and then it didn't have to work again. [laughs] It didn't need to work because you were onto 
the next thing. The Akamai internship I feel like was one of the turning points, where I was like, "Oh, in real systems, it doesn't work just in that point in time," it's like, "for here and forever more" type of work.

Kevin: What additional kinds of work, what additional magnitude of work is required to take this from something that works day of, when you're standing in front of the sponsor, to something that can be 
operating 24x7 at global scale.

Willie: You start to realize that there is all of this other tooling and scaffolding and expertise 
and process that goes into it, and that for a whole set...I've got a lot of Lyft war stories as well.

Kevin: That's a whole 'nother episode. [laughs]

Willie: That's a whole 'nother episode.

When you're a builder and you're starting out as a builder, you're like, the building is the thing. There's a transition point I think that most folks should go through as software engineers where you realize that cost-wise, maintenance is the thing. Uptime and understanding, and keeping it up and around is actually the dominant cost. There's a lot of expertise there.

Kevin: As the system becomes more successful and spends more and more time in production, that prototype period will become a smaller and smaller fraction of its total operational life.

There was something you said, a thread I wanted to pull on about...How also did you find the difference between working in a place like Akamai and being a CS undergrad?

Willie: All adult jobs have a very different feel to them than when you've never had one before. It's always something amusing, but I also have to talk a lot of first-job mentees or interns through around, you show up and you work in college and in the labs, it's a little like bursty work.  I'm just going to drink Red Bull—well there's no Red Bull back then, but drink Mountain Dew Code Red and stay up all weekend and work on this project.

It was a different experience to be like, "OK, you're in at this time. You're expected to be out at this time," and you need consistent throughput. There's lots of philosophical things I can get into [laughs] around that. I'm not sure that's the right way to do it, but that's the way it was 
at Akamai. That's the way professionals do it.

That was one of the biggest changes was because sometimes you're just there. I actually have a semi-funny story about this, which was, as I was going through this transition, I rebelled. I was like, "Literally, I have no good ideas right now." It's like one of these days where I was like, "I had no good ideas."

Of course, I'm really poor at sleep habits. I'm really tired. At the time I didn't drink coffee. I was like, "I need a nap." I'll wake up, I'll wake up refreshed, and there was a chair, that was a reclined, lean-back chair, right in front of Tom Leighton's office.

Kevin: [bursts out laughing] He was not the CEO at that point, but he was something else.

Willie: No. I think he was the CTO, and he was always gone and whatnot. He was always gone.

Kevin: High muck-a-muck.

Willie: [laughs] I mean, whatever. I was tired. There was a chair, I'm going to go take a break. [laughs] I'd never seen him. He'd been gone traveling, whatnot. Very turbulent period.

Of course, I'm sleeping when he returns to his office from some piece of travel and looks down 
[laughs] at me and he's like, "Rough day" or something along those words.

I'm like, "Yeah, something like that."

Kevin: [laughs]

Willie: Then he goes into his office and I'm like, "I feel like I should not be here right now." [laughs]

Kevin: It's time to get back to work. All right. Nice.

Willie: It's time to get back to work. [laughs]

Kevin: I took a couple naps. I took some naps at Akamai. We had a big beanbag in InfoSec, and that was occasionally the only way I was going to get the rest of the day's work done. Also similarly, again, to Akamai's credit, he didn't string you up by your toenails for slacking off on the job.

Willie: Exactly. [laughs] [crosstalk] Go ahead.

Kevin: There was a social, like, "ah," [laughs] about the expectations.

Willie: It was general and there were so many smart people there that I enjoyed it. It was cool. It was cool and it was fun in a way that 20 years ago, it was kinda cool to be a programmer, but not like it is today. Here were a bunch of people who cared about cryptography, which was the thing I really cared about back then and cared about computer security. As an intern, you want to bathe in it.

Kevin: Also, there were not that many places you could go in 2004, which were working at the scale that Akamai was at. Google was starting to get there, but... Maybe one of the other CDNs, I think Limelight was still maybe a going concern at that point, but otherwise there just wasn't...Webscale was not nearly the thing that it became even five years later. And certainly today. Everybody is running globally distributed systems.

The kind of like, "Oh, you have servers in where?" thing was not nearly such a common thing as it has become.  There weren't all that many places to go to...

[crosstalk]

...experience that kind of work at that kind of scale.

Willie: Today it's still amazing because, like many things I feel like I anchor a little bit on my youthful experiences. Sometimes I look up and I'm like, "The world is magical." [laughs] We get globally distributed software and you barely have to work for it. This is amazing.

Kevin: I was on a call, I scheduled it via Slack, and I used my Calendly link. We log onto the call and I'm like, "Oh, where are you based out of?" The guy is like, "Oh, I'm based out of Istanbul, Turkey."  I'm like, "I had no idea, you could have been next door." That's wild. When it's good, it's really good. [laughs]

Willie: Actually, probably for good reasons, several of the startups I've worked on, I now see the versions of...because they didn't work, and I see the versions of what we have now, and I'm like, "The world's amazing." We got there, it just didn't happen to be us type of thing.

Kevin: Wrong place, wrong time. The number of companies that were 5, 10 years too early for something for some reason is like...they could see where the world was moving, it just wasn't there yet. [crosstalk] Go ahead.

Willie: I was going to say, the other interesting thing about Akamai that I think back on now but I didn't think too much about then was the feeling of what technology felt like in the dark winter of 2003 and 2004.

I remember walking. I didn't have a context space, but I remember walking around the Akamai office. A lot of people had the Akamai stock price up. [laughs] At one point, I'm sure the price was many multiples of where it was then, but they were tracking it. I remember thinking, I was like, "That doesn't feel healthy." [laughs]

Kevin: No, it's not.

WIllie: When the stock price goes up and down.

Kevin: It's not. That was a very '90s boom thing, and Akamai got kicked in the teeth on that one a couple different ways, but also a story for a different episode.

That was an interesting time. There was a long moment in there after the '90s boom. There was a long hangover of still '90s approaches to building technology as we were busy trying to figure out how we do 
this better and open source is maybe a thing, and all these, but hadn't yet taken off again and not till, what was it? 2009, 2010, after the financial crisis, did we start to see the next big upswing that we are just coming off now.

It was a different time then. I'm both nostalgic for parts of it, the transparent electronics and the computers that weren't connected to the Internet and so didn't need continuous software updates. The Internet was a small village rather than a metropolis to put New York City to shame.

Willie: I actually have a map of one of the things in the '90s where they sold traceroute maps, but the clustering. There was like... the military network over here...

Kevin: I remember that!

Willie: I remember getting it in probably '95, '96. I was like, "Oh, this is so cool." I have it at home. I haven't looked up at what the multiple is for number of people online in '95 versus 2023, but it's interesting to see 
if that would even appear as a pixel. [laughs]

Kevin: Right, right. [Like] the Internet of that era is the size of a quarter of the map of the Internet as it is today [or something]. I don't think you can even necessarily build a map like that because if you do traceroute, you lose all of the carrier-grade NAT stuff and everything behind people's home 
NATs and blah, blah, blah, blah, blah.

Willie: That was the other thing I remember about this job was...The thing they told us on the first day of this project was ...AOL is a black hole. They were like, "We get nothing."

We have the graphs and there's this huge chunk of the bar chart of where we get traffic from. [laughs] The biggest chunk was, we see nothing. We know nothing about AOL. The traffic goes in and comes 
out, and we don't have any visibility.

Kevin: Interesting. Do you know why that was?  Was it just that they had turned off the ICMP or... I forget what packets it is that traceroute uses, or...

Willie: I don't remember now. I knew back then why they had done it, but I don't remember now what the 
cause was. I'm sure there's a Reddit post somewhere [laughs] that explains it. I'm refraining my search hand, like... [laughs]

Kevin: Exactly. Maybe one of the listeners to this podcast will be able to fill us in on the details. That 
would be, if you have some details, or you feel like we nerd-sniped you and you can dig up that Reddit post...

Willie: I'd love to hear it. [laughs]

Kevin: Do please post it at the comments. Awesome. Willie, this has been great.

If people want to connect with you, where can they find you online?

Willie: Probably the best place to find me is on Twitter @Bigwilliestyle.

Kevin: Great, love it. Twitter @Bigwilliestyle.  I am Kevin Riggle, @kevinriggle on Twitter as well. Complex Systems Group is my little cybersecurity consultancy. If you want to talk about how your systems break or how to avoid them breaking, do please reach out.

With that, this has been the "War Stories" podcast on Critical Point with Willie Williams. Thank you so much for your time, Willie. Really enjoyed this.

Willie: Pleasure, thank you.

Kevin: Til next time folks, take care. Thank you so much for listening.

If you enjoyed that, please like and subscribe down below. We're still getting the channel going, and we'd love to know if people want to see and hear more from us. If you have an incident story you'd like to tell here, please email us at hello@complexsystems.group.

Also, we're always looking for people who aren't cis White dudes like me to tell stories. If that describes you, please feel particularly encouraged to reach out.

Now, this one's a bit of a long shot, but I just finished reading a book about DEC, about Digital Equipment Corporation, which was a major computer manufacturer back in the minicomputer days when computers were the size of refrigerators and coincidentally, where my grandfather worked in the '70s and '80s.

If you worked for DEC, or Data General, or any of the other computer manufacturers of that day, and you have an incident story to tell, please reach out, because software incidents didn't just start happening after the Internet.

You can find me on Twitter @kevinriggle and on Mastodon @kevinriggle@ioc.exchange. My
consulting company, Complex Systems Group, is on the web at complexsystems.group,

And with that folks, til next time.

[music]