He Won George Hotz's Money - Andrey Petrov
Download MP30:00
(Andrey) You up for one more?
(Kevin) Yeah. Absolutely.
(Andrey) All right. So the last one features George Hotz, which some people might remember from hacking the original PlayStation 3 and the iPhone...
(Kevin) I remember him from, like, telling Elon he would solve all of Twitter's problems in a week.
(Andrey) He tried.
One thing most people might not know is he's also been on and off involved in some Ethereum stuff.
One of the interesting attributes of Ethereum is this collectively-owned computer, essentially. We have this -- these guarantees where all the code executed on it, it will be 100% right because every node verifies it.
(Kevin) For some values of right.
(Andrey) One of the, kind of, the constraints, in order to maintain these decentralization properties, and for everyone to be able to run a node at home on their laptop, there's a very -- there's a fairly small limit on how much throughput it can handle.
People often make fun of it for having something, like ten transactions per second, but turns out we can solve it by this -- this innovation called rollups, and essentially, we can run the transaction somewhere else, and we basically commit to a batch of transactions on Ethereum, and we can prove that they executed accurately.
So, back when these rollups were first, kind of, designed slash discovered, the theory of it was fairly well established, but the way -- but implementation was still kind of a long road ahead.
The people working on these rollups, they're hanging out in, like, a San Francisco cafe, and they're just, like, bashing their heads, like, how are we gonna build this. They turn and George Hotz is there next to them.
They know he's really smart, so they just, like, start, like, brainstorming with him, and he gets nerd sniped, and they basically convince him to build a proof of concept of a rollup in exchange for a very generous amount of money, but the term that they came up with is that they're gonna do an initial bug bounty, and a, like, $50,000 bug bounty, but the money's gonna come out of his compensation if there's bugs.
Long story short, we got $50,000.
(Kevin) Howdy, folks.
We are back, once again, on the War Stories Podcast on Critical Point.
Today, I am here with a friend of a friend, who I met through Twitter, Andrey Petrov.
Chat about that time he broke, uh, production. Uh, 'cause that's what we do here. Or several specific varieties of production. I think we have a few stories here, maybe, to chat here. So, very little preamble today.
As always, we are available at WarStories.CriticalPoint.TV on all your podcast apps. If you are watching this on video and on YouTube at CriticalPoint.TV, if you would like the video, but I'm excited to get stuck in, and don't have too much more to say to that.
So, we will roll the titles and get started.
[intro music begins]
(Kevin) And we're back! Uh, once again, this is the War Stories Podcast on Critical Point. I'm Kevin Riggle, and I'm here with Andrey Petrov.
Andrey, can you tell us a little bit about yourself, and how you found yourself in a place where you might break production?
(Andrey) Hi. I am an independent open-source developer. I've been largely building open-source things since... 2007 or so. Uh... One of my most known things is urllib3. It's an HTTP library in Python that's used by a lot of people.
(Kevin) That lies under, like, the Requests library, which is used by everybody, which is now baked into the Python standard library, if I recall?
(Andrey) Not super much, um --
(Kevin) Okay.
(Andrey) But, like, PIP uses urllib3, as well, so basically anytime you've done any kinda Python stuff, you've probably run some of my code.
(Kevin) I think, like, yeah, every time I see a Python exception, half the time, it's got your urllib3 in the somewhere, so yes -- yes.
(Andrey) Although, there's a bunch of other people working on it and, like, maintaining it these days, so I -- I know more do meta-maintaining.
(Kevin) Okay. Okay. Community management kind of things, or?
(Andrey) Something like that. Yeah.
(Kevin) Okay. Yeah. What got you into doing that?
(Andrey) Um, my first job, they've...
So, basically, we were uploading a lot of images to S3. At the time, we were actually one of the largest S3 customers. Uh, this was 2000... 8, maybe? And turned out, the state of HTTP libraries at the time was not great.
So, every request was, like, a separate socket, uh, separate everything.
(Kevin) Yeah. It was dire. I remember this. It was bad.
(Andrey) Yeah.
I estimated how long it would take us to upload all the images we had, and it was, like, three months or something? Uh, and they're, like, I can write a new HTTP library in less than three months. So, while that's going, I basically wrote what later became H -- urllib3, uh, and largely support for multithreading, support for using sockets, a whole bunch of stuff like that.
Uh, and then we got that down to, I think, like, three or four days. And they very generously let me open source it under my own name, and kind of
5:00
one thing led to another from there.
(Kevin) And so -- but that is not -- that's -- that comes sort of after the story --
(Andrey) No.
(Kevin) -- that we're talking about. Like, yes... Uh, because this story, if I recall from our conversation, starts in high school.
(Andrey) Yeah. So, originally... So, I started coding in my early teens. Um, and my high school had this really -- I went to this, like, big public high school. And they had this half-assed computer programming class that was organized by our gym teacher, who has never written any kind of code.
(Kevin) Okay. Oh, goodness.
Okay. Sure. Why not? Great.
(Andrey) Yes. But -- yeah, they kinda handed us a book, which I think was... Uh, we did some Turing, and I think at some point C? I chose to learn C. Uh, and that was going on. And then I was also, at the time, was working on my personal website, which I ended up using PHP for. And I was kind of... doing some work for our high school.
Our high school needed a -- they -- we didn't have any kind of software, and their rooms were constantly -- their, like, booking system was kind of this... just like a... piece of paper on the wall, and people were constantly running into each other.
So, I built them this kind of booking suite, and that they really liked. Um...
(Kevin) So I'm getting, uh, the sense that you're, like, maybe about my age, and you're doing this maybe in the early aughts? I'm -- yeah.
(Andrey) Yeah. This is, like, started around '99, and kind of the incident happened around 2000.
(Kevin) Okay.
(Andrey) Uh, and -- very important context. This is the age where people are using modems. So this is, like, 36K modems, maybe -- sorry, 33.6K modems, maybe 56K modems are just -- have just been released.
(Kevin) Okay. Yeah. Yeah. So, we're at this moment where the Internet is taking off in a big way. A lot of people are coming onto the Internet for the first time.
Like, I had been on the Internet for... you know, since the mid-'90s at that point, you know, with dial-up account, and so... Uh, well, yeah, a lot of people were coming on for the first time.
(Andrey) Yeah. I -- I would say, we're probably a couple years behind you there. Uh, like, at least back then.
So -- and -- this is, like, almost everyone is on like a, uh, 33.6K modem, and just, like, a few people got a 56K, just now.
So, I -- basically, I'm kind of unofficially just kind of building software for my high school.
I think, at one point, they, like, paid me $200 bucks or something, kind of. Just, like, he gave me some cash.
(Kevin) Kind of an honorarium.
Yeah. You're probably, like, 15, 16. They maybe can't even officially employ you yet. Like...
(Andrey) Yeah.
I -- and then one day they approached me, and said, oh, we wanna send a newsletter to all of our alumni. They have this, basically, this CSV of around 1,000 email addresses, and they wanna start sending these monthly newsletters. And they don't have -- like, newsletter software doesn't exist back then.
Uh, so they're like, how do we do that? Can you, like, write us a thing to do it?
So, I go to my PHP, and I start -- that's mainly -- the reason I wrote it in PHP is 'cause I knew how to run it and deploy it, and it had kind of the, like, send mail API built into it somehow.
So, I'm writing my own PHP, I got my loop, I go over every email address, I send the email, I run it locally... It's slow, but it works great because it's a modem. That was okay. And then I bring it to them. Oh, wait, we want also attach a PDF to it. Uh, and it's, like, a 1-and-a-half meg.
(Kevin) Ooh-kay. Yes.
(Andrey) It's really big. And I don't know. I might have mentioned something at the time. It's, like, oh, that's really huge. It's, like, yeah, whatever. It doesn't matter. Um... [laughter]
So... I tried running it locally, and I'm on, like, a 36K modem. And it's really slow 'cause uploading 1-and-a-half meg, uh, 1,000 times is gonna take me a week.
Uh, so I had this brilliant idea. Instead of sending a bunch of two emails separately, I'm just gonna BCC everyone --
(Kevin) Right.
(Andrey) -- with one email so that way I only have to put it once, and let them --
(Kevin) The mail server will figure it out.
Yes. Yes. Exactly.
(Andrey) Yeah.
(Kevin) Yes.
(Andrey) Uh, so... I have this very vivid memory. I have my loop, and then inside, I'm gonna write -- I'm gonna keep that loop for now. I'm gonna write another loop underneath it, to kind of accumulate the BCC string.
So, I'm looping over all the email addresses. I'm building up a big string to put it in a BCC, [laughter] and I don't know if you can guess what --
(Kevin) I have some guesses as to where this is going, yes, yes, yes.
(Andrey) Yeah, so... I wrote this. Uh, I tested it with, like, one email. Worked great. Uh... I get my -- the gym teacher had this server that they rented. He gave me the log-in and password and their mail server and everything.
I log in. It's really fast. I think it was, like, running on, like, a T3 or something. Like, I've never attached a server this fast. I was really excited.
(Kevin) Ultimate power. Yes.
(Andrey) Yeah. So I upload my PHP in there. I uploaded my inputs. I go to the PHP endpoint, uh,
which is how I would run PHP back then. I just open it in my browser, and I just kind of, like, let it spin. And it's, like, taking a while, it's taking a while. I'm like, oh, it's taking a while. I would think it would be really fast. It's, like, a T3. So it should be really fast.
Uh, and it's sitting there, and then I'm kind of, like, as it's going, I'm kind of replaying all my lines of code in my head that it should be running. And then I had this realization: I never took out the outside loop.
(Kevin) Right. Yes.
(Andrey) So, basically, what happened is I sent a 1-and-a-half meg PDF 1,000 times on a 33K modem --
(Kevin) Oh, no.
(Andrey) -- in their inbox. So they got over a gig of email blocking their inbox. A thousand people. It was just -- it's, like, everyone who ever went to the school.
(Kevin) And this is in the era where people's inboxes were generally limited to, like, 10 megs. Uh...
(Andrey) Well, there's that. But also, there wasn't any kind of server-side mitigations. Like, the idea of, a -- what I accidentally did is this... denial of service attack, basically. And maybe I was the one that encouraged them to build some mitigations for that. Maybe not. But there weren't any at the time.
So...
(Kevin) You were certainly part of the wave of, like, "adversarial behavior," even though, you're -- the adversary was as much Murphy as anybody, to, yes, encourage the formation of those, yes, controls.
(Andrey) Yeah. So basically, for the next 2 weeks, our school was just getting non-stop calls from everyone that ever went to that school. It's like, I can't get my email. I can't get my email. People are, like, leaving their computers on for days at a time, trying to download everything.
Like, uh, brought down to the principle's office. Had to give a, like, "formal apology," whatever that means.
Yeah...
(Kevin) Were there -- besides, sort of, like, making you -- calling you out on the carpet and making you sort of promise to never do it again, were there any other, like, repercussions, or...?
(Andrey) No. The way I remember it is, I -- they wanted to keep using it after I fixed it. So, they -- I built them a little UI, and I think they kept using it for some number of years afterwards.
I think they paid me around 80 bucks for that one.
(Kevin) Okay. Great. [laughter] Okay.
Yes.
(Andrey) So... Maybe they got what they paid for. I don't know.
(Kevin) Uh, yeah. I mean, we always do, um...
Yes. We have the best humans available. Uh...
(Andrey) I did the math on this. It would have taken them 3 day -- 3-and-a-half, non-stop days to download everything in their inbox in order to get access to their later emails.
(Kevin) Okay. Okay.
Or -- I -- could they call up the ISP and be, like, hey, I -- please delete everything in my inbox? I don't know.
(Andrey) I mean... I'm sure some have tried. I can only imagine what the level of competence was back then.
(Kevin) Yeah. Yeah. Like, you have to get through -- I mean, I don't know, we were on a little rural phone company, so, like, the odds of actually getting through to somebody, uh, with, root privileges on the email server were actually fairly high.
(Andrey) Fair. Yeah...
We're, like, in a little city with kind of a... national telecom most people using, yeah, so I don't know. Anything's possible, but I think most people just left their devices on for the half a week and then let it plow through.
(Kevin) Oh, my god. Okay. Well, and this is -- like, we had a gig's worth of hard drive space. Well, we had a gig's worth of hard drive space, so, like, uh... That's, uh, yeah, quite a substantial amount of stuff to download and then delete. Goodness.
Um... And this was also in an era where, I don't know about you, but on my dial-up line... Well, okay, so first of all, we were lucky to have the dial-up line before we got DSL on a separate line because otherwise, it would just tie up this main phoneline, and --
(Andrey) Right.
(Kevin) -- you know, that was the main point of contact for everything. You know, my dad's a pastor -- so that was how his, you know, congregation --
(Andrey) Livelihood.
(Kevin) Yes. That was how his livelihood, like, you know, for the longest time, when I was young, I could only be on the Internet 30 minutes because that was the only -- that was the longest my parents were willing to have the phoneline tied up for dad's work stuff.
Um... But, whence -- even once we got the second line, it was still the case that, uh... it would kick you off after 12 hours, which I remember because I was trying to download my first-ever set of Linux ISOs. Uh...
(Andrey) Yeah.
(Kevin) Like Red Hat 7 or something. Uh... And was, like, oh, I -- you know, it's free. We can, technically speaking, download it, but, uh... Yeah. It was going to take several days. And after 12 hours, we got kicked off of, like --
(Andrey) And also, back then, resuming was very unreliable, too.
(Kevin) Extremely unreliable, yeah. Yeah.
(Andrey) Yeah. I remember downloading the Diablo 1 demo. It was, like, 54 megabytes.
(Kevin) Okay. Yeah.
(Andrey) Which was, like, 16 hours of downloading or something like that.
(Kevin) Right. Yeah. Yeah.
(Andrey) And then my computer would blue screen right before it finished.
(Kevin) Oh, no!
(Andrey) And -- but that was kind of the -- the impetus to get me to -- like, I -- I ragequit Windows so hard when that happened. That it actually, like, I got to reinstalling operating systems, and, like,
15:00
formatting things, and, like, I got to the next level, and then eventually using Linux and all that, so...
(Kevin) Yeah, same.
(Andrey) Guess it was a good thing in the end.
(Kevin) Yeah, yeah. Same.
My first Linux came out of a CD at the back of one of those books. I think it was Caldera OpenLinux. And it was, like, learn Linux in 30 days or something, you know.
Um... Yeah. That was wild. So, cool. Yeah. Okay. So very much that that early, scrappy era of the Internet when everybody was still figuring it out. Like, nobody's quite sure yet if it's gonna be a big deal or not.
(Andrey) Yeah. I mean, in retrospect, like, I was kind of writing down notes to learn and kind of reflecting that experience, and on -- on the one hand you can say it was irresponsible of the gym teacher to enlist me this way, but in any hand, that was really useful, very hands-on experience.
(Kevin) Oh, absolutely.
(Andrey) I feel like has had very substantial impact on me moving forward, so I really wish more -- more students would have that kind of opportunity to kind of make these kinds of mistakes.
(Kevin) Yeah. For sure. Yeah.
I was very... I don't know. I was very desperate in that era for any kind of real thing to do, uh, with my computer programming experience, my, you know, computer interests. And I really struggled to find people who were willing to let me, you know... make those mistakes.
(Andrey) Yeah. Yeah. I remember, a lot of our kind of "computer class" assignments were just glorified interview questions, basically. Like, yeah.
(Kevin) All, like, little programming puzzles, like --
(Andrey) Yeah.
(Kevin) Yeah.
We did some computer competition, programming competition stuff, which were all structured as, like, take this as input, do a little algorithm on it, you know, produce this output. And it was fine.
(Andrey) But you don't learn, to, like, to actually comment out your extra for loop before writing your other loop.
(Kevin) Right. Right.
(Andrey) Until you're actually do something in production.
(Kevin) Yeah. Exactly. Yeah.
Well -- and if you hadn't done that, somebody else would've, and did, probably. And probably fixed several, you know, rounds of that before the... uh, administrator is implemented whatever, you know, send mail, you know, limits were available in whatever, you know, uh... mail server they were running. Whatever postfix limits were available on the receiving end, and like, started to think harder about that.
'Cause, yeah, spam was only just taken off, and spam filtering was not yet a, uh... substantial thing. We were starting to -- I was starting to play around with, like, SpamAssassin, or whatever the built -- stuff that was built into, like, the Linux email client was in that era, or just a little bit past that era.
So... Yeah.
So what happened -- where did you go after that -- after -- after drawing the attention of the entire alumni community of your high school, and not in a particularly positive way?
(Andrey) Yeah... Ultimately, it was kind of a non-event. I don't think anyone kind of remembers it except me.
(Kevin) Yeah.
(Andrey) Yeah. And then I went off to university, studied computer science, did the urllib3 stuff. Moved to Silicon Valley to do -- start a pilgrimage that I'm obligated to do.
(Kevin) As one does. Yes, yes. Excellent.
(Andrey) Uh... Yes, so around then, I was working on these social analytics applications. Um...
So, the next story. Uh... This one's really not my fault, but also kind of a war story. Uh, so I built this Twitter analytics tool called Tweepsect. And basically, what it did is, it loaded your graph, and it a set intersection, so everyone, the -- you're, like, following, everyone's following you. It'd do the intersection. It'd give your mutual followers, who's following you only, and who you're following only. And then I referred to them as stalkers and stalking.
(Kevin) Okay. Right.
(Andrey) Which is kind of -- got people excited.
(Kevin) Yeah.
(Andrey) Uh... And one cool thing about the way that app was built, is it was pure client-side JavaScript.
(Kevin) Okay.
(Andrey) So, there was a small service-side, uh, shim to do the Auth hand-off. But after that, it was entirely run in browser, which is really cool 'cause we basically have infinite scaling without any kind of costs. It was, like, literally, just like a -- a static website, otherwise, um... And... Uh, so this was, like, back when Twitter had an open API that everyone could use.
(Kevin) Right. Yes. Yes.
(Andrey) Um... Anyways -- and then the other thing is kind of relevant is I also had built another social analytics. I was building, like, this, like, social analytics start-up called SocialGrapple, and I was going through these acquisition talks. And... Uh, one of the talks was actually with -- with Twitter.
So I had just met their director of platform just a few weeks earlier. Uh, so I built this Tweepsect thing kind of the year before, and then a year later, this is happening.
So, I met their director of platform. We talked about acquisition stuff. It's kind of things to work out. Uh...
And if -- fast forward a few weeks later, I used Google Analytics to track who's using my stuff, so I could see, kind of, like a real-time thing of people kind of searching, and I would get maybe
20:00
a few dozen people a day. Uh, maybe, like, a few thousand. Or, like, 1,000 people a month, using this thing. Uh, and then one day, I opened up Google Analytics, and I see all of a sudden, there's, like, a thousands of people there at the same time.
(Kevin) Okay.
(Andrey) I'm, like, what? What's happening?
(Kevin) What just happened? Yes.
(Andrey) Uh, yeah... And I assume it's a bug in my analytics, so I go in the app, check that, everything looks fine. I search, uh... on Twitter, people mentioning it, it's being mentioned. And then every so often I see a mention with my app's name but with a different link. It had a different link to it.
Uh, and I opened it in incognito and I checked, and it's, like, exactly my app. It even had my, like, analytics code in it and everything, but it's under a different URL.
(Kevin) Interesting. Okay. Yep.
(Andrey) And when you clicked log-in, it goes to, like, this, like, Twitter log-in form, but it's not on Twitter.com. It's on the iTwitterer.com or something like that. They had -- they kind of bought another domain that looked similar.
(Kevin) Right. Classic.
(Andrey) Uh... Yeah.
So what happened was, they basically -- they were -- stole my app, and they built this, like, phishing website where they would harvest people's log-ins, and then they would redirect back to my app, so I would still get all the traffic, which I really -- which I appreciate.
(Kevin) Okay. Great. Yes.
Glad that worked out. Yeah. It doesn't always.
[laughter]
(Andrey) But then once they stole the log-ins, they would go through everyone who's following you, and it would DM everyone that link. So, it would spread virally.
Everyone would get a DM. It's, like, oh, check this out. It just -- I looked up who's, like, stalking me. You should try it. Or something along those lines. There's -- yeah.
(Kevin) So it was running a viral marketing campaign for you.
(Andrey) Exactly. But they were also stealing everyone's log-ins, passwords.
It's kind of weird 'cause I had this, basically, live dashboard of every account that's being stolen in real-time. So just, like, the numbers, just, like, going up like crazy. Uh, so I... mentioned the, like, Twitter support team and stuff, and like, hey, can you help? Can you do something about this?
No one answered. But luckily, I just met the director there, and I emailed them, and then even though it was, like, 11... PM at night, he, like, responded a few minutes later.
Um... And allegedly, they were on it.
Uh, but the interesting thing is that it kept going for kind of about a month after that.
Like, I would look at my analytics, and it would, like, drop for a few hours, which I'm assuming they were playing some weird kind of... some cat-and-mouse game.
(Kevin) Maybe. Yeah.
(Andrey) With their, like, OAuth -- with their authentication flow to kind of block -- try to block them, and then it would go back up, like, a day later 'cause they would just change their method.
(Kevin) Okay. Interesting.
(Andrey) And the other funny thing about that one was that they were live. They had a live proxy of my website, so I could make changes on mine. It would be reflected on theirs. So I put up a big, fat warning, and it showed up on theirs as well.
But unfortunately, that also prompted them to eventually stop doing that method, and then they just kind of... Yeah. They just stole it wholesale.
Um... And yes, it's kind of interesting. I mean, based on my analytics, some, like, hundreds of accounts were harvested, and even made into mainstream news at some point.
If you searched for Stalktrack, it was, like, mentions on it -- 'cause they, like, kept it, ow, doing stuff with it. They kept trying to, like, harvest more and more addresses more ways.
And I think up to, like, 2015, there was some mention of it, to say, like, this is a scam, don't log-in with this app, and stuff like that, so...
Yeah. That was really wild.
Um... But yeah, I -- it -- I -- ever since after that, my -- my -- I went from having around a thousand visitors a month to having about a hundred thousand visitors a month, and it kind of, like, stayed that way for... What they say, a year or two, and then I put some ads on there. I made a few hundred bucks.
(Kevin) Yeah, yeah. For sure. Yeah.
I wonder, did you ever figure out who was behind it or have any guesses as to what the attribution was?
(Andrey) Uh, I mean, given that they kept going with it, and kept kind of, it -- just kept, like, building on other scams, it feels like some -- some kind of team somewhere that's been doing this for a while. They're gonna keep doing it. So, no, I don't know who's there behind it, but it feels like one of those things that just kinda... There's an adversarial group out there, and then there always will be, and they just happened to launch with my app --
(Kevin) With your app, yeah.
The sort of carrot, yeah. That's interesting. I, you know --
(Andrey) Seed.
(Kevin) Yeah, yeah. Exactly.
You hear about all these influence operations and... And this is a little bit of a, you know, window into how they work, you know. It's very convenient of them if there's already something in the environment which they can appear to be like, you know, already something legitimate, which they can, yeah, copy like this and leverage because that means, you know, people are coming into this from a place of, like, oh, yes, this, you know, looks legitimate. This looks like it's, you know, done by somebody rather than...
(Andrey) Yeah. Even if it doesn't look super legit, it's just -- it's an interesting use case that you can kind of use as a distraction.
(Kevin) Yeah. Yeah. Yeah.
(Andrey) I think, for sure.
(Kevin) Was there --
(Andrey) Uh...
(Kevin) -- ever any sense of what they were doing with the accounts, like...
25:00
Were they using them to send other kinds of spam, or...
(Andrey) That's a good question. Uh... Yeah, that's a good question. Um, it -- this was back before the day of password managers, so I'm assuming they were also reusing the log-ins on other websites.
(Kevin) Oh, for sure.
(Andrey) That would be my guess.
(Kevin) Okay.
(Andrey) The other thing that's interesting is, like, there was basically no mention of this. Like, probably half a million accounts got harvested, and it wasn't really in the news or anything. And like, Twitter didn't make an announcement. I mean, it's not technically their sole responsibility, but also, like, it's a substantial number, especially for, this was 2011.
(Kevin) Yeah. Okay. Yeah. Yeah. Um...
(Andrey) Twitter wasn't that huge.
(Kevin) We hadn't -- yeah. We hadn't yet quite figured out how big a problem account takeovers were gonna become, and quite exactly what the that was gonna look like at that era, but yeah.
(Andrey) But I think of it -- it's still happening on other services in some way, shape, or form. And then we just kind of never hear about it.
So, it's -- it's kind of interesting to be in... the eye of the storm almost. Just, like, having a real-time analytics dashboard, like how, what are the odds of getting that, right?
(Kevin) Yeah, yeah, yeah. That's actually a very interesting data set. I don't know if you still have access to that, but... uh, seems like something that could be interesting for one of these, yeah, accounts security researchers to take a look at, and maybe do some, uh... yeah, work on, so if anybody listening to this, or, uh, in the comments wants to get in touch with Andrey, the -- his contact info is available there.
(Andrey) Yeah. Hit me up.
(Kevin) Yeah. Will be in the description and in the show notes.
Um... That's also... I don't know, the -- on the Twitter side, I feel like the managing the OAuth, uh... security, like, managing your OAuth endpoint, being able to blacklist apps which are misusing it, it's something that we didn't really understand was important until relatively recently.
There was, I think, a Google thing which went around in, like, 2017 -- actually, when I was at Stripe. Uh... And we were very lucky because the lady who ran our internal phishing tests had also run an internal attack, testing whether she could get people to approve a malicious OAuth app, which worked great.
And so, was what, uh... The way she ran -- this is -- her name's Karla. She's got a talk up on that she had a BlackHat a few years ago called phishing -- or called Ichthyology: Phishing as a Science, which is brilliant, and I will link in the description end notes.
But she always used the internal phishing tests, which we were required to do by someone of our compliance regimes, not to, like, you know, go shame people for clicking links, or whatever, but to drive internal change in the business. And so, Karla got us to set up single sign-on. Karla got us to block -- to set up a allow list for OAuth apps on our Google Workspace rather than doing a block list. Uh, Karla is... I forget what the third thing was... Uh, but Karla is also the reason that AWS supports Yubikeys now.
(Andrey) That's the key. 'Cause I feel like OAuth namespacing wouldn't've mattered 'cause they're just harvesting usernames, log-ins, but having --
(Kevin) Okay.
(Andrey) -- multi-factor auth would've helped, for sure.
(Kevin) Okay. Very interesting -- on the part of Twitter.
(Andrey) Yeah. Just in the example that I was using. Uh, yeah. Just being able to log-in username and -- with your username and password and just be able to DM everyone with a link. Like, no amount of OAuth would've helped.
(Kevin) Okay. Yeah. Yeah.
(Andrey) Yeah.
(Kevin) I guess... Was it actually OAuth that they were using, or were they actually popping up a, like, a fake Twitter or log-in form --
(Andrey) Yeah. It was -- they actually duplicated the actual log-in form, and they were just harvesting, straight up, and then they were just logging in, yeah.
(Kevin) So, yeah... Then their... yeah, chief factor, especially of the, like, Yubikey kind where not everybody recognizes this, but it's actually checking the URL that you are sending your credentials to for you because it turns out that humans are really bad at bitwise string comparison, at catching that iTwitter.com rather than Twitter.com, whereas computers are really good at bitwise string comparison. And so, we should make our computers do bitwise string comparison for us, and Yubikeys, too. Uh... Or all the other little U2F tokens, and so...
Yeah. That is the kind of thing that would've helped their I.
I think that that attack is still possible with, like, a --
(Andrey) Yeah.
(Kevin) -- phone-based 2FA?
(Andrey) I mean, at the end of the day, also, Twitter should've been noticing that the same IP address is probably logging in a hundred thousand times to a hundred thousand different accounts.
(Kevin) Yes.
(Andrey) Yes. Also stuff like that.
(Kevin) -- whereas the rate limiting on Twitter's log-in form -- actually maybe those drops you saw in the analytics were, like, Twitter hastily implementing rate limiting, and were turning the rate limiting knobs, yeah, on their log-in form.
(Andrey) They probably blocked an IP, and they got a different IP, and then they went back up, and that kind of thing. Yeah.
(Kevin) Yeah. Yeah. Exactly. Um... Folks who -- I think, like, when I talk with software developers, you know, software engineers, who are working on the sort of, like, more features side, I think... Folks just, like, have never been on the receiving end of this stuff, or
30:00
if they have a log-in form on the Internet, and have maybe, like, looked to see what is being sent to their log-in form. And if you have a log-in form on the Internet, and you do not have rate limiting on it, and you have not looked at it recently, maybe go -- check your logs.
See what people are sending.
(Andrey) Yeah. I mean, yeah. I was going to say, dealing whatever scrypt or, like, bcrypt thing you're using is kind of an implicit rate limiting just by overloading your system. But...
(Kevin) Well...
(Andrey) Yeah.
(Kevin) Up to a point, and we really hope that you're using scrypt or bcrypt, and not, I don't know, MD5. There's still a number of, uh, bad log-in forms out there on the Internet.
(Andrey) That's true.
(Kevin) It's nice to have extra rate limiting because even though, like, the -- those are the password hashes which you're -- you should be using along -- long with a Salt -- this gets into a whole -- I should do a... I -- I'm actually contemplating doing a video about some of this stuff.
But this gets into the weeds. But, like, if somebody is sending you more than, like, 1 log-in request a second, even 1 log-in request every 10 seconds, that is not a human typing their password. That is an automated system.
(Andrey) Yeah. I mean, maybe we're just a few years away from not wanting to discriminate between computers and humans, so we'll see.
(Kevin) Maybe. I'm still very skeptical. If Chat-GPT is sending me more than one log-in request a second, that is also too many, thank you.
(Andrey) Get a different IP address. Come on.
(Kevin) Get a different IP address. Yeah.
Um... That is going to change things in interesting ways, and I'm, yeah, very curious to see what happens on the account security side, I'm not super looking forward to it. I'm not thinking that it is going to change things for us in positive ways. It is going to introduce new and interesting adversarial patterns that we are going to have to deal with. So...
(Andrey) For sure. Yeah.
(Kevin) Yeah. I did a talk, actually -- so, we met via our mutual friend, Patrick McKenzie, I think?
(Andrey) No. We met through... Willie, I think.
(Kevin) Oh, okay!
(Andrey) Yeah. But I have met, also, Patrick McKenzie, but --
(Kevin) Okay. Cool.
(Andrey) -- we're not close.
(Kevin) Okay. Cool. Never mind then. Um...
(Andrey) Yeah.
(Kevin) Uh, I did a talk at MicroConf, which is the conference that Patrick McKenzie goes to and talks about a lot. Um...
(Andrey) My wife actually works there. She's -- she's the... Yeah. Tracy Osborn. She's the director of TinySeed.
(Kevin) Oh, oh, brilliant. Okay. Cool. I hadn't realized that connection. Okay. Uh, it's a small world. Um...
(Andrey) Yeah.
(Kevin) I gave a talk at MicroConf this year about "The Three Simple Cybersecurity Changes Every East Bootstrapped Software-as-a-Service Business Needs to Make." And one of them was implementing rate limiting on your primary -- or on your -- on all your authentication endpoints. So, yeah... Um, was well received.
I had a number of people come up to me afterwards and be, like, so I -- you know, the CEO's asking the CTO, like, are we doing that? And the CTO is like, let me double check. I'm like, great. My work here is done. Um...
(Andrey) Nice. Well done.
(Kevin) I think it was yes for most people, but yeah, still. Important to know that and to know why because, yeah, until you have run something at-scale, you've probably never had the experience of somebody, you know, trying a bunch of password dumps, at you or, in this case, like, you know, cloning your site and, you know, setting up a phishing version of it, and trying to trick your users into giving, you know, away their credentials. And, uh --
(Andrey) Exactly.
(Kevin) Yeah...
(Andrey) You up for one more?
(Kevin) Yeah. Absolutely.
(Andrey) All right. So, the last one features George Hotz. Uh, which some people might remember from hacking the original PlayStation 3 and the iPhone.
(Kevin) Yes.
(Andrey) Uh, and a bunch of other stuff. Now, he -- he also started, uh, Comma.AI and now he's started another AI company.
(Kevin) Okay. Most recently, I remember him from, like, telling Elon he would solve all of Twitter's problems in a week. Coming on and, like, yeah, he tried, and like four days in noping out, and yeah, and having a long, like, Twitter Spaces with Elon about how horrible things were inside, which to me, sounded like someone who had never encountered a large software system before.
(Andrey) Yeah. I think they had a mismatch of expectations, both of them.
(Kevin) Yes. Yes. Yes.
(Andrey) Yeah. Yeah. So, one thing most people might not know is he's also been on and off involved in some uh... Ethereum stuff, which is just, like, a cryptocurrency thing that I also happen to work on on and off. Um...
And... So some back story. One of the interesting attributes of Ethereum is this collectively-owned computer, essentially, where we can all execute -- we have this -- these guarantees where all the code executed on it will be 100 percent right because every node verifies it.
(Kevin) For some values of right?
(Andrey) It'll -- like, it'll match the spec. Um... Like, it'll always execute accurately.
(Kevin) It will -- the computer will always do exactly what you told it to do. Um, yes... For better or for worse.
(Andrey) And it also has a lot of interesting censorship resistant properties where, basically,
35:00
in order to stop a transaction from happening, someone would have to burn billions of dollars at a time for an extended period of time, for as long as they wanna stop a transaction from happening. Um...
(Kevin) This does also make security a bit hard. Unfortunately.
(Andrey) Yeah. It does. It's a huge orchestration that happens once every year or two, where it's, like, many hundreds of people involved all over the world. It's very magical to watch.
(Kevin) Okay.
(Andrey) Um... [laughter] But yeah, so one of the things -- one of the, kind of, the constraints, in order to maintain these -- decentralization properties and for everyone to be able to run a node at home on their laptop, there's a very -- there's a fairly small limit on how much throughput it can handle.
People often make fun of it for having something like 10 transactions per second or something like that. Uh, and then --
(Kevin) I believe Patrick McKenzie calls it a slow database. Um...
(Andrey) Yeah.
(Kevin) Yes.
(Andrey) But turns out we can solve it by this innovation called rollups. And essentially, we can run a transaction somewhere else, and we basically commit to a batch of transactions on Ethereum, and we can prove that they executed accurately.
Um, so back when these rollups were first, kind of, designed slash discovered, people, like, the theory of it was fairly well established, but the way -- but implementation was still kind of a long road ahead.
This is 20... 19, maybe?
(Kevin) Okay.
(Andrey) 2020? Uh... Actually, I have a date. 2022 is when this story happens. So the way this -- this story goes, my, I'm retelling someone else's story, and then my story comes in later.
Uh, the people working on these rollups, they're hanging out in, like, a San Francisco cafe. And they're just, like, bashing their heads, like, how are we gonna build this, how are we going to build this? And then they turn, and
George Hotz is there, next to them.
And... They know him 'cause he's kind of experimented in the ecosystem, and they know he's really smart, so they just, like, start, like, brainstorming with him, and he gets nerd sniped, and they basically convince him to build a proof of concept of a rollup in exchange for a very generous amount of money.
But the term that they came up with is that they're gonna do an initial bug bounty, and a, like, $50,000 bug bounty, but the -- the money's gonna come out of his compensation if there's bugs.
(Kevin) Okay.
(Andrey) Which is kind of an interesting arrangement. Basically, he has to build this, they're gonna pay him, but if there's bugs in it, he's gonna pay for the bugs.
(Kevin) You have an immutable piece of software where, once you deploy it, it's gonna be very hard to update it, and it is going to be responsible for millions of dollars. So... Okay. Yeah. Better get it right the first time.
(Andrey) Yeah. And the nice thing about it -- these kinds of immutable softwares is, you could still deploy more instances of it, and just say this version is the newer version. You can disregard the old version.
So, it's immutable, but also there's kind of a, like, social layer where we can still orchestrate things around.
(Kevin) Yeah. For sure.
It's the new version of the contract, not the old version of the contract.
The old version of the contract. Uh...
(Andrey) Yeah. Exactly. And this is be -- this is before it actually may have made it onto main.
In fact, they just launched some version of this onto "mainnet," as we call it, last week, so --
(Kevin) Oh, okay.
(Andrey) It's taken long to have -- to actually productize this.
(Kevin) Got it. Okay.
(Andrey) Yeah. But this was kind of an initial, very rough proof of concept. There's, like, a lot of bash.
(Kevin) Sure. Okay.
(Andrey) It's all --
(Kevin) Okay.
(Andrey) -- kind of gum and duct tape. Um...
But anyway, so, George goes and hacks away for days and maybe a couple weeks and then gives them this thing, and like, takes their money, and then they do this bug bounty.
And I have this business partner that occasionally do bug bounties and security audits. His name is Max, and we saw it, and, like, oh, let's try to earn $50,000.
(Kevin) Yeah. Absolutely. Take George Hotz's money.
(Andrey). Yeah. Exactly. And... So, we kind of allocate a few days to this. We sit down, we pull the -- all the source, the instructions, we try running it, and there's an error. Uh... Just, like, right off the bat.
(Kevin) Okay. It's, like, you can't get the thing -- you can't get the source code to build?
(Andrey) No. It builds and it runs, but then it gives you an error.
(Kevin) Oh, okay.
(Andrey) Uh, my... coauditor, Max, he's like, oh, I'm just gonna post this error on a Discord, like, ask them what's going on, like, is this supposed to happen, and I'm, like, sitting there, and I'm like, wait a minute. If this is erroring out, this might actually be a bug --
(Kevin) Yeah.
(Andrey) -- that is a $50,000 bug, and he just posted it on the Discord.
(Kevin) Oh, no.
(Andrey) And I'm, like, smacking him, and getting him to delete the message as quickly as possible.
(Kevin) Yeah.
(Andrey) Um... And... Yeah, he -- so it turns out we're the only ones that tried to run it on live data. When they were developing it, they ran it on, like, a, like, testnet, basically. So, it's a -- it's like
40:00
a test environment, and no one really bothered to run it, and there's certain types of transactions that didn't work with the... types of proofs that it had to generate.
(Kevin) Okay. Interesting. Okay.
(Andrey) So, we -- we -- yeah...
So we basically spent a day or two debugging it, and identifying the problem, and, like, documenting it, and doing, like, a -- like, a proper security reports, submitting it to Immunefi and all that. Um... And... Yeah, uh... Long story short, we got $50,000 --
(Kevin) $50,000 of George Hotz's money. Congratulations!
(Andrey) Yeah, which is -- it was -- I mean, to be fair, he's a -- honestly that experience has reaffirmed a lot of, like, respect for him. He's obviously extremely smart. He's really good at, like, building a mostly working system extremely fast. It's extremely complicated. And yeah, like, just, like, a quick summary of how it works, just to kind of give you an impression.
It -- he built an implementation of the MIPS VM that runs on the blockchain, and you basically -- create --
(Kevin) So this is an older processor architecture.
(Andrey) Yeah.
(Kevin) Um... Uh...
(Andrey) So --
(Kevin) Is that an IBM thing?
(Andrey) No, it's an old RISC -- it's a RISC architecture that's extremely simple. I think it only has, like, like, 30-something instructions.
(Kevin) Okay. Cool.
(Andrey) Um, it's -- it's extremely reduced --
(Kevin) Ah, okay. Great. That's what RISC stands for reduced instruction set computing.
There was a big fight back in the '90s about whether Intel and the CISC architectures were going to win, or Sun and DEC and the CISC -- the RISC architectures were gonna win, and eventually, we combined them, is my understanding.
(Andrey) I mean... I mean, RISC does seem to be kind of making a very strong comeback, lately.
(Kevin) Well, yeah. Pure-play RISC are ARM now, is pure-play RISC, and does seem like it is eating Intel's lunch, with the way that Apple's deploying it, and the way that ARM themselves are deploying it, and
it gets used in a lot of mobile stuff, and yeah, it's just good in --
(Andrey) Yeah.
(Kevin) -- a lot of ways. Uh... That...
(Andrey) Yeah. So to kind of give you, like, a -- like, a little taste of how this system works. So he built a MIPS VM --
(Kevin) Sorry, the Ethereum blockchain already has its own VM.
(Andrey) VM. Yeah. It runs the -- yeah, the EVM, the Ethereum VM.
(Kevin) So, do we have a VM in a VM, right?
(Andrey) Yeah. So, he built a -- MIPS VM inside of Ethereum VM because one of the main Ethereum node implementations in Go, which is able to build into MIPS. So basically, you could just -- you could build the entire Ethereum node, build it into MIPS, and execute it on-chain using the MIPS VM, using a binary search to have a challenge system to figure out where you disagree on the execution of any given transaction.
So... Basically, you -- you have a challenger and a defender, and it would both do this. And in order to avoid executing probably millions of instructions or, like, billions of instructions, you would do a binary search of instructions, which is log-in. So, instead of billions of instructions, you would end up running, like, 90.
Uh... And then you would kind of go back and forth until you agree on which instruction you disagree on, and it would only execute this one instruction on-chain.
(Kevin) Okay.
(Andrey) I would -- which is, like, reasonably expensive. But it's just one instruction instead of, like, billions of instructions.
(Kevin) Okay. Okay.
(Andrey) It's a really clever design, and I have a lot of respect for it. And he built this whole thing in weeks. That mostly worked except for a small nuance.
(Kevin) The small nuance being that the, like... How do you ensure that you're running against the same data on...?
(Andrey) So you, like, pre-commit on-chain, so you, like, post a hash, basically, on-chain of your data, and then all of the inputs is based off of that hash.
(Kevin) And so the issue here then was not that, like, the data was different between the different Ethereum executions, so -- so, okay. So backing up, from my understanding, for just a second.
So this is -- we can think of this as a way, kind of, of running Ethereum on Ethereum in the way that we run, like, Windows in VM-ware on Windows for testing purposes, if you wanted to have a virtual Windows box, you know, for development or debugging or security auditing or whatever.
(Andrey) But you're not running all of Windows. You're only running one instruction that you disagree on.
(Kevin) Okay. Yeah. Yeah.
(Andrey) But you're -- but in theory, you've -- you would be able to run everything.
(Kevin) You could run all of it, but you have this disagreement about this one instruction, which is probably the, like, commit of some block, maybe, or...?
(Andrey) So, okay -- so another kind of important thing to -- the way these rollups work -- so, there's a few different kinds of rollups. There's zero knowledge rollups, and there's optimistic rollups. Uh, zero knowledge rollups use this -- a branch of
45:00
really interesting cryptography called zero knowledge cryptography to essentially prove instructions, so you can run an arbitrary set of instructions, and then you can create a small proof that you can just verify, and you can have a mathematical guarantee to some amount of certainty that it was executed accurately.
(Kevin) It's a way of giving me a thing that I can hand to somebody else where -- they're -- of proving that I did the computation that I say that I did, is my understanding.
(Andrey) Yeah. So -- like, a -- imagine something not now, but in, like, 15 years, Amazon Lambda, for example, that, like, runs arbitrary functions would not only run a function but also give you a -- kind of a, like, proof that you can verify that it ran it accurately. Uh, so that -- that's kind of what you can do with zero knowledge stuff, but um...
And then there's optimistic rollups, which work differently. They rely on -- in order to communicate state from the rollup to the Ethereum main network, they rely on a fault proof and a fault proof window.
So basically, it means -- because Ethereum itself can't run the whole rollup to verify the that it's running accurately, it basically has this assumption that if there weren't any fault proofs within the last week, then the transaction is by default accurate. Um... Which is kind of an interesting trick.
(Kevin) And a fault proof is an adversarial sort of thing where somebody else is coming along, trying to verify this, and if they find an error, they post a fault proof, which invalidates the transaction or can be used to?
(Andrey) Yeah. And the nature of that fault proof is actually what we mentioned earlier. It's basically the state route, which is kind of, like, your hash pre-commitment on the state of the blockchain, as you saw it in the beginning, and also -- which is kind of the, like, pre-commitment of the execution, and once you provide that proof, you go into that kind of binary search to figure out what -- where the disagreement is, and then only that one instruction of disagreement gets executed on-chain in order to resolve the -- to basically get -- to, like, decide whether the transaction is good or not.
So, basically, I could challenge any transaction that's kind of in the "inbox" of the optimistic rollup on-chain, but I have to go through this kind of a back-and-forth, and one side wins, one side doesn't.
Um... But, really, it's --
(Kevin) It presumably costs money.
(Andrey) Um, I mean, the amount to execute a single transaction, so...
(Kevin) Okay.
(Andrey) Have maybe a few dozen dollars or maybe a hundred bucks. But there -- there's also a whole, like, bonding process, so in order to challenge, you have to put down a bond that's strictly more than how much it would cost to challenge it. And there's, like, a whole kind of economic security system 'cause it's cryptocurrency, and it deals with money.
(Kevin) So this bug that you found then, where you were getting an error as soon as you tried to start executing this --
(Andrey) Yeah.
(Kevin) -- on live data. Was it just that it, like, there were transaction-types in the blockchain that it had to interact with that it wasn't expecting, or...?
(Andrey) Yeah. Yeah. Basically, it was -- yeah, it wasn't able to create a proof for certain types of transactions, and the reason that bug was important is because there's that seven-day window --
(Kevin) Right.
(Andrey) -- for... someone to give a proof, and if you can break the prover, if you can basically make it not be able to execute any proofs after some transactions, that means you can put incorrect transactions, and no one will be able to actually prove you wrong.
So, being able to break -- inject a transaction that a, like, prover won't be able to -- it'll get stuck on, is actually really bad.
(Kevin) Interesting. Okay. Fascinating.
And this was just, sort of, like, again, that, you know, that there was a type of transaction that the prover wasn't expecting. It didn't have a branch in its sort of case statement for it to handle it, and so it was just throwing up its hands, being like, I don't know how to -- I don't know what to do with that. And that broke the whole system?
(Andrey) Something like that. It was -- it was something to do with the -- the memory oracle, so this was a whole other thing. We can get in the weeds if you like.
(Kevin) Goodness. Yeah. Well, we got time.
(Andrey) So -- okay. So... Because you're running a VM inside of a VM --
(Kevin) Right.
(Andrey) Uh... The VM at the very bottom doesn't have access to all the memory it would otherwise need.
(Kevin) Okay. Sure. Right.
(Andrey) Like, it can't just, like, be, like, oh, give me memory address this, this, and that, in a kind of load state, so instead --
(Kevin) In fact, you really don't want your VM to have access to all of the memory. That's one of the reasons you run things in VMs is so that they only have access to the memory that you give them inside the VM, and they can't touch the memory outside the VM.
(Andrey) Well, the interesting thing is they don't have -- this particular VM has no access to any memory. It only has access to proofs of memory.
(Kevin) Okay.
(Andrey) So, it has the a pre-image oracle, which basically means, uh... The way, like, proofs work, generally, is just like, a -- these, like, Merkle trees
50:00
of hashes, so it's just, like, a -- hashes of hashes of hashes of hashes, and you have some kind of signature over them, and then you've -- you could prove that it exists somewhere in this tree.
So... They have this -- might be mangling this -- it's been a few years -- but in order for the VM and the insight to work, it needed to -- any time it needed to access data, it needed to query the hash of the data, rather than the data itself, and then -- this pre-image oracle that will -- would take the hash and map it, do the operation on the actual data, and then kind of get -- just give it the result, essentially.
And it was -- basically, the way the pre-image oracle worked to generate these hashes, it wasn't working for all kinds of transactions, um... It, like, turned out it needed to do a fairly substantial rearchitecting to make it more -- more generally, but.
(Kevin) Oh, interesting. Okay. And it sounds like they had not run this on live data before they, yeah, published this. Did they, actually, like, ship this to production?
(Andrey) No, no, no, no. This is, like, pre-pre-pre-pre-pre-preproduction. This is just, like, George Hotz being a -- an amazing cowboy, whipping this out into couple weeks, writing, probably, like, six months, went to work in a couple into weeks, and then dumping it and then leaving, and then, like, okay, let's see if this works.
(Kevin) Great. Okay. And it turned out that they had another six months' worth of work to do.
(Andrey) Well... That and, like, another couple years' worth of work, but yeah, as you said, there's, like, billions of dollars that it's -- is on the line, so... Uh...
(Kevin) Okay. Yeah.
(Andrey) It has -- it's -- it has to be extremely careful.
(Kevin) I guess -- I mean, I guess they got the proof of concept that they wanted.
(Andrey) Yeah. A bit -- yeah, exact, 'cause the whole thing is they wanted to figure out an approach that they could work to -- towards actually formalizing, and I think George helped them, kind of throw them in the right direction, essentially.
(Kevin) Sure. Yeah. Okay. Yeah. At least, here is a thing we know works on this subset of problem, and now how do we generalize it to the full problem, even if that is, I guess, several more years' worth of work, they still, like, getting from 0 to 1 was pretty important.
(Andrey) Yeah. I think using MIPS was -- was strictly his idea 'cause I think he has a lot of experience with RISC processors, back from his, like, PlayStation 3 days and all that.
(Kevin) Yeah. I very much think of him as, like, more of an embedded person than a -- yeah, application person.
Um... Okay. It's -- and that is, like, just from a, like, technical from a technical standpoint, from a, like, you know, wild-eyed, kind of, like, hacking, like, standpoint, the idea that you can run Ethereum an -- you know, an Ethereum VM on the Ethereum blockchain by the way running a MIPS VM in Ethereum, and then to run that is... yeah, kind of wild.
It -- it's like -- it's, like, it's computers all the way down.
(Andrey) Well, it's not just that, but basically the implications of this is anything that you can build into MIPS, you can run on Ethereum now. Or at least not run per se, but you can actually prove on Ethereum, and then there's a bunch of other -- there's a bunch of other rollups that do other stuff that that you can prove basically anything like, using WASM, or, like, whatever.
So, kind of, from, like, in the abstract, nerd sniping type of perspective is, like, being able to, like, write a program and relatively cheaply have, a hundred percent guarantee that's worth many billions of dollars that was executed accurately is something that, like, any random person can do now is really wild.
(Kevin) Right.
Yeah. And, like, there are whole Linux distributions which are, you know, for MIPS. Like, that was one of Debian's -- or there were. I don't know if Debian still builds to MIPS, but it used to. So you could have conceivably anything that you can run in Debian now, I guess, you know, proved on the Ethereum blockchain, and yeah. Wild.
(Andrey) It's a neat environment, and like, I think kind of at the end of the day, we will see some, like, traditional, kind of the AWS's Lambdas and stuff offering similar guarantees eventually, but I think there's just not enough market demand for it yet.
(Kevin) I feel like, you talk about the social, you know, controls, which help manage that, and I feel like that's where we get a lot of that kind of stuff from AWS Lambda. It would only be in really you know, niche, narrow, sort of, like, extremely high-value use cases around things like cryptography, around things like money where we would want these kinds of guarantees.
(Andrey) What I think, also just for, like, offloading operations. Like, I think a lot of people aren't comfortable, for example, like, running AI. People, like, would you trust
55:00
all your AI to be run not locally? Like, what if they're -- they're influencing your results?
(Kevin) I mean, clearly people do. Clearly people do.
(Andrey) They do, but it's also a really big argument around it, whether people feel okay with it, whether they trust the results, like --
(Kevin) I mean, I don't trust the results, but I don't trust the results for reasons outside of, like, whether it -- they -- you know, the computations got performed correctly. I don't trust the results because there's something higher in the stack which is does not care about the truth value of its output.
(Andrey) But once we solve that one, then we're gonna work our way down the stack, and then we're gonna have other kinds of issues. So, it -- it's --
(Kevin) Yeah. We'll see.
Also, like, given the amount of compute -- I guess it -- the compute is going as -- is going to training as well as to deployment with the amount of compute that it takes certainly to train these things, these, like, you know, data centers.
(Andrey) Yeah. Well, I mean, just as an examples, like, if you wanted to, well, okay, so, like, one...
I'm trying to think of a good kind of mainstream example... Like, maybe in some games that we currently rely on client-side DRM to -- to, like, guarantee that you're not, like, modifying the game in some way. Instead, we could have instruction proofs, maybe.
(Kevin) Right. Yeah. Yeah. Yeah. Right now, that seems to be done mostly from a client verification standpoint rather than, yeah, verifying with the actual computations that the client is performing, although...
(Andrey) Yeah. Well, that, and it's also being -- everything's being replicated server-side as well, which, if you found instruction proofs, then you don't need to do that.
(Kevin) Mm... Mm... Interesting. Okay.
We're now getting, like, deep in the weeds. Uh, and you also get a preview of some of the stuff that if you do not live in the Valley, are kind of conversations that I have around the Valley a fair bit with people, but...
Yeah, and so there -- there is yeah, like, open areas of research, and things to build, and ideas to explore, but, yeah, very cool stuff.
Any other stories to share with us, today, Andrey?
(Andrey) That's all I got for now.
(Kevin) Okay. Okay.
(Andrey) But I'll try to manifest some more in the future.
(Kevin) Okay. Yeah. Well, yeah. It would be lovely to have you back on. Um... And, yeah, this has been real lovely.
Um... Where can people find you online?
(Andrey) Uh, I'm usually -- I'm Shazow everywhere. Uh, it's Shadow but with a Z.
(Kevin) Great. Okay. Cool.
Uh...
(Andrey) Yeah.
(Kevin) Twitter, Bluesky, Mastodon...
(Andrey) Um...
(Kevin) All the usual.
(Andrey) Yeah, and like Farcaster, if you've heard of it, and uh...
(Kevin) I have not.
(Andrey) Shazow.net on Google, on Bing, and so on, and so forth.
(Kevin) Okay. Okay. Cool. Yeah.
(Andrey) On Keybase.
(Kevin) Great. Love it. Gotta -- oh, yeah. Gotta be on Keybase. You're doing crypto.
(Andrey) Gotta be on Keybase.
(Kevin) Yes. Awesome. Awesome. Uh, this has been the WarStories podcast on CriticalPoint. I'm Kevin Riggle. This is Andrey Petrov. Oh, and Andrey, what is it that you're doing now? Like, uh --
(Andrey) Um... I mostly do lot of random open-source stuff. I build a lot of Ethereum infrastructure and stuff. I work on some Ethereum libraries.
(Kevin) Cool. Yeah. Cool.
Okay. So you've -- okay. So you've moved on from Python infrastructure to Ethereum infrastructure.
(Andrey) Yeah. I mean, I've been working it for almost a decade now, on and off, but yeah.
(Kevin) Okay. Okay. Cool.
Cool. Very cool. Um... Yes.
This has been the WarStories Podcast on CriticalPoint, uh... Kevin Riggle, Andrey Petrov.
Thank you so much, Andrey. It's been a ton of fun.
(Andrey) Thank you.
(Kevin) We will see you all next time.
(Andrey) Take care.
[outro music plays]
Creators and Guests
