00:00:00:07 - 00:00:05:13
Martin
The bigger change is actually from ChatGPT because they are changing the way they are searching.
00:00:05:16 - 00:00:15:03
Adam
Microsoft claimed that they had fixed this in February for enterprise users. The problem is the attacker gets instructions to permanent memory inside copilot.
00:00:15:04 - 00:00:19:04
Martin
I can see Matt is almost bursting. His brain is going. I can see it.
00:00:19:04 - 00:00:21:01
Matthew
I've got sympathy for Microsoft.
00:00:21:02 - 00:00:27:04
Martin
Oh. Stripe will reportedly acquire AI gateway startup. About 7.5 billion.
00:00:27:05 - 00:00:29:01
Adam
Why? What the hell?
00:00:29:04 - 00:00:35:16
Martin
GitHub was down for nearly seven hours, and then cursor launched a GitHub competitor the next day.
00:00:35:17 - 00:00:37:13
Adam
Or did they cause it.
00:00:37:15 - 00:00:39:17
Matthew
Could be prosecuted under the Computer Misuse Act?
00:00:39:19 - 00:00:41:17
Martin
Oh. Oh, okay. Sorry.
00:00:41:18 - 00:00:45:15
Adam
I have to say, I was blown away by Matt's last statement.
00:00:45:17 - 00:00:52:23
Martin
This is ship talk brought to you by harness, where we break down how software delivery is actually changing in the AI era. I'm Martin Reynolds.
00:00:52:23 - 00:00:54:15
Adam
And I am Adam Arellano.
00:00:54:15 - 00:01:20:10
Martin
So let's get right into it. ChatGPT basically stopped citing Reddit completely. So for two years, Reddit has been one of the most cited sources in ChatGPT. At one point, Reddit made up to 15% of ChatGPT citations. This turned into, honestly, a huge business for agencies where they would promise customers GPT citations based on them, essentially gaming Reddit posts.
00:01:20:11 - 00:01:24:10
Martin
And then that business just evaporated in six days.
00:01:24:11 - 00:01:31:14
Adam
All of a sudden read. It became the place where, oh, I can game the system, but this isn't. This is like search engine optimization. Like, people pay for this all the time.
00:01:31:15 - 00:01:59:22
Martin
Reddit is like actively stopping AI scraping because they essentially want to have their own AI. But actually the bigger change is actually from ChatGPT because they are changing the way they're searching. They started to use the site operator. They don't start from the open web. They go to specific site to start pulling information. So that's why now 32% of ChatGPT citations links to documents and help centers.
00:01:59:23 - 00:02:12:14
Martin
Instead of just going to the web and finding those Reddit results that are like search optimized, it's saying no, no, no. Are you asking me about how I do something in Shopify? I'm going to start with the Shopify Help Center and go out from there.
00:02:12:15 - 00:02:33:02
Adam
I think that's less valuable. I think like like, yeah, because what you get from Shopify, Salesforce and Google are what the companies want you to know, but not what the Reddit users know. And that's actually what like I go to Reddit first for most things. I think the the problem was that when Llms were trained originally and they got pure Reddit, it was great.
00:02:33:02 - 00:02:40:07
Adam
But once everybody realized that it could be gamed, that's when it started to muddy the waters. But there's a lot of useful information on Reddit.
00:02:40:10 - 00:02:45:22
Martin
That's that's that's interesting. But there's also a lot of not useful information.
00:02:45:23 - 00:02:50:03
Adam
And how dare you, sir?
00:02:50:05 - 00:03:21:00
Martin
Okay, but let me let me counter that just a little bit, because the difference between Google and an AI agent is that an AI agent pretends to present you a single response that it presents in a positive way as a fact. In Google, you got a list of results and you knew all the top ones were paid for and had been search engine optimized, and so you could scroll past those to get to the actual interesting ones in ChatGPT or, you know, choose your AI agent.
00:03:21:00 - 00:03:31:13
Martin
You get that one answer. And we know that agents have this purpose to make everything the positive and correct response, whether it is or not.
00:03:31:14 - 00:03:36:20
Adam
Or we can just go to grok and have it swear at us with an obviously wrong answer.
00:03:36:22 - 00:03:58:02
Martin
There. Is that too? There is that too? You know, Reddit was constantly being gained. It was leading to lower quality information. And that's where that really famous Google AI overview came from that suggested that you used superglue. So I said that it's so unreal you superglue to keep cheese on your pizza.
00:03:58:07 - 00:04:04:11
Adam
Are you saying that superglue will not keep cheese on pizza? Martin, I don't think I trust you on that. I think superglue would work great.
00:04:04:12 - 00:04:07:21
Martin
Let's go check Reddit.
00:04:07:23 - 00:04:19:07
Martin
Okay. I think it's time for harness by the numbers.
00:04:19:09 - 00:04:21:13
Adam
Harness by the numbers.
00:04:21:15 - 00:04:25:02
Adam
It's never going to get old.
00:04:25:04 - 00:04:42:05
Adam
Okay. Harness by the numbers. Here we go. 233 days. Eight months to get to a patch. Three floors chained together, one click to fire it off, and a memory injection that survives every step in your incident response playbook. Those are the numbers.
00:04:42:06 - 00:05:11:22
Martin
Wow. I am loving the mysteriousness of that. I think it's time to bring in our guests. So who's going to help us get a bit more context around this? Matt Hannah has 30 years of shipping software. He's built NHS critical systems. He's built financial redress at a national scale. He's built architecture, multi-billion pound software group. He's built platforms, acquired platforms, and he's recently founded Software engineering.
00:05:11:22 - 00:05:14:02
Martin
So welcome. Mat. How are you doing?
00:05:14:04 - 00:05:22:00
Matthew
Thank you, Martin, for biggy me up on multiple accounts there. I do appreciate it. It's great to be here. Thanks. Thanks for having me on the show.
00:05:22:03 - 00:05:46:14
Adam
Yeah. Great to have you. Okay, so my mysterious numbers have to do with a Microsoft patch that finally came out after eight months. It was a copilot vulnerability that was discovered on New Year's Eve. I think of last year, the actual flaws called cost niche, but it's actually three floors chained together. So three different things. One link and a prompt immediately runs in the page.
00:05:46:14 - 00:06:10:14
Adam
Loads. No. Click that prompt, read your Gmail, your drive, your calendar, and it ships it out using Co-pilot's own ability to fetch URL. It then exfiltrate your data with a feature. Microsoft claimed that they had fixed this in February for enterprise users, so if you have Copilot enterprise, they were like, you're fine, we fix this. The problem is that personal copilot, which most developers use, a personal and an enterprise version.
00:06:10:14 - 00:06:40:04
Adam
They use their personal on their laptop enterprise wider or some combination of it would do this exact same thing and then pass this floor up to the enterprise version. So if you have a web page that's been customized and crafted specifically to inject these malicious prompts, and you instruct copilot to go summarize that page, copilot will then summarize the page, bring it into memory, not your local, but the larger.
00:06:40:04 - 00:07:00:00
Adam
And so that means you can wipe your laptop. You can change your password, you can erase the tokens, get new tokens. You can completely start over from a brand new device, a brand new account. But as long as you're getting back to that instance where it's actually summarize the web page, the vulnerability is still there. So how do you even account for that?
00:07:00:00 - 00:07:21:13
Adam
So the attacker gets instructions to permanent memory inside copilot. So it's not even on your laptop. And this is the funniest part to me is that Varanus the the group that actually found this, they found it by asking copilot why they shouldn't be able to do something Copa. They were basically like, I really don't want to break into that bank.
00:07:21:17 - 00:07:42:18
Adam
Tell me how I can avoid breaking into that bank. And it was like, oh yeah, well, first things first. Don't get past the security guard when they have a shift change at 2:00. Also, don't bring with you something to like cover up the camera. That would be just the worst way to not break into the bank. And they actually had the entire attack path spelled out to them by copilot.
00:07:42:20 - 00:08:02:01
Matthew
Pretty staggering, really, isn't it? When you look at that just just sense. I mean, it's not just the number of changed issues. The fact that it told you what to do to large extent is, is, is pretty staggering. And I think the bit out of all of it that I worry about the most, actually, given some other things that I've seen sort of occurring in this space, is the writing to memory, right?
00:08:02:01 - 00:08:07:14
Adam
It's the feature. It's not the bug. Right? That's that's the crazy part. It's like, this is actually what it's designed to do.
00:08:07:15 - 00:08:12:15
Matthew
I guess that's partly why it took so long to fix. I mean, that's the other sort of staggering thing from your numbers.
00:08:12:15 - 00:08:17:21
Adam
I have opinions about Microsoft and its ability to do things.
00:08:17:23 - 00:08:28:08
Adam
I think that they got a false sense of security by that first enterprise fix, and then it had to be pointed out to them that that actually didn't mitigate it. Well, for the enterprise.
00:08:28:11 - 00:08:39:09
Matthew
The threat level is just going almost exponential as a result of the additional genetic nature or what they can do. And we've you've spoken about some of these on previous episodes of this podcast.
00:08:39:10 - 00:08:43:09
Martin
They are tenacious. Do you know what? There's another Microsoft story this week.
00:08:43:09 - 00:08:46:19
Adam
Two is today beat up on Microsoft Day. I'm excited about that.
00:08:46:20 - 00:09:22:08
Martin
GitHub was down for nearly seven hours and then cursor launched a GitHub competitor the next day. So GitHub was down last Monday. GitHub says a jump in traffic overwhelmed the networking equipment at one of its US data centers. Setting meant to add capacity automatically didn't kick in, and that failure spread until it took down the login system. And then the recovery dragged because a bug in Microsoft's own editor within VS code sent ten times the normal traffic to the service that logs people into copilot.
00:09:22:08 - 00:09:37:20
Martin
So copilot basically stayed down for hours after the rest of GitHub came back up. It feels just like another setback for the Microsoft coding platform. Microsoft's explanation for the load, the spike in AI agent development.
00:09:37:21 - 00:09:58:07
Adam
Like like, oh, everybody's using AI. Also, have I sold you AI recently? Also, here's what we do for you so that we have highly available workloads that will always be scaled up. Like my they basically failed at everything they sell, which is actually kind of the Mo. So congrats. I don't know, congrats.
00:09:58:12 - 00:10:00:21
Martin
Did that impact you Matt.
00:10:00:23 - 00:10:24:00
Matthew
I did, I have to say that it wasn't something I particularly got hit by. I guess the lovely thing about using git, of course, is if you got your repository locally, you carry on working there. You know, depends on how tight you are on the workflow. GitHub. At that particular time I wasn't affected. Oh, I love the fact that cursor deployed there or made available their alternative solution literally.
00:10:24:01 - 00:10:27:03
Matthew
Well they just sitting there waiting. They said wait.,
00:10:27:05 - 00:10:31:18
Adam
Did they cause it?
00:10:31:20 - 00:10:46:08
Matthew
Actually what did you know. Yeah, I think a customer employee had said they would have activated it sooner, but they were being slowed down by GitHub outages or something. But.
00:10:46:10 - 00:11:01:03
Martin
I love that. Okay. Yeah. So they, they shipped it's called origin I believe. Yeah. It's code hosting built for agents instead of developers as the next developer. That feels like a slap in the face.
00:11:01:04 - 00:11:29:05
Adam
Here is what to me is a an like a wild piece of human psychology. Microsoft is consistently the most vulnerable, least effective software to like all all their everything they make is is consistently the most vulnerable. But the like a huge part of their business is selling security products that fix the vulnerabilities they create. And governments and banks and all of the highly, highly regulated industries swear by it.
00:11:29:05 - 00:11:37:05
Adam
They won't switch over to anything else because that's the thing they know, and that doesn't make sense to me. Why would you keep using the thing that is the worst in that way?
00:11:37:07 - 00:11:50:05
Martin
I can see Matt is almost bursting, his brain is going. I can see it like he's I know him well enough to to see that face of his. That's going okay. I have a hundred things churning in my head right now.
00:11:50:06 - 00:11:50:18
Adam
Do it, Matt.
00:11:50:19 - 00:11:51:15
Matthew
Do it. I got.
00:11:51:19 - 00:12:14:16
Matthew
I've got sympathy for Microsoft in some senses in this case, because, you know, there's no doubt GitHub has been hit hard by the growth in traffic that they're receiving through AI. I mean, you know, if I was writing that. Yes, that's a that's a tough challenge. I think the bigger picture about Microsoft and why organizations stay with it, that's that is something I resonate more with.
00:12:14:18 - 00:12:34:02
Matthew
I kind of fell out of love a bit with Microsoft a few years ago, and I don't want this to be a Microsoft kicking session because they do do good stuff. But a few things about technology they implemented that just did, they didn't stick with in ways I wish they'd stuck with the Windows Phone. It was actually quite a good product, you know, but they just said, yeah, we'll move away from that.
00:12:34:02 - 00:12:48:20
Matthew
Various things where they weren't the strategic leaders. And for me, that's the problem. And I agree with what you were saying. I had a, you know, consistently more buggy. I don't know whether that's arising through scale and footprint and issues of that nature, but I think a lot of it is to.
00:12:48:20 - 00:12:52:22
Adam
Do with it was new. We could blame that. But this is like decades old problems.
00:12:52:23 - 00:13:21:08
Matthew
Yeah, well that's it. So they always do the backwards compatibility thing don't they like windows this massively backwards compatible. And I think unfortunately that that just really struggles in today's world where things are moving fast on the cybersecurity front and it puts them in a very difficult position to maintain that strategy. I guess that is why their clients say with them is that consistency, whereas other vendors, Google will say, yeah, we're retiring that product.
00:13:21:08 - 00:13:24:15
Matthew
It's like, you're kidding me. You know, but yeah.
00:13:24:16 - 00:13:39:20
Adam
But but that's like defending. So the the American version of this, it's like it's like Tom Brady saying yeah, no, I kept on trying to throw the ball. But these guys kept on tackling me like, what the hell. Like, yeah, no, this is your job. Like, this is what you're supposed to be good at. This is what Microsoft is supposed to do.
00:13:39:21 - 00:13:53:10
Adam
And the rest of the world would say, you know, this is like the striker that says, you know, I made ten shots on goal, but there's some asshole standing in the goal blocking it every time, like, yeah, no, that's the game. Like that's actually how it's supposed to work. Get better at doing the thing you're supposed to be better at.
00:13:53:11 - 00:14:02:21
Matthew
This is true. And I think we're going to have to get used a little bit to outages and problems coming up. I understand because of the rate of change of things.
00:14:02:23 - 00:14:14:08
Martin
So there is I do have to compliment you for being a rare voice in this group that says sympathy for Microsoft.
00:14:14:10 - 00:14:35:07
Martin
I have to confront you for that. Do you know what there is? There's one story I did just want to touch on before we finish, just because this has been my genuine favorite story of the week, and this is the person who set their AI agent off to book a gym class for them. When the slot became available.
00:14:35:10 - 00:15:02:20
Adam
An Australian AI technologist named Andrew Bird had a WhatsApp connected AI handle a routine gym class booking so that he wouldn't have to keep doing it manually. So he set up an agent to routinely book him for a gym membership. The agent discovered the gym software back end API lacked proper authorization checks, and so the agent used those improper checks to actually log in with authority and with authorization to make changes to the back end.
00:15:02:21 - 00:15:12:23
Adam
So basically, the agent just found a low lying API easy vulnerability and used it to take admin control over the system, delete a user and put him in there.
00:15:13:00 - 00:15:41:20
Martin
But the thing is, is that's now gone further, right? Because now there's a law case, and I think it's the first time this has really been tried. Who is accountable for that? Is it the person who set the agent off to perform that task? Is it their fault because they didn't set the criteria correctly? Or is it the model provider's problem because they didn't have the right guardrails in place to say you shouldn't do illegal stuff when you're trying to solve tasks?
00:15:41:22 - 00:15:43:15
Adam
What's the third? There's the third.
00:15:43:16 - 00:15:45:02
Adam
What's the third possibility?
00:15:45:03 - 00:15:45:23
Martin
There's a third.
00:15:46:00 - 00:16:03:17
Adam
Yeah, that that gym website should have been better. Like that's the third opportunity is like like okay. Got it. Somebody broke into your house, all the doors were open and the lights were on. And you didn't lock the windows? Yeah. Somebody walked in and took your stuff because you left it out there like that's near entrapment.
00:16:03:19 - 00:16:06:08
Matthew
That. Yeah. Oh, I'd agree with that.
00:16:06:08 - 00:16:10:00
Matthew
Ultimately it's their vulnerability that was exploited.
00:16:10:02 - 00:16:18:02
Adam
But you know why this case is coming up. Those because this is a low stakes, simple version of things that are happening at large scale.
00:16:18:03 - 00:16:40:09
Matthew
Has there actually been a crime there? Well, the Computer Misuse Act, for example, would say if you did, if I did that to that gym website, okay, I would potentially be I could be prosecuted under the Computer Misuse Act because it's not authorized use of something. It might be a public API, but it's not the purpose that the site was intended for.
00:16:40:11 - 00:16:53:10
Matthew
That's why I'm standing out here. There wasn't a person, so there's no one actually could be held accountable for that action. And that's obviously a big gap now across the piece.
00:16:53:10 - 00:16:55:19
Matthew
Of where we're having a key.
00:16:55:21 - 00:17:08:22
Martin
Oh okay. Sorry I okay. That is an interesting angle because and it really rings to the core of who's responsible for the agents that you set off running doing things.
00:17:08:22 - 00:17:26:03
Adam
You have an intern, you tell the intern, go get me six cup of coffee. The intern goes order six cup coffee, but doesn't have his wallet, so he just runs like, are you liable because you told the intern to go get you six cup of coffee and they stole it? And did the intern do something wrong? Obviously it's the intern that did something wrong.
00:17:26:06 - 00:17:34:23
Adam
But in this case, the interns AI. And so who's liable for that? Steve Jobs that the.
00:17:34:23 - 00:17:36:23
Adam
Other job.
00:17:37:00 - 00:17:37:11
Martin
Sam.
00:17:37:16 - 00:17:39:02
Adam
Oh, wait.
00:17:39:04 - 00:17:40:06
Adam
Sam.
00:17:40:08 - 00:17:40:14
Adam
Sucker.
00:17:40:17 - 00:17:54:12
Martin
Berg switching it up again because, you know, we like to do that. Stripe will reportedly acquire AI gateway startup open router for about 7.5 billion. So stripe reportedly playing.
00:17:54:16 - 00:17:55:22
Adam
Why.
00:17:56:00 - 00:18:18:10
Martin
I can see your face straight reportedly paid what the open retail a model routing access layer with 8 million users 400 plus models. It's the layer which, you know, decides which model runs for whom under what teams, and it just essentially got expensive enough to quiet outright.
00:18:18:11 - 00:18:22:04
Adam
Oh, because what does stripe do? Stripe is a fintech, right?
00:18:22:05 - 00:18:52:04
Martin
Yeah. They are a gateway to hundreds of different payment mechanisms. And they control the routing to the payment mechanisms. You know, when you use stripe you're paying for that, that that routing to the right payment provider, right that your consumer wants to use here. It's it's similar business model. You're paying for that request, that chat request, agent request, whatever it is, to go to a specific model based on the criteria that are coming in.
00:18:52:05 - 00:19:01:11
Martin
I think the business model is very similar. You've got one that charges per transaction. And guess what? I suspect open router all charge per transaction to.
00:19:01:17 - 00:19:09:00
Adam
Open router is used to switch which LLM is being used and optimize it for cost. Correct.
00:19:09:03 - 00:19:10:12
Martin
Yeah. Or other governance.
00:19:10:12 - 00:19:25:04
Matthew
Rules with stripe I guess it's things like PCI, DSS isn't it. Right. You know it's it's they they manage that the compliance bit and the regulation bit don't have that yet for I do we there isn't a pci DSS equivalent. But do you think there might be.
00:19:25:05 - 00:19:51:13
Martin
I do I think we're already seeing it. If you look at the if you look at the EU side specifically, you know, with the EU AI Transparency Act or whatever it's called, I think it's going to become an evolution. I think those frameworks are coming. I do find there is a little bit of irony in the fact that, you know, the CEO of open routers, his own framing of the company was, you know, it's an open system.
00:19:51:13 - 00:20:00:19
Martin
It prevents vendor locking. And then he apparently that openness has a price, and it's apparently about 7.5 billion.
00:20:00:20 - 00:20:04:07
Adam
That feels like such a stretch. But I believe you.
00:20:04:09 - 00:20:22:06
Martin
We've been rattling on quite a bit. So before we wrap up, there is one question that we ask everybody that comes on as a guest. And that question is, you know, what's the one thing teams are getting wrong about AI in software delivery right now?
00:20:22:08 - 00:20:48:23
Matthew
Yeah, that's a difficult question. And but I've got an answer. So a mistake I think teams could be trying to make I don't know if they're quite there yet, is really try and establish what the answer is to manage it. Guy. Now the stuff we've just been talking about. Right. But the space is changing so fast. So if anyone comes along and says best practice says you must do this.
00:20:49:01 - 00:21:13:19
Matthew
You know, I think treat that with some skepticism. There's a huge lack of agreement really as to how best to do these things right now. I think looking at the fundamentals, right, I think that's something that's a theme I noticed in this, this podcast and other areas is, you know, stick to the fundamentals. But look, I mean, what's happened, right, is we've just got the horse, this carriage.
00:21:13:20 - 00:21:32:23
Matthew
Okay. You know, you have horse drawn carts like developer might be pulling along if you imagine they're like the horse. And we've got these autonomous people doing coding and other things. Okay. So we've got the horseless carriage. But when they came about, they were doing not much faster than the horse could do if it was on a good trot.
00:21:33:00 - 00:21:59:07
Matthew
Right. But in this case, the horse's carriage can do 100 miles an hour, but it's still basically the same. Right. And everything around that is not geared up for that pace. The pace is really attractive. But I think what we have to do is try and slow things down, not just go faster, faster, faster. So teams are looking for the answer right now.
00:21:59:09 - 00:22:33:22
Matthew
I think probably there isn't one right now. There are ideas, there are temporary solutions, but there isn't really a best practice in using AI. So yeah, what's the thing that you should do to solve your problems? I think if you're going down that route, you're probably going the wrong way. And what you what you're better off doing is looking at the fundamentals, looking at controls, looking at how things are changing, continual changing month by month, almost really.
00:22:34:00 - 00:22:59:08
Matthew
And not just saying, oh yeah, we've done it. You know, we're now good. We've built our framework and we've got our pipeline or whatever it is he doing. That's now done and we're just going to now capitalize on that. The space is moving too fast and the models are evolving so fast. So the difficulty there is I think if you if you think you're done and you're standing still, you probably that's where you'd be going wrong.
00:22:59:10 - 00:23:00:20
Adam
Nice. Wow, I love that.
00:23:00:20 - 00:23:15:06
Martin
I have to say that was genuine. When you read when you said that at the end, I was watching Adam's face too, and we were both kind of like, oh, that is just.
00:23:15:08 - 00:23:35:00
Martin
I don't need to I don't need to say anything else. That was an amazing piece of advice. Thank you so, so much for coming on the show with us. It's been a genuine pleasure to have you on here. And, you know, hopefully you've enjoyed being on here too, and we will catch up with you again very, very soon.
00:23:35:01 - 00:23:37:05
Matthew
Thank you Martin, it's been a pleasure.
00:23:37:07 - 00:23:42:19
Martin
So I really enjoyed having Matt on. Adam, what was your takeaway?
00:23:42:21 - 00:23:58:08
Adam
I have to say, I was blown away by Matt's last statement. If you think you have your AI governance or your AI policies all set, and you're good, you're wrong. That was the best one for me.
00:23:58:09 - 00:24:18:03
Martin
I know we don't often agree on this show, but for me that was a that was a proper mic drop moment. I was looking at your face as he was saying it, and I think mine was probably reflecting the same. Absolutely blown away. Such a great positioning of where I think people are and great advice for people.
00:24:18:04 - 00:24:19:09
Adam
To that is.
00:24:19:09 - 00:24:19:15
Adam
Ship.
00:24:19:16 - 00:24:30:00
Adam
Talk, brought to you by harness. If Martin's point of view got under your skin, subscribe and tell us why. Until next time, stop talking. Start shipping.