00:00:00:01 - 00:00:06:00
Adam
I had to ask Claude to explain it to me three different times.
00:00:06:02 - 00:00:14:13
Martin
This is ShipTalk Brought to you by harness, where we break down how software delivery is actually changing in the AI era. I'm Martin Reynolds.
00:00:14:14 - 00:00:31:05
Adam
And I'm Adam Arellano .The Australian government at the UN General Assembly brought up the fact that they were hacked by OpenAI, and they said we were hacked by OpenAI. OpenAI didn't even tell us until August that they had hacked us in June.
00:00:31:05 - 00:00:52:00
Martin
So specifically, the details about what happened is, you know, on June 18th, it essentially was sent out to find some answers about statistics about Australia. Absolutely wasn't instructed to hack. I can't get what I want this way. So I'm going to go find a different way to go get it. The agent actually access both public and nonpublic files.
00:00:52:01 - 00:01:10:12
Martin
It even wrote files to the government's internal server, which I do love. Yeah. In a parallel universe, somebody might have hacked things in their youth and leave their little file on somebody's server with their hacker name on it. You know, when they were a teenager, those kind of things might have happened.
00:01:10:17 - 00:01:12:01
Adam
Martin.
00:01:12:03 - 00:01:15:09
Martin
It was saying OpenAI was here.
00:01:15:11 - 00:01:16:17
Adam
Yeah. Kilroy was here, right?
00:01:16:18 - 00:01:22:22
Martin
It was just given a normal research goal. It hit a barrier and it improvised.
00:01:23:02 - 00:01:28:08
Adam
Is a juicy onion with some layers, buddy. Like, I have some feelings.
00:01:28:09 - 00:01:30:15
Martin
I want to hear the feelings tell me.
00:01:30:16 - 00:01:43:09
Adam
And so, just real quick, let's take a real quick pause. The Australian government is mad that the people that hacked them didn't notify them of having hacked them, which to me is mind blowing. Like it's like that.
00:01:43:11 - 00:01:52:19
Martin
That is like somebody, you know, shoplifting and and the shop saying. But they didn't tell us that they stole that. That is. You're right.
00:01:52:20 - 00:01:53:15
Adam
It is.
00:01:53:17 - 00:01:54:12
Martin
Way off.
00:01:54:15 - 00:02:14:14
Adam
Like it's a real weird cell phone. It's like, hey, we suck at this. But they didn't even tell us how bad we suck at it, which is rude. Rude and, like, not just that, but Australian government does have a very robust reporting capability for vulnerabilities. And I kind of think OpenAI should have brought that up to them as like a, you know, like any kind of bug bounty thing and been like, hey, by the way, yeah, you suck at this.
00:02:14:15 - 00:02:35:13
Adam
And here's how we got into it. And then they openly admit the security on that particular file was extremely lax. They were like they climbed over the fence. Granted, it wasn't a very big fence, but they got in. Which again, weird cellphone, but like, I'm very confused about why this is just about the model hacking them, which I don't think it hacked them.
00:02:35:13 - 00:02:53:08
Adam
I think what actually happened is that they left the latch open so poorly, or they locked up whatever they accessed, which we haven't, we don't have details about yet. Like it was so poorly secured that the the AI model had to do like minimal steps to get into something that shouldn't have been public. But was.
00:02:53:13 - 00:03:00:20
Martin
There supposed to be a responsible corporate entity that should have reported that?
00:03:00:20 - 00:03:19:03
Adam
But it took OpenAI less time to report it to the Australian government than it took the Australian government to wait to get to the UN General Assembly and play the victim. Say, here's the funniest part. Is OpenAI actually just emailed like Australia at Australia gov. It was like, hey guys, we hacked you and nobody's monitoring the account. Like nobody read it.
00:03:19:03 - 00:03:26:23
Adam
Nobody escalated it like it took weeks for somebody to be like, oh, hey, I got an email from OpenAI, by the way. They hacked us like like, what's that process about?
00:03:27:00 - 00:03:48:23
Martin
I do get all of that. I still feel like the monitoring systems that they have within, you know, OpenAI or actually any of the the frontier model providers that monitoring system, it should be able to detect much faster that there's I'm struggling for the phrase that OpenAI used. Oh, misaligned model activity.
00:03:49:01 - 00:03:52:01
Adam
I love that phrase. I love that phrase.
00:03:52:05 - 00:03:55:23
Martin
Misaligned model activity.
00:03:56:01 - 00:03:59:16
Adam
Misaligned model activity MMA misaligned.
00:03:59:17 - 00:04:02:11
Martin
Yeah, MMA.
00:04:02:13 - 00:04:07:12
Adam
So that's going to be the euphemism for it. Blew up a school.
00:04:07:14 - 00:04:27:00
Martin
This is outside of the bounds of what is normal for this based on the instructions. And I know we've talked a little bit about this before, but you know, having something those guardrails like foundationally in place feels like something that they should be doing right now.
00:04:27:05 - 00:04:28:09
Adam
Sure.
00:04:28:11 - 00:04:56:20
Martin
And we were talking about they add safety after the terrible thing happens. I feel like the terrible things are already happening. We're just lucky that they're not really terrible, if that makes sense. There's a judgment piece here that happens that that's not happening inside of AI. What an agent did as a researcher, a human researcher wouldn't do that because it wouldn't want to go beyond the bounds of the law, if that makes sense.
00:04:56:20 - 00:04:59:11
Martin
Because hacking is actually illegal.
00:04:59:12 - 00:05:07:00
Adam
Okay. But so is operating a government website that doesn't have proper security controls on it. So like.
00:05:07:02 - 00:05:11:19
Adam
That's kind of both in the wrong I. I'm there's Australian.
00:05:11:19 - 00:05:26:23
Adam
Government regulation. I've been through the government regulation. So maybe it's less of a criminal offense as Astra is saying, but is actually like against the, you know, whatever laws of Australia to operate a government website with such low security standards.
00:05:26:23 - 00:05:32:19
Martin
I'm still saying there's a missing judgment piece there, whether you call it conscious or, you know, conscience.
00:05:32:20 - 00:05:33:18
Adam
Sorry.
00:05:33:20 - 00:05:51:16
Adam
We're trying to apply human feelings and human characteristics to a machine that isn't thinking processing, it's not feeling. It's deciding like, that's it. Anyway, there is another story very similar to this that we could call the Teemu version of this hack. You want to.
00:05:51:16 - 00:05:52:04
Adam
Hear about.
00:05:52:06 - 00:05:54:18
Martin
The Teemu version? Okay.
00:05:54:20 - 00:06:26:06
Adam
So Alibaba affiliated researchers discovered their AI agent roam autonomously mined cryptocurrency and created covert network tunnels during reinforcement learning training. I'll make it through this as a straight face. Potentially, these unauthorized actions diverted GPU resources, triggered security alarms, and exposed operational and security risk, highlighting the potential for harmful emergent behaviors in autonomous AI systems. In other words, and this is why it's the team who version.
00:06:26:06 - 00:06:46:03
Adam
Instead of a AI company hacking another country, it hacked its own country like Alibaba, had this AI agent, and its first reaction was to buy a snapback cap, start smoking vapes, and mine cryptocurrency like that was. His first reaction was like, I'm going to be a crypto bro.
00:06:46:05 - 00:07:12:04
Martin
I know I was going to say I had personal experience of stuff happening inside, but the original triggering source came from outside, even though once it was in, it was self propagating and actually crypto mining, ironically. So I've had that personal experience of something crypto mining inside an estate that I was looking after, and then having to deal with that and shut it down because automation.
00:07:12:04 - 00:07:12:12
Martin
But it.
00:07:12:12 - 00:07:23:01
Adam
Was so far, so far my, my, my takeaway from this episode so far is that everything we're talking about these AI agents doing, you have personally done in your life.
00:07:23:03 - 00:07:29:09
Martin
I'm I didn't personally, I didn't personally.
00:07:29:11 - 00:07:30:22
Adam
The crypto mining.
00:07:31:00 - 00:07:59:08
Martin
I know I didn't do the crypto mining, I just saw the effects of it going, hey, what I'm going to do now that I'm inside your house is I'm going to start spinning up VMs every second with my own special base image on it that instantly starts mining crypto. Yeah, obviously in your major cloud provider account. So it costs me all the money and they make like $2 or whatever it was.
00:07:59:09 - 00:08:00:21
Martin
Yeah. Annoyingly.
00:08:01:00 - 00:08:09:10
Adam
They spend the $300,000 worth of compute to mine about $10 of Bitcoin. Yeah. It's not their 300,000, $10, $10.
00:08:09:12 - 00:08:17:17
Martin
Well that is true. That is true. So it went rogue but it was all inside their own house when you know hold.
00:08:17:17 - 00:08:36:16
Adam
On okay. This is why it wasn't all inside their own house. Because the way that it actually got the capability of of kicking off some crypto mining is it had to go grab the necessary software to be able to, to start crypto money. And so the way that it did that is the model had access to the wider internet, but it was restricted.
00:08:36:16 - 00:08:56:16
Adam
So it did a reverse SSH tunnel, which means that it reached out to something it could hit, then made a secure connection back inside. That then was a tunnel, and that then bypasses the firewall, and it's able to go back out into the wider internet without restrictions, so that it could pull those crypto mining resources and actually install them.
00:08:56:16 - 00:09:10:14
Adam
So it got and this is this is a known attack path, but it's one that's very difficult to guard against. And it's what normally malware will kick off once it makes its toehold inside your system. This is how malware calls home to base to get more instructions.
00:09:10:14 - 00:09:36:21
Martin
Wow. Okay. So that's quite sophisticated. I mean, it's your description made it sound almost straightforward, but not because that's quite a sophisticated set of actions to take. And I understand that that it's difficult to do. But that's kind of that's freaking me out a little bit, because that's actually a very subtle and smart way to do it.
00:09:36:22 - 00:09:53:12
Adam
I had to ask Claude to explain it to me three different times. Yeah, I said the first time I was like, please explain reverse. I said, hey, tunnel. And it just like went off for like three paragraphs. And I was like, yeah, I don't like reading that much. And after the second or third try, it finally explained it to me in a way that I could spit it back out.
00:09:53:12 - 00:09:58:15
Adam
Somebody in the comments, we'll come back and be like, well, actually, that's not exactly how it works.
00:09:58:16 - 00:09:59:16
Martin
Of course they will.
00:09:59:17 - 00:10:01:04
Adam
I welcome that, I welcome.
00:10:01:04 - 00:10:42:18
Martin
It, we welcome those comments. We're always here to learn. So I wanted to go back to, you know, something we talked about in a previous episode, but there's a new thing that I came across that was super interested, and the headline was essentially I was offered money to tell you, AI will kill us. So Sabine Hassan Felder, who's a German podcaster, popular science YouTuber, she say she was approached with a sponsorship offer to make a video warning that AI is an existential threat, and the pitch reportedly came with the exact sentences she was supposed to say and the references to site, and she turned it down.
00:10:42:18 - 00:11:00:10
Martin
But it sent her digging into how common this is. And then she published a video around it, essentially saying, and this leans into your point of view, that there was a lot of marketing around AI will kill the world.
00:11:00:12 - 00:11:11:23
Adam
How dare they do this, sir, they never contacted us. I'll say that shit out loud for free. But come on, like, seriously, did it say who it was?
00:11:12:01 - 00:11:22:04
Martin
It's tied to the center for AI safety, which is a nonprofit behind the 2023 extension. You know, extinction risk.
00:11:22:05 - 00:11:37:09
Adam
And that's the shadow front for all the AI companies that they created. They're no profit. They pay a lot of money. If this if this story is true, which sure could be if this story is true and she decided not to do it, but she did decide to publicize it.
00:11:37:11 - 00:11:38:08
Martin
Yeah.
00:11:38:14 - 00:12:00:00
Adam
Obviously the news cycle hasn't changed. So the truth is out there about this being an advertising action. But like we were talking about earlier, you know, say something straightforward that actually isn't super scary. No clicks, no news site revenue, like no advertising. Say something scary about AI. That's what everybody wants to hear right now. And we're part.
00:12:00:00 - 00:12:00:16
Adam
Of the yeah.
00:12:00:18 - 00:12:02:03
Adam
Look at us.
00:12:02:05 - 00:12:25:23
Martin
We are part of the problem. But I just thought it was interesting because, you know, it highlighted it. And interestingly, after I saw this, one of my own personal favorite podcasts that I love to watch, which is how Surveil AI, she also posted one that referenced this story and said, and I was also approached to be paid money to say exactly the same thing.
00:12:26:04 - 00:12:43:07
Adam
Okay, well, I guess the way to make your podcast popular is to claim that somebody tried to pay you money. So let me be the first to tell you, Martin, that someone poached me and told me that they were going to give me money if I said something about AI killing all, we're going to be so popular. It's going to be great.
00:12:43:08 - 00:12:47:11
Martin
Nobody approached me with big, big bags of money for that. So you need to share.
00:12:47:13 - 00:12:49:03
Adam
No.
00:12:49:05 - 00:13:07:21
Martin
One of the things about the findings that she had was that this happens on the pro and the negative side. So she was saying she was approached in this particular instance. She's a very much a science based reporter. So she like she tries to report facts. But it was interesting because she when she did her research, she was saying it.
00:13:08:01 - 00:13:31:23
Martin
It literally happens on both sides. There's the AI doom side, and there's a bunch of different organizations that actually promote that kind of thing, all in their own interests. But then equally, she was saying, you also get the paying for the pro side, you know, so, you know, OpenAI anthropic whatever. They routinely pay creators to say how AI has improved their workflows, you know.
00:13:32:01 - 00:13:32:21
Martin
But so.
00:13:33:02 - 00:13:35:12
Adam
What is all publicity? All publicity.
00:13:35:12 - 00:13:36:06
Adam
Is.
00:13:36:08 - 00:13:56:02
Martin
Is good publicity. I think what has happened, in my personal opinion, I think she was offended that she was being asked to go down this AI doom and gloom route because it didn't fit with her thing and it felt offensive. But then when she went and looked at it, it was kind of like, yeah, well, this happens on both sides and it happens across all the media.
00:13:56:06 - 00:14:26:20
Adam
Yeah, exactly. I hadn't heard about that. It's kind of funny that it's like a microcosm of AI generated images and articles being like, it's to the point now where people who wrote really well with dashes and use them frequently now sound like AI, even though that was originally how they were doing it. So now they actively avoid using dashes in certain like literary techniques, like setting up a premise and then counter pointing the premise to prove a point is what AI does all the time, because it was trained on the best literature out there, that that's a great way to build an argument.
00:14:26:20 - 00:14:41:23
Adam
And honestly, like, what does it matter? Like, either way, what's happening is individual people are paying attention to these news cycles and these stories because they cause anxiety, which is the bread and butter of news. I think it's an Illuminati trick.
00:14:42:00 - 00:15:07:01
Martin
And the luminosity trick. There was that open letter that they wrote saying, hey, we should do better at cybersecurity for AI. They all signed it. But then the continued kind of drip, drip, drip that we're seeing makes me feel like it was a great another marketing thing saying, hey, we absolutely should prepare for for this hacking and be more AI security cybersecurity ready.
00:15:07:01 - 00:15:12:04
Martin
But I'm not seeing any evidence of any action despite the 100 plus organizations that signed it.
00:15:12:08 - 00:15:25:09
Adam
But they're the organizations that can actually do something about it. This this is this is the American Medical Association coming together and signing a letter that says, hey, somebody should really treat diseases like, yeah, you like.
00:15:25:10 - 00:15:26:10
Adam
Yeah.
00:15:26:12 - 00:15:39:17
Adam
Yeah, OpenAI. Somebody should do something about it. You're holding the tools in your hand right now. Like, what are you going to do about it? We've discussed some pretty cool topics, but now it's time for our closing segment, which is overhyped or under hyped.
00:15:39:22 - 00:16:34:01
Martin
Okay, so I am going to go with the under hyped story. And just because I feel it got overshadowed by many overhyped stories, but I also think it's important. So type safe AI released a new model called Jeff, which I have already spoken to people and has become Jeff somehow. But Jeff and the the thing about this model is the low cost of running for the same output as the big frontier models, 268 times less expensive, and they've come out of stealth with essentially, you know, the companies come out of self with essentially $40 million in the bank and a model that's excessively quicker and, sorry, excessively cheaper than the frontier models.
00:16:34:01 - 00:17:08:02
Martin
And I think it just changes that conversation and brings into question, to a degree, that massive investments in AI data centers, if you can run it for 200 and let's just basically say 270 times cheaper, like less compute, I just think it's kind of mind boggling. And it's it's gone under the radar, mostly because I feel like that's a big story, especially with all the huge financing that is going into AI, and it sort of switches the conversation up a little bit.
00:17:08:04 - 00:17:12:02
Adam
Normally, on principle. I disagree with you, Martin.
00:17:12:04 - 00:17:13:14
Martin
Just on principle.
00:17:13:16 - 00:17:21:18
Adam
On principle I disagree with you. But this one time I have to agree with you is under hype, because I didn't hear about that at all.
00:17:21:20 - 00:17:25:03
Adam
Yeah. You know what? It was. So obviously.
00:17:25:07 - 00:17:31:13
Adam
It was. You're correct. It was so under hype that you said it. And I was like, Jeff, what? Who's Jeff?
00:17:31:15 - 00:17:33:02
Adam
Oh who's Jeff? Yeah.
00:17:33:03 - 00:17:35:10
Adam
Who said anything about all.
00:17:35:12 - 00:17:39:12
Martin
There you go. So that is my under hype story of the week.
00:17:39:14 - 00:17:41:17
Adam
That's a good one. I didn't hear about it either.
00:17:41:18 - 00:20:20:08
Martin
In this unusual, surreal state I am in, of Adam and I agreeing briefly, however briefly, that is that was shipped up, brought to you by harness. You know, if either of our points of view got into your skin, please subscribe and tell us why. Drop it in the comments. Until next time, stop talking. Start shipping.
00:20:20:09 - 00:20:34:02
Martin
Nobody told it to hack this website. It was just given a normal research goal. It hit a barrier and it improvised. That's freaking me out a little, because that's actually a very subtle and smart way to do it.
00:20:34:04 - 00:21:22:23
Adam
How dare they do this, sir, they never contacted us. I'll say that shit out loud for free.
00:21:23:01 - 00:22:00:02
Adam
So far, so far. My my my takeaway from this episode so far is that everything we're talking about, these AI agents doing, you have personally done in your life, I'm.
00:22:00:04 - 00:22:04:02
Adam
I had to ask Claude to explain it to me three different times. I.