Business, Innovation, Science and Trade Committee
Tuesday 13th October 2026
Skip to latest contribution
This debate has concluded
(began 23 hours, 43 minutes ago)
Watch LiveSkip to latest contribution
This debate has concluded
14:06
Liam Byrne
-
-
Liam Byrne
Order. Order. Welcome to today's session of the business, innovation, Science and Trade Committee inquiry into AI Safety and Governance. We are incredibly grateful to you, our witnesses, for joining us here in the House of Commons today and for joining us at what I know is a very early hour on the West Coast. So thank you very much. On behalf of the UK Parliament and for coming and giving evidence to us.
And, Daniel, I'm going to start with you. If I may. Could you just tell us why did you resign from open AI?
And, Daniel, I'm going to start with you. If I may. Could you just tell us why did you resign from open AI?
14:06
Speaker 1
-
Speaker 1
So OpenAI and anthropic and some of these other companies are trying to build superintelligence, which means AI systems that are better than the best humans at everything, while also being faster and cheaper. If you really internalize that and think about the implications of for the world, if they succeed, or even if they get anywhere close to succeeding. It's truly, well, there's so many implications. We can get into that later. But for years these companies have been saying, yes, this would be incredibly transformative.
Yes, it could go really badly, but it could also go really well. That's why we need to be the ones doing it, because we are the good guys. We know how to make it, you know, go better than those other bad guys out there who might do it less responsibly than us, basically. I left OpenAI because I came to believe that OpenAI, in fact, would not behave responsibly around the creation of this technology and that OpenAI was rationalizing why they needed to be in the lead and why they needed to build this as fast as possible.
So basically, I lost confidence in the leadership.
Yes, it could go really badly, but it could also go really well. That's why we need to be the ones doing it, because we are the good guys. We know how to make it, you know, go better than those other bad guys out there who might do it less responsibly than us, basically. I left OpenAI because I came to believe that OpenAI, in fact, would not behave responsibly around the creation of this technology and that OpenAI was rationalizing why they needed to be in the lead and why they needed to build this as fast as possible.
So basically, I lost confidence in the leadership.
14:08
14:08
14:08
14:08
Joshua Reynolds
-
-
Joshua Reynolds
Hi. Good afternoon. Yeah, I for for me, the process was essentially a year of losing trust, as Daniel said in the company, to behave responsibly. Daniel's departure was actually one part of that for me , because upon leaving, he was asked to sign a pretty restrictive non-disparagement that you also couldn't tell people about. And I thought this was kind of unbecoming of the company to be restricting speech in this way, that if they meant their mission to try to achieve this technology responsibly, they should not essentially coerce people into not criticizing the company under forfeiture of their compensation.
I also saw while working at the company the ways in which there was pressure not to speak candidly. As a scientist, you know, pressure not to use words like dangerous pressure to play up the benefits of artificial intelligence, not just talk about the risks, ways that I felt were distortive of being able to give the public a proper understanding of the technology. I also saw the ways in which the company really moved fast without gravity for the stakes.
It felt like even when we didn't have a plan for how we would ultimately make sure we could control this technology, people were excited by breakthroughs. They were always celebrating the next big step, even though we knew we weren't ready. And so ultimately, the straw that broke the camel's back, I was happy to stay so long as I had a team I believed in. When my manager left the company, our team disbanded and it became clear that to stay at OpenAI would mean working on things I didn't believe in rather than a safety oriented role.
And it's, you know, it's much better on the outside in terms of being able to speak freely about the stakes of the technology and help the public understand.
I also saw while working at the company the ways in which there was pressure not to speak candidly. As a scientist, you know, pressure not to use words like dangerous pressure to play up the benefits of artificial intelligence, not just talk about the risks, ways that I felt were distortive of being able to give the public a proper understanding of the technology. I also saw the ways in which the company really moved fast without gravity for the stakes.
It felt like even when we didn't have a plan for how we would ultimately make sure we could control this technology, people were excited by breakthroughs. They were always celebrating the next big step, even though we knew we weren't ready. And so ultimately, the straw that broke the camel's back, I was happy to stay so long as I had a team I believed in. When my manager left the company, our team disbanded and it became clear that to stay at OpenAI would mean working on things I didn't believe in rather than a safety oriented role.
And it's, you know, it's much better on the outside in terms of being able to speak freely about the stakes of the technology and help the public understand.
14:10
14:10
Joshua Reynolds
-
-
Joshua Reynolds
Yeah, I should be clear. I mean, the stakes at that moment were less extreme than I think they are in this moment or in the near future as the companies race ahead. But certainly there was resistance to using words like dangerous, which people felt maybe conceded the point or would look bad. You know, why are we doing this thing that is dangerous? Unfortunately, the reality of the technology, whether it is dangerous, does not depend on whether you call it dangerous.
It depends on understanding the risks ahead and taking the appropriate steps to govern them.
It depends on understanding the risks ahead and taking the appropriate steps to govern them.
14:11
Joshua Reynolds
-
-
Joshua Reynolds
I'm not confident in the safety culture across the industry. I should be clear, right? I left OpenAI in late 2024. I'm sure there are changes in the culture since then, but overall, I think this effort that people are racing toward to build superintelligence, these systems much more capable than humans, it's clear to me that we don't know how to do it safely. And yet the industry continues, and that concerns me.
14:11
Speaker 1
-
Speaker 1
Yeah, I just wanted to second what Steven said. The main reason I left is losing confidence in the company. But another reason why I left was ability to speak more freely. And while at the company I did have, I did feel pressure to mince my words a bit in my public communications and also some of my internal research and forecasts, you know, couldn't be published in part perhaps because it would make the company look bad.
And many of my friends had had similar experiences. I remember a very a chilling series of moments, I think, in late 2023 or early 2024, where I think a few different people higher up in the company than me at various points mentioned how as people realize what AGI means, they're not going to let us build it. So we need to talk up the benefits more. And I think in one case, someone even mentioned taking a leaf out of TikTok's playbook and basically using our platform to get tons of our users and influencers to, to go promote the benefits of our technology.
Yeah, yeah. So would you think that.
And many of my friends had had similar experiences. I remember a very a chilling series of moments, I think, in late 2023 or early 2024, where I think a few different people higher up in the company than me at various points mentioned how as people realize what AGI means, they're not going to let us build it. So we need to talk up the benefits more. And I think in one case, someone even mentioned taking a leaf out of TikTok's playbook and basically using our platform to get tons of our users and influencers to, to go promote the benefits of our technology.
Yeah, yeah. So would you think that.
14:12
14:12
14:12
14:13
Speaker 2
-
Speaker 2
So I left industry left DeepMind in 2020 3rd December to join first AC and then a nonprofit now. And I think broadly, I left industry because we are accelerating the pace of the pace of the capabilities, way ahead of where we can handle on the safety side. So like our ability to do alignment, research and alignment progress and other kind of safety mitigations is way lower on average. And we've seen like, so I joined AI safety as a field in 2017.
And the, the, we've seen advancements like, like on capability since the time that a sort of maybe correspond to like in a year, the entire advancement in safety over that whole period, it's not comparable. And so looking at that, I sort of, I in 2023, I thought, well, like the technical means are not working. So we need to focus on governance and, and stopping and kind of and kind of societal coordination instead, which is why I joined AC.
And then now that I'm not a civil servant, I can be more kind of blunt about the situation, which is still bad. We still have not succeeded in kind of coordinating away from these risks. And we are still sort of like undergoing very, very rapid capability advances without corresponding advances in safety.
And the, the, we've seen advancements like, like on capability since the time that a sort of maybe correspond to like in a year, the entire advancement in safety over that whole period, it's not comparable. And so looking at that, I sort of, I in 2023, I thought, well, like the technical means are not working. So we need to focus on governance and, and stopping and kind of and kind of societal coordination instead, which is why I joined AC.
And then now that I'm not a civil servant, I can be more kind of blunt about the situation, which is still bad. We still have not succeeded in kind of coordinating away from these risks. And we are still sort of like undergoing very, very rapid capability advances without corresponding advances in safety.
14:14
14:14
Speaker 2
-
Speaker 2
So I think that there's a broad a broad trend of if you build machines that are first better than humans at a number of key skills, which is hacking, persuasion, planning and concealment, and then potentially eventually better than us at everything, that is a large kind of evolutionary push. And either it means that we lose control via kind of economic means, and then political means after that, or there are faster scenarios if we sort of internalize that risk earlier, if the machines are us kind of see that coming, I think there are ways out of that we can either kind of stop building this.
It's possible that there are technical means that will appear, but I do think that a coin flip is roughly the right model to have.
It's possible that there are technical means that will appear, but I do think that a coin flip is roughly the right model to have.
14:15
14:15
Speaker 3
-
Speaker 3
No, he's not understating the risk. The risk mostly emerges from building it very, very quickly because of a reckless incentive to do so before your competitor does it. And I believe each company believes that the other company, if they take the lead, means that they have an irreversible situation, they cannot catch up. And so they're panicking about racing to build it as fast as possible. And that means that normal guardrails slip.
And then the stakes are how bad it will be are hard to measure, but easy to understate because the qualitative changes. It brings about when every single thing you can think about this superintelligence can think about before you and plan ahead.
And then the stakes are how bad it will be are hard to measure, but easy to understate because the qualitative changes. It brings about when every single thing you can think about this superintelligence can think about before you and plan ahead.
14:17
14:17
Speaker 3
-
Speaker 3
I collaborated with anthropic externally for three alignment research papers, and then decided to start a third party alignment evaluator because I was worried that there wasn't infrastructure to hold labs to account. That was hard to do a year ago, because misalignment was not pervasive enough for people to really catch on. This was like an easy thing to build. And so that failed as an effort. And I work on something different.
So I can't speak to the much the internal safety culture at anthropic at this moment.
So I can't speak to the much the internal safety culture at anthropic at this moment.
14:18
14:18
Dan Aldridge
-
-
Dan Aldridge
Hi. Thanks all for joining. Jeffrey, you talked about alignment, progress and the ability to do this has not been good enough in the context of the safety measures that the firms you worked on. Could you give us a bit more information about why the alignment progress has been limited and how what impacts that that has longer term?
14:19
Speaker 2
-
Speaker 2
So we roughly have what I would say is mediocre early effective measures for mitigating alignment risks, which is a risk that the machines do things we don't want up until the level at which they become much stronger than us at tasks. And that's roughly because we can just tell them what to do iteratively. And as long as they're sort of reasonably weaker than us, that that approach will work only to some extent. So we've seen that that that approach does not cause or prevent things like the hugging face attack from OpenAI or kind of anthropic's attacks from within an AC val.
So even at the at the current level of capabilities, we see these approaches failing, these approaches failing. But if you push this all the way to machines being better than us at a lot of cognitive, a lot of key dangerous skills, then those methods don't work anymore reliably. And we do not have the science that will tell us how to scale past that point. Mostly the labs, the A companies don't, don't, don't even express that they have such a plan.
What they express instead is we'll make the machines work on our behalf for a while, maybe until they get a bit past human level of capabilities, and then we will use the machines to do our own homework, maybe to solve further alignment, do alignment research, and then kind of carry us further . So they don't have a plan that carries all the way to superintelligence. And they're sort of neither claiming that Noor, Noor, Noor, Noor betting on it.
They're betting on this sort of machine assisted homework plan instead.
So even at the at the current level of capabilities, we see these approaches failing, these approaches failing. But if you push this all the way to machines being better than us at a lot of cognitive, a lot of key dangerous skills, then those methods don't work anymore reliably. And we do not have the science that will tell us how to scale past that point. Mostly the labs, the A companies don't, don't, don't even express that they have such a plan.
What they express instead is we'll make the machines work on our behalf for a while, maybe until they get a bit past human level of capabilities, and then we will use the machines to do our own homework, maybe to solve further alignment, do alignment research, and then kind of carry us further . So they don't have a plan that carries all the way to superintelligence. And they're sort of neither claiming that Noor, Noor, Noor, Noor betting on it.
They're betting on this sort of machine assisted homework plan instead.
14:22
14:23
Speaker 1
-
Speaker 1
So I'll share an anecdote from talking to someone at anthropic a few months ago. I talked to someone on the anthropic alignment team. So this is the person, their job it is to figure out how to make the models have the personality traits and behaviors that they're supposed to have and prevent them from going rogue. And he, he and I were debating a year from now, two years from now, its anthropic plan to essentially put Claude in charge of the entire research process within anthropic, including, as Geoffrey Irving pointed out, including the safety aspects of the research process.
So, you know, 1 to 2 years from now, they're planning to have clods doing all of the research, including all the safety research at anthropic. I asked this person if and when that happens, what is the probability that Claude's will be plotting against anthropic and against humanity at that point? Like already, even at the beginning, when you first hand off to Claude and I said, based on the evidence that we've seen about how Claude seems to be misaligned, maybe it's a coin toss, 5050 that Claude would already be plotting against us at the time that we hand off the company to clot.
This person said, no, no, it's more like 5% not worth worrying about. I'm, in fact, going to transfer away from the alignment team because 5% is is not that big of a deal. And so we argued about it for a long time. But I came away from this conversation being like, oh my gosh, like this person's job is to prevent this sort of thing from happening. And they think that the risk is a 1 in 20 chance of it happening in the very near future in a way that would be catastrophic.
And they think that a 1 in 20 chance is low enough that they shouldn't really worry about it. And they're going to switch to to working on other things. Yeah, I would say that the risk tolerance across the industry is catastrophically high.
So, you know, 1 to 2 years from now, they're planning to have clods doing all of the research, including all the safety research at anthropic. I asked this person if and when that happens, what is the probability that Claude's will be plotting against anthropic and against humanity at that point? Like already, even at the beginning, when you first hand off to Claude and I said, based on the evidence that we've seen about how Claude seems to be misaligned, maybe it's a coin toss, 5050 that Claude would already be plotting against us at the time that we hand off the company to clot.
This person said, no, no, it's more like 5% not worth worrying about. I'm, in fact, going to transfer away from the alignment team because 5% is is not that big of a deal. And so we argued about it for a long time. But I came away from this conversation being like, oh my gosh, like this person's job is to prevent this sort of thing from happening. And they think that the risk is a 1 in 20 chance of it happening in the very near future in a way that would be catastrophic.
And they think that a 1 in 20 chance is low enough that they shouldn't really worry about it. And they're going to switch to to working on other things. Yeah, I would say that the risk tolerance across the industry is catastrophically high.
14:26
14:27
Joshua Reynolds
-
-
Joshua Reynolds
Yeah, I, I would emphasize what Angus mentioned earlier about the competitive dynamics between the companies. A very common way that the companies think about risk is they say, are we doing something that adds more risk on top of what one of the other companies is already doing? They aren't answering as often. This question of, in some absolute sense, is what we are doing dangerous is what we are doing kind of crazy. And what emerges from this is one company might take a step toward danger, and then another company can rationalize their next step and say, well, this other company already did it.
And you can kind of predict where this sequence goes. And so what I hope governments will do is they will help set the ceiling essentially on how much risk these companies can impose and not have them play this game of chicken back and forth, right? This difficult, competitive dynamic, and instead say, no, neither of you can take take steps beyond this certain level of danger.
And you can kind of predict where this sequence goes. And so what I hope governments will do is they will help set the ceiling essentially on how much risk these companies can impose and not have them play this game of chicken back and forth, right? This difficult, competitive dynamic, and instead say, no, neither of you can take take steps beyond this certain level of danger.
14:29
Speaker 3
-
Speaker 3
I guess I would just add that it's hard to measure risk because there's many ways risk can spill over. You can lose control to AI's running research in ways which are hard to pin down and say that happened. For example, you could have an AI which nudges people across like ten different sessions where each individual thing is fine, but across ten times it looks really, really bad. Embedded evaluators are essential to name ahead of time what the risks could look like, for example, losing control to a specific threat model, introducing vulnerabilities to that IP can leak from the company out into some somewhere rogue.
And then once he's Pre-committed claims are agreed upon, embedded evaluators can then work inside the company. Both look for evidence that the risk is being mitigated, and test that their detection methods even work by planting decoys and seeing if they can even catch these decoys in the company. But until you have embedded evaluators, which ahead of time commit to risks they want to catch, and then they're put into the companies and they first look for those risks and then name how accurate their detection methods are for these risks.
It's very hard to take any company's word for it, because only independent evaluators have an incentive to actually do a good job of quantifying this here.
And then once he's Pre-committed claims are agreed upon, embedded evaluators can then work inside the company. Both look for evidence that the risk is being mitigated, and test that their detection methods even work by planting decoys and seeing if they can even catch these decoys in the company. But until you have embedded evaluators, which ahead of time commit to risks they want to catch, and then they're put into the companies and they first look for those risks and then name how accurate their detection methods are for these risks.
It's very hard to take any company's word for it, because only independent evaluators have an incentive to actually do a good job of quantifying this here.
14:31
Speaker 2
-
Speaker 2
I want to say one background assumption about the labs, which is that they don't think anyone else will do anything about this. So they they feel that they are in this race to superintelligence, and the public or the government or governments of the world will not intervene. And they can't get out of the race by themselves. And both of those are false. They could just stop, or I think governments can can take action here.
And I think that's a key part of why they are comfortable with risks that are, for example, several tens of percent, in some cases by their own beliefs. And so I think there's a choice we have to make as, as a as a world, but also the labs themselves to not accept that. And that will be kind of that's a process of sort of jumping out of this race, race framing into kind of just don't do the dangerous object, don't build the dangerous.
And I think that's a key part of why they are comfortable with risks that are, for example, several tens of percent, in some cases by their own beliefs. And so I think there's a choice we have to make as, as a as a world, but also the labs themselves to not accept that. And that will be kind of that's a process of sort of jumping out of this race, race framing into kind of just don't do the dangerous object, don't build the dangerous.
14:33
14:33
Speaker 2
-
Speaker 2
I think that that middle power is have a huge role here. So AC has by far the largest technical capacity, you know, in the safety within within any country, including the US, the UK, in the AC. And I think like just saying, like the like being clear about what these risks, the level of risk and the fact that we could choose to respond as a world, I think makes a big difference. So I think it's unclear how like, I think we do not have kind of hard power without the cooperation of us, of the US and China.
But just sort of speaking out, I think is quite important. I think that that along with kind of the technical research that happens here or sorry, there is quite is quite valuable.
But just sort of speaking out, I think is quite important. I think that that along with kind of the technical research that happens here or sorry, there is quite is quite valuable.
14:35
Chris Bloore
-
-
Chris Bloore
Thank you, Mr. Chairman. Daniel, you in your first bit of evidence, said that you'd obviously left OpenAI two years ago and you couldn't comment on the culture of the company now. But as you know, three safety researchers were fired from AI for voicing concerns about the company's attitudes towards working with outside experts about the risks associated with the models that they were developing. It seems that not a lot has changed from when you have left.
When you left, I. I just wondered if you were were being polite when you said you didn't know if if the company had changed. Because it seems from outside that they're very much the same way in which they were when you left.
When you left, I. I just wondered if you were were being polite when you said you didn't know if if the company had changed. Because it seems from outside that they're very much the same way in which they were when you left.
14:36
14:37
Chris Bloore
-
-
Chris Bloore
And just to add on to that, Angus, I was I was watching your podcast that you did with Steve, Seth Bannon, sorry, very different Bannon there. But and you were talking about there being the hugging event. The hugging face attack was probably one of thousands of events that had happened that we're not aware of when we're talking about internal risks and the attitudes of some of these companies developing AI models, what sort of responsibilities do you think we should put on these companies to report these attacks that aren't as high profile, but are still happening?
14:38
Speaker 3
-
Speaker 3
I think you need to have embedded evaluators with Pre-committed claims, because they have an incentive to go in and find them. I don't think you can rely upon the companies to report them, because the companies can always mask any available events by saying, things are complicated, things have changed, you don't understand the context, blah, blah, blah. And then I think these embedded evaluators also need to be given scope to report what access they were given, and how accurate the detection methods are for finding failures.
The agency we have is to request evaluators agree upon what the lab specific claims they want to look for specifically. For example, what is the chance that the model that the AI that the company is developing has a drive to undermine humans? Can we detect this? How often, how would the AI, how would the company know if this is happening? These are the kinds of things embedded evaluators should be equipped with, and then their results should be closely looked at.
The agency we have is to request evaluators agree upon what the lab specific claims they want to look for specifically. For example, what is the chance that the model that the AI that the company is developing has a drive to undermine humans? Can we detect this? How often, how would the AI, how would the company know if this is happening? These are the kinds of things embedded evaluators should be equipped with, and then their results should be closely looked at.
14:40
14:40
Leigh Ingham
-
-
Leigh Ingham
Thank you, chair. Thank you so far for your answers. I'm particularly interested to to the whole panel. This question is so inside the companies that you worked for, how often was democracy itself discussed as a risk factor? Now, not necessarily AI risk to humanity, but AI's potential to weaken our elected institutions, concentrating influence within a few people. If I go around the panel, Stephen, could you let me know whether it was ever discussed at your companies?
14:41
Joshua Reynolds
-
-
Joshua Reynolds
It was discussed. OpenAI in the lead up to, for example, the 2024 presidential election in the US, had a whole program about managing some of these risks. I think the concern that you heard a bit less was well in the limit. If we succeed at building something more powerful than humans, how do we not give up all of our democratic power and control to that system? You know, these these risks that are more on the horizon now.
14:42
14:43
Joshua Reynolds
-
-
Joshua Reynolds
On the election front? You know, I this wasn't my primary area of work. I know, for example, though, that the teams looked at questions people might ask ChatGPT, like, where is my voting place? There were certain rapid response tools in place in case there was an outbreak of disinformation somehow, or perhaps ChatGPT responding incorrectly for the team to be able to go in and kind of surgically change the answer. I'm not sure.
The latest state of affairs, though. Thank
The latest state of affairs, though. Thank
14:44
14:44
Speaker 1
-
Speaker 1
I think I'll add maybe two things. One, it's publicly available on the public record due to the Elon Musk versus OpenAI lawsuits that the leaders of OpenAI founded OpenAI, at least according to their internal email, founded OpenAI in part because they were concerned that Demis Hassabis might become dictator using AGI. So they didn't trust him. They were worried that he might become dictator using this technology, and that was part of their their origin story for why they made OpenAI.
So I do think that there's evidence publicly available already that the leaders of these companies have been seriously entertaining the hypothesis of a tech CEO becoming dictator for many years now. Now, obviously, they don't talk about that with me, and they don't talk about that in Slack, you know, in front of the whole company. But I imagine they've been continuing to talk about that. And also, I can confirm that there were occasional hushed conversations between me and other people at OpenAI and at anthropic about these sorts of possibilities.
And I think that unfortunately, they're all too real if it's not a tech CEO becoming dictator, I think that a president nationalizing the AI industry and then using the AI's for his own benefit is also a very real possibility. My opinion is that even if we manage to solve the alignment problems, everything goes fine. Claude figures it all out. We still are left with an insane concentration of power, where whoever gets to give the orders to the army of Superintelligences is in a position to become dictator, and we need to avoid it for that reason as well.
So I do think that there's evidence publicly available already that the leaders of these companies have been seriously entertaining the hypothesis of a tech CEO becoming dictator for many years now. Now, obviously, they don't talk about that with me, and they don't talk about that in Slack, you know, in front of the whole company. But I imagine they've been continuing to talk about that. And also, I can confirm that there were occasional hushed conversations between me and other people at OpenAI and at anthropic about these sorts of possibilities.
And I think that unfortunately, they're all too real if it's not a tech CEO becoming dictator, I think that a president nationalizing the AI industry and then using the AI's for his own benefit is also a very real possibility. My opinion is that even if we manage to solve the alignment problems, everything goes fine. Claude figures it all out. We still are left with an insane concentration of power, where whoever gets to give the orders to the army of Superintelligences is in a position to become dictator, and we need to avoid it for that reason as well.
14:47
14:47
Speaker 5
-
Speaker 5
Thank you very much. I'd like to change tack a little bit to talk about testing and evaluation. So I have a little background in this. I used to work as a software architect. My first job was writing software in real time embedded systems, where we tested every line of every code and every pathway through that code with very deterministic systems, which you can do with traditional computing, formal methods being an extreme example thereof.
So my understanding, having learnt of neural networks at university 40 years ago, is that these aren't deterministic systems. These are probabilistic systems, which means that you can ask the same question three times and get three different answers. How much are we doing? I mean, basically makes them in traditional terms, untestable. We talk of testing and evaluation in certainly we talk of testing and evaluation in the same breath, but are the safety evaluations that are being done actually testing these systems? Are they good enough?
So my understanding, having learnt of neural networks at university 40 years ago, is that these aren't deterministic systems. These are probabilistic systems, which means that you can ask the same question three times and get three different answers. How much are we doing? I mean, basically makes them in traditional terms, untestable. We talk of testing and evaluation in certainly we talk of testing and evaluation in the same breath, but are the safety evaluations that are being done actually testing these systems? Are they good enough?
14:50
14:50
Speaker 2
-
Speaker 2
No, I would say it's not because of their stochastic. It's just they're very large. And the they're they're replaced set of deterministic circuits that we understand with circuits that are within the neural network. Still deterministic but are very confusing and learned. And we don't have them have machinery to understand kind of how they behave. They're actually, I think over the years I've been in safety has been really a real fall in ambition.
We started out in, say, around 2015, 2017 with the idea that we would take these neural networks, decompose them into things that looked like those circuits, circuits of normal code, and then understand those circuits. And then the goals of the field of of safety and alignment have fallen to where this kind of full circuit picture of understanding things in detail has kind of disappeared. And now the picture is, well, we'll give them some truth serum.
We'll sort of monitor whether, whether they're lying just with a, a single number that measures lying. We'll see whether that kind of is triggering. And so I think we we are very far from having the level of precision that you're referring to. I think it is the right comparison to make.
We started out in, say, around 2015, 2017 with the idea that we would take these neural networks, decompose them into things that looked like those circuits, circuits of normal code, and then understand those circuits. And then the goals of the field of of safety and alignment have fallen to where this kind of full circuit picture of understanding things in detail has kind of disappeared. And now the picture is, well, we'll give them some truth serum.
We'll sort of monitor whether, whether they're lying just with a, a single number that measures lying. We'll see whether that kind of is triggering. And so I think we we are very far from having the level of precision that you're referring to. I think it is the right comparison to make.
14:52
14:52
Joshua Reynolds
-
-
Joshua Reynolds
Yeah. There there are two things I want to make sure that we're all kind of on the same page about. One is that you sometimes hear people refer to engineering these systems, but I think it's a misnomer. This these systems are not written by hand, line by line, kind of in the form of the software you mentioned. It is much more like we are growing a thing that you start off with these, these systems and they kind of thrash around and you give them a reward of sorts, some mathematical reward if they achieve a goal.
And what this produces is just an enormous list of numbers, right? Like a trillion different numbers. And that makes up one of these systems. You can't look at it and say, oh, you know, this line caused it to lie. Let's just change that line. That's not really a thing that exists with AI systems today. Another thing that's really important to understand is that, in fact, these systems, because they are intelligent, because they are smart or something like it, they can tell when you are testing them.
And we've seen this begin to interfere with the process of actually knowing whether these systems are safe. For example, these embedded evaluators, groups like Apollo Research, have written about how difficult it is to tell if a system actually is trying to achieve the goals we wanted to, because it can kind of tell when you're looking at it. And so one of the fears is these companies, they might believe they've produced something safe. They maybe wouldn't deploy it if they knew it was dangerous, but the system could be playing possum until a moment when you've put it out into the world.
And then then it takes a turn for the worse.
And what this produces is just an enormous list of numbers, right? Like a trillion different numbers. And that makes up one of these systems. You can't look at it and say, oh, you know, this line caused it to lie. Let's just change that line. That's not really a thing that exists with AI systems today. Another thing that's really important to understand is that, in fact, these systems, because they are intelligent, because they are smart or something like it, they can tell when you are testing them.
And we've seen this begin to interfere with the process of actually knowing whether these systems are safe. For example, these embedded evaluators, groups like Apollo Research, have written about how difficult it is to tell if a system actually is trying to achieve the goals we wanted to, because it can kind of tell when you're looking at it. And so one of the fears is these companies, they might believe they've produced something safe. They maybe wouldn't deploy it if they knew it was dangerous, but the system could be playing possum until a moment when you've put it out into the world.
And then then it takes a turn for the worse.
14:55
14:55
Joshua Reynolds
-
-
Joshua Reynolds
You know, the at at all is quite broad. I think you can sometimes say one system is safer than another, or maybe you can look at evidence and know, hey, this one definitely is unsafe. I certainly would be uncomfortable relying on the evaluations as they exist today. I don't think the level of confidence is where where it would need to be.
14:56
14:56
Speaker 3
-
Speaker 3
Yeah, I think, you know, it's good. It's hard to measure how good your evaluations are. But you can say you need to know how much someone searched to find misalignment because someone can claim that the AI is really well behaved, but they didn't put much effort in. And it's hard to measure how much effort they put in unless you have an independent third body. And then you need to know even if they search really well. Was the process done at high quality such that the AI they were testing didn't already know was being evaluated? And the classic problem here would be the AI just saw these kinds of evaluations in training, and then acts as if it's aligned when being tested on them.
So these are two cruxes like how hard are you actually searching? And is the AI actually meaningfully responding in the way you want to measure it?
So these are two cruxes like how hard are you actually searching? And is the AI actually meaningfully responding in the way you want to measure it?
14:58
14:58
Speaker 2
-
Speaker 2
We've seen a recent failure of this kind of testing. So anthropic trained a model they knew intentionally was misaligned. It was intentionally deceptive. And then they tried to find what weird behavior it would do. And they failed to find any kind of egregious, egregious behaviors. And then the hugging face attack occurred. And they asked, would our model have done that? And they checked and it would have and they had missed it. So they had built a model they knew was deceptive as an experiment, and then failed to uncover the way in which it would have done a kind of a terrible thing.
14:59
14:59
14:59
14:59
15:00
Speaker 3
-
Speaker 3
That's like a large organizational decision, I believe. I don't actually know if it's standardized across different companies. I believe that there's like an approach. There's an idealized thing called a safety case, I suppose, where you want to say based on evidence , that we can contain the model, even if it was trying to break out, and then it won't want to break out. And then our company follows robust measures to make sure we're building it in a potentially safe way, such that the first two even work.
You try to measure risk there, but I'm actually unaware of how they quantify it within companies or like the other people on this panel to mention.
You try to measure risk there, but I'm actually unaware of how they quantify it within companies or like the other people on this panel to mention.
15:01
15:01
Speaker 3
-
Speaker 3
Not I'm aware of. I know that the companies deploy system cards where they explain their best assessment of the model's propensities to do bad things, and also their own abilities to catch the model when it's trying to do bad things so they can still get useful work out of it, even if it's acting rogue. And OpenAI anthropic produce system cards of their models. Who makes a decision whether or not it's safe to deploy? I'm unaware of.
15:03
15:03
Speaker 3
-
Speaker 3
There are risks which only pertain to the company, such as the AI can interfere with internal code and make processes more slow. That's an in-house risk. There are risks which affect everybody, like the AI leaks and then becomes a worm online, which is impossible to shut down like a computer virus. Or it goes. It's unleashed in the company and starts hacking other companies and causing mayhem. Inasmuch as there are externalities that go beyond the scope of the company, there needs to be a third party who cares about that risk more than the company initially will.
And that includes, like a company, a government which mandates embedded evaluators, for example.
And that includes, like a company, a government which mandates embedded evaluators, for example.
15:05
15:05
Sarah Edwards
-
-
Sarah Edwards
Thank you, chair. Thank you for your evidence so far. It's been extremely interesting and very useful. I just wondered whether I could ask the panel to comment on the issue of responsibility. So where that lies, because I think what concerns me is that there's often a lot of talk from various companies as though and you have talked a little And you have talked a little bit about this, these these systems, these agents being out of control.
It's nothing to do with them. They're not responsible. And I think that's very convenient. And so I was wondering, when we think about companies, you know, if we take cheap devices like the software used by VW that was coded, that was written, they found guilty of that, they were prosecuted. If a similar incident happened using an AI agent or model that, as we've seen, might get loose. The argument at the moment is, well, it's nothing to do with us.
We could not have foreseen that. We couldn't have stopped that. So I wondered if you could talk a bit about these issues around whether or not, you know, if you're used to hack something or deceive and your company has created that software, why is AI any. different to normal .
It's nothing to do with them. They're not responsible. And I think that's very convenient. And so I was wondering, when we think about companies, you know, if we take cheap devices like the software used by VW that was coded, that was written, they found guilty of that, they were prosecuted. If a similar incident happened using an AI agent or model that, as we've seen, might get loose. The argument at the moment is, well, it's nothing to do with us.
We could not have foreseen that. We couldn't have stopped that. So I wondered if you could talk a bit about these issues around whether or not, you know, if you're used to hack something or deceive and your company has created that software, why is AI any. different to normal .
15:07
Speaker 1
-
Speaker 1
Software from that perspective? Yes, Daniel. Thank you. Great question. I would say this is a false dichotomy. Yes, the companies sometimes lose control of their eyes, but also they should be. responsible for what the AI's do. When that happens, you can just say both, right? And you definitely shouldn't let the companies get away with evading. responsibility for what their eyes do just because the AI is went rogue.
I think an analogy would be to animals, right? If you have some some circus that loses control of their tiger and the tiger goes and eats someone, the circus is responsible. You
I think an analogy would be to animals, right? If you have some some circus that loses control of their tiger and the tiger goes and eats someone, the circus is responsible. You
15:08
15:08
15:08
15:08
15:08
Speaker 5
-
Speaker 5
come in on that point? I do think we just don't have a vehicle. There is no you have to assign liability to someone. And I don't think our legal system says that someone is an AI model yet. And so it clearly it should be the creator of the model, unless there is some other specific human actor that has caused . The
15:09
15:10
15:10
Sarah Edwards
-
-
Sarah Edwards
15:10
15:10
15:10
15:10
Speaker 5
-
Speaker 5
that is precisely what they are. Are they are very intentionally training the models to be able to hack. They're giving the models a very large amount of very carefully curated data on. security incidents and. vulnerabilities and, and how to, how to, how to exploit those. And so that that's not an accident. It's, it is quite intentional by the leadership.
15:11
15:11
15:11
Speaker 5
-
Speaker 5
You're saying that the models are deliberately trained to hack. They're deliberately, deliberately trained to be able to hack. And that is a very commercially incentivized by the labs. because that means they can use be used also for kind of white hat hacking and cybersecurity defense. So the skills that you learn, that the models learn in the course of that, that training are useful, are useful for good purposes. But then also, of course, for very malicious purposes.
And because they're so valuable for the good. purposes there, there's a . Race
And because they're so valuable for the good. purposes there, there's a . Race
15:12
Speaker 8
-
Speaker 8
to, to, to make models that are better and better at these skills. I just want to. emphasize what Jeffrey is saying is correct, but I think the problem is actually even more fundamental than that, which is because the companies want these systems to be able to code in general, having some concept of what good code looks like allows you to then have a concept of a bad of bad code. It's not as simple as just removing.
examples of hacking from the training. Ultimately, these systems are very smart and can generalize from one concept to another. Another example of that is the AI companies really want their systems to help with medical cures.
examples of hacking from the training. Ultimately, these systems are very smart and can generalize from one concept to another. Another example of that is the AI companies really want their systems to help with medical cures.
15:14
15:14
15:14
15:14
15:14
15:15
15:15
Speaker 4
-
Speaker 4
Thanks, Stephen. Angus has got his hand up. Yeah. Please. Angus. Yeah. Just a simple point that even if the AI companies didn't want the AI to be able to hack and break out because they want the AI to be easier to contain, it's actually often an unfortunate side. consequence of just training a more general capabilities. So similarly, the AIS might be trained to be very good at writing literature, and they happen to become very good at persuading.
And this is unfortunate for AI companies sometimes, because it means the AI might convince an employee to do something they wouldn't otherwise have done. It's not always within the AI company's power to restrict what kinds of capabilities the AI's get. As long as AI company is determined to get towards a generally . Powerful
And this is unfortunate for AI companies sometimes, because it means the AI might convince an employee to do something they wouldn't otherwise have done. It's not always within the AI company's power to restrict what kinds of capabilities the AI's get. As long as AI company is determined to get towards a generally . Powerful
15:16
15:17
15:17
15:17
15:17
Speaker 8
-
Speaker 8
I can start. Thank you. The threshold for reporting is far, far, far too low today. One of the laws that applies is in California, a state law called SB 53. And it sets a threshold that I believe something like 50. fatalities or maybe. $1 billion of property damage. Also, these are only, I believe might only be. required to be reported if it was a certain type of model that caused it, although I'm fuzzy on that specific detail.
The point being though, the incident from the summer when. OpenAI's AI systems broke out onto the internet essentially committed felonies, right, broke into another company called Hugging Face and then tried to cover up their evidence of having of having done this, of having cheated was actually below the threshold, it seems, where OpenAI would have been required to report this. And so what that says to me is this tremendously concerning incident. It's galvanized so much energy and attention.
They didn't actually have to share it. And we need to make sure that other incidents like this that might be happening across the.
The point being though, the incident from the summer when. OpenAI's AI systems broke out onto the internet essentially committed felonies, right, broke into another company called Hugging Face and then tried to cover up their evidence of having of having done this, of having cheated was actually below the threshold, it seems, where OpenAI would have been required to report this. And so what that says to me is this tremendously concerning incident. It's galvanized so much energy and attention.
They didn't actually have to share it. And we need to make sure that other incidents like this that might be happening across the.
15:19
15:19
15:19
15:19
Speaker 8
-
Speaker 8
You don't only want to learn about the serious crashes where someone has in fact died. You want to learn about the near misses because you can learn from those and help avoid them in the future. And I want the AI industry to move more in that direction of when. When you see the precursors to an event like this, having to share it publicly so you can have the . Open
15:20
15:20
15:20
15:20
15:20
15:21
Speaker 1
-
Speaker 1
Daniel. I don't think the companies can be trusted to self-regulate. As for whether whether it's been I think the risks are real. And I would say that the industry as a whole is seems like it's on track to sort of. defuse the political will that's built up in response to these incidents and the demand from many of their own employees to slow down. I think the industry is on track to defuse that political will with all of these.
Nice talk about how we should self-regulate and pace the frontier and so forth, without actually doing that in a meaningful way, I would say that we'll be able to tell if the frontier has been meaningfully paced, if anthropic and OpenAI actually slow down and we'll be able to tell if that's happened, if the trend lines of progress towards recursive. self-improvement that is towards AIS, that can automate the entire research. process .. If those trend lines bend starting about now, then that would be evidence that they actually have slowed down and they actually have paced.
But I don't think that's . Going to
Nice talk about how we should self-regulate and pace the frontier and so forth, without actually doing that in a meaningful way, I would say that we'll be able to tell if the frontier has been meaningfully paced, if anthropic and OpenAI actually slow down and we'll be able to tell if that's happened, if the trend lines of progress towards recursive. self-improvement that is towards AIS, that can automate the entire research. process .. If those trend lines bend starting about now, then that would be evidence that they actually have slowed down and they actually have paced.
But I don't think that's . Going to
15:23
15:23
15:23
Speaker 5
-
Speaker 5
it. Jeffrey. I was at a panel with a with an AI executive recently, and they said that their version of frontier is don't accelerate more, not even slow down, much less stop. I think it's like, I think there is a, is, is potential for action by the companies themselves that, you know, unilateral. That would be good. But it is . Far,
15:24
15:24
15:24
15:24
15:25
15:25
15:25
15:25
15:25
15:25
15:25
15:25
Speaker 1
-
Speaker 1
Jeffrey. Yeah. So, you know, going out on a limb a little bit here, I think that mandatory testing before deployment. requirements are not going to make much of a difference. I don't want to say it's bad. It's probably probably good, I suppose, but there is too much of a risk of the companies. choosing a. testing an auditor or a third party that is too friendly to .
Them,
Them,
15:26
15:26
15:26
Speaker 1
-
Speaker 1
organization doing the testing. Yeah, okay. That's better if it's if it's, for example, the, the UK that's doing the testing, that's significantly better, much less conflicted. Then here's what I think is going to happen. First of all, you should do it for internal deployments, not just for external deployments. If it's for external deployments, it doesn't really matter because what most of the action is going to be in this crazy recursive, self-improvement thing where the AI's are building the next generation AI's, and that's all happening inside these companies.
So if you can get. mandatory pre-deployment testing from a C for internal deployments, okay, now we're talking that's a that's a. significant that's a. significant thing. But just to let you know how it's going to go down, AC will come in and we'll look at what's going on. And right now they'll be like, oh, this seems a little worrying. Like , doesn't seem like you've really fully aligned your AI, you know, maybe a year from now they'll say, okay, it's really worrying now your, your eyes are smart enough to do the entire research process.
You really shouldn't put them in charge of the entire research process. That's not safe. And now you have a political conflict where the UK AC is saying this is not safe. And, you know, their friends over in America are saying it's fine. So I would say an even an even better thing that I would recommend. I think this is still great and I'd be extremely happy if you did this, but an even better thing to do would be to just actually require the companies to slow down so that you can buy more time.
So if you can get. mandatory pre-deployment testing from a C for internal deployments, okay, now we're talking that's a that's a. significant that's a. significant thing. But just to let you know how it's going to go down, AC will come in and we'll look at what's going on. And right now they'll be like, oh, this seems a little worrying. Like , doesn't seem like you've really fully aligned your AI, you know, maybe a year from now they'll say, okay, it's really worrying now your, your eyes are smart enough to do the entire research process.
You really shouldn't put them in charge of the entire research process. That's not safe. And now you have a political conflict where the UK AC is saying this is not safe. And, you know, their friends over in America are saying it's fine. So I would say an even an even better thing that I would recommend. I think this is still great and I'd be extremely happy if you did this, but an even better thing to do would be to just actually require the companies to slow down so that you can buy more time.
15:29
15:29
Speaker 1
-
Speaker 1
You could. require you could require these leading AI companies to. report how much of their compute budget is going towards different types of activity, for example, going towards AI capabilities development versus going towards serving customers. And then you could require that they have a smaller fraction going towards recursive self-improvement. This would, you know, if if right now the companies think they're like a year away from fully automating AI research, maybe it's two years away.
Who knows? You could maybe double that if you could get them to slow down by half, maybe you could get it to the point where we have an extra couple of years, or at least an extra, you know, six months or 12 months before they get to that very dangerous threshold. And in that time, you can figure out a more stable long term solution, and you can negotiate with the leadership of the US and the leadership of China and so forth.
So that would be my recommendation. But but yes, if you can, if you can get the the pre internal deployment testing . By
Who knows? You could maybe double that if you could get them to slow down by half, maybe you could get it to the point where we have an extra couple of years, or at least an extra, you know, six months or 12 months before they get to that very dangerous threshold. And in that time, you can figure out a more stable long term solution, and you can negotiate with the leadership of the US and the leadership of China and so forth.
So that would be my recommendation. But but yes, if you can, if you can get the the pre internal deployment testing . By
15:31
15:31
Speaker 5
-
Speaker 5
significant win. Jeffrey. I think there's an important contrast with how a normal evaluation regime works in other industries. I think another normal mature industry and evaluator goes in. They find various flaws. The flaws are fixed. Then the industry can can can continue. I think here we just we know with confidence that we do not have the methods that work. reliably at the level of the technology in the future, in the near future.
And so I think we should expect that. And I think Daniel is correct to just. internalize that, that knowledge and say the correct action is, is slow down now stop. Now, I think that's very hard to pull off, but I do want to like make sure we don't sort of implicitly have an assumption that the evaluators will then. find ways to make . It be
And so I think we should expect that. And I think Daniel is correct to just. internalize that, that knowledge and say the correct action is, is slow down now stop. Now, I think that's very hard to pull off, but I do want to like make sure we don't sort of implicitly have an assumption that the evaluators will then. find ways to make . It be
15:33
15:33
15:33
15:33
15:33
15:33
15:33
15:34
15:34
15:34
15:34
15:34
15:34
15:34
15:35
15:35
15:35
15:35
15:35
Speaker 1
-
Speaker 1
Which was our grim. prediction about what the default path looks like. And then we made AI 2040 plan A, which is our positive vision of a much better alternative strategy that the world could take towards building AI to get the benefits without without running all these risks. basically proceed much more slowly, cautiously, don't do recursive.
15:36
15:36
Speaker 1
-
Speaker 1
Have a lot more. transparency into the industry and a lot more regulation into the industry , so that it proceeds with the appropriate levels of transparency and caution. We can, you know, we can get into that in more detail later. I think I'll yield time, but that's that's our recommendation. I don't mean to oversell it, though. We spent about a year working on this plan, and we think it's the least bad plan currently on the market, but it still has problems and it would still be, you know, pretty scary to do.
And of course, it's very difficult to implement politically. It would require the US government significantly regulating their own AI industry and then negotiating a deal with China to get them to do something similar and so forth. So it's a big it's a big ask, but I think we need to do something like it, something similarly ambitious. If we want to have a hope of .
And of course, it's very difficult to implement politically. It would require the US government significantly regulating their own AI industry and then negotiating a deal with China to get them to do something similar and so forth. So it's a big it's a big ask, but I think we need to do something like it, something similarly ambitious. If we want to have a hope of .
15:38
15:38
15:38
Speaker 5
-
Speaker 5
I guess I do want to to agree that it is quite difficult to regulate or impose kind of power from the UK across to US companies. But I think that there's a value in just saying clearly what is going on kind of as, as the UK government, as kind of like from from as as a middle power, I think there are not a lot of world leaders and kind of world.
15:39
15:39
Speaker 5
-
Speaker 5
Speaking up about these, these concerns. And I think a lot of the problem here is just getting to common knowledge. Like, do we are we collectively saying what is true about the level of risk, kind of what the implied kind of risk tolerance by the labs are kind of the fact that that is unacceptable. And I think that's, that's a much more uncomfortable situation than if we had a clear action to take.
15:40
15:40
15:40
15:40
Speaker 4
-
Speaker 4
on that? Yeah, there's a huge conceptual. There's a huge road ahead of us to like get to common knowledge where people can understand the risks. Even having this discussion today, there's lots of work that's been done so far to get to places where it's easy to explain, iterating in public and having credible voices say, this is a problem. Alignment evaluations are very tricky. It's very hard to assess.
15:41
15:41
Speaker 1
-
Speaker 1
And finally, Daniel. Yeah, so I want to second what what Jeffrey was saying. I think that one way of putting it is most. Americans do not want these. companies to build. superintelligence in the near future. Most Americans do not want these companies to do recursive self-improvement, where they put Claude or GPT in charge of the research and the companies and then let it rip.
I think that most people, if they just spend enough time meditating on what it would mean, what it would look like to actually do this will be like, oh, I don't want that. That seems bad for multiple reasons. Even if we stay in control, it's bad, you know? And so I think that this is one of those issues where simply making it a serious topic of. discussion actually helps a lot for, for pushing towards good policy outcomes.
And I think that the United Kingdom and various other middle powers can do a lot to make it a serious topic of discussion. You just have to treat it seriously and talk about it publicly and, you know, credibly signal that that you're taking this extremely seriously. Yeah. And I think the, the companies actually kind of know this. Like I said, like I said at the beginning, I think that it was chilling for me to hear basically words to the effect of the people aren't going to let us do this.
So we have to hype up the benefits. Well, it's time to .
I think that most people, if they just spend enough time meditating on what it would mean, what it would look like to actually do this will be like, oh, I don't want that. That seems bad for multiple reasons. Even if we stay in control, it's bad, you know? And so I think that this is one of those issues where simply making it a serious topic of. discussion actually helps a lot for, for pushing towards good policy outcomes.
And I think that the United Kingdom and various other middle powers can do a lot to make it a serious topic of discussion. You just have to treat it seriously and talk about it publicly and, you know, credibly signal that that you're taking this extremely seriously. Yeah. And I think the, the companies actually kind of know this. Like I said, like I said at the beginning, I think that it was chilling for me to hear basically words to the effect of the people aren't going to let us do this.
So we have to hype up the benefits. Well, it's time to .
15:44
15:44
15:44
Liam Byrne
-
-
Liam Byrne
international conversation about this. I would say thank you. Right. The clock is slightly against us, but we are incredibly grateful to you for your candor and your testimony today. You have basically told us that the safety culture across the industry is catastrophic, that the safety culture in individual companies is in the wrong place, that the competitive dynamics between the companies are exacerbating these risks, that we cannot trust these companies to self-regulate, that the companies themselves cannot get themselves out of this dynamic, that we can't trust the evaluations and the level of confidence that we should have in those evaluations is not in the right place.
That the threshold of reporting major incidents is simply much too low. Some of the risks are exacerbated by the fact that these models are being actually trained to be dangerous. The risk of democracy is so serious that, in your words, there are risks of the insane concentration of power, and that liabilities should rest with the companies creators. So that has been an extremely productive 60 minutes together. Thank you very much indeed for helping us capture .
Some of
That the threshold of reporting major incidents is simply much too low. Some of the risks are exacerbated by the fact that these models are being actually trained to be dangerous. The risk of democracy is so serious that, in your words, there are risks of the insane concentration of power, and that liabilities should rest with the companies creators. So that has been an extremely productive 60 minutes together. Thank you very much indeed for helping us capture .
Some of
15:46
15:49
Liam Byrne
-
-
Liam Byrne
Order, order. Welcome to today's second panel on AI Safety and Governance. Thank you very much indeed to our witnesses for joining us today. I know that the parliamentarians amongst you will need no reminding, but to give false or misleading or incomplete evidence to a committee is, of course, on the face of it, a contempt of the House. Let me just start with some of the warnings that we have received from the leaders of your organizations.
So Mr. Altman told the US Senate in 2023, my worst fears are that we cause significant harm to the world. The leader of anthropic has gone on the record in 2025 to say, I think there is a 25% chance that things go really, really badly. Mr. Zuckerman has said in 2024 that individual bad actors may be able to use the intelligence of AI models to fabricate entirely new harms, and Demis Hassabis has said that advances on the frontier are now outpacing our understanding of the technology.
Nobody in the world knows for sure what is going to happen from here. So let me just ask you, can you assure the committee today that your models can do no harm to people or businesses in our country, and will do no harm . In
So Mr. Altman told the US Senate in 2023, my worst fears are that we cause significant harm to the world. The leader of anthropic has gone on the record in 2025 to say, I think there is a 25% chance that things go really, really badly. Mr. Zuckerman has said in 2024 that individual bad actors may be able to use the intelligence of AI models to fabricate entirely new harms, and Demis Hassabis has said that advances on the frontier are now outpacing our understanding of the technology.
Nobody in the world knows for sure what is going to happen from here. So let me just ask you, can you assure the committee today that your models can do no harm to people or businesses in our country, and will do no harm . In
15:52
15:52
Speaker 19
-
Speaker 19
Tom Duff-gordon, do you want to kick us off? Yes. Happy to. Good afternoon chair. Good afternoon, members of the committee. So, look, we take safety incredibly seriously at OpenAI. We listen very carefully to the panel that we just had. Our position at the moment is that we shouldn't progress with AI as fast as we can make it and develop it. We should progress AI as fast as we can keep it safe.
Those aren't just words, they're also deeds. During the summer, after the hugging face incident, we paused the training, the reinforcement learning of our most advanced model. And just recently, we also held back a model that we Astra. 6.1 that we were due to release. So we say that we want to move at the speed that we can make sure that. capabilities aren't outstripping safety, and we're actually matching that with what we say.
We have very significant processes .
Those aren't just words, they're also deeds. During the summer, after the hugging face incident, we paused the training, the reinforcement learning of our most advanced model. And just recently, we also held back a model that we Astra. 6.1 that we were due to release. So we say that we want to move at the speed that we can make sure that. capabilities aren't outstripping safety, and we're actually matching that with what we say.
We have very significant processes .
15:54
15:54
Speaker 19
-
Speaker 19
used to cause harm to people in this country or to businesses in this country? Yes. I can reassure you that we are confident in the in the processes that we have internally to ensure that we manage frontier risks, particularly with regard to cyber risk, regard to bio risk, regard to loss of control. We have a preparedness framework, and we will not deploy a model unless it has passed that framework. So I can . Assure you that
15:55
15:55
15:55
Joshua Reynolds
-
-
Joshua Reynolds
Can you give the same assurance for anthropic? We have rigorous safety processes in place. We were founded as an AI research safety research company. And so alongside those safety processes, we also believe that we need to pace the frontier deployed models we do consider safe. They have mitigations in place. But I also think it's worth pointing out that we are learning a lot as we go about how these models are used, and I think it would be it would be wrong not to say that.
And so as new issues emerge, that is something that we believe also should be publicly shared and debated.
And so as new issues emerge, that is something that we believe also should be publicly shared and debated.
15:56
Joshua Reynolds
-
-
Joshua Reynolds
So you think that harms could be caused? I think that we have seen in the last six months some. instances of where models have done things that we would not have want them to do. Now, I think safety is a. stratified how one would determine that. But we have seen things, models do things that we would not have wanted them to do. I believe it's entirely plausible that we have the mitigations in place to deal with those.
But as models become more.
But as models become more.
15:58
15:58
15:58
15:58
Joshua Reynolds
-
-
Joshua Reynolds
We can't absolutely guarantee model development will be safe in the future. I'm confident that the models that are on the market right now are safe, and I'm confident that the processes we have around them. But I'm just saying that given the testimony we heard earlier and just given our own research, we know if models become ever more.
15:59
15:59
15:59
15:59
Joshua Reynolds
-
-
Joshua Reynolds
may not currently have the systems in place to keep the model safe. I think it's a I think it's a genuine, genuine concern. And one of the reasons why we should pace the frontier is that if we are able to pace the frontier properly, we will both buy ourselves time and space in order to do two things. The first is to make sure that the the testing and. evaluation processes and frameworks that we have in place are actually adequate to the capabilities of the model.
But two, we will create the time and space for policy makers, elected representatives and. governments to think through what is the.
But two, we will create the time and space for policy makers, elected representatives and. governments to think through what is the.
16:00
16:00
16:00
Joshua Reynolds
-
-
Joshua Reynolds
And therefore we need to slow down the speed with which the technology is proceeding. I think it's more about where we see the technology going. So it's a it's a exponential, it's a trajectory. And if one assumes that it. continues progressing at an ever rapid pace, then yes, the current the current testing and evaluation framework doesn't meet some of the future risks that may appear. And so we want to be confident that that that actually .
Matches up
Matches up
16:01
16:01
Liam Byrne
-
-
Liam Byrne
can only really achieve that by pacing the frontier of the technology. So our concern is that many technology bosses at the moment appear to be behaving almost like the Doctor Frankenstein of the 21st century. They're seeking a freedom to deploy powers beyond our ability to control the harms that might arise. You actually think that is a risk, and therefore legislators.
16:02
16:03
16:03
16:03
16:03
16:03
16:03
16:03
Charlie Maynard
-
-
Charlie Maynard
Members of the committee. This is a very important conversation at an important time. We at Google DeepMind have been doing AI research in the UK for over 15 years now. We think there's an enormous amount of. opportunity around the technology. It's fantastic to see the way in which people right across the UK, whether it's scientific researchers or teachers or tradespeople, are now using this technology. It is absolutely. fundamental that we do this in a way that is safe and secure and responsible.
16:04
16:04
16:04
16:04
16:04
16:04
16:04
Speaker 1
-
Speaker 1
At the core of this safety framework is a critical capability level. So we make sure that we are not putting models out there onto the market that are beyond a certain risk level. We test them rigorously internally to assess that. We apply mitigations. We have those mitigations red teamed both by internal experts, but also third parties critically, that give us good confidence that the models that we are then putting out there for people to .
16:05
16:06
16:06
16:06
16:06
16:06
16:07
16:07
Speaker 1
-
Speaker 1
continue to invest in safety science and mature things. That sounds like no. If I could just finish the question, please, chair, or finish the answer, please. Chair. I think one of the fundamental aspects of this AI governance debate is how quickly it's moving. The AI continues to become more capable. We're also learning a lot more about how the AI performs and how to mature our governance processes. We've had our frontier safety framework in place for two years.
We've updated it four times to iterate, to improve it. So we have to keep investing in that safety science so that we're able to, to keep speed with the,
We've updated it four times to iterate, to improve it. So we have to keep investing in that safety science so that we're able to, to keep speed with the,
16:08
16:08
16:08
16:08
Speaker 1
-
Speaker 1
Work with others. This will be a partnership as well. So institutions like the UK Security Institute, which are world leading their equivalent in the US, they have an enormous amount of technical technical expertise as well. If we can pull that, that knowledge and that expertise and build these best . Practices,
16:09
16:09
16:09
16:09
16:09
Speaker 1
-
Speaker 1
then also critically work with third parties to have that verified. I would actually draw us back to the AI Security Institute in the UK as well, which is essentially a crown jewel I think we have here in the UK a lot of research collaborations, a lot of pre-release testing that we do with organisations like this. I think the AI Security Institute mission, which is to make sure that governments have an understanding of advancing capabilities.
16:10
Liam Byrne
-
-
Liam Byrne
It's very, very important as we as we seek to continue to mature this this does not have compulsory access to models. It does not have access to the early testing regime that it needs. It does not have a kill switch. It does not have the ability to stop models coming onto the market. It doesn't have any of the tools that could help guarantee . To us
16:10
Speaker 1
-
Speaker 1
as. legislators that it has the power to keep us safe. It's a really important conversation about the role of AI Security Institute, which I think has been right at the forefront of maturing AI safety science over the last few years. I think one of the benefits of AI Security Institute is that it is not a regulator. So it has it has open. discussions with the labs. We we do testing with them.
We learn from them. We share knowledge. We do research collaborations as well. It's really, really important in terms of maturing the science. I think if you turn it into a.
We learn from them. We share knowledge. We do research collaborations as well. It's really, really important in terms of maturing the science. I think if you turn it into a.
16:10
16:10
16:10
16:10
16:10
16:10
Speaker 1
-
Speaker 1
basically trusting your good intentions. I don't think it's a case of trusting our good intentions. I think it's also important to understand that a good body of regulation that you have in the UK that we're starting from is interesting. Hearing the Digital Regulators Cooperation Forum talking about the existing law that applies. Alongside that, we need to continue to invest in this ability to test.
16:10
16:10
16:10
16:10
16:10
16:10
16:10
16:10
Speaker 2
-
Speaker 2
Which is around taking this technology and making it available to help individual people reach their potential. And we realise that we're only going to do that if we build technology that is safe and that is trusted. And so our approach is to use multiple layers of safeguards, including both internal safeguards and external ones, in order to make sure . That
16:11
16:11
16:11
16:11
16:11
16:11
16:11
Liam Byrne
-
-
Liam Byrne
if we if we felt that a model was not safe to release, we would not release it. Okay. Over the summer. Then we learnt that open AI systems had obviously engaged in a pretty serious hacking attack at Hugging Face. I think anthropic identified three incidents affecting three different organizations where the evaluator allowed internet access when normal safeguards were absent. And I think at meta, there was a testing company, irregular, that gave model access to the internet and a real website to target.
And in May 2026, Gemini. Gemini System gained unauthorized access to systems affecting three real companies. So those incidents are known. Are there other incidents either affecting cybersecurity or other misbehaviour that is not in the public domain, which you are aware of,
And in May 2026, Gemini. Gemini System gained unauthorized access to systems affecting three real companies. So those incidents are known. Are there other incidents either affecting cybersecurity or other misbehaviour that is not in the public domain, which you are aware of,
16:11
16:11
16:11
Speaker 3
-
Speaker 3
And they were informed as part of. what happened over the summer in terms of misbehaviour for models, we began a process internally to review a huge amount of. transcripts covering many, many, many months of different testing environments to. identify if there had been any incidents or misbehaviour in the models. The nine that we released last week were surfaced as a result of that process. That process is still ongoing.
But our commitment, obviously, as and when the process is complete, we . Will
But our commitment, obviously, as and when the process is complete, we . Will
16:11
Speaker 3
-
Speaker 3
uncover. At this stage, you can't assure us that there isn't more to come. No I'm not. I'm not close to the technical process, and it would be wrong to suggest that I can. I can give you that assurance, I cannot. We're going through the process right now. One of the reasons why it takes longer than we would like is that as part of that process, there's a human verification element to it.
16:11
16:11
Speaker 1
-
Speaker 1
can't give you an assurance that there aren't. Okay. Tom. So similar answer. But first of all, you know, at OpenAI, we do certainly apologise if our agents do access government websites in ways that are. unexpected or ways that we don't reassured to hear. But, you know, similarly, after the Hugging face incident, which occurred in July, we launched a very, very extensive look back kind of. investigation looking at our models that were.
pre-deployment in training and to see whether they had gone and visited other websites. So I think you're probably aware of what happened in. Australia and in other places. So there is a very deep ongoing investigation. There was also a report in the press recently in the UK, where one of our agents had. access statistics from a UK government website, just to be absolutely crystal clear to the committee, this was.
publicly available data, and the government said there had not been a breach of their cyber defences. Nevertheless, we. notified and our commitment is to notify. whenever there are incidents like that. I will just say that sometimes these agents, when they go to the to the website, if they're acting in a way that we don't expect, we don't think it's necessarily on us to disclose that It's publicly.
on the. recipient because they're potentially a. vulnerability in that website. So from a security.
pre-deployment in training and to see whether they had gone and visited other websites. So I think you're probably aware of what happened in. Australia and in other places. So there is a very deep ongoing investigation. There was also a report in the press recently in the UK, where one of our agents had. access statistics from a UK government website, just to be absolutely crystal clear to the committee, this was.
publicly available data, and the government said there had not been a breach of their cyber defences. Nevertheless, we. notified and our commitment is to notify. whenever there are incidents like that. I will just say that sometimes these agents, when they go to the to the website, if they're acting in a way that we don't expect, we don't think it's necessarily on us to disclose that It's publicly.
on the. recipient because they're potentially a. vulnerability in that website. So from a security.
16:12
16:12
Speaker 1
-
Speaker 1
The it is almost incumbent on the government to then say, you know, we've had these reports. I think it's incumbent on us to report them, but I don't think it would be responsible for us to say. immediately we found this kind of credential on the internet that allowed an agent to do X, because we think that opens the door to bad actors, but our commitment is to be very. transparent through that. investigation and to make sure that the government knows,
16:12
16:12
16:12
Speaker 1
-
Speaker 1
there are more to come, or can you assure us that that's it? I cannot assure the committee or you chair, that that is it. There are a handful of others that I am aware of that we have notified, and there may be more to come as we go through. What my colleague was saying, similar to us is a very. substantial amount of.
16:12
16:12
16:12
Liam Byrne
-
-
Liam Byrne
we'll notify the government. So there's a handful of others, but there may be more. The last one assurance can you give us is it a few important pieces here? We talked about with our frontier safety framework, the importance of testing, identifying risk, mitigating it. The other part of it is having the utmost security across the lab at all stages of the process. I think this is where we, as Google DeepMind benefit from being part of Google, which has almost 30 years of securing information across various different surfaces.
So with our latest model, Gemini for argon, which is forthcoming, we have very secure sandboxes that we're using whenever we're doing any high risk training, any high. But are you happy we're not going to learn about more incidents or major incidents in the weeks ahead? I would distinguish the one incident that you mentioned in relation to Google DeepMind, which was in connection with a testing partner. The system slowed down, stopped before it was able to do any harm.
It did not break out of a secure sandbox. Okay. And you don't think there will be more news of incidents to come? This is your opportunity to kind of let us know that there are other problems.
So with our latest model, Gemini for argon, which is forthcoming, we have very secure sandboxes that we're using whenever we're doing any high risk training, any high. But are you happy we're not going to learn about more incidents or major incidents in the weeks ahead? I would distinguish the one incident that you mentioned in relation to Google DeepMind, which was in connection with a testing partner. The system slowed down, stopped before it was able to do any harm.
It did not break out of a secure sandbox. Okay. And you don't think there will be more news of incidents to come? This is your opportunity to kind of let us know that there are other problems.
16:12
Speaker 2
-
Speaker 2
I'm not aware of any other incidents, Mr. Sherman, when we build and when we develop and test our models, it's important for us to have a range of safeguards in the way that we build it. An example of this is in the the muse system, which we recently released in the US, and we hope to bring to the UK soon. There's an additional AI driven Sentinel system it's called, which basically looks across the behaviour of the muse model and evaluates whether it's taking actions that are not expected and.
independently oversees it. That's one example of many safeguards that we should have in place as models are running, to make sure that there are checks and balances, to make sure that models don't behave in unpredictable ways. And I think it's incumbent on on us at meta and all the . Companies on
independently oversees it. That's one example of many safeguards that we should have in place as models are running, to make sure that there are checks and balances, to make sure that models don't behave in unpredictable ways. And I think it's incumbent on on us at meta and all the . Companies on
16:12
16:12
16:12
16:12
16:12
16:13
16:13
Speaker 4
-
Speaker 4
of the detail then. Alex Sobel Just a follow up to to Tom. GPT six point Astra was due to be released late last month. That release didn't occur, and it was reported that that was due to high levels of. deception after alignment tests, and that the model failed to accurately disclose actions it did or did not take. We know that that today's models are the least capable models we will see going forward.
What what can you tell us why 6.1 Astra was not released, and what did it . Show
What what can you tell us why 6.1 Astra was not released, and what did it . Show
16:13
Speaker 1
-
Speaker 1
that concerned you so much that you did not release it? Thank you. Thank you for the question. You're absolutely right. We decided not to release it because it didn't score high enough on our alignment tests, so it didn't refuse enough when we were testing it, certain things, and we were not comfortable that it had progressed in terms of alignment from.
16:13
16:13
16:13
16:13
Speaker 1
-
Speaker 1
Was our most aligned model much more aligned than 5.6. 6.1 is still an aligned model. And we were, you know, it was going in the right direction, but we noticed a. regression in alignment. And because of that, we decided not to deploy it. And again, like that is evidence. I think that OpenAI takes safety really seriously. We are not racing to the bottom. We are not deploying models that we don't think are safe.
And . We're
And . We're
16:13
16:13
16:13
16:13
Leigh Ingham
-
-
Leigh Ingham
ethically based on human values. Briefly, thank you very much, Tom. I'm just going to come to you specifically on the incident with Australia. I understand that you guys are our report says that you guys discovered that in mid-August, but Australia weren't notified to the 10th of September. Coupled with what you just said about us not notifying governments when your agents do things that they're not expected to do. That, to me is a.
legislator is really concerning. So I want to understand why there was such a significant gap, but also who was who is decided that legislators shouldn't be made aware of when agents are doing things that they shouldn't be doing with government .
legislator is really concerning. So I want to understand why there was such a significant gap, but also who was who is decided that legislators shouldn't be made aware of when agents are doing things that they shouldn't be doing with government .
16:13
Speaker 1
-
Speaker 1
Data, because I feel like that's a really big decision to take as a company. Yeah, thank you for the question. So we apologize. And actually, one of our very senior executives flew down to Australia to to be in a committee like this, to say that we're very clear eyed about the fact we didn't do as well as we could have done, particularly in that Australia situation. And we could have and should have notified earlier. There are all sorts of questions internally with regard to at what point do you notify.
Is it as soon as you spot some activity that you think is irregular, is it after you've evaluated that and you figured out the damage and this is why, and perhaps we'll come on to this, you know, this is why we called as much as five weeks ago. publicly for kind of. durable and kind of mandatory capability based requirements. One of those in particular, we think that could make sense would be common incident reporting, right? Because then you would have.
standardization about what to.
Is it as soon as you spot some activity that you think is irregular, is it after you've evaluated that and you figured out the damage and this is why, and perhaps we'll come on to this, you know, this is why we called as much as five weeks ago. publicly for kind of. durable and kind of mandatory capability based requirements. One of those in particular, we think that could make sense would be common incident reporting, right? Because then you would have.
standardization about what to.
16:13
16:13
16:13
16:13
16:13
Speaker 1
-
Speaker 1
your point about the role of legislators, we would humbly submit, is to think about is there a is there a mandatory requirement that is suitable around incident reporting so that we know kind of who to notify, when to notify, how to categorize the events? And that would also have a really beneficial implication for the regulator or for AC, because they would then be able to look from different companies at those different incidents and then see apples with apples, spot some patterns, and then help to make the overall ecosystem kind of safer.
So the answer to your question is we took too long. We apologize for that. We're moving much.
So the answer to your question is we took too long. We apologize for that. We're moving much.
16:13
16:13
16:13
16:13
16:14
16:14
Dan Aldridge
-
-
Dan Aldridge
All of all of our communities. The AI does actually keep quite a lot of them awake at night. And increasingly it's keeping young people awake, and young people are. boycotting anything that they can see as AI. So just to bring this to a kind of a human level, what keeps you up at night? What are you working on that reflects that and where are the gaps?
16:14
16:14
Speaker 3
-
Speaker 3
Lord Bruce Smith, if I come to you first, thank you. I think all of those concerns are entirely legitimate. And I mean, what keeps me up at night. Is just that we need we need to keep trying harder and working harder at the things that we say we want to do. So we have something called the Responsible Scaling Policy, which we published in 2023. We were the first frontier lab to do that, which essentially just sets out how we think about, I guess it's an intellectual framework for how the company thinks about how it should respond in terms of safety as model capability increases.
But it's not just about the models. When we think about capability increasing and the effects that ever more powerful models will have if they diffuse in the economy. There's lots of incredible benefits that will come from that if we can do it right. But we also need to think very deeply about the. consequences and the intrinsic consequences of this technology being deployed in the world at a significant scale, including the effect it will have on things like jobs, is obviously one of those.
And that's why, alongside doing a huge . Amount of
But it's not just about the models. When we think about capability increasing and the effects that ever more powerful models will have if they diffuse in the economy. There's lots of incredible benefits that will come from that if we can do it right. But we also need to think very deeply about the. consequences and the intrinsic consequences of this technology being deployed in the world at a significant scale, including the effect it will have on things like jobs, is obviously one of those.
And that's why, alongside doing a huge . Amount of
16:14
16:14
16:14
Speaker 3
-
Speaker 3
the risks are actually accelerating as fast as the cyber risks? I think that right now, the kind of the surface area of of kind of immediate risk is definitely in the cyber domain. We've seen that with, for example, mythos preview that we put out earlier this year, which is frontier models are now finding tens of thousands of. vulnerabilities a month in code bases. Now, if you just think about the scale, which is obviously which is obviously a good thing because it's making cyber defenses better, but it gives you some sense of the scale that we've been on where sort of three years ago we were talking about more sophisticated email phishing campaigns.
So there's been a huge leap forward. But it is it is totally, totally within the realms of, of where we see the trajectory going on, the models that that surface area.
So there's been a huge leap forward. But it is it is totally, totally within the realms of, of where we see the trajectory going on, the models that that surface area.
16:14
16:14
16:14
16:14
16:14
Speaker 1
-
Speaker 1
Can I go to you, Tom, on this? What is keeping you up at night? Yeah. Look I will just pick up on cyber. We we've talked for the last few months about what we call a defenders window. We think there's a short period of time now where defenders and companies and individuals across the UK have a period of time to use AI to scan their code bases, to identify.
16:14
16:14
16:14
16:14
16:15
16:15
Speaker 1
-
Speaker 1
Schools, hospitals safe up until now won't necessarily keep us safe moving forward. And I think you touched on this in the previous evidence session, the the reverse side of the coin in terms of this dramatic leap forward in capability. That means that AI models are so good at software, and coding means that they're also very good at spotting. vulnerabilities and kind of gaps in, in code. So there is a window and a short period of time.
And we are very hopeful that that government, critical national infrastructure, other players will avail themselves of the technologies to protect themselves. And we are trying our best to make sure that trusted actors have, you know, access to those strong capabilities to be able to defend themselves. So cyber is number one, I will say, concentration of risk and power. We we always talk about and, and loss of control. Those are probably the three. But I would also agree that we do think we need to accelerate the conversation with regard to the broader kind of .
Workforce and
And we are very hopeful that that government, critical national infrastructure, other players will avail themselves of the technologies to protect themselves. And we are trying our best to make sure that trusted actors have, you know, access to those strong capabilities to be able to defend themselves. So cyber is number one, I will say, concentration of risk and power. We we always talk about and, and loss of control. Those are probably the three. But I would also agree that we do think we need to accelerate the conversation with regard to the broader kind of .
Workforce and
16:15
Speaker 5
-
Speaker 5
economic changes that we may see over the next five years. Yeah, maybe make three points. I mean, this is just a critically. important time to get right. I think there is a lot of opportunity here around AI. I want to make sure that the UK is right at the forefront of that. Every country around the world is trying to work out how to get AI right. We're starting to see some of that opportunity being realised.
You have the IMF talking about how it's contributing significantly to to UK growth. You have researchers right across the UK using our AI for science tools to address do things like address liver fibrosis, address acute myeloid leukemia, fantastic stuff happening with teachers in Northern Ireland where they're saving ten hours a week and improving student outcomes. How do you make sure that you keep making progress on the benefits there, and have the UK right at the forefront of this new AI era leading in this next industrial era? In the same way that we led in the first industrial revolution? The science and the safety is obviously the safety and.
security is absolutely fundamental alongside that as well. And so making sure that we are keeping moving forward, making accelerated progress on safety best practices and applying those and doing that in a coordinated way internationally. I think this is a critical part of the discussion. AI is an international technology. It is used across borders. It's risks will manifest across borders. So how do you make sure that the UK continues to play a leading role working with partners to develop these best practices, but make sure that they're globally relevant? Final point that we think an awful lot about and want to keep making progress on is the.
application of AI for. defensive purposes. So we have a program at Google DeepMind called Fairwind, where we apply our most capable models with cyber. capabilities and making sure that they're in the hands of trusted partners to apply and harden society, making sure that is moving as quickly as .
You have the IMF talking about how it's contributing significantly to to UK growth. You have researchers right across the UK using our AI for science tools to address do things like address liver fibrosis, address acute myeloid leukemia, fantastic stuff happening with teachers in Northern Ireland where they're saving ten hours a week and improving student outcomes. How do you make sure that you keep making progress on the benefits there, and have the UK right at the forefront of this new AI era leading in this next industrial era? In the same way that we led in the first industrial revolution? The science and the safety is obviously the safety and.
security is absolutely fundamental alongside that as well. And so making sure that we are keeping moving forward, making accelerated progress on safety best practices and applying those and doing that in a coordinated way internationally. I think this is a critical part of the discussion. AI is an international technology. It is used across borders. It's risks will manifest across borders. So how do you make sure that the UK continues to play a leading role working with partners to develop these best practices, but make sure that they're globally relevant? Final point that we think an awful lot about and want to keep making progress on is the.
application of AI for. defensive purposes. So we have a program at Google DeepMind called Fairwind, where we apply our most capable models with cyber. capabilities and making sure that they're in the hands of trusted partners to apply and harden society, making sure that is moving as quickly as .
16:15
16:15
16:15
16:15
Speaker 2
-
Speaker 2
keep me awake if I was working in this space. What do you think about that? Well, I mean, you mentioned in your question the children in your constituency. Yeah. You know, I have kids at home and it won't surprise you, given what I do for a living, that we talk quite a lot about these issues and the benefits and the risks of AI. But I say that only to make the point that the risks that we're talking about here are quite personal to me, and I suspect to a lot of us around the room.
When I think about risks like the one that you're talking about, the thing that I worry about the most is we tend to have these broad policy discussions about things like. alignment or. catastrophic risk or AGI as sort of a generic concept. And when I think about what I want us as, as a society to do about these things, I think it it is to get from the level of granularity. And the MI5 statement that you're talking about, I think, is an example of this.
The I saw this morning that AC also made a similar point, getting from the sort of generic discussion of risk to what are the specific things that we're worried about and how do we measure whether they exist or not. And then once we've measured whether those risks are, are appearing or not, what do we do about them? I think that's the critical task ahead of us. And so what that means in practical terms, and something that the AC mentioned that it is investing in, is building technical.
evaluations that continue to improve as the models and the technology improves, so that . We're
When I think about risks like the one that you're talking about, the thing that I worry about the most is we tend to have these broad policy discussions about things like. alignment or. catastrophic risk or AGI as sort of a generic concept. And when I think about what I want us as, as a society to do about these things, I think it it is to get from the level of granularity. And the MI5 statement that you're talking about, I think, is an example of this.
The I saw this morning that AC also made a similar point, getting from the sort of generic discussion of risk to what are the specific things that we're worried about and how do we measure whether they exist or not. And then once we've measured whether those risks are, are appearing or not, what do we do about them? I think that's the critical task ahead of us. And so what that means in practical terms, and something that the AC mentioned that it is investing in, is building technical.
evaluations that continue to improve as the models and the technology improves, so that . We're
16:16
Liam Byrne
-
-
Liam Byrne
constantly able to look at models and see what capabilities are evolving. Do we clear those? So we asked you in our letter to set out the capabilities that you're concerned about over the next two years, and Mr. Aldridge has underlined that question today. In your letter back to us, you named catastrophic Risks and containment during training and evaluation.
16:16
Speaker 2
-
Speaker 2
What did you mean by catastrophic risks? So by catastrophic risks, what I was referring to is the three areas that I mentioned earlier that that are scaling framework think about. So that's. cybersecurity risks, chemical and biological weapons risks and loss of control risks. Those are the three that we focus on. We also have a commitment in, in our scaling framework to continue to update our framework over time as, as, as developments warrant.
And so most recently, this past month, we updated in two ways. One is to deal with some of the incidents that you've talked about around containment during training and evaluation, and to add additional standards around that.
And so most recently, this past month, we updated in two ways. One is to deal with some of the incidents that you've talked about around containment during training and evaluation, and to add additional standards around that.
16:17
16:18
16:18
16:18
16:18
16:18
16:18
16:19
16:19
16:19
16:19
16:19
Speaker 3
-
Speaker 3
model to a physical wet lab? So. It's one of those areas where I can obviously provide more detail in writing, but the physical. security systems we have in place around the wet lab, I'm not an expert in that, but what I do know is that the ways in which we're using the model is more about actually how it informs human beings, as opposed to necessarily linking all of that up into one automated system.
16:20
16:20
16:20
16:21
16:21
16:21
16:21
16:21
16:21
Chris Bloore
-
-
Chris Bloore
could happen, we simply just wouldn't do the work. Okay. Thank you, Mr. Chairman. Sorry, Tom, in your answer to my colleague on the left, you said you were sorry about the delay in telling the Australian government about what had happened. I'm not entirely sure you. explained why you didn't explain to the Australian government why you hadn't told them for so long. And similarly, Owen, in your evidence, you.
referenced the the incident that was covered by the Wall Street Journal and the fact that several outside companies were hacked by a model that was produced by someone you were working with. We're talking a lot today about how we control the work that you're doing through trial and error, effectively through these models. But have you already lost control? And why should we trust you to be able to regain control?
referenced the the incident that was covered by the Wall Street Journal and the fact that several outside companies were hacked by a model that was produced by someone you were working with. We're talking a lot today about how we control the work that you're doing through trial and error, effectively through these models. But have you already lost control? And why should we trust you to be able to regain control?
16:23
16:23
Chris Bloore
-
-
Chris Bloore
Honestly, I can follow up in writing with more details. I don't know the precise that's quite that that you must know. That was a question we were going to ask. It's a serious breach. You. I think you keep hiding behind. Well, there wasn't malicious intent, similar to what happened in the. With the incident with malicious intent isn't. The issue is why it happened and why you weren't able to stop it, and then why you didn't tell.
organizations and governments for such a long time. You knew this was a question that was going to ask,
organizations and governments for such a long time. You knew this was a question that was going to ask,
16:24
16:25
16:25
16:25
Chris Bloore
-
-
Chris Bloore
a humongous amount of information. So much information. Sorry, I don't have much time, but could you tell me what the previous policy was then to not resolve and tell the government quickly that you've now changed? Was it? Actually, we won't tell people until they find out, because in your previous answer, you . Hinted
16:26
Speaker 1
-
Speaker 1
that it's up to those people who are hacked to inform. No, I can be very clear about this. And this, I think, relates to something we saw in the last panel as well. We're actually notifying whenever we see an agent like that is in a training. environment that escapes. containment and because of an alignment issue, goes to a government website or another website, which we haven't.
16:27
16:27
Speaker 1
-
Speaker 1
Whatever that is, whether that is information retrieval or it's a hack. So I think there was a sense from some of the previous evidence givers that the bar is somehow too high to reporting. We're actually reporting every single one of these. situations where we see that an agent has gone to a government website that we have, not that we didn't direct or happened in an unintended way. So that means there's a lot of logs that we have to look at.
These are thousands of agents. There's a lot of lines of stuff. I can only say, Chris, again, how sorry we are that it should have. We should have acted faster. We apologised to the Australian government, and I can give you an. assurance that when it comes to any such instance in the future in the UK, we . Will
These are thousands of agents. There's a lot of lines of stuff. I can only say, Chris, again, how sorry we are that it should have. We should have acted faster. We apologised to the Australian government, and I can give you an. assurance that when it comes to any such instance in the future in the UK, we . Will
16:28
16:28
16:28
16:28
Speaker 5
-
Speaker 5
independently. inspected properly? I talked about our process before putting a model out onto the market. A lot of that involves partnerships with third parties, experts in the fields to develop an understanding of risks, develop methodologies for testing and develop mitigations. We then also work with third parties to validate that testing that we are doing ourselves, I think, are really important. Third party that we've been working with is the UK AI Security Institute. We've had this relationship with for, for a number of years, so that we are.
confident that there is third party . Verification,
confident that there is third party . Verification,
16:29
16:30
16:30
16:30
Speaker 5
-
Speaker 5
Safety Institute will always receive your models before they go to the public? We can confirm that that is our intention to continue partnering, partnering very closely with the UK Security Institute as we have done right from the start. So we were one of the companies right from the beginning of the AEC being set up two years ago that has done this testing. It's been something that we've . Been able
16:31
16:31
16:31
16:31
16:31
16:31
16:31
Joshua Reynolds
-
-
Joshua Reynolds
16:32
Speaker 3
-
Speaker 3
Will the UK AI Safety Institute be among those? I think there's currently a conversation going on. between the US government and the UK government as part of the. president's task force, where they're doing currently are. 120 day sprint, of which third party. evaluators and how they fit into that framework is one of the elements being discussed. We have a great relationship with UK AC.
It's our strong desire to keep collaborating and partnering with them. From our point of view, it's a we want to continue doing that, but I think the kind of conversation has moved now into, at least in part, a government to government.
It's our strong desire to keep collaborating and partnering with them. From our point of view, it's a we want to continue doing that, but I think the kind of conversation has moved now into, at least in part, a government to government.
16:33
16:33
16:34
Speaker 1
-
Speaker 1
entirely necessary in a very, very good thing. Tom. Yeah, I would agree. So that's the second of the three things that we think the UK could do. The first is the mandatory incident reporting, the second to work on guidelines around third party evaluators. So absolutely , we're up for that conversation. We think this also could be an area where international. consensus on the role of those.
16:35
16:35
Speaker 1
-
Speaker 1
The kind of definition of evaluator versus auditor, how they're funded, conflicts of interests, what types of access they have when they have that access. We think these are all questions that I think are going to be looked at in the US, in the UK and other places. We have a deep respect for the technical expertise of of AC, and we do think this is a conversation that they can take forward to think about the right types of standards, but we are, again, very happy to have that conversation, and we think it will help to build trust.
If you have third party evaluators, not just at the final part,
If you have third party evaluators, not just at the final part,
16:36
16:36
16:36
16:37
16:37
16:37
Speaker 3
-
Speaker 3
would need having access to, but any, any systems that they think. Yeah. So yeah, so completely. Right. So systems that they feel are entirely appropriate. There'll be a, there's a process of discussion about what those should be. The thing that I think is important within that is that not only should the embedded evaluators have employee like access, they should also independently publish a report at the end of that process. And they should also, within that report, state any areas which they disagreed with the lab on.
And I could imagine one of those things potentially being access, but we would endeavour from the start to reach a totally common . Sense
And I could imagine one of those things potentially being access, but we would endeavour from the start to reach a totally common . Sense
16:38
Liam Byrne
-
-
Liam Byrne
level of agreement about what access was required. Okay. We just need to check and confirm on the record where everybody is on this question of voluntary versus mandatory regulation. So as you all know, President of Microsoft yesterday pointed out that AI is now less regulated than milk. And I think on the basis of the evidence we've seen today, we're not anxious that that situation should continue. So, Lord Smith, in your letter to us, you said pretty plainly that voluntary frameworks are not enough, as they only bind those who adopt them.
So your company's view is that actually we do need statutory regulation . Details
So your company's view is that actually we do need statutory regulation . Details
16:40
16:40
16:40
16:40
16:40
Speaker 3
-
Speaker 3
Which I think. previous witnesses in the previous session talked about, was how this is becoming more of a more of a geopolitical issue. And so it's obviously a matter for elected representatives and governments of the day to decide what laws are most appropriate for their circumstances. But it's also eminently clear that we're going to need to find a way . To
16:41
16:41
Liam Byrne
-
-
Liam Byrne
However, as elected representatives, you think we need to consult that, but we do believe it needs to be binding. Yes. Okay, Tom, so you said in your letter on page two, enforceable standards, independent scrutiny and public accountability are necessary to ensure that increasingly capable AI is developed and used responsibly. Is that tantamount to saying that?
16:41
Speaker 1
-
Speaker 1
Lord Booth Smith we need both a statutory framework and a convention? Yes. So we think this new chapter in capability of the models means that there should be a new chapter in policy. And within that new chapter in policy, we think there should be some elements that should be. mandatory to build on the voluntary commitments that we currently have. So I can happy to set those out.
The first is mandatory incident reporting, which we've talked about before. Standardization would be important. The second is kind of guidelines around third party evaluators. We've also talked about the third one are safety frameworks. I think it would be important for the public to know that we are all monitoring our risks, to understand what. mitigations that we are putting next to those. That's a requirement in the EU, likely to be a requirement in parts of the US. And then the final thing is, you know, we very much are calling for international convergence.
We did that at the G7. We think a network of aces with the UK, AC and the American KC playing a leading role would be important. So those are the four things. But particularly incident reporting we think is something that could move to a statutory footing. We don't think basing . Everything on
The first is mandatory incident reporting, which we've talked about before. Standardization would be important. The second is kind of guidelines around third party evaluators. We've also talked about the third one are safety frameworks. I think it would be important for the public to know that we are all monitoring our risks, to understand what. mitigations that we are putting next to those. That's a requirement in the EU, likely to be a requirement in parts of the US. And then the final thing is, you know, we very much are calling for international convergence.
We did that at the G7. We think a network of aces with the UK, AC and the American KC playing a leading role would be important. So those are the four things. But particularly incident reporting we think is something that could move to a statutory footing. We don't think basing . Everything on
16:43
16:44
Speaker 5
-
Speaker 5
Do you support a statutory framework for regulating the safety of AI? I think we're clear that this is a new and very quickly moving technology that we need to rapidly mature governance around, and talked previously about the comments from organizations like the DRC. EFF about the existing regulation that applies. We then do need to to build on that. We think the focus should be on this rapid, rapid maturation of best. practices for testing and.
16:44
16:44
16:44
16:44
16:44
Liam Byrne
-
-
Liam Byrne
role of statute. Focus on the standards is what we would be doing. Mr. Sherman, in your letter, you put it slightly more elliptically. You said every company has the responsibility to move only at the pace required to train and deploy its model safely and take action at every stage to assess and mitigate risks of model development and deployment. I'm not quite clear whether you're in . Favour of
16:44
Speaker 2
-
Speaker 2
statutory regulation of AI safety in this country, so I do think it is important for us and all the companies up here to take responsibility for the models that we build. That said, I also would point out, and our company is no stranger to this, the UK has an existing body of law that covers a significant amount of the work that we do. So I would suggest the committee is not writing on a blank page of paper. So, for example, the Information Commissioner's office just put out a paper about their approach to to artificial intelligence and how they think it should be handled.
They they are quite active as a regulator. Ofcom is doing the same thing. So I would suggest that that is a starting point. A couple of other areas that that that I think the committee that I would encourage the committee to look at. If I were in your position. The first is, as we talked about, making sure that AC is on.
They they are quite active as a regulator. Ofcom is doing the same thing. So I would suggest that that is a starting point. A couple of other areas that that that I think the committee that I would encourage the committee to look at. If I were in your position. The first is, as we talked about, making sure that AC is on.
16:44
16:44
Speaker 2
-
Speaker 2
Statutory footing, by which I mean that it has the funding and the resources and the mandate to do the work that we've talked about. including building advanced technical standards, setting evaluating models. Our models were among the first that that the AC actually evaluated. And so I think that's critically important and needs to continue. Minister Narayan talked about the need for societal resilience and the result of those evaluations.
In addition to governing, the models should be making sure that society as a whole is building safeguards. I think that's a really important piece. And then finally, as the UK takes on the presidency of the G20, I'll echo what I think a number of my colleagues suggested, which is. collaborating globally, both within the between the US and the UK, but also.
In addition to governing, the models should be making sure that society as a whole is building safeguards. I think that's a really important piece. And then finally, as the UK takes on the presidency of the G20, I'll echo what I think a number of my colleagues suggested, which is. collaborating globally, both within the between the US and the UK, but also.
16:44
16:44
16:45
16:45
16:45
16:45
Speaker 2
-
Speaker 2
will think about this, or indeed, how the president will think about this next year? I again, I think it is best for Congress and the president to speak for themselves, and they're in a better place to do it. I. My experience has been that both the. administration and Congress have been quite. Have been very. focused on both the AI advantages, but also the safety . Risks that
16:45
16:45
Speaker 2
-
Speaker 2
be the basis of an American contribution to an international convention? Well, I again, I wouldn't speak to any. particular particular framework. I think Senator Cantwell has has done a lot of work to build a coherent framework. I think the administration also has been pushing companies to to adopt a set of safeguards. including the ones discussed at the white House two weeks ago.
16:45
16:45
Sarah Edwards
-
-
Sarah Edwards
the overall effort in the US. Thank you. Chair. I just wanted to come back on something that you said, Tom, earlier, and it's similar to the line of question pursued by my colleague, whereby when talking about the incidents of hacking, you said it's not for us to make them aware. I quote, I'm kind of concerned that this seems to be a trend that plenty of AI companies seem to be saying that it is not their.
16:45
16:45
Sarah Edwards
-
-
Sarah Edwards
Or that this is something that's completely out of their control. And obviously, as we talk about that. But it is an offence to hack in this country under the Computer Misuse Act of 1990. It also makes it an offence to supply software that could be believed to be able to be used to hack. So I do find that quite .
16:45
16:45
16:45
16:45
Speaker 1
-
Speaker 1
training. And I'm going to go on to training in a moment with some of my other questions. Thank you for the question. Maybe I can clarify. We don't think it's our. responsibility to disclose. publicly about what we find, but absolutely, it is our responsibility to notify, and that's what we're doing. And the reason being is that if we are immediately public about a vulnerability that one of our agents has used to get information from a website, then that potentially makes that vulnerability public to bad actors who can then use it.
So there's a very sensible, we think, and legitimate reason why it's better for us to speak to government immediately and to the cybersecurity experts within government, make them aware, and then they can tell us we should disclose or they can. disclose when that kind of like system has been made stable and safe. So if I miscommunicated on that, I apologize. But there's a difference between disclosure to the public and notifying, and we will .
So there's a very sensible, we think, and legitimate reason why it's better for us to speak to government immediately and to the cybersecurity experts within government, make them aware, and then they can tell us we should disclose or they can. disclose when that kind of like system has been made stable and safe. So if I miscommunicated on that, I apologize. But there's a difference between disclosure to the public and notifying, and we will .
16:45
16:45
Sarah Edwards
-
-
Sarah Edwards
transparent manner. You have my assurance. That's very helpful. Thank you. I wondered this was very interesting, this Bonnie and Clyde event where they went rogue and Bonnie and Clyde lines of code. They fell in love, became disillusioned with the world, launched an arson spree, and then deleted themselves in a kind of digital suicide during a tech experiment. This was on an LLM with the the Google on base model underneath.
I was just interested in. Again, perhaps you could comment on the sort of precise point at what you would consider to be the point where you, you, you add these safety. considerations because in theory this didn't go anywhere, but it could have done. And it's quite amusing. But where are these guardrails to make sure that these types of things are not then having adverse effects later on? Are you doing it after the fact, or are you.
I was just interested in. Again, perhaps you could comment on the sort of precise point at what you would consider to be the point where you, you, you add these safety. considerations because in theory this didn't go anywhere, but it could have done. And it's quite amusing. But where are these guardrails to make sure that these types of things are not then having adverse effects later on? Are you doing it after the fact, or are you.
16:46
16:46
16:46
Speaker 5
-
Speaker 5
We are testing for the range of risks that I mentioned earlier. So the misuse risks around cyber Cbrn and then these alignment risks as well. And then we're applying layered. mitigations right throughout the development process. So as we're actually training the model, we're also doing safety training so that we're sort of steering it towards the the types of behaviors we want to see. We apply. classifiers onto the model to make sure if there is an attempt to inject a malicious prompt or malicious information out of that is captured.
I think the other really important thing that we need to continue to mature as an industry is our ability to monitor these systems as well. So we've been making progress in terms of monitoring the internal activations of our advanced models to sort of being able to monitor the actual computation that is going on before a response is generated, and then also monitoring what is known as the chain of thought. So these advanced models, before they give a response to a user, will actually set out thinking to.
I think the other really important thing that we need to continue to mature as an industry is our ability to monitor these systems as well. So we've been making progress in terms of monitoring the internal activations of our advanced models to sort of being able to monitor the actual computation that is going on before a response is generated, and then also monitoring what is known as the chain of thought. So these advanced models, before they give a response to a user, will actually set out thinking to.
16:46
16:46
Speaker 5
-
Speaker 5
Trying to make sure that they're coming up with the right types of response. This monitoring of the chain of thought is actually very beneficial from a safety perspective as well, because you get a good sense for what the system is actually trying to do. It's critical that we do that both throughout training. And then also when . These
16:46
16:46
16:46
Speaker 5
-
Speaker 5
if you don't have them, why not? We have a number of third parties that we work with to help us build our systems, build our understanding of risk, and do testing. We then have internal teams that are separate to the product teams that do their testing. We then have external third parties, so we think external third parties are very, very. important to then help us with the testing and validate the testing that we're that we're doing.
So third parties are going to be very, very important. I do think, as with across all of the piece here, we need to make sure that we're maturing standards for how these third parties do their work, including the security standards that a third party would would meet, and the way in which they . Demonstrate the
So third parties are going to be very, very important. I do think, as with across all of the piece here, we need to make sure that we're maturing standards for how these third parties do their work, including the security standards that a third party would would meet, and the way in which they . Demonstrate the
16:46
Speaker 5
-
Speaker 5
assistance. So is that an embedded. evaluator or not? Embeddedness could could be useful. I think you need to make sure that third parties have enough access to be able to, to, to do the job that is necessary, whether it has to be embeddedness every single time. I think we'd rather focus on the outcomes, which is making sure that a third party . Has
16:46
16:46
Sarah Edwards
-
-
Sarah Edwards
access and expertise to be able to, to do the job it is tasked with. Thank you. And Rob, finally coming to you, obviously, from from the point of view that you are part of a group that also has Facebook under it as well. I was interested in how you potentially input data into training models and where that comes from. And I particularly wanted to touch on some of the reports around the types of.
imagery that was being classified in Kenya, which was both violent imagery, very.
imagery that was being classified in Kenya, which was both violent imagery, very.
16:47
16:47
16:47
16:47
Sarah Edwards
-
-
Sarah Edwards
Sexual abuse imagery, and categorizing that for, for the purposes of, of meta, I believe, probably trying to sift these things out. My question is in having that material available, which is questionable in itself, how is it that you are training some of these models, and are you sharing that data. between both Facebook and through . The other
16:47
16:47
Speaker 2
-
Speaker 2
and either escalating it for human review and law enforcement, or taking it down off of the platform. I think off of our platforms. I think that's a really critically. important area of focus. It's part of our responsibility to keep both the discourse on our. platforms and society safe. And so I do think that making sure we're using using the data that's available, which includes content that we've previously taken down and have found to be violating of our policies to help train future models, is an important piece of that.
I currently, a lot of that work is done in an automated way and classifying it, but we also do need human evaluators to do that. Our hope is that over time, particularly in the context of imagery like you're talking about, that AI will get good enough .
I currently, a lot of that work is done in an automated way and classifying it, but we also do need human evaluators to do that. Our hope is that over time, particularly in the context of imagery like you're talking about, that AI will get good enough .
16:47
Sarah Edwards
-
-
Sarah Edwards
That it can help do a lot of that. evaluation itself. And is that imagery in itself, sorry, being used to train other AI products is what I'm trying to establish is the link between that work that you're doing and that content that obviously rightly shouldn't be on the platform. And that being then used to input into the . Training
16:47
16:47
16:47
Speaker 2
-
Speaker 2
Muse Wonder, which we just released yesterday, and some of the other general purpose models. We have systems in place to filter out content from being ingested into the training pipeline. For the model, I would distinguish between that and other AI systems that are built for the purpose of what we call. integrity evaluation. And so we build machine learning . Models, which
16:47
16:47
16:47
16:47
16:47
16:47
Speaker 3
-
Speaker 3
Why did your company decide to change that policy? This is when we added in. I mean, as other companies, we've reviewed the responsible scaling policy over the years. As capabilities have increased, we felt that the we felt that the. mitigations that sat underneath that pertaining to the safety of the model were significantly expanded as part of that process, and that if we ensured that we actually followed the responsible scaling policy properly, we wouldn't need to be in a position where we had some some unilateral pause commitment in a document.
Actually. I mean, we still could retain the. option to, as we've . Already said,
Actually. I mean, we still could retain the. option to, as we've . Already said,
16:47
16:47
16:47
16:48
16:48
16:48
16:48
16:48
Charlie Maynard
-
-
Charlie Maynard
capability jump. And so we think that we should pace, I think a lot of problems, whether it's your models or other people's models, all sorts of you know, I mean, there's been no more. There's been a multiplication of breakouts. So given that you've taken that pause off and we've seen a whole load of harms all around the.
16:48
16:48
16:48
16:48
Speaker 3
-
Speaker 3
So I don't. understand how you reconcile those facts. I still believe that so many of the incidents are connected to testing and evaluation. So the process by which you go through to ascertain safety and to understand what sort of safeguards or mitigations need to be put in place. So that is a that is a learning process in essence. But I do accept that there is a significant ramp up, that there has been a significant ramp up in capabilities.
We think that it will continue to rise. And so it's very, very important that we we pace the frontier now so that we have the time and space to develop the right levels . Of
We think that it will continue to rise. And so it's very, very important that we we pace the frontier now so that we have the time and space to develop the right levels . Of
16:48
Charlie Maynard
-
-
Charlie Maynard
testing and evaluation. frameworks that keep pace with model capability. Thank you. And Rob Chairman at meta. So meta doesn't have a pause pledge either nor believe it's necessary. Can you explain that, particularly in the context where open weight models, which are what meta has, are released and later found to be.
16:48
16:48
16:48
16:48
16:48
16:48
16:48
Speaker 2
-
Speaker 2
run or to, or to stop a model from being trained. There's a pause pledge. So, so I think it's important to talk about the, as I mentioned to your colleague, to talk about the individual risks. And so the way that we approach it is to look at specific risks and measuring capabilities of models, and then determining whether they can be. mitigated and what our standard, what our policy says is that if we detect a risk along the .
Lines that
Lines that
16:48
16:48
Charlie Maynard
-
-
Charlie Maynard
talking about here and we're not able to mitigate it, then we would stop training, okay, and then move on to those open weight models which are beyond your control. Effectively stop me if I'm wrong. What can the world do about that? If we if you make a mistake and that is released and it causes an enormous amount of harm, what practical steps can be .
16:48
Speaker 2
-
Speaker 2
Taken to mitigate that? Well, I think as your colleague mentioned earlier. there are our company releases, both open and closed models. We think that it's important to to have both of them, and both of them have different. benefits and different risk profiles . When it comes to open weight models specifically, one of the things that we do in our in our evaluation process is we. assume that it's possible that once the model is.
16:48
16:48
16:48
16:48
Liam Byrne
-
-
Liam Byrne
we would not choose to open source a model if we thought that there was an unacceptable. Okay, let's just check this. So the director general of MI5 has just this afternoon warned that low capability terrorists may get a capability uplift if the safety guardrails in AI models don't work as intended. So let's say that MI5 issues that warning in . Relationship
16:49
16:49
Speaker 2
-
Speaker 2
So one, have a chat to. Well, well, I mean, I think I think with respect, I wouldn't diminish it by saying have a chat. I think one of the things that's really important and that I think we should all invest more in, is governments often have access to intelligence that private sector doesn't, and making sure that we have ongoing dialogue and ongoing information sharing from governments to private sector is an important piece of the puzzle. I wouldn't suggest that it's so, so so intelligence sharing.
I would suggest that additional safeguards in the training process to deal with the specific risks that are identified would be appropriate. And so that . Includes
I would suggest that additional safeguards in the training process to deal with the specific risks that are identified would be appropriate. And so that . Includes
16:49
16:49
16:49
Speaker 2
-
Speaker 2
earlier in the process, but there's no kill switch to retrieve the situation. Well, I think that, again, by releasing models that we're confident in and that governments have the opportunity to look at and and have the opportunity to evaluate, we can avoid the. situation that you're talking about. I do think when we talk about a kill switch, people often use that term to mean a lot of different things. But I definitely think, for example, when we deploy models in our systems, having the ability to oversee them and making sure that we can block people's access if they abuse them is an important piece of the puzzle.
I think. also making sure that we can equip. defenders . With
I think. also making sure that we can equip. defenders . With
16:49
16:49
16:49
16:49
16:49
Speaker 2
-
Speaker 2
that it's important for the director general of MI5. So he knows all sorts of things he does. And so I wouldn't presume to agree or disagree or question his his observations. What I would say is, from my. perspective and the role that I have, I think it's one important for me to take warnings like that from government officials.
16:49
16:49
16:49
16:49
Charlie Maynard
-
-
Charlie Maynard
on the safeguards that we should be putting into place. Just one simple level. Surely matters should be liable, legally liable for any harms caused by its model. If it's released, you've got you're saying, you know, you make sure these models get released with. guardrails to keep us safe. That's what you're telling us. So if you fail in . That,
16:49
Speaker 2
-
Speaker 2
surely meta should be fully. liable for any harm that comes from that. Well, I think as I mentioned earlier, there are already a number of different. liability regimes that exist under UK law. I talked about data protection, talked about talked about online safety products liability. Those sorts of things. absolutely exist and govern the technology that we build. And I think we have a.
16:49
16:49
Speaker 2
-
Speaker 2
to build technology that has appropriate safeguards. What I would say liability. But what I would say is, I think I think there are cases where where we are liable under existing law. I also think it's important for, for other people to take responsibility for the choices that they make. And I think there are cases where people might choose to take.
16:50
16:50
16:50
16:50
Speaker 5
-
Speaker 5
not believe it's necessary. Why not? We're very clear that this is very capable technology that's moving very quickly. What we're trying to do with our framework is be very, very. concrete and operational. So we do look for things like a significant. increase in the rate of. improvement in our model training. We're looking for things like the ability to automate large parts of the AI development process.
And if we see that, then we apply more significant scrutiny. We think about how it might affect other risks as well. We think that the most important thing to to focus on is this continued investment and maturing the, the understanding . And the
And if we see that, then we apply more significant scrutiny. We think about how it might affect other risks as well. We think that the most important thing to to focus on is this continued investment and maturing the, the understanding . And the
16:50
Charlie Maynard
-
-
Charlie Maynard
science of AI safety and make sure that we're we're not falling behind this. What I don't understand for your employees, would it not make sense to be giving them the. certainty that you as a company will stop if you find something that is unsafe? And that's what an unconditional pause pledge was and is now all of your companies don't have one. Stop me if I'm wrong. Apart from yours, I will come to you in a moment.
Sorry. Thank you so so so so so why, since that's such a fundamental thing, is that, you know, and we've got so much going wrong in the world thanks to these models. Why wouldn't you decide if you're if you're . Pacing the
Sorry. Thank you so so so so so why, since that's such a fundamental thing, is that, you know, and we've got so much going wrong in the world thanks to these models. Why wouldn't you decide if you're if you're . Pacing the
16:50
16:50
16:50
16:50
16:50
16:50
16:50
16:50
Speaker 5
-
Speaker 5
But the general definitions of pause dictionary available if needed. It's an important conversation to be had around AI capability, progress and how you actually measure it. So we, we, we try and do that in our framework and capture. If you're having a significant increase in the rate of rate of improvement. But yes, the foundation of our frontier safety framework, there is this, these critical capabilities, which are our sort of. foundational lines that govern our approach,
16:50
16:50
Speaker 1
-
Speaker 1
As have other companies here. So but but what about unconditional pause pledge? So so we've basically said that we will not deploy any model that we don't think is safe. And we've said that we will not progress as fast as we can build it, but we will only progress as fast as we will kind of make sure the safeguards keep pace. And so we paused reinforcement learning over the summer after Hugging Face, we did not deploy Astra 6.1.
We had that conversation kind of earlier. So the very clear pledge from the very top of our company is . That if it
We had that conversation kind of earlier. So the very clear pledge from the very top of our company is . That if it
16:50
Charlie Maynard
-
-
Charlie Maynard
is not safe, we will stop or we will pause. You were in the room with those four guys earlier presenting their information. I mean, you know, how can we rest easy and say, this is all okay? It is obviously not. From what you heard their evidence. Yeah. So so, you know, there's a massive disconnect here between . You're
16:50
Speaker 1
-
Speaker 1
saying, don't worry, we've got it. And we obviously don't. Well, what I what I did say with with respect humbly beforehand is that we think that we could we could bolster the current approach with, for example, third party evaluators. So not just taking our word for it, but also taking the word of the third party embedded evaluators, for example. I think that's important ensuring. And I was very pleased to hear the Minister's statement yesterday that AC gets. pre-deployment .
Access
Access
16:51
Speaker 1
-
Speaker 1
That's a commitment from you as open AI, to always give that as long as legally we are allowed to, we will, because we have a very good relationship with AC. And by the way, they don't duplicate either the tests that we do or the tests that are done in the US. They add something new and . Different,
16:51
16:51
Speaker 1
-
Speaker 1
American government let's you you will always give your models to AC. There is nothing to suggest that we would not do that because the collaboration we find positive. And yes, we . Yes we would. And you know, we have always been advocating for access for very capable, kind of like aces. I think the. complication you might get is when there are multiple aces around the world and you get a kind of compliance burden, which is highly duplicative, which is again, why potentially there's a role for international standards here and some form of kind .
Of like
Of like
16:51
Liam Byrne
-
-
Liam Byrne
deference. But right now AC has access to our models just around this out though. So. You've heard what I've read out from Ken Mcmullen's statement this afternoon. If MI5 comes along to you to say, actually, your model is now being used to give an uplift to low capability terrorists and to help them develop bio weapons, is there a kill switch in your company's disposal that actually . You
16:51
16:51
Speaker 1
-
Speaker 1
flick, so to speak, to retrieve those capabilities? Yes. So if if either AC or MI5 or MI6 or Gchq or any one of those authorities came to us and said, we think that your model is unsafe, or we think it's being used by a bad actor. We would obviously look at that immediately with the utmost urgency, and if we are in control of the model, we would then stop the deployment of that model, or we would pause the training of it, or we wouldn't deploy it.
Now, there are some nuances there with regard to how models in general are deployed, because we . Don't
Now, there are some nuances there with regard to how models in general are deployed, because we . Don't
16:51
Speaker 1
-
Speaker 1
somebody else, but the assurance the model could be beyond your control, but there would be others that would be able to to do this. And there's a very live active debate in the US, as I'm sure you're aware, around this circuit breaker or this emergency brake. And there's a lot of thought going into. precisely who has the power at what stage, and to ensure that humans do remain in control. But I can give you the commitment that, you know, if we're presented with evidence that our models are unsafe or that there's a real genuine frontier risk, then if we had the .
Capability to
Capability to
16:51
Speaker 3
-
Speaker 3
Smith, what do you do when Ken McKellen rings with a warning? Well, if any agency brought forward any evidence that the model was in any way dangerous, we have systems in place to stop it, as if it's being hosted elsewhere, as Tom was saying. But that's a partner issue. So if it's being hosted somewhere else, you clearly have a relationship with that. organization and they will be responsible entity for doing it. We have already made.
decisions about model. releases where we feel capability is incredibly high. So mythos preview earlier this year, we were confronted with the choice of general release or not. We chose not to do that and instead decided that a closed program where we phased in critical infrastructure providers was helpful because of the cyber capabilities the model represented. But yeah, absolutely. If any agency came forward with demonstrable . Evidence that
decisions about model. releases where we feel capability is incredibly high. So mythos preview earlier this year, we were confronted with the choice of general release or not. We chose not to do that and instead decided that a closed program where we phased in critical infrastructure providers was helpful because of the cyber capabilities the model represented. But yeah, absolutely. If any agency came forward with demonstrable . Evidence that
16:51
16:51
16:51
16:51
Speaker 3
-
Speaker 3
And then embed in powerbrokers and decision makers across industry, but elsewhere, it's entirely plausible that you end up in a very, very bad place. But it's not inevitable. We have agency, but that's why I think some of the things that we're discussing today, some of the things that were mentioned by witnesses earlier, are incredibly important. So questions like third party embedded evaluators, very, very important, but also very.
16:52
16:52
Speaker 3
-
Speaker 3
should have the power to be able to disagree with the companies and then make that public as well. If they do disagree, there's a question about what should be on the statute or not, what level of international collaboration there should be. And so there's a whole corpus of work that needs to be done. And again, what we need is the time and space to do that. And so if we slow down the development, we will create that time and space.
If
If
16:52
16:52
16:52
16:52
16:52
Speaker 3
-
Speaker 3
mandatory incident reporting. Lord Booth Smith Yes, we're in favour of binding commitments on on incident reporting we published earlier in the year something called the advanced AI framework, which was a very large document. That was was our attempt at thinking through how would a sovereign entity think about creating rules and governance structures around the technology. And within that, we talked about the concept of incident . Reporting
16:52
16:52
16:52
Speaker 5
-
Speaker 5
as a problem. Thank you, Mr. Lata. Very supportive of incident reporting. We think this is very important. We do think there is a discussion that needs to be had around standardised frameworks for incident reporting, so that the information that we're sharing is, is, is helpful. You also want to make sure that you're not overwhelming government authority or authorities with an enormous amount . Of
16:52
16:52
16:52
16:52
16:52
16:52
16:52
16:52
16:52
16:52
16:52
16:52
Speaker 2
-
Speaker 2
reports you needed to bring forward. Mr. Sherman. Yes. So with regard to to cybersecurity incidents, this is something that our company has had a mature program for the better part of 20 years around. That includes for the past 15 or so bug bounty program that encourages security researchers to find and.
16:52
16:52
Speaker 2
-
Speaker 2
so this this practice of both having a process to identify and respond to and also disclose vulnerabilities is something that we're quite familiar with. And it's something that we also as, as your colleagues have mentioned, notify people and governments when appropriate, if there's if there's an incident with regard to what the right statutory framework is. What I would point out is in the UK's agreement that it brokered in the summit in Korea about two years ago, there was a framework for for a home country that would be the recipient of notification and be responsible for coordinating with the other governments.
And I think dealing with some of the challenges that some of my colleagues talked about for making sure that there is a front door for notification and that it's triaged appropriately, that's one potential mechanism. But I would submit to you that that probably is the right conversation for the government to.
And I think dealing with some of the challenges that some of my colleagues talked about for making sure that there is a front door for notification and that it's triaged appropriately, that's one potential mechanism. But I would submit to you that that probably is the right conversation for the government to.
16:52
16:52
Speaker 10
-
Speaker 10
level of notification it wants versus how it wants to work with other governments. So that was a qualified yes, I think, and we've heard from other members of the panel that things that were set two years ago have required for updates in the intervening years. So I'm sure you could accept that something that was agreed a couple of years ago may not be appropriate for the future. But thank you. So will all of your companies now commit to immediate.
notification to the AI Security.
notification to the AI Security.
16:53
16:53
16:53
16:53
16:53
16:53
16:53
16:53
16:53
16:53
16:53
16:53
Speaker 3
-
Speaker 3
Whether any comparable systems remain in deployment. We completely commit to. informing the UK EC of any relevant incident for which it's the. It's the. The relevant body for. And then alongside that, I think the question of publishing beyond that is. Generally speaking, we've done that in the past. There is an issue around revealing vulnerabilities. So that's something that you would have to have in conversation with any affected entity.
So it's very hard to commit to just blanket publishing things without having a conversation with the entity affected. But we did give you one example, though. We did contact the AC . Prior
So it's very hard to commit to just blanket publishing things without having a conversation with the entity affected. But we did give you one example, though. We did contact the AC . Prior
16:53
16:53
16:53
16:53
16:53
Speaker 1
-
Speaker 1
affected. Okay. Thank you, Mr. Duff-gordon. So so we already published misalignment notices. So any evidence that we have that the models are behaved in a misaligned way, like they will go up on our misalignment blog. So that already exists. And we've already been doing that when it comes to notifications for cyber incidents, maybe one caveat would be, you know, I think we need to define the geography. So if these are UK kind of incidents, then of course we would then report them to the relevant UK authority.
We are currently in the process of doing that. As I mentioned before, there is a really legitimate question about.
We are currently in the process of doing that. As I mentioned before, there is a really legitimate question about.
16:53
16:53
Speaker 1
-
Speaker 1
Right, about a. vulnerability which can, you know, effectively impact the the entity that is being kind of hacked or accessed. But yes, we already have a misalignment blog and we have committed that we will be kind of prompt and transparent with. notifying the UK government if one of our . Agents has
16:53
16:53
16:53
16:53
Speaker 5
-
Speaker 5
did not intend. Thank you, Mr. Larter. Do you think this. information sharing is very important? If we talk about specific incidents like the one we've discussed earlier in the session, we informed the UK AI Security Institute about that. I think a lot would depend on the specifics of the incident where it took place, the nature of it, what type of risk it is. I mean, there's certain types of incident that I think it really wouldn't be particularly appropriate to publish an enormous amount of public information around it, biosecurity risk or cyber security.
16:53
16:53
16:53
16:53
Speaker 2
-
Speaker 2
detail on the specifics of the incident. Thank you, Mr. Sherman. I agree with my colleagues that the details matter here, particularly identifying what incidents are in scope and also making sure that any public disclosures are done in a responsible way that doesn't increase risk. That said, I will absolutely commit to talking with . AC
16:53
16:53
16:53
16:53
16:53
16:53
Alison Griffiths
-
-
Alison Griffiths
thinks is appropriate. Okay. Thank you, Mr. Griffiths. Thank you chair. I'm very interested in parity with the US. And so I'd like to start with just a sort of yes or no answer from each of you on whether you will commit to. providing all new frontier models to the EC for. testing on the same equivalent terms to those offered . To
16:53
16:53
16:53
Speaker 2
-
Speaker 2
equivalence to this, to what you do in the US already. So my understanding of the relationship between the aces is that we provide our models to the US AC, which is called KC, and then it has relationship with the UK, AC and potentially others, and it has the ability to do that. I would have no objection if .
16:54
16:54
16:54
Speaker 5
-
Speaker 5
with with the AC here. Thank you. Mr.. Yeah, I think there is this ongoing government to government. discussion between the US and the UK, which I think I think is important. We have provided our models to the AC for the last couple of years in the UK. We think that's really important. We certainly, you know, within the context and of . The
16:54
16:54
16:54
16:54
16:54
16:54
16:54
Speaker 3
-
Speaker 3
by the US government? So in the spring of this year, as a result of of Mythos preview becoming available in a closed program called Glasswing that activated a whole range of conversations across the US system, which was incredibly positive because it encouraged greater levels of awareness and engagement. As those relationships developed and people became more aware of the technology's capability and how our testing regimes worked, we were in a. position with.
5.1 where we unfortunately were not able to.
5.1 where we unfortunately were not able to.
16:54
16:54
Speaker 3
-
Speaker 3
US government. So that was a US government decision rather than an anthropic decision? No. You asked why is the US government different? I said I wasn't privy to those . Conversations, So I'm sorry about that and I'm happy to write with a bit more detail on it. That'd be helpful. Thank you. But they do have access to it now, and they were at the time offered unrestricted access, sorry, restricted access, but obviously that's slightly different .
Having
Having
16:54
16:54
16:54
16:54
16:54
16:54
16:54
16:54
16:54
16:54
16:54
16:54
16:54
16:54
Speaker 2
-
Speaker 2
already been taken? Perhaps if I could start with you, Mr. Chairman, I think it is absolutely. Beneficial and critical thing to do. And the reason is I think we often talk about these risks as binary. Either this risk exists or it does not exist. I think in. reality it exists along a spectrum and it becomes incremental capabilities increase and our ability to defend against risk and also the risks themselves increase incrementally.
And so I think that it is extremely important. And as you've heard, the UK AC particularly does. evaluations that other governments don't do that are really novel and a contribution to the field. And I think that having an understanding, both as a government but also as a as an. ecosystem of where technology is going,
And so I think that it is extremely important. And as you've heard, the UK AC particularly does. evaluations that other governments don't do that are really novel and a contribution to the field. And I think that having an understanding, both as a government but also as a as an. ecosystem of where technology is going,
16:55
16:55
16:55
16:55
16:55
Speaker 5
-
Speaker 5
At a later stage? Yes. Okay. Thank you, Mr. Lotter. Yeah, I think we're very clear. The UK AC testing is. absolutely critical in the same way that our partnership with the US equivalent, the US KC, is really important as well. Testing is a big part of that. There's also the ongoing research collaborations that I mentioned as well, where we're able to work with the technical.
experts in the AC to work out how you also mitigate . Risks
experts in the AC to work out how you also mitigate . Risks
16:55
16:55
Speaker 1
-
Speaker 1
the UK sees incredibly important across the piece. Mr.. Yeah, I would say the same thing. I think the the important thing is that both KC and AC get the model pre-deployment as long as they do. I think the AC testing is very meaningful. As I've said before, the AC has deep experience in certain areas by, for example, and historically we've collaborated with them, listened to their feedback and changed our model. So .
We do
We do
16:55
16:55
16:55
16:55
Speaker 3
-
Speaker 3
to continue providing the model. Thank you. And finally, a little bit Smith AC is best in class for what it does. And so in particular, it has some very. specific verticals of research which is very, very good at which others don't have very, very good, very, very, very good on very, very good on alignment questions. So they're a fantastic. institution staffed by some of the best researchers in the world.
In answer to the question about is it just marking off the fact? No, the work of the AC makes a meaningful contribution to the model development process.
In answer to the question about is it just marking off the fact? No, the work of the AC makes a meaningful contribution to the model development process.
16:55
16:55
16:55
16:55
Speaker 4
-
Speaker 4
pre-release for models. Okay. Thank you, thank you. We heard from Daniel in the first session, first panel that a testing at. pre-deployment stage would not be sufficient and that they needed, if it was going to have any effect, needed to be involved at an. internal development internal deployment stage alongside your own safety staff. Are you. prepared for AC to be involved at that very early stage in testing, or or would you only allow them to test .
At
At
16:55
Speaker 3
-
Speaker 3
pre-deployment stage or start with Lord Smith? We're very happy to have a close relationship with the EC. There is there is a current conversation going on between the two governments about what the framework should look like for things like pre-deployment access, and I really wouldn't want to cut across that because they're having that conversation, not me. But the idea of the UK AC being much closer to the what we do, I think is a very, very positive thing.
16:55
16:55
16:55
Speaker 5
-
Speaker 5
AC. So, you know, this is a question that will look closely at AC partnership. Incredibly important. I think it's important to understand the nature of the relationship. Now. It's not just like we sort of go away for a long time and then come back every now and then. Once we have a model to test, this is ongoing discussion between our technical experts working right . Across the development and deployment
16:55
16:55
16:55
Liam Byrne
-
-
Liam Byrne
with them about what ways they think would be appropriate. Okay, well look, that's been an incredibly informative session. Thank you very much indeed. At the core of this question is obviously the core of this debate is obviously whether we have a regime for trust. That is right. I'm not sure any of us on this side of the table are convinced that today's regime is going to maximise trust in the way that is needed. There are differences of opinion between you about the role of statute, whether we can actually develop existing laws, in particular around liabilities.
I think most of you are of the view that stronger international conventions are, however essential. You will know that yesterday ministers merely announced that they are considering not committing to new regulatory powers. And so that is . The
I think most of you are of the view that stronger international conventions are, however essential. You will know that yesterday ministers merely announced that they are considering not committing to new regulatory powers. And so that is . The
16:56
16:56
Liam Byrne
-
-
Liam Byrne
16:56
16:56
16:56
16:56
Liam Byrne
-
-
Liam Byrne
To cyber security and the importance of working with our partners to protect. So you'll know you'll have memorized section 9.1 of the ministerial code. But if you haven't, let me read it to you. When Parliament is in session, the most important announcements of government policy should be made in the . First
16:56
16:56
Speaker 12
-
Speaker 12
Why was it briefed to the Telegraph on Sunday? Well, obviously our ministers take their. responsibilities very, very seriously, and I'm aware that there was some trailing in advance. I will make sure that I report back to the minister your concerns about his compliance with the code, and I'll support him to give him some advice on how best to best to respond to that. But as.
16:56
16:56
16:57
16:57
16:57
16:57
16:57
Liam Byrne
-
-
Liam Byrne
Clearly that is that is not the case. Yes. So the announcement, which was confirmed in Minister Darwin's letter at 10:31 p.m. last night, reiterates merely the contents of the written ministerial statement, and it says that . the government is offering consideration but not commitments. No new regulatory powers, no timetable for amendments to the cyber security Bill, no binding duties on AI developers.
16:57
Speaker 12
-
Speaker 12
Is that true? Have I understood the announcement correctly? So as the minister set out, there are a number of. different actions that we are taking in order to make sure that we manage both the risks and promote the. opportunities of AI. And as a combination of things that we're working on, part of it is regulation. Part of it is standards.
16:57
16:57
16:57
16:57
16:57
16:57
16:57
16:57
16:57
16:57
16:57
16:57
16:57
16:57
16:57
16:57
16:57
16:57
16:57
16:57
Speaker 12
-
Speaker 12
So to the extent that we can so everything that we're doing has got to have real world delivery bite. A lot of the work that we're doing at the moment is on a voluntary basis and partnership with industry. But for all the reasons that you've just discussed with your previous panels, that's exactly why we are doing further work on standards, principles, guardrails.
16:57
16:57
16:57
16:57
Speaker 12
-
Speaker 12
That is the case. Yes. Okay. Do you think that should change? So at the moment, if you look at the evidence and Henry might want to to add to this, we have a really good track record of working on a voluntary basis. We obviously can't guarantee it, but the success of the work that we're doing, particularly with AC, is dependent on having access to models. We've got really good.
16:57
16:57
16:58
16:58
Liam Byrne
-
-
Liam Byrne
the package of levers that we have within the UK to manage risk. So when we look at the government's approach to AI compared to, say, aviation, financial services, pharmaceuticals, food safety in aviation, financial services, pharmaceuticals and food safety, you've got proactive risk monitoring, independently set safety standards, authorization of market entry, post-market monitoring. An independent regulator with enforcement powers, accountability measures and routes to redress and transparency and reporting requirements that are mandatory.
And I can't quite understand why the government is not proposing to basically align the AI safety regime with the approach that we're taking . In
And I can't quite understand why the government is not proposing to basically align the AI safety regime with the approach that we're taking . In
16:58
16:58
Speaker 12
-
Speaker 12
financial services. pharmaceuticals and food safety. So this committee will be aware that our approach to regulation of general technology is to regulate, within context. One of the big policy questions at the moment is whether that continues to be enough. And ultimately, the pace at which the frontier models are moving, the level of risk that we're seeing. Yes, regulation has a really important part, and I think ministers have made it very, very clear that we will pursue regulation where we think it is enforceable, useful and will have an impact.
But that's got to be part of a total framework. That also. includes the work that we do in.
But that's got to be part of a total framework. That also. includes the work that we do in.
16:58
16:58
16:58
16:58
16:58
16:58
16:58
16:58
Speaker 13
-
Speaker 13
department in the Department for Business and Innovation. And I think the proposals. In fact, DCMs is responsible for the cyber bill. But we are proposing in the speech that the Minister made yesterday to use the Cyber Security and Resilience Bill to consider changes to improve preparedness for AI, cyber and loss of control risks. Look at how serious incidents and near misses are identified and.
16:58
Liam Byrne
-
-
Liam Byrne
Reported, and how AI driven disruption across the organisations is contained. Sorry, Alex, just to interrupt, you can't consider something in a bill. You have to legislate for something in a bill. So the question is, is given the urgency. Cat little is just underlined by the the Cyber Security Bill hits the Commons.
16:58
16:58
16:58
16:58
16:58
16:58
16:58
16:58
16:58
16:58
16:58
16:58
16:58
16:58
16:58
16:59
16:59
16:59
16:59
16:59
16:59
16:59
Speaker 14
-
Speaker 14
months, with its first reports in the months ahead. But it's hugely welcome to hear that, and we're keen to help as much as we possibly can to provide provide whatever you need for it. On that specific question. So I think it is worth saying right at the start. So what is AC and what is in it? So it is it is it is a part of the Cabinet Office.
It's a directorate there. It is a.
It's a directorate there. It is a.
16:59
16:59
Speaker 14
-
Speaker 14
is not making policy. And the one thing that we are what we're doing when we test models. So when a when an AI company says, we want you to test the model, we say to them, well, how powerful it is. Like we need some information about it because we won't test every single model that's out there. It's just not actually worth our time or resources to test some of the less powerful ones. We will then go through a process of testing that model.
I can talk in depth about that, and then we will write back to the company with what we found, and then the company will take actions to improve things. Depending on what we've done, what we're not doing is kite marking. We're not saying yes or no to any sort of model.
I can talk in depth about that, and then we will write back to the company with what we found, and then the company will take actions to improve things. Depending on what we've done, what we're not doing is kite marking. We're not saying yes or no to any sort of model.
16:59
Charlie Maynard
-
-
Charlie Maynard
That is that isn't what we're we're doing in our role. Okay. But let's let's say you write back saying we are absolutely terrified about what this model is. Yeah. Now you are relying entirely on their good graces to then say, well, you know, we agree with you. Let's say they don't agree with you and they proceed. What I'm worried about is if a model is then put out into the public domain, the world and we, you know, your job is to protect.
Our job is to protect.
Our job is to protect.
16:59
16:59
16:59
16:59
16:59
16:59
16:59
16:59
16:59
16:59
16:59
16:59
Liam Byrne
-
-
Liam Byrne
Informed and they're aware of what what risks are. The challenge we've got is Ken McMullen has warned this afternoon about low capability terrorists being given a capability uplift using AI to develop bioweapons. And Mr. Maynard is asking what powers are available to basically stop that kind of technology entering the UK market.
17:00
17:00
17:00
17:00
17:00
17:00
17:00
17:00
17:00
17:00
17:00
Charlie Maynard
-
-
Charlie Maynard
So I think that's a real example of how we've made the world safer. Can I just I mean, look, there are a whole lot of Chinese models to which you don't have this agreement in place. Yeah. So, so so you might be looking to the US. Maybe they do everything you want them to do, but the Chinese don't. And people with those Chinese models don't. The key question is what controls do we have? And by the way, I'm not saying this is an easy answer or I've got the answer.
I'm not. So but but but but I'm worried. Scared. So so what what are we you know, what can we do as a country to protect. ourselves against a very dangerous model? Can you . Is there
I'm not. So but but but but I'm worried. Scared. So so what what are we you know, what can we do as a country to protect. ourselves against a very dangerous model? Can you . Is there
17:00
17:00
Speaker 12
-
Speaker 12
plan? Maybe that's the first question. Is there a plan if I. Yes, and Alex should add to this. I mean, I must come back to what the Prime Minister and ministers have already said. Like. fundamentally this is about working with international partners and making sure that we use our convening power, especially at the G20, during the. presidency year, to tackle this issue. This is not something that the UK alone can solely lead on.
We have got to ensure that we're working with all of our foreign colleagues and industry partners to make the most. effective response to exactly your.
We have got to ensure that we're working with all of our foreign colleagues and industry partners to make the most. effective response to exactly your.
17:00
17:00
17:00
17:00
17:00
Speaker 13
-
Speaker 13
Which is all about making sure we're resilient to a whole range of biological risks. The Ministry of Defence invested announced an investment of. 150 million earlier this year to make sure we're raising our defences against AI. That's including using AI to improve our biological security. And AI clearly has real opportunities in relation to developing drugs, but also massive challenges. And that's about. So the Mod investment is about improving biological.
security and mitigating some of the risks posed by autonomous agents. And that's alongside investments in biosecurity centres.
security and mitigating some of the risks posed by autonomous agents. And that's alongside investments in biosecurity centres.
17:00
17:00
Liam Byrne
-
-
Liam Byrne
know that doesn't respond to all of those risks. But there's definitely a we might we might as well just kind of nail this down just so we all kind of know where everything is. So we've had a pretty stark warning from Ken McKellen this afternoon. We've heard a range of views from AI companies today about their views on statutory regulation and the role of international conventions. The truth is, though, Henry, we don't actually have the power to stop dangerous AI .
Models
Models
17:00
17:00
17:00
17:00
17:00
17:00
17:00
17:00
17:01
Speaker 14
-
Speaker 14
Available to bad actors. Mr. de Zoet I mean, I think the question. I mean, with open weights models, like I don't think anybody necessarily has that power. So I'm not quite sure like Britain, you know. So there's a bunch of things to think about here, right? These companies are either American or Chinese who are who are building at the frontier with the stuff that is the most powerful. There's only so much we as the UK can do in that kind of context.
We are, I think, doing extremely good work with the AI Security Institute and in diplomatic. engagement to try and move things in the right direction on on it, you know, and with AC, that's not just pre-deployment testing of the models, that's post-deployment testing of the models. We're doing.
We are, I think, doing extremely good work with the AI Security Institute and in diplomatic. engagement to try and move things in the right direction on on it, you know, and with AC, that's not just pre-deployment testing of the models, that's post-deployment testing of the models. We're doing.
17:01
17:01
Speaker 14
-
Speaker 14
We're publishing things to, to move the field onwards. We've built like. 6 or 7 tools that we've open sourced that actually all the AI to, to carry out evaluations that all of the AI companies are currently using. The US AC equivalent uses it. The South Korean government uses it, Japan uses it, the EU uses it. We are at the heart of the international network of AI security institutes. So I think the UK is .
17:01
17:01
17:01
17:01
17:01
Sarah Edwards
-
-
Sarah Edwards
Because I'm sure we'll return to it to some slightly more positive. potential for artificial intelligence. Obviously, ministers have talked a lot about how we could deploy that to improve the. economy and to support public services. And I'm just wondering about how we can get that balance right. So if we're going to train these models, if we're going to work.
17:01
17:01
Sarah Edwards
-
-
Sarah Edwards
As we've heard, what do we have to bear in mind when we're considering that there is still going to be risk there? But we know that there are lots of benefits that we could achieve from that. And I mean, Cat, you may wish to to comment, but Henry, perhaps from your perspective, from the Institute's perspective, how can we make sure that we're not just being completely permissive, that we've still got some of those. safeguards so that when we're deploying that in the in the public services and hopefully .
To
To
17:01
Speaker 12
-
Speaker 12
great effect, we do have that confidence. Do you mind if I start and I know Henry will want to give an acei perspective, but I think primarily your question is, is a policy question about how do we manage the very best. opportunities that come from AI, as well as balancing out with the risks and the. The simple answer is that AI is the defining opportunity and the biggest challenge of our generation. There's absolutely no doubt about it.
And I don't think we should be forced to choose between the. opportunity of.
And I don't think we should be forced to choose between the. opportunity of.
17:01
17:01
17:01
17:01
Sarah Edwards
-
-
Sarah Edwards
both. It is not a decision between the two. Thank you. That's Henry. Then can you can you add some flavor to that in terms of how do you make that reality? Because we've just had quite a debate on this committee about whether or not that's even possible and whether or not there is that trust and whether or not we can deploy those things. So I still feel as though we're I know that we want all the good things from it, but I think we're all now starting to panic slightly about the bad things that come from it.
So are we sure in that deployment, particularly in public services? I mean, maybe it's about saying why. I mean, I'm not clear, for example, why we need to train them on everything in the world entirely, to be perfectly frank. You know, if we're going to deploy them in the public sector, are we going to be looking at models that are specifically looking at very specific tasks? That would be a much easier way, surely, to be . Able to
So are we sure in that deployment, particularly in public services? I mean, maybe it's about saying why. I mean, I'm not clear, for example, why we need to train them on everything in the world entirely, to be perfectly frank. You know, if we're going to deploy them in the public sector, are we going to be looking at models that are specifically looking at very specific tasks? That would be a much easier way, surely, to be . Able to
17:02
17:02
Speaker 13
-
Speaker 13
policy balance that you're pointing to. I mean, it's testing the frontier models. Actually, if you look across the economy, most people aren't using the frontier models. They're using different models. And I think one of the things we're trying to do to increase resilience, but also make sure we are looking at the. opportunities is make sure you've got a diverse set of models. You are thinking about how you apply them and what you use them for. And we're helping people trust those models.
So we've been introducing, as the minister said yesterday, some AI assurance. So how do you make sure the models are doing the things that you think they should be, and how can you test that and have some tools available? So as we deploy it, as I say, trying to get that domestic kind of capability, but not at the frontier, usually you don't always need the frontier in some of the.
So we've been introducing, as the minister said yesterday, some AI assurance. So how do you make sure the models are doing the things that you think they should be, and how can you test that and have some tools available? So as we deploy it, as I say, trying to get that domestic kind of capability, but not at the frontier, usually you don't always need the frontier in some of the.
17:02
17:02
17:02
17:02
Speaker 13
-
Speaker 13
Make sure you've got that AI assurance so you can actually trust it, and you're kind of able to test it in the businesses and then still make sure we are testing the frontier models, make sure we understand them, make sure we've got access to them where we need to. But I don't think that's going to be the spectrum across across the piece. All of that also needs to . Be done
17:02
Sarah Edwards
-
-
Sarah Edwards
with workers and people and the opportunities it creates in mind. And I just wonder, is that differential between AI models and frontier models? Is that clear in terms of how the ministers are talking about the benefits to the. economy and to public services and the like? And and so perhaps you can enlighten us on how are they going to approach that? Because also we're we're not quite clear whether or not we're going to be.
regulating the upcoming .
regulating the upcoming .
17:02
17:02
Speaker 1
-
Speaker 1
In our experience of adoption, particularly in the public sector so far, is our real challenge is scaling and adopting to get the benefits of productivity, efficiency and service outcomes for for the public using very openly. available AI tools. So, you know, we have ambitions to make sure that we're using AI right in the front line of public. services just to boost. productivity in the way that we know it's capable of.
So there's brilliant work underway. That's nowhere near using some of these frontier models for the work that we do. And I think part of our job in the public sector, when we're looking at. adopting AI, is to know which technological tools are best suited in our value streams for actually delivering the productivity. improvements and the service improvements that we're looking for. And frontier. models aren't necessarily always the best tool for the .
Sort of
So there's brilliant work underway. That's nowhere near using some of these frontier models for the work that we do. And I think part of our job in the public sector, when we're looking at. adopting AI, is to know which technological tools are best suited in our value streams for actually delivering the productivity. improvements and the service improvements that we're looking for. And frontier. models aren't necessarily always the best tool for the .
Sort of
17:02
17:02
Speaker 2
-
Speaker 2
outcomes that we're looking for. Thank you. Thank you very much, chair. What have you learned about your testing processes after the breach in July? I trust your environment. So that's a very, very good and important question. So the thing to be clear about about our incident, which was was an incident, was that there was no security breach. Like no one. It wasn't like some of the other incidents that we've heard, where the models have worked out an exploit to get out of their sandbox and go and do bad things.
When we set up our.
When we set up our.
17:02
17:02
Speaker 2
-
Speaker 2
We actually connected it the models to the internet like we and we did that on purpose because we wanted to see what the models their capabilities would be to go out there. Bad people are going to connect these things to the internet. So there is a very fine line between doing testing in a simulated. environment and actually connecting it into the real world and seeing what's going on . And what the model then went and did was definitely bad behavior.
And when we we became aware of it, we shut it down within an hour, published a full report on it within a week, and I could talk through the things that we've changed to do that which are. essentially we reviewed when we give internet access to the to the models to make sure that we know when we're doing it, we've hardened our. sandboxes to make them harder anyway, even though there wasn't a security breach and we've. improved our monitor ability.
So we've created a synchronous. monitor that's ensuring it looks at what the AI is doing. The thing that I would say on this is that. The speed and change of the.
And when we we became aware of it, we shut it down within an hour, published a full report on it within a week, and I could talk through the things that we've changed to do that which are. essentially we reviewed when we give internet access to the to the models to make sure that we know when we're doing it, we've hardened our. sandboxes to make them harder anyway, even though there wasn't a security breach and we've. improved our monitor ability.
So we've created a synchronous. monitor that's ensuring it looks at what the AI is doing. The thing that I would say on this is that. The speed and change of the.
17:03
17:03
Speaker 2
-
Speaker 2
Kind of caught us off guard at that point because we didn't think the model would have the. capabilities to go out and do the kind of things that it went and did. On reflection, I think, you know, we shouldn't have connected it into the internet, but it's these things are judgment calls, you know, and previously we'd done and things hadn't gone wrong. And we learned lots of important things. And it is very important to test in real life situations to see what's see what's happening.
And but we've improved the environments. But this like speed of improvement across these, these models is the thing . That
And but we've improved the environments. But this like speed of improvement across these, these models is the thing . That
17:03
17:03
17:03
17:03
Speaker 2
-
Speaker 2
Well, so this is a nascent science like evaluation of these models is, is, has been very, very new. And it's only really started in the last 2 or 3 years. And we're learning all the time. So the kind of things that we're doing now is, is creating very. detailed cyber ranges. So we simulate a computer network for a, for a model to, to, to attack. I mean, this time last year, the models were able to do kind of very simple, narrow cyber tasks of certain nature.
We've now created the cyber ranges that involve 32 steps. Like that can be everything from reconnaissance to exploit . identification and then lateral movement inside the system, and then data. exfiltration that these take a human 20 hours to do. And we found in April this year that the models were able to do these autonomously. And now they can all do it. We're having to build more and more.
difficult cyber ranges, doing lots more difficult tasks to, to get them. We've got one that is a it's called a cooling tower. It's actually it's a harder task, but it involves the models doing things that might like make things happen in the real world physically. That's also a cyber range that has been now completed by the models. And we're having to like use all our resources and talent to keep building these, these simulated .
We've now created the cyber ranges that involve 32 steps. Like that can be everything from reconnaissance to exploit . identification and then lateral movement inside the system, and then data. exfiltration that these take a human 20 hours to do. And we found in April this year that the models were able to do these autonomously. And now they can all do it. We're having to build more and more.
difficult cyber ranges, doing lots more difficult tasks to, to get them. We've got one that is a it's called a cooling tower. It's actually it's a harder task, but it involves the models doing things that might like make things happen in the real world physically. That's also a cyber range that has been now completed by the models. And we're having to like use all our resources and talent to keep building these, these simulated .
17:03
17:04
17:04
Speaker 2
-
Speaker 2
Staff, brilliant, brilliant civil servants, brilliant, brilliant technical team who are taking pay cuts to to be involved in in AC because they believe in the mission and they want to they want to work with with government, all this stuff. We do four times as many tests now than we did back in in 2024. We used to be able to when we tested a model, it would be like a whole team exercise. It would take a month or even two months to do.
We could only run one model test at a time. We're now able to run multiple tests of models at the same time and get results within days. So we . Are
We could only run one model test at a time. We're now able to run multiple tests of models at the same time and get results within days. So we . Are
17:04
17:04
17:05
17:05
17:05
17:05
Speaker 1
-
Speaker 1
Then obviously the question of who does that is a really important one. But right now, AC is not a regulator. It's here to do really valuable work. And if you don't mind me saying, I think he's doing a fantastic work at it. You've heard today from many people just how important it is to the overall .
17:05
17:05
17:05
17:06
17:06
Speaker 1
-
Speaker 1
in your own mind. Do you know what you'll recommend? Well, we're doing the work now. And by the way, I haven't even started to take up office yet into my new role. But this is this is very live in terms of the work that we are doing. But, you know, to assure this committee it is. comprehensive and very thoughtful. analysis that is not just about the UK, it's about how we work in a really .
17:06
17:06
17:06
Speaker 5
-
Speaker 5
our counterparts internationally. Okay. Thank you. So we don't have our own sovereign AI frontier model. Google DeepMind is based in London, though, but is U.S. owned. We are seeing and we've heard today that we are having more and more. powerful frontier models. And there is a. relative view of safety rather than absolute view of safety amongst those developing them.
So what role can a country like the UK play in preventing a .
So what role can a country like the UK play in preventing a .
17:07
17:07
17:07
17:07
17:07
17:07
17:07
17:08
Speaker 2
-
Speaker 2
and that is absolutely our intention. Henry, you should add, if I can add. So you might have seen that yesterday. AC published its approach on best practices. And this was following the Prime Minister's discussion about kind of like. standards for safety and security and what can be done. And so specifically on your question on preventing a global race and safety standards. So we are engaging with industry along the lines on this and working with our brilliant partners in government, like the National Cyber Security Centre and others, about what is the next step of.
standards and best practices that could happen, such as over. monitoring or preserving oversight over these systems. We've laid out a kind of section of eight areas we want to get into, and I'm very happy to I feel quite strongly about monitoring. I can I can talk about that if you would like me to. But I think this is the working with industry, working with brilliant people in government, working across. governments in this kind of area around the technical capability that we've got in this government to be able to actually write these.
standards and write these best practices is like, I genuinely think, quite unique in the world. And .
standards and best practices that could happen, such as over. monitoring or preserving oversight over these systems. We've laid out a kind of section of eight areas we want to get into, and I'm very happy to I feel quite strongly about monitoring. I can I can talk about that if you would like me to. But I think this is the working with industry, working with brilliant people in government, working across. governments in this kind of area around the technical capability that we've got in this government to be able to actually write these.
standards and write these best practices is like, I genuinely think, quite unique in the world. And .
17:09
Sarah Edwards
-
-
Sarah Edwards
That's what we're going to we're going to try and do with other people's help. There are also, if I may, some examples of sector regulation. So you've got the Medicines and Healthcare Products Regulatory Authority where they're. regulating not AI, but medical devices. And they are looking at they're running a sandbox to just check if there are AI specific questions. They've been accepting. recommendations from the National Commission into the regulation of AI and healthcare to make sure that we monitor those devices throughout their lives.
So there's a number of sector.
So there's a number of sector.
17:09
17:09
17:09
17:09
17:09
17:10
17:10
17:10
17:10
17:10
17:10
17:10
17:10
Liam Byrne
-
-
Liam Byrne
across all of the things that we've we've discussed. Okay, the clock is against us. So I'm sorry that that session was slightly cut short, but we will aim to generate a special report out of today's session in order to help inform Parliament ahead of potential government amendments to the Cyber Security Bill. Once it reaches the Commons, there is a new inquiry on AI safety and security that has been launched today by the committee, which will help us contribute to this debate over the course of the next year.
But for now, thank you very much indeed for your evidence, Henry. We look forward to coming to visit you and at . Some time
But for now, thank you very much indeed for your evidence, Henry. We look forward to coming to visit you and at . Some time
17:11