Can society primarily rely on AI companies' own decisions and existing incentives to prevent serious harms from advanced AI systems, without substantially stronger external regulation?

“It really is a, a form of protection. They understand that they have to self-police. ... But there's going to be a tremendous self-policing aspect.”

“these are, these are, companies with agency, agency. These are CEOs with agency. ... if a car company, competing with all bunch of other car companies, which, which they are, I'm competing with all kinds of companies, which I am. If I believe that I'm about to launch a product that is unsafe, It is completely in my ability, my”

“If I believe that I'm about to launch a product that is unsafe, It is completely in my ability, my power, and my responsibility, and I'm incentivized to do so, to not launch the product.”

“I completely agree that safety is paramount. I completely believe safety is paramount. I completely believe companies ought to ship safe products. I believe that CEOs and leaders of companies and the board of directors of companies have the responsibility and should have the courage to do the right thing.”

“there are so many laws, there's so many obligations, they're so incentivized to ship safe products. If they ship unsafe products, their customers go away. If they ship unsafe products and they harm somebody, they could have a civil lawsuit.”

“what we need is for you guys to make smart decisions about what to release. If a product is unsafe, which is to say it's unreliable or behaves unpredictably, then don't release it. And stop looking to some sort of global governance as the answer.”

“If a car company, competing with all bunch of other car companies, which they are, I'm competing with all kinds of companies, which I am. If I believe that I'm about to launch a product that is unsafe, it is completely in my ability, my power, and my responsibility, and I'm incentivized to do so, to not launch the product.”

“I agree with you. That market forces play a huge role here. Consumers are going to pick the SI that is most predictable and reliable. This is a point Mark Zuckerberg has made. So I do think market forces play a role here.”

“in our other scenario ai 2040 plan a where we describe like What we think the government should do to regulate this industry and how they should negotiate with China to get China to do similar things”

“the company is biased and shouldn't be trusted to make all these judgments appropriately because of their incentives. And then the government auditor, well, they might just be limited. Even if they're trying their best, there might not be that many of them. They might have limited experience. They might be like busy, stretched between monitoring different companies and so”

“the way to do it is to have different AI companies spread out over maybe some different countries, but have extreme levels of transparency and regulation so that they're not, so they're not in this sort of prisoner's dilemma where if I don't do it, the other guy will.”

“that's why we didn't go for like a more normal, like there should be a regulatory agency that gets to come in and monitor what the companies is doing.”

“You can't rely on the industry to self-regulate here. I mean, it's just insane.”

“You can't rely on the industry to self-regulate here. I mean, it's just insane.”

“the FTC is now looking into OpenAI and Anthropic, which feels a little bit like Does the government want them to self-regulate or does it not? Because the way they seem to want to self-regulate is by effectively forming a cartel.”

“I think it's really weird that they all have been, not all, but many have been begging the government to regulate them and enforce them to slow down. And then they all get together and sign this pinky swear that's like morally binding, that has no teeth to it, that says like, we're going to police each other.”

“You know, one potentially positive vision is, you could have the AI industry be something like the pharmaceutical industry, which moves very conservatively and, like in my opinion, often too conservatively. and the incumbent pharmaceutical companies are, are pretty positive on this regulatory regime that imposes so much conservatism because that creates like a moat.”

“one way you can align corporate incentives with the public good is regulation, and the regulators are currently refusing to do that. I mean, we just saw them come out with a voluntary sort of Semi-agreement between the AI labs, it is not going to be legally enforceable”

“I don't think self-regulation works, and I would agree that we will need to find whether it's local, national, or international, real rules about, about how these systems work and the harms and the damages that they might cause.”

“if we just prosecute OpenAI in the way that we would if somebody intentionally hacked hugging face and make sure that there are real consequences there, then that is something that I think will actually really help move the needle towards safer practices of using AI.”

“I don't give a shit if another company or country might go ahead, take responsibility, and do not be the ones who bring disaster down on all of our heads.”

“If the government won't act, the companies cannot wash their hands of responsibility. If they can't make a convincing case to impartial external experts that the next training run or internal deployment will be safe, they must delay it.”

“Governments must now require companies to demonstrate that they can keep their current systems fully under control before they train a new, more powerful and dangerous one.”

“There's also competitive pressure. If one company slows down or country, then others won't.”

“We are forcing you to go ahead because of the boogeyman of China.”

“releasing this information publicly is probably a lot better than releasing it just to a government body. Basically because, you know, we're, we're like building the plane of like AI safety research, like as we're flying it. and it's not like there's a box checking exercise, that any kind of government agency that's like often understaffed, especially with like technical”

“if you had this sort of requirement where you can just, like, publish it, or all the, all the training, the whole life cycle of every AI as it's trained is just, like, visible to everybody, then someone trying to insert a hidden bias like this, well, everyone would see that they're doing it, you know? So I think it”

“I really hope that there can be some sort of regulation or requirement when incidents like this happen to Like let people in to study what happened and run variations of it and things like that.”

“that would distribute much more the power over AI development. And I also think it would be much better from an alignment perspective, because it would just, you would just get more open debate, more expertise going into the decision.”

“I think it could still be a big win, even if it's not like the case that everybody instantly signs on to it. I do think eventually you want to have a regime that has some teeth to it, but I think it could be like a pretty big change, even if a few AI labs signal that they take”

“I think the vision would be, I think at first through self-regulation and then probably hopefully through something that's more binding. You have companies agree that, you know, if they're thinking of training and deploying any model bigger than, say, GPT-3, then they need to involve an external evaluation or auditing team, which could be ARC or could be some”

“I think it could still be a big win, even if it's not like the case that everybody instantly signs on to it. I do think eventually you want to have a regime that has some teeth to it, but I think it could be like a pretty big change, even if a few AI labs signal that they take”

“if you had this sort of requirement where you can just, like, publish it, or all the, all the training, the whole life cycle of every AI as it's trained is just, like, visible to everybody, then someone trying to insert a hidden bias like this, well, everyone would see that they're doing it, you know? So I think it”

“I think the vision would be, I think at first through self-regulation and then probably hopefully through something that's more binding. You have companies agree that, you know, if they're thinking of training and deploying any model bigger than, say, GPT-3, then they need to involve an external evaluation or auditing team, which could be ARC or could be some”

“I don't think self-regulation works, and I would agree that we will need to find whether it's local, national, or international, real rules about, about how these systems work and the harms and the damages that they might cause.”

“one way you can align corporate incentives with the public good is regulation, and the regulators are currently refusing to do that. I mean, we just saw them come out with a voluntary sort of Semi-agreement between the AI labs, it is not going to be legally enforceable”

“that would distribute much more the power over AI development. And I also think it would be much better from an alignment perspective, because it would just, you would just get more open debate, more expertise going into the decision.”

“releasing this information publicly is probably a lot better than releasing it just to a government body. Basically because, you know, we're, we're like building the plane of like AI safety research, like as we're flying it. and it's not like there's a box checking exercise, that any kind of government agency that's like often understaffed, especially with like technical”

“in our other scenario ai 2040 plan a where we describe like What we think the government should do to regulate this industry and how they should negotiate with China to get China to do similar things”

“We are forcing you to go ahead because of the boogeyman of China.”

“I really hope that there can be some sort of regulation or requirement when incidents like this happen to Like let people in to study what happened and run variations of it and things like that.”

“that's why we didn't go for like a more normal, like there should be a regulatory agency that gets to come in and monitor what the companies is doing.”

“the way to do it is to have different AI companies spread out over maybe some different countries, but have extreme levels of transparency and regulation so that they're not, so they're not in this sort of prisoner's dilemma where if I don't do it, the other guy will.”

“You know, one potentially positive vision is, you could have the AI industry be something like the pharmaceutical industry, which moves very conservatively and, like in my opinion, often too conservatively. and the incumbent pharmaceutical companies are, are pretty positive on this regulatory regime that imposes so much conservatism because that creates like a moat.”

“Governments must now require companies to demonstrate that they can keep their current systems fully under control before they train a new, more powerful and dangerous one.”

“There's also competitive pressure. If one company slows down or country, then others won't.”

“If the government won't act, the companies cannot wash their hands of responsibility. If they can't make a convincing case to impartial external experts that the next training run or internal deployment will be safe, they must delay it.”

“the company is biased and shouldn't be trusted to make all these judgments appropriately because of their incentives. And then the government auditor, well, they might just be limited. Even if they're trying their best, there might not be that many of them. They might have limited experience. They might be like busy, stretched between monitoring different companies and so”

“I agree with you. That market forces play a huge role here. Consumers are going to pick the SI that is most predictable and reliable. This is a point Mark Zuckerberg has made. So I do think market forces play a role here.”

“I completely agree that safety is paramount. I completely believe safety is paramount. I completely believe companies ought to ship safe products. I believe that CEOs and leaders of companies and the board of directors of companies have the responsibility and should have the courage to do the right thing.”