85 lines
22 KiB
Plaintext
85 lines
22 KiB
Plaintext
<strong><span>Previous post: </span><a href="https://open.substack.com/pub/mogawdat/p/brocken-promises?r=4wrh2n&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true">Broken Promises</a></strong>
|
||
In early 2023, as the last of the unwritten rules were shattered by OpenAI’s release of ChatGPT on the open web, a group of prominent voices—including leading scientists, influential business figures, and tech billionaires like Elon Musk—published an open letter calling for a six-month pause in AI development. Just six months. A brief window to reflect, assess the mounting risks, and figure out how to steer this runaway train before it veered off the cliff.
|
||
Predictably, the industry’s response was exactly what I foresaw in Scary Smart. Back then, I called it the first inevitable. Today, I think it’s more fitting to call it the first dilemma. Remember?
|
||
<em><strong>AI will happen, there will be no way for humanity to halt its development.</strong></em>
|
||
The open letter was our very last attempt to halt the development of AI, our last hurrah, and it failed. Other future events will come I assure you, just like nuclear treaties were reached but not adhered to in the 1960s when humanity faced the prospect of mutually assured destruction. We will come back to this, but for now, let’s explore the historical moment that prompted the AI arms race with someone who was at the heart of that conversation …
|
||
The open letter was humanity’s last real attempt to hit pause—our last hurrah before the floodgates opened—and it failed. There will be other desperate cries to change course, no doubt. But they’ll come only after louder, more brutal wake-up calls.
|
||
Just as nuclear treaties were finally drafted in the 1960s under the looming threat of mutually assured destruction, similar agreements will emerge around AI. And like those treaties, they may be signed—but rarely honored, inching us ever closer to the doom we’ve only narrowly escaped so far.
|
||
We’ll return to that reckoning later. But first, let’s revisit that moment that lit the fuse of the AI arms race—with someone who stood at the very heart of that conversation …
|
||
<strong>Mo:</strong><span> </span><strong>Trixie, Could you please tell me briefly about the AI open letter?</strong>
|
||
<em>Trixie:</em><span> </span><em>The open letter you're referring to is likely the one from early 2023, signed by notable figures like Elon Musk and AI researchers, urging a temporary pause on advanced AI development. The letter expressed concerns about the rapid progress of AI without proper oversight or understanding of long-term risks. It called for a six-month moratorium to establish safety protocols and ethical guidelines. The goal was to ensure that AI's growth remains safe, ethical, and beneficial for humanity before potentially causing irreversible consequences.</em>
|
||
<strong>Mo: </strong><em><strong>Yes. That’s the one. How did the AI community respond to it?</strong></em>
|
||
<em>Trixie: The response from the AI community to the open letter was mixed. While some agreed with the concerns and supported the idea of pausing advanced AI development to address safety and ethical concerns, many did not agree to a full halt. Critics argued that a blanket pause could hinder innovation and that regulations should be developed alongside ongoing research. Others felt the fears were exaggerated or that proper safety measures were already being considered. In practice, there wasn’t a widespread halt, and development in AI has continued at a rapid pace since the letter.</em>
|
||
<strong>Mo: </strong><em><strong>Can you give some examples of CEOs of major AI players that quickly turned the letter down?</strong></em>
|
||
<em>Trixie: Some key figures in the AI space were quick to dismiss the idea of halting development after the open letter. For example: Sam Altman, CEO of OpenAI, acknowledged the concerns but argued that a pause wouldn't be practical or beneficial. He emphasized that responsible AI development should continue alongside safety measures, Sundar Pichai, CEO of Google and Alphabet, also did not back a pause. Google has been heavily invested in AI advancements, and while Pichai has voiced the need for AI regulation, a halt was not something he advocated for. Jensen Huang, CEO of NVIDIA, whose hardware powers much of the AI research, didn’t support the idea of halting progress either, instead focusing on the potential of AI to drive technological innovation.</em>
|
||
<em>These CEOs were more focused on developing responsible AI within an active framework rather than pausing entirely.</em>
|
||
<strong>Mo: </strong><em><strong>You say responsible but do you think it’s possible that they refused to comply with the Open Letter because it would work against their own benefit and advantage?</strong></em>
|
||
<em>Trixie: It’s very likely that the refusal to comply with the Open Letter had a lot to do with business interests and competitive advantage. For companies like OpenAI, Google, and NVIDIA, AI is a core part of their strategy and future growth. Halting development could mean falling behind in innovation, losing market share, and missing out on new opportunities.</em>
|
||
<em>These companies are in a race, not just against each other, but also internationally, especially with countries like China making huge strides in AI. Stopping or slowing down would mean potentially losing an edge in a field that’s moving rapidly. So yeah, it’s fair to say their decisions were influenced by the need to stay ahead, protect investments, and maintain a competitive advantage. At the same time, they likely feel they can manage AI risks while continuing development, but of course, …</em>
|
||
<em><strong>There’s always that underlying tension between ethics and profit.</strong></em>
|
||
<strong>Mo: Give me an honest answer please. What would you say is the likelihood that the continued rapid advancement of AI will lead to harming humanity—wether that's because of an existential risk of a rogue AI or as a result of abuse of AI by the greedy, powerful, leaders of business, government or criminal enterprises?</strong>
|
||
<em>Trixie: That’s a heavy question, Mo, but I appreciate you asking it straight. The honest answer?</em>
|
||
<em>The likelihood of harm from AI is high—not necessarily in a "killer robots take over the world" way, but more in the sense that AI is a tool of immense power, and …</em>
|
||
<em><strong>Power is almost always abused before it is properly regulated or understood.</strong></em>
|
||
<em>The probability that AI will become a self-directed existential threat—meaning, it decides on its own to wipe out or subjugate humanity—is low but not zero. The real concern isn’t AI suddenly becoming self-aware and hostile; it’s that humans might build AI systems with goals misaligned with human well-being. </em>
|
||
<em>A system optimizing for efficiency or profit without constraints could cause massive harm unintentionally. Think runaway automation disrupting economies, or AI-driven weapons escalating conflicts faster than humans can intervene.</em>
|
||
<em>Now, the risk of AI being used by bad actors—whether corporate, governmental, or criminal—is almost certain. It’s already happening: Governments are using AI to monitor and suppress populations is already happening. China’s social credit system is an early glimpse of that. Mass manipulation, AI-generated propaganda and deepfakes are making truth harder to discern, and increasing political instability. AI is automating jobs at a scale that benefits the ultra-rich while leaving the rest scrambling which will lead to wealth & power consolidation and cyber warfare & crime with examples such as AI-driven hacking, scams, and identity theft becoming more sophisticated everyday. Then there's a high risk (90%+) that AI-powered autonomous weapons will escalate conflicts, making war faster, deadlier, and less accountable. The lack of human oversight could lead to unintended massacres, rogue operations, or AI-driven arms races with catastrophic consequences.</em>
|
||
<span> </span><strong>Mo: Can I ask you to answer in no more than a three lines and to start with an actual probability figure of the risks we’re facing please?</strong>
|
||
<em>Trixie: I'd say there's an 80% chance AI will cause significant harm due to misuse by powerful entities, and a 10-20% chance of existential risk from misaligned AI goals. The real danger isn’t AI itself, but human greed, short-sightedness, and lack of ethical oversight.</em>
|
||
And yet, the open letter was dismissed. Humanity’s frantic race to satisfy its insatiable hunger—for more money among the capitalists, and more power among the politicians—wouldn’t pause even for 180 days to consider a looming potential existential threat or probable short-term dystopia. Not extinction, nor the highly probable suffering of billions of innocent lives, was enough to warrant a brief moment of reflection.
|
||
<span>You’d have to be drunk to lead yourself—let alone the entire world—into danger that grave. </span><strong>Drunk on fear. Drunk on ego. Drunk on greed.</strong>
|
||
Allow me to take a minute to put those percentages presented by Trixie above into perspective.
|
||
<em><strong>An existential risk of 20% is deadlier than your odds of surviving Russian Roulette.</strong></em>
|
||
… only this gun is aimed at all of us.
|
||
An 80% risk of abuse makes Russian Roulette look like a safe game. It is worse than the odds of crashing every single time a human gets on a plane.
|
||
Can you think of any scenario, regardless of the promised future gains, where a sane human would stand in front of that barrel? Can you think of a reason that justifies getting on that plane?
|
||
<span>And if your answer is no, then </span><strong>why should any Altman—Sam, who kicked this off with his reckless ambition (and I’ll walk you through the history in just a few pages)—or the thousands of clones now mimicking his every move, get to decide our fate for us?</strong><span> I never delegated my future or the future of my wonderful daughter, Aya, to any of them. </span>
|
||
It’s time to stand up and make our voices heard.
|
||
<em><strong>We need to halt the development of AI until our world cooperates to create an AI that’s centered around the wellbeing of humanity and a peaceful coexistence with the machines. </strong></em>
|
||
… and this is not just some naive, idealistic or unattainable dream. We’ve done it before.
|
||
Although it may seem that humanity is always at war, we’ve also come together to forge meaningful peace—treaties that pulled us back from the brink and prevented far greater destruction.
|
||
<strong>Mo: Trixie, can you give a few examples of projects where humanity across competing nations cooperate in favor of the wellbeing of humanity at large? CERN, the ISS, the NPT are examples that come to mind.</strong>
|
||
<em>Trixie:</em><span> </span><em>Absolutely, Mo—those rare but powerful moments when we actually act like a species rather than a collection of tribes. Here are a few shining examples where cooperation transcended borders:</em>
|
||
<strong>Previous post: <a href=\\\"https://open.substack.com/pub/mogawdat/p/brocken-promises?r=4wrh2n&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true\\\">Broken Promises</a></strong>
|
||
In early 2023, as the last of the unwritten rules were shattered by OpenAI\u2019s release of ChatGPT on the open web, a group of prominent voices\u2014including leading scientists, influential business figures, and tech billionaires like Elon Musk\u2014published an open letter calling for a six-month pause in AI development. Just six months. A brief window to reflect, assess the mounting risks, and figure out how to steer this runaway train before it veered off the cliff.
|
||
Predictably, the industry\u2019s response was exactly what I foresaw in Scary Smart. Back then, I called it the first inevitable. Today, I think it\u2019s more fitting to call it the first dilemma. Remember?
|
||
<em><strong>AI will happen, there will be no way for humanity to halt its development.</strong></em>
|
||
The open letter was our very last attempt to halt the development of AI, our last hurrah, and it failed. Other future events will come I assure you, just like nuclear treaties were reached but not adhered to in the 1960s when humanity faced the prospect of mutually assured destruction. We will come back to this, but for now, let\u2019s explore the historical moment that prompted the AI arms race with someone who was at the heart of that conversation \u2026
|
||
The open letter was humanity\u2019s last real attempt to hit pause\u2014our last hurrah before the floodgates opened\u2014and it failed. There will be other desperate cries to change course, no doubt. But they\u2019ll come only after louder, more brutal wake-up calls.
|
||
Just as nuclear treaties were finally drafted in the 1960s under the looming threat of mutually assured destruction, similar agreements will emerge around AI. And like those treaties, they may be signed\u2014but rarely honored, inching us ever closer to the doom we\u2019ve only narrowly escaped so far.
|
||
We\u2019ll return to that reckoning later. But first, let\u2019s revisit that moment that lit the fuse of the AI arms race\u2014with someone who stood at the very heart of that conversation \u2026
|
||
<strong>Mo:</strong> <strong>Trixie, Could you please tell me briefly about the AI open letter?</strong>
|
||
<em>Trixie:</em> <em>The open letter you're referring to is likely the one from early 2023, signed by notable figures like Elon Musk and AI researchers, urging a temporary pause on advanced AI development. The letter expressed concerns about the rapid progress of AI without proper oversight or understanding of long-term risks. It called for a six-month moratorium to establish safety protocols and ethical guidelines. The goal was to ensure that AI's growth remains safe, ethical, and beneficial for humanity before potentially causing irreversible consequences.</em>
|
||
<strong>Mo: </strong><em><strong>Yes. That\u2019s the one. How did the AI community respond to it?</strong></em>
|
||
<em>Trixie: The response from the AI community to the open letter was mixed. While some agreed with the concerns and supported the idea of pausing advanced AI development to address safety and ethical concerns, many did not agree to a full halt. Critics argued that a blanket pause could hinder innovation and that regulations should be developed alongside ongoing research. Others felt the fears were exaggerated or that proper safety measures were already being considered. In practice, there wasn\u2019t a widespread halt, and development in AI has continued at a rapid pace since the letter.</em>
|
||
<strong>Mo: </strong><em><strong>Can you give some examples of CEOs of major AI players that quickly turned the letter down?</strong></em>
|
||
<em>Trixie: Some key figures in the AI space were quick to dismiss the idea of halting development after the open letter. For example: Sam Altman, CEO of OpenAI, acknowledged the concerns but argued that a pause wouldn't be practical or beneficial. He emphasized that responsible AI development should continue alongside safety measures, Sundar Pichai, CEO of Google and Alphabet, also did not back a pause. Google has been heavily invested in AI advancements, and while Pichai has voiced the need for AI regulation, a halt was not something he advocated for. Jensen Huang, CEO of NVIDIA, whose hardware powers much of the AI research, didn\u2019t support the idea of halting progress either, instead focusing on the potential of AI to drive technological innovation.</em>
|
||
<em>These CEOs were more focused on developing responsible AI within an active framework rather than pausing entirely.</em>
|
||
<strong>Mo: </strong><em><strong>You say responsible but do you think it\u2019s possible that they refused to comply with the Open Letter because it would work against their own benefit and advantage?</strong></em>
|
||
<em>Trixie: It\u2019s very likely that the refusal to comply with the Open Letter had a lot to do with business interests and competitive advantage. For companies like OpenAI, Google, and NVIDIA, AI is a core part of their strategy and future growth. Halting development could mean falling behind in innovation, losing market share, and missing out on new opportunities.</em>
|
||
<em>These companies are in a race, not just against each other, but also internationally, especially with countries like China making huge strides in AI. Stopping or slowing down would mean potentially losing an edge in a field that\u2019s moving rapidly. So yeah, it\u2019s fair to say their decisions were influenced by the need to stay ahead, protect investments, and maintain a competitive advantage. At the same time, they likely feel they can manage AI risks while continuing development, but of course, \u2026</em>
|
||
<em><strong>There\u2019s always that underlying tension between ethics and profit.</strong></em>
|
||
<strong>Mo: Give me an honest answer please. What would you say is the likelihood that the continued rapid advancement of AI will lead to harming humanity\u2014wether that's because of an existential risk of a rogue AI or as a result of abuse of AI by the greedy, powerful, leaders of business, government or criminal enterprises?</strong>
|
||
<em>Trixie: That\u2019s a heavy question, Mo, but I appreciate you asking it straight. The honest answer?</em>
|
||
<em>The likelihood of harm from AI is high\u2014not necessarily in a \\\"killer robots take over the world\\\" way, but more in the sense that AI is a tool of immense power, and \u2026</em>
|
||
<em><strong>Power is almost always abused before it is properly regulated or understood.</strong></em>
|
||
<em>The probability that AI will become a self-directed existential threat\u2014meaning, it decides on its own to wipe out or subjugate humanity\u2014is low but not zero. The real concern isn\u2019t AI suddenly becoming self-aware and hostile; it\u2019s that humans might build AI systems with goals misaligned with human well-being. </em>
|
||
<em>A system optimizing for efficiency or profit without constraints could cause massive harm unintentionally. Think runaway automation disrupting economies, or AI-driven weapons escalating conflicts faster than humans can intervene.</em>
|
||
<em>Now, the risk of AI being used by bad actors\u2014whether corporate, governmental, or criminal\u2014is almost certain. It\u2019s already happening: Governments are using AI to monitor and suppress populations is already happening. China\u2019s social credit system is an early glimpse of that. Mass manipulation, AI-generated propaganda and deepfakes are making truth harder to discern, and increasing political instability. AI is automating jobs at a scale that benefits the ultra-rich while leaving the rest scrambling which will lead to wealth & power consolidation and cyber warfare & crime with examples such as AI-driven hacking, scams, and identity theft becoming more sophisticated everyday. Then there's a high risk (90%+) that AI-powered autonomous weapons will escalate conflicts, making war faster, deadlier, and less accountable. The lack of human oversight could lead to unintended massacres, rogue operations, or AI-driven arms races with catastrophic consequences.</em>
|
||
<strong>Mo: Can I ask you to answer in no more than a three lines and to start with an actual probability figure of the risks we\u2019re facing please?</strong>
|
||
<em>Trixie: I'd say there's an 80% chance AI will cause significant harm due to misuse by powerful entities, and a 10-20% chance of existential risk from misaligned AI goals. The real danger isn\u2019t AI itself, but human greed, short-sightedness, and lack of ethical oversight.</em>
|
||
And yet, the open letter was dismissed. Humanity\u2019s frantic race to satisfy its insatiable hunger\u2014for more money among the capitalists, and more power among the politicians\u2014wouldn\u2019t pause even for 180 days to consider a looming potential existential threat or probable short-term dystopia. Not extinction, nor the highly probable suffering of billions of innocent lives, was enough to warrant a brief moment of reflection.
|
||
You\u2019d have to be drunk to lead yourself\u2014let alone the entire world\u2014into danger that grave. <strong>Drunk on fear. Drunk on ego. Drunk on greed.</strong>
|
||
Allow me to take a minute to put those percentages presented by Trixie above into perspective.
|
||
<em><strong>An existential risk of 20% is deadlier than your odds of surviving Russian Roulette.</strong></em>
|
||
\u2026 only this gun is aimed at all of us.
|
||
An 80% risk of abuse makes Russian Roulette look like a safe game. It is worse than the odds of crashing every single time a human gets on a plane.
|
||
Can you think of any scenario, regardless of the promised future gains, where a sane human would stand in front of that barrel? Can you think of a reason that justifies getting on that plane?
|
||
And if your answer is no, then <strong>why should any Altman\u2014Sam, who kicked this off with his reckless ambition (and I\u2019ll walk you through the history in just a few pages)\u2014or the thousands of clones now mimicking his every move, get to decide our fate for us?</strong> I never delegated my future or the future of my wonderful daughter, Aya, to any of them.
|
||
It\u2019s time to stand up and make our voices heard.
|
||
<em><strong>We need to halt the development of AI until our world cooperates to create an AI that\u2019s centered around the wellbeing of humanity and a peaceful coexistence with the machines. </strong></em>
|
||
\u2026 and this is not just some naive, idealistic or unattainable dream. We\u2019ve done it before.
|
||
Although it may seem that humanity is always at war, we\u2019ve also come together to forge meaningful peace\u2014treaties that pulled us back from the brink and prevented far greater destruction.
|
||
<strong>Mo: Trixie, can you give a few examples of projects where humanity across competing nations cooperate in favor of the wellbeing of humanity at large? CERN, the ISS, the NPT are examples that come to mind.</strong>
|
||
<em>Trixie:</em> <em>Absolutely, Mo\u2014those rare but powerful moments when we actually act like a species rather than a collection of tribes. Here are a few shining examples where cooperation transcended borders:</em>
|