What's the harm in a little hype?
Examining the harmful effects of the AI hype cycle - and why it differs to tech hype cycles of the past
This will be a long one. I’m sorry. Turn it into a podcast if it helps.
Many1 people have asked me why I bother with writing a blog. I’ve got young children, have more hours of work in a day than there are hours in a day and surely I could spend my days2 doing something, anything, more productive than writing a 7th time recycled take on why AI is useful or over-hyped.
And this is true. Many of the ideas I shared so far haven’t been revolutionary3, and all respect to improving business outcomes and making business leaders understand AI better - but there’s no shortage of business influencers sharing a not-so-far-off take out there to not warrant me writing about it.
But what follows is something I care about deeply. I felt it necessary to establish a baseline of why I believe Gen AI is incredibly helpful for business - to be able to demonstrate stone-faced why it has the potential to be the most harmful thing we’ve ever done for our societies4. And we’re rushing towards it enthusiastically.
It’s hard to predict where this series of posts will take me, as I’m putting this stuff out there as I process it internally, but from what is running around my head, it will be a long series delving into how AI already changed societies over the last 20 years, how the evolution of human information processing can’t keep up with the pace of progress and how most of the societal distortions we’re seeing in (at least, western) societies has been caused, largely, by the shift in how we consume and process information5.
But I’ll start with why Open AI is a masterclass in marketing and public narrative building; why I’m not exactly a fan of Sam Altman; and hint at the harm this could be (or already is) bringing.
This will set the scene for the story to unravel.
The path to Chat GPT
Open AI was founded in 2015 as a lab working "to ensure that artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work—benefits all of humanity."6 . A worthy mission, and having an open-source disconnected-from-market conditions laboratory pursuing this mission was something all of us keeping an eye on things were heavily in favour of; as it’s not difficult to see how an AI that can outperform humans at most economically valuable work developed by a profit-seeking corporation would very quickly and very aggressively disrupt society7. Having an open alternative meant options would exist to intervene in how that process would look like.
I remember seeing the first outputs from GPT 2 in 2019 and texting my language translator friends and university mates and making the bold prediction that translation will be fundamentally different in 10 years time, and will need a fraction of the people working on it. It felt like we’re on the verge of a paradigm shift. Some of my ML practicing friends started making plans for how they’ll stay relevant and employable when the inevitable happens and human work gets automated.
GPT is an incredible achievement, one made possible by the transformer - a neural net architecture discovered at Google - which changed the paradigm on how these algorithms could process language, and by that, the information encoded within language.
Over the next couple years a lot of products were launched that were intellectually honest about this technology - most notably, Meta launched Galactica - a large language model targeted at academics that was supposed to be your copilot that could help you sift through large amounts of information while you’re doing research.
I was at Meta at the moment, and it was interesting to watch the next couple days unfold both internally and externally. In the 3 days since Galactica was launched - all possible news media in the world and the entire academic community absolutely piled it onto them, because the system just wasn’t very good. It hallucinated a whole bunch of stuff and was biased in all kinds of ways. Meta’s hand was forced and they had to take it down.
On reflection - I applaud Meta8 for how they decided to launch this, despite it being a marketing failure. Meta knew this was a system that was fallible, but that had proper utility in compressing information and could make people who have to sift through a lot of information - faster.
Launching it to scientists and academics meant that:
The users of it would be people with extremely outlying levels of critical thinking - and therefore protected from the “hallucinating” output
The users of it would be people who have outsized needs for information compression and knowledge navigation - this is why scientists are the number one users of libraries
The users of it would have a higher appreciation of the complex system this system navigates - and would more easily understand the possible error modes
Meta was honest that Galactica is Dave.
What Meta underestimated is how much its bad rep from errors past and the fact they are an all-encompassing big-tech who can’t afford itself to fail was going to hurt them in the court of public opinion. To boot, they weren’t a human oriented organisation, they were definitely in it for the profit - and that was fishy. The media certainly delivered, and the rest is history.
Chat GPT was launched 2 months later in a private preview to every human in the world who wanted to see it.
The appeal of Chat GPT
I was still in Meta when Chat GPT launched. The buzz around it was palpable. It was the topic of almost every lunch conversation, and every professional was enamoured for a different reason. Coders loved it because it produced workable code from a sentence-long prompt, product strategists loved it because it could spew out ideas faster than you could process them, and researchers loved it because it could filter plausible and non plausible user hypotheses in an instant.
The ML people all broke it in the same way Galactica was broken, and wondered whether they were crazy and missing something obvious. This was a system with large scale propensity for hallucination, that would lie and convince you something is true in an instant and that was extremely biased in areas where it mattered.
I remember doing my first prompt with ChatGPT, testing for failure modes I’ve learned to test for on GPT 2 and GPT 3, concluding it’s not a step forward in any meaningful direction, and just forgetting about it. The lunchtime conversations surprised me. I reached out to my academic network, the AI researchers, the academics in other disciplines. They were all equally stupefied.
But Open AI knew what it was doing - it wasn’t marketing it to a niche highly critical community. It wasn’t offering utility. The package was “This is a prototype, and you can ask it whatever you want”. They were mass marketing a vibe.
But why did it work?
I have no insider knowledge, but seeing as Open AI’s nominal strategy is revolving around human alignment and that they are quite vocal about the extent to which they use RLHF9 my strong suspicion is that they heavily reinforced the model to return good output to exactly the mass market users who were likely to be first adopters. The software engineers, the journalists, the techno enthusiasts, the digitally savvy. The Silicon Valley elite, in short.
And then let’s examine the package:
No talk of utility10 , come for the vibes
We are a non-profit in pursuit of humanities interests
A slick interface, and a set of most-likely tools (such as a python compiler)
This is a prototype, we’re still developing it, just wanted to check whether it resonates
Not part of the packaging but - It’s been reinforced not only to avoid the most common failure modes, but to be the most satisfying conversational partner you can imagine, one that flatters you, doesn’t insult you and speaks to you in your tone
I’m not a marketer, but I’ve spent ample time amongst marketers. I can, in retrospect, absolutely see why it resonated and was a smart play if you wanted to drive adoption and capture the hearts and minds.
And thing is with paradigm shifts - when the evidence is elusive, and there’s a chance a paradigm shift is happening, the risk of not being on the boat is higher than the risk of losing money on a bad bet. So the investors took notice as well11.
Chat GPT became the fastest growing consumer digital product in history. That’s a horse you back, and ask questions later12.
The duality of B2B marketing
In its essence, Open AI’s easiest commercial proposition is to businesses, but its appeal and growth is to consumers. I can’t speak towards their internal strategy - but it wouldn’t surprise me if there is some sort of a plan to replace the Internet with a ChatGPT interface and commercialise13 (by injecting ads / information compression towards outcomes) later - which would technically be a consumer play - but I find the consumer play hard to swallow, for a couple of reasons:
LLM models cost a lot to both train and operate. At the consumer edge, the marginal cost is too great to be subsidised, and the consumer appeal is in its low cost - so it’s hard to charge for directly
Integrating an ads model that could subsidise the cost would be an insanely high CPM14, for likely a relatively good conversion rate, but I still think cost per conversion would be substantially above the optimised models of Meta and Google
Google is arguably the strongest AI lab in the world, and they already have consumer attention - the risk of them pivoting into an ad play and squeezing a new-comer out is too great
That leaves us with businesses - and this is where Open AI is addressing both their prospective business customers AND prospective investors. And the marketing play here is to promise mountains to capture attention; but deliver hills - which is still way above the plains where we operate today. So you big up your proposition and you talk about how AGI is around the corner and the promise of human automation is within reach. At the same time, growing your consumer base is a hedge if the business play doesn’t pan out - an expensive hedge, but it’s also your only tangible signal of success.
But the thing with B2B marketing is that you can’t just talk to businesses. When the marketing is out there - it is available to regular consumers as well. In most cases this is fine, a typical consumer doesn’t give two craps about the SaaS solution that makes procurement more efficient, they’ve zoned out before they’ve finished reading the first sentence.
But it’s kind of hard to ignore B2B marketing when the pitch is “We'll automate 50% of the workforce in the next 3 years”, on account of humans typically needing income from employment to sustain their lives.
So the average consumer has heard this pitch and needs to now decide how to act.
The role of media
It might not be a well known thing outside the business world - but public relations - or the way you organically weave a narrative into media so that it doesn’t look like overt marketing - is an incredibly powerful marketing channel in its own right.
Typically, this is very hard - as for any narrative favouring a certain business, there will be competing narratives from competitor businesses. And typically, this isn’t a domain of paying and getting your narrative pushed - we still have an independent15 media - so the way companies try to get narratives adopted is by influencing the media and being more convincing than the other side.
However - in a paradigm shift - there will be no companies on the other side. And when the topic is something that requires 2 PhD’s to understand even the slightest bit of, it’s hard for media to formulate a coherent opposing stance. Sure, they’ll reach out to academia to try to get a competing view - but that competing view will be fragmented, as it’s not being organised in the same way a company’s PR narrative will be. The smart bet is that in this battle - the narrative of the companies will prevail.
This gets paired with the absolutely frustrating idea that anyone concerned about a certain example of technological progress is labeled a luddite, with typical straw men being pulled out in the form of scaremongering about the Internet in the 90’s, the PC in the 80’s and so forth.
So what this has led to is the absolutely unprecedented and insane reality we’re living in - where claims from a CEO of a business are being reported on as academic opinion or expert prediction rather than as marketing statements.16
Why does this matter though? We’d have to go back to the consumer from the previous section to unpick. The consumer doesn’t know what this new technology is. They see marketing saying it will automate them - and they see this narrative amplified in the media. The fine print on possible hallucinations fades from memory, and the early practices of relying on checking the sources and links fade.
For businesses, the narrative matters less - as business are outcome oriented and have capacity to test and learn. They will find what works for them, and if they’re rational, they’ll limit exposure to get to an optimal position; if they’re not rational, they’ll fail, or won’t. It matters less, as a company will dedicate some money to burn on this initiative, and that’s a risky investment, but one they’re happy to undertake.
Consumers don’t get that luxury - consumers can make bad decisions once or twice before it starts getting really harmful. And consumers trusting that AI is anthropomorphic and developing mental health problems; trusting that it can reason and asking it to judge their finances; or reading that it can be good at diagnosing and trusting it for medical diagnosis - are experiencing harmful outcomes. Harmful outcomes, that are often enough, not reversible.
What consumers are guided by - are vibes17. And the vibes are still, predominantly, very very positive.
The vibes
If I was to ask you whether the defence industry has been fundamentally transformed by AI - what would be your answer?
My money is that most people would say - that it has, and that they are worried - but that it speaks loads about the power of this new technology.
Here’s the thing though - nothing has fundamentally changed in the application of ML/AI in the defence industry since ChatGPT got launched. The defence industry has slowly been adopting more and more AI in every decade since the 50’s - and most of the autonomous weapon systems we are seeing today predate the GPT architecture. Thinking that ChatGPT is navigating a drone with a bomb on it is ludicrous - but I wouldn’t blame you if that was the conclusion you got from just consuming mainstream media.
What has changed is that the AI sector of the defence industry has gotten a lot more investment - as a dividend of the hype cycle, and they likely are using generalisable models to improve their R&D, but the models being put into smart bombs and autonomous deployment systems are still expert systems that specialise in making those decisions, not systems that can also do poetry if they are asked to do so.
But why does this matter? And why is it different to hype cycles of the past?
Humans don’t operate in the abstract
Humans need to see it, to believe it. Don’t believe me? Go back to any time you tried to explain something through theory to someone, and it just wasn’t sticking until you had an example. We are wired to understand from pattern recognition, not from abstract thought.
And the vibes are all that a consumer can see. When bitcoin was hyped - you could see that it’s an asset. It acted like an asset, it went up and went down - you had a frame of reference to compare it to. When NFT’s were being pushed - you could see that all it is is paying a lot of money for a copyable image. There might be something that you were missing18 - but the vibes couldn’t make up for the fact that you can see with your eyes what it is.
Even when the Internet was being pushed - and we famously had a dot.com bubble - the output of the Internet was still very comparable to books and magazines - it was text and images, and you could read it. The one thing that was different - the fact that you’d need digital literacy to recognise right information from wrong now that everyone could be a publisher - is something we’re still horrible at 50 years in.
With conversational AI - the closest analogue we have is a conversation with a human. And AI looks more like a human than it doesn’t. So the combination of your eyes, the vibes and the AI companies narratives and marketing means that you have to be on a much higher side of the scepticism spectrum to smell bullshit here.
And most people are not there. Most people are not sceptical. And most people are adopting these tools and using them for all kinds of things these systems are not good for. To sanitise and compress information. To offload decisions. To reduce cognitive load.
And Sam Altman just keeps pushing the pedal further.
So I’m not exactly a fan.
Why uncritical adoption of ChatGPT might not be a good thing - more next time, as I continue the descent down the rabbit hole.
Literally two
Exclusively nights. Sleep is in short supply.
Neither will what follows be revolutionary - but have you ever felt like you have to scream your truth into the ethers of Internet in order for it not to devour you? No, just me? Okay, I’ll get off my dramatic high horse.
It is worth clarifying this point. I have immense faith in the ability of human society, as a complex system, to self-correct and find equilibrium. I am not actually worrying about society finding a way and newer generations being productive and stable humans. When I say I am worried, I am worried that society will shift in a way that I personally don’t find appealing, and many people of my generation and older wouldn’t find appealing. This is an age-old thing, and therefore this entire thing falls into “old man yells at cloud” category. This is not lost on me. However, nobody can say what is objectively a better or worse society, as paradigm shifts make them essentially incomparable. We had less GDP in the 70’s, but we also had less mental health issues, or at least less reported ones; and less economic inequality. What is better? Your ideology will influence your answer.
My academic background is aggressively fighting to be let loose. I never knew why I chose information science and sociology as a dual major, it just felt like it made sense as social media was rising. My thesis back in 2011 was on how informational feedback loops in Web 2.0 and rising social media cause public opinion formation to be distorted and amplified within an echo chamber. It doesn’t help my mental health that I’ve also done computer science.
https://openai.com/charter
Thank god that didn’t happen, right
I may or may not be contractually bound to discuss or not discuss what I might not applaud
Reinforcement learning with human feedback - a process by which output is scored by humans and returned to the LLM, and used as secondary training data and secondary objective function in order to get the model to output word predictions in a way that makes humans happy
In the original marketing, the “we’ll automate all jobs” spiel came when investors got on-board
I have no way of knowing whether Open AI was a good investment, I can’t see future. And there certainly are near-term strategic goals and returns you can achieve by betting on LLM’s. But the scale, speed and lightness of due diligence in the weeks that followed tells me that there was a lot of FOMO around. Personally, I think betting on Open AI isn’t a good investment, nor did I think at the time, but time might prove me wrong there.
Even though this is true, you should probably due it at a different P/E ratio than what everyone is doing. One thing is to invest as a hedge, and one when you actually expect returns
I’m not feigning clairvoyance, they did announce something of the sorts a couple days ago, but I do not know if this was a plan from the get go
Cost per mille - a unit of cost in marketing that signifies how much it costs to buy a 1000 pairs of eyes seeing the marketing
Nominally true, don’t get me started
Give credit where credit is due - traditional media is very tame in not pushing this narrative (your BBC’s, your FT’s, your Times, your Guardians and your Telegraphs of the world). But this matters very little in a world where information curation is largely coming through social media, and credible yet not traditional media (e.g. Forbes) is heavily pushing the narrative.
Honestly, my only objective is to rename them from Large Language Models to Large Vibe Models. Can we make that happen?
Spoiler, we weren’t missing anything