LLMs will eventually, some day in the future, be as smart as humanity was at some point in the past.
As of the summer of 2024, the pace of Large Language Model development has slowed. The concept of the eventual emergence of artificial general intelligence is waning in both its scale and popularity. Which means that it’s probably just about time for AI’s “killer app” to emerge.
LLMs are mostly a tool, by now. People smarter than I are starting to apply these tools to actual problems they might be good for, so for the rest of us, here’s a quick break-down of how these things work.
LLMs are born when fed a huge set of text data. For the most part, the data sets are big outdated portions of the internet with some of the garbage and useless nonsense scooped out. The “black box” is the analysis of the text which figures out the mathematical relationships between the words.
Afterwards, the models are “fine tuned” using sets of inputs in the form of a question and a known good answer. The “black box” is then told to use those question-answer pairs to refine its output to create answers that align with the known good questions and answers.
If the question is about something that doesn’t exist in the initial data sets, most of the big LLMs are able to acquire new text data. For example, if ChatGPT doesn’t know what you’re talking about, it’s allowed to use Bing Search to find new documents on the internet to ingest. This is called “Retrieval Augmented Generation” or RAG.
So that’s why I’m not scared, because LLMs will never (seemingly ever) know anything that someone hasn’t said before. LLMs will eventually, some day in the future, be as smart as humanity was at some point in the past. Even if LLMs are ingesting text generated by LLMs, if you dig deeply enough into the trail of documents parsed by any given LLM eventually you’ll find my angsty LiveJournal posts from 2001, or an essay about Roy Orbison completely wrapped in cling film.
And for what it’s worth, it’s only gonna take a few more of these before the industry backs away from these tools significantly.