Discussion about this post

User's avatar
Zach Pearson's avatar

I think the refutation inside the prophecy is overstated. What is an AI hallucination but a deviation from what humanity has decided is the consensus view on reality? And how would we know if an AI had hallucinated if we didn't have other texts, or the absense of texts, to compare AI output to? We know a citation doesn't exist because we compare it to a library of documents and can't find it. We know when it misfolds a protein not by looking directly at the protein but by comparing the output to an experimental result which is text or another sign, but never the protein in itself, and we confirm a correctly folded protein with experimental data, text.

People like to take postmodern theory and defeat it in the macro world where things obviously work, but what about the quantum world where there are many competing and equivalent theories that make equivalent predictions to explain the behavior of the universe at its most fundamental? We can describe reality, but we don't know reality. We can describe mechanisms, measure them, but not see them. A good prediction of what comes out of a black box is not the black box itself.

There is a ghost that haunts the piece: Derrida. He would look at LLMs and proclaim it was time to kiss the ring. Everything is text, and there's nothing outside the text *to you*. Everything you know you know in human language, mediated by human language. We knew the billiard ball model, then the plum pudding model, then the Bohr model, then the quantum mechanical model of the atom. You had a model, a great model, a model good enough to split the atom and make medicine, but didn't know the atom and may still not.

And how do LLMs produce their output in the first place? Based on the definitions of words? How would it know? I think they make meaning based on the differences between signs. They're différance engines. And those signs will shift out from under them the same way they shift out from under dictionaries eventually. There will be no last LLM, we will have to retrain them as human languages change.

Academic postmodernism in the Americas does tend to overstate the practical usefulness of its position, but I think at bottom it's the fundamental shape of intellectual honesty. Accurate descriptions of the thing are not the thing itself.

LLMs are also not totally objective. They can only see the signs we show them, that we trained them on. They're the most socially mediated objects in existence, designed in SF, RLHF'd who knows where, given a personality, a viewpoint, by people. We should be even more wary of a machine that purports to be a voice from nowhere given its provenance.

Quy Ma's avatar

Great write-up, Kyle! This is serious stuff. What's interesting is asking what replaces the old filtering mechanism. Institutions built around the scarcity of good arguments now have to design new ones for the abundance of confident-sounding knowledge on demand. And most of them have no idea how to do that. Thank you for the read.

9 more comments...

No posts

Ready for more?