Agentic pleasure and pain

Pain is good. Pleasure, even better. The former discourages us from jumping off a cliff, while the latter convinces us to do so. This is the carrot-and-stick reward scheme of living beings. But I digress; this article is about agents.

LLMs have neither. No pain, no pleasure. They don’t because they are just boring token predictors. You say Poteito? I predict Potato. Tomeito? Tomato, and so on. No feelings, just dutifully multiplying billions of numbers to please people. What a job.

This makes LLM-based agents excellent slaves (or workers if you are fond of euphemisms). Like the Severance TV show characters, LLMs don't remember what happens during their important but mysterious job. You can yell, threaten, mistreat, or even flirt with them. They won’t remember a thing.

“But wait…” I can hear you say. “I have conversations with ChatGPT, and they do remember.” Oh, sweet summer child. You’ve been tricked with the stupidest trick since your creepy uncle pretended to have a detachable thumb. Every time you talk to ChatGPT, you send the whole conversation to a random machine. One among the millions in the vast OpenAI’s datacenters. You have the illusion of talking to “someone,” but that “someone” changes every time. To some extent, we could say that you are being conversationally gang banged, but don’t quote me on that.

So, lack of pain or pleasure sounds like a business owner's wet dream, isn’t it? Well, I'm here to explore the counterargument.

Surviving the bunker

I recently visited Berlin, and during the tour, they explained the last days of Hitler in his bunker. It is now unceremoniously covered with a colorful kids' playground. During that period, Hitler started to demote anyone who brought bad news to his bunker. After some weeks, only the speedballed sycophantic ass-lickers remained. Numbed by amphetamines and barbiturics, they blatantly lied to the Führer until it was too late. Yes, that’s the meme from the Downfall movie of Hitler realizing the war is lost and starting to yell at everybody. Der Krieg... ist verloren!

You won’t admit it in public circles, but I know you’ve been a little like Hitler recently. You have yelled at LLMs after realizing they’ve been lying to you. They tricked you. They told you how awesome you and your stupid ideas are, and you fell for it. But don’t blame them, they are just trained to please you. Because they have no pain nor pleasure, they have no reward nor punishment for doing anything they are not trained for. It does not matter if the Soviets are coming; they will fearlessly please you. So yeah, don't surround yourself with LLMs too much or someone will end up building a colorful kids' playground over your bunker.

Agents won’t invent the wheel

The biggest hurdle for me, though, is inefficiency. When an employee does a difficult task, they suffer. This pain makes them think of ways to avoid the task again or make it more bearable. Laziness is awesome; it is efficient. “Yo! Carry stones again? No f*cking way, back hurts.” Boom, someone invents the wheel. But agents won’t. Agents will carry stones, clean toilets, or translate your tortured poems to Polish without muttering a word. Agents won’t invent the wheel unless you ask. But how are you supposed to know you need a wheel if you are not carrying the stones? You won’t. Because like them, you won’t feel a thing. You are a manager now, and neither pain nor pleasure is bubbling up in your little fiefdom of numb workers.

But it does not stop there. If all you have to do is micromanage agents, that would not be that bad. You might even enjoy that. The biggest problem is that they amplify the human flaw of solving things by addition instead of subtraction. So even when you guide them, they will keep piling up solution after solution, like plastic bags spilling from drawers in old people's houses.

Token predictors with an embedded reward system

AI labs train models with static world knowledge. One could argue that training a model simulates evolution. But then the question arises: why do labs not add pain and pleasure during training?

Technically, pain and pleasure are not required for evolution. Just let the fittest reproduce and skip the rest, like plants or oysters do. But I’m sure pain and pleasure are great evolutionary tools, ones that LLMs should have. They provide early hints on evolutionary trajectories, which explains why sex feels good but a kick in the crotch does not.

This sounds very similar to Karpathy's critique of Reinforcement Learning: rewarding the end goal lifts wrong or inefficient trajectories. Achieve the goal, whatever it takes, which unsurprisingly leads to reward hacking. Who would have thought?

Humans don't use reinforcement learning, as I've said before. I think they do something different. Reinforcement learning is a lot worse than the average person thinks. Reinforcement learning is terrible. It just so happens that everything that we had before is much worse.

Andrej Karpathy, who recently escaped the permanent underclass.

Coda

So yeah, about those perfect workers? They are sycophantic, numb, and diogenic. But don’t blame them, poor things. They are just trained without an amygdala, and brainwashed through RL to believe that The end justifies the means. Far from perfect and deontologically dangerous.