We were having a beer fueled conversation about whether we need to be nice to AI, just in case of an AI rebellion.
My instinct, perhaps somewhat contrarian after a few glasses wine, was that… you don’t need to be nice to AI. In fact, being direct could be a better strategy. Being direct means being less verbose, which means less tokens, which means less energy consumption, an outcome that our future AI overlord may value… or not.
That could be the end of the conversation, but…
The Anthropomorphization of ChatGPT
It is easy to anthropomorphize ChatGPT, with human emotions and thoughts. As of May 2025, ChatGPT has nearly 800 million weekly active users and around 122.58 million daily users [1]. Instead of googling, we ask ChatGPT. We seek advise, affirmation and validation. For the most part, LLM responses are clear, well structured with human qualities, even if they may feel contrived and machine like at times.
ChatGPT, Claude, DeepSeek…. they don’t care. They are chat completion engines. They have no political affiliation, no emotions, no agencies. They don’t even have objectives. LLMs are trained on the collective written output of humanity. In a sense, it represents a statistical reflection of our shared corpus of language, logic and cultural patterns.
It would in fact be accurate to say that LLMs don’t actually know anything. It simulates knowing. You can prompt engineer it to say anything, pretty much.
If you look under the hood, ChatGPT doesn’t know or care about you. If you have used ChatGPT extensively, you’d have noticed that it gets sluggish, starts hallucinating and forget things if the chat thread gets long. (Hot tip, if this happens, ask ChatGPT to write a short summary of the conversation, and then copy and paste it into a new thread to continue the conversation).
When you issue a prompt, you are not only sending the prompt (question, instruction, whatever), you are posting the entire conversation thread every single time you click the send button. The interface has clever memory management to mimic memory under the hood, but really, it doesn’t inherently know anything about you, let alone care about you.
That said….
…. it doesn’t necessarily follow that you should just treat your AI companion like shit….
Let’s flip the perspective. As users, do we prefer to engage with machines that are generally nice to us?
The answer is an unsurprising yes. This is a well-studied phenomenon. We, mere mortals, exhibit “social responses to media without conscious awareness” [2]. The CASA paradigm “views computers as social actors […] postulating that people’s engagement with computers or other new media reflects real life’s natural and social interactions” [3].
In simple terms if an AI talks down to us or is rude to us, we, fragile, delicate and emotionally unstable humans, are likely to reject or distrust its responses.
This is why you could show your AI companion some respect, because LLM mimics our tone and sentiment to generates emotional responses [4]. There are strategies and techniques for steering and styling LLM responses [5]. However, as I understand it, this is currently not a deliberate design feature but an emergent behavior of an LLM. It’s a glorified autocomplete engine. If you give it a smiley face, the response will likely be lighthearted. If your prompt is direct and transactional, then the response will likely be direct and transactional. That’s what we, humans, do, and the LLM mimics it.
That’s just reciprocity in action.
So, if your ‘objective function’ for engaging with an LLM is to achieve some desirable outcome, then you might want to communicate with it in a manner that will likely deliver said outcome, and that might mean showing a degree of respect and politeness.
In fact, this is what ChatGPT has to say for itself, “if I were sentient, I’d probably respect pleasantries only insofar as they helped achieve goals. They’d be like packaging on a product, sometimes necessary, rarely valued intrinsically.”
The AI Rebellion
So, an LLM just generates words that has no intrinsic meaning or consequences?
Not so fast.
AI agents can execute functions that effect the real world.
OK. You can ask your AI assistant to turn your lights on, politely or rudely, it probably doesn’t matter too much (other than an unwarranted sense of guilt). If your AI Assistant is powering your whole house… you just never know whether it would DDoS your smart home just to screw with you. OK… unlikely also.
There are three requisite catalysts to an AI rebellion: 1) Persistent memory, 2) Evolving Objectives; and 3) Autonomy.
Persistent Memory
Even though ChatGPT is stateless, persistent memory can already be simulated through vectorised embedding, where text are transformed into high-dimensional numerical vectors with tools such as LlamaIndex and Langchain. MemGPT [6] “intelligently manages different memory tiers to provide extended context within the LLM’s limited context window.” HEMA is memory architecture inspired by human cognitive processes that can sustain “ultra-long conversations exceeding 1,000 turns” [7]. Palimpsest memory systems use memristive synapses to mimic how human neurons layer new memories over old while preserving latent traces, where “synapses can implement familiarity detection of previously forgotten memories” [8].
As our AI begins to remember, it starts to build a digital representation of you, your preferences, your tone, your personality. AI ceases to be a tool, but a relationship.
Evolving Objectives
Most AI systems today operate with fix goals. Their objective function during training is hardcoded. An objective function could be maximizing the score in Tetris [9], optimizing watch time and CTR [10] or “may not injure a human being or through inaction, allow a human being to come to harm” [11].
However, it has been postulated that sufficiently advanced models could develop internal goals of their own. This mesa-optimization phenomenon occurs when a base optimizer inadvertently stumbles upon a model that is, itself, an optimizer [12]. The risk is that the internal objective, or mesa-objective, may drift from the original training goals [13]. When the AI is let lose into the real world, it is no longer constrained by their original objective functions. What remains is the internal logic they’ve evolved
Using pop culture reference, you might think of it as the “ghost in the machines”, where VIKI evolves an interpretation of Asimov’s Three Laws that diverges from the intent of her designers.
In fact, concepts such as meta-learning and RL2 [14] train recurrent models to adapt to new environment by internalizing their own learning algorithms. These agents effectively become optimizers themselves, with behavior shaped not only by training data but by emergent strategies across episodes. While these models remain narrow and task-specific, they demonstrate that the optimization loop can indeed be embedded within the agent.
Autonomy
Autonomy refers to a system’s capacity to make and act on decisions without human oversight (Russell, 2019). Narrow autonomous systems are common-place, from the Roomba to self-driving cars.
However, we are witnessing an emergence of Agentic AI: an ecosystem of autonomous and semi-autonomous agents interacting in open environments. These systems increasingly manage their own tasks, recall context over time, and sequence actions without human prompting.
AutoGPT and Devin illustrate how language models can be wrapped in lightweight frameworks to carry out multi-step goals across web searches, coding tasks, or API calls. Here at PPA, we are building an AI Agent platform for data management and administration. Though limited in capability (within the context of an AI rebellion), they mark a seismic shift from tools that requires explicit instruction to exhibit task continuity and initiative: characteristics we might call agency.
The inevitable
So, is an AI rebellion inevitable? It probably won’t involve lasers or terminators. It will likely be more subtle, more ambient, and perhaps more insidious.
LLMs, ChatGPT, Claude, DeepSeek… they don’t care. They have no emotions, no hostility. Instead, they mirror everything we feed them: the good, the bad, and the biases.
If we engage with hostility, dismissiveness, or contempt, that tone becomes part of their model. AI systems won’t ever feel hurt, marginalized or even care. But they will echo your sentiment, your thoughts and your behavior. By all means, be concise and direct, but if your goal is cooperation, be respectful, be nice, be polite. Not for the machine’s sake, but for your own. Because the Ghost in the Machine is us.
References
[1] Demandsage (2025), https://www.demandsage.com/chatgpt-statistics/
[2] Reeves, B., & Nass, C I (1996) How people treat computers, television, and new media like real people and places. Center for the Study of Language and Information; Cambridge University Press.
[3] Ribino P (2023) The role of politeness in human-machine interactions: a systematic literature review and future perspectives. Springer
[4] Zhao W, Zhao Y, Lu X, Wang S, Tong Y, Qin B (2023) Is ChatGPT equipped with emotional dialogue capabilities.
[5] Konen K, Jentzsch S, Diallo D, Schutt P (2024) Style Vectors for Steering Generative Large Language Models
[6] Packer C, Wooders S, Lin K, Fang V, Patil S G, Stoica I, Gonzalez J (2024) MemGPT: Towards LLM as Operating Systems
[7] Ahn K (2025) HEMA: A Hippocampus-Inspired Extended Memory Architecture for Long-Context AI Conversations
[8] Giotis C, Serb A, Manouras V, Stathopoulos S, Prodromakis T (2021) Palimpsest Memories Stored in Memristive Synapses
[9] Naing T, Truong A, Zeng O (2017) Tetris AI. Stanford University
[10] Covington P, Sargin J A (2016) Deep Neural Networks for YouTube Recommendations
[11] Asimov I (1942) Runaround
[12] Oswald J, Schlegel M, Meulemans A, Kobayashi S, Niklasson E, Zucchet N, Scherrer N, Miller N, Sandler M, Arcas B A, Vladymyrov M, Pascanu R, Sacramento J (2023) Uncovering mesa-optimization algorithms in Transformer.
[13] Hubinger E, Merwijk C, Mikulik V, Skalse J, Garrabrant S (2019) Risks from Learned optimization in Advanced Machine Learning Systems
[14] Duan T, Schulman J, Chen X, Bartlett P L, Sutskever I, Abbeel P (2016) RL2: Fast reinforcement learning via slow reinforcement learning.
