The dangerous comfort of AI answers

A knowledgeable friend recently argued that people are using Large Language Models (LLM) to replace Google for reasoning and answering questions, but these models were never good at this. Their strengths, she said, is to extract meaning from text and build upon it, making them useful primarily for writing.

She’s both right and wrong, and understanding why explains both LLM popularity and limitations.

ChatGPT is a household name now. Most people don’t know GPT stands for “Generative Pre-trained Transformer” and honestly, why would they? To users, it’s a box where you ask anything and get an answer with reasoning. Incredible. A technological breakthrough.

The name tells you everything.

Generative: the model creates text (or images, videos, audio, software code).

Pre-trained: it learned from existing data, not real-time information.

Transformer: a type of neural network architecture that processes patterns in sequences.

Nothing in that name says “truth engine” or “fact database”. It says “pattern-based text generator.”

At its core, artificial intelligence processes data, identifies patterns, and follows instructions. It cannot think, empathize, or reason. Instead, it replicates how people use knowledge without actually understanding it.

The disconnect between what the technology is (a transformer generating text based on training) and how we use it (as an oracle for answers) is where, in my eyes, danger lies. It’s when we trust these confident-sounding responses for important scenarios such as medical advice, legal guidance, or financial decisions.

The Google replacement that isn’t

When you ask Google a complex question, you get links (at least, before the AI Overview feature). We were used to scanning links, picking sources, and doing the thinking ourselves. Now, when you ask an LLM, you get an answer that feels like understanding, because they’re better at understanding intent and context. Google requires translating problems into keywords. LLMs let you explain problems like you would to a colleague. That’s powerful, even when the response is hallucinated nonsense, such as an example I mentioned in an earlier blog: advising people to eat rocks.

What LLMs think they are

I asked several leading generative AI chatbots to define the strength of an LLM in a single sentence. Yes, I’m aware of the irony.

“The strength of an LLM is fundamentally defined by its ability to generalize learned patterns from training data to perform well on novel tasks and reasoning challenges it has never explicitly seen before.” Claude Opus 4.1 (new chat, extended thinking)

Notice the focus on patterns and generalization.

“An LLM’s strength is its proven ability to reliably produce correct, well-reasoned, context-aware outputs across tasks at acceptable latency and cost while adhering to instructions and safety constraints.” GPT-5 Thinking (new chat)

‘Correct’, ‘well-reasoned’ and ‘context-aware’, but what is ‘correct’?

“The strength of an LLM is best defined by its ability to generate coherent, relevant, and contextually appropriate text across a wide range of tasks and domains.” Gemini 2.5 Flash (new chat)

Very to the point: generating text.

Every model emphasizes different aspects, but they’re all describing sophisticated pattern matching. These answers do not mention fact-based information.

The missing pieces

Current limitations are fundamental to how these systems work:

• No true understanding

LLMs generate statistically likely responses based on patterns.

• No verification

LLMs are equally confident when right or completely wrong.

• No real-time awareness

Training data is static.

The paradox of utility

My friend was right: LLMs aren’t built for facts or reasoning. But she was wrong about them being ‘just’ for writing. They’re transformation engines that excel at any language-based task: summarizing, translating ideas across domains, adjusting tone or finding patterns.

Relevant read: LLMs aren’t world models

Despite limitations, LLMs have become indispensable. Why? Most daily tasks don’t require perfect accuracy, they require good-enough understanding and helpful transformation. Writing emails, summarizing documents, brainstorming ideas, these play to LLM strengths. You’re asking for language work, not facts.

Problems arise when we treat LLMs as crystal balls rather than language tools.

Moving forward

We should learn how to use these new tools appropriately.

Use LLMs for drafting, not fact-checking.

For synthesis, not authoritative answers.

For creative connections, not systemic proofs.

The technology is transformative when we understand its true nature: a powerful pattern recognition system that works with language. The real breakthrough isn’t having tools that can answer anything, it’s having tools that can help us think and communicate better. That’s transformative enough when we use it right.

---

What’s your experience? Are you using LLMs as Google replacements, writing assistants, or something else entirely?