10 Dangerous Misconceptions About Vibe Coding: Characteristics of Language Models / OSWARLD

"Our company doesn't need developers anymore. We just need ChatGPT!"

Have you ever heard something like this? Amidst the recent AI craze, many people are packaging LLMs (Large Language Models) as if they were an all-powerful silver bullet. In particular, those who shout "AI Revolution" while selling unverified solutions go out of their way to hide the limitations of LLMs.

But what is the reality? LLMs are undeniably powerful tools. However, if you blindly trust this tool without properly understanding its essence, serious issues can arise—ranging from security vulnerabilities to outright project failure.

In fact, after writing the post below, I had a lot going on—many people reached out to me, and I even received book publishing offers. But as I wrote in the book and have reiterated multiple times, it is practically impossible to deliver vibe coding in the form of a lecture. A workshop or guide format might be feasible, though.

Here are a few common misconceptions + propaganda? agitation? that I've heard from people I personally met who use vibe coding or similar AI-assisted coding services.

Misconception 1: "Code generated by an LLM always/sometimes works"

The False Belief

"It's code generated by AI, so of course it works, right?" or "Sure, it fails occasionally, but it works most of the time, doesn't it?"

The Truth

Whether code generated by an LLM works or not is a hit-or-miss gamble. According to recent research, more than 40% of AI-generated code contains security vulnerabilities. Just because code compiles or runs does not mean it "works properly."

This is much like asking a 5-year-old child to prepare dinner. The child might have memorized the recipe flawlessly, but they don't truly understand the dangers of flame control or how to handle knives properly. An end result comes out, but whether it is a safe and proper meal is a completely different matter.

# LLM이 생성할 수 있는 "작동하는" 코드
while True:
    pass  # 무한 루프 - 컴파일도 되고 실행도 되지만...

The Right Approach: LLM-generated code must always be treated as a draft. It must undergo thorough review, testing, and security checks.

Misconception 2: "An LLM can judge whether code is correct"

The False Belief

"The AI can verify the code, so it must be right."

The Truth

LLMs cannot judge the correctness of code. This also ties into a fundamental limitation of computer science, the Halting Problem. Because an LLM cannot know in advance how many tokens it will generate, it cannot pre-verify its own output.

An LLM is a pattern recognition machine. It merely generates statistically plausible code; it cannot judge whether that code is logically sound or handles every edge case.

It's like how a parrot can mimic human speech without understanding the meaning of the words. The same goes for LLMs.

The Right Approach: Code verification must be done by humans. Unit tests, integration tests, and code reviews remain indispensable.

Misconception 3: "If it works the way I expected, it's correct code"

The False Belief

"I tested it, and it turned out the way I wanted. Doesn't that mean it's fine?"

The Truth

This is one of the most dangerous misconceptions. Your expectations themselves might be incomplete.

# 사용자의 요청: "숫자를 두 배로 만드는 함수 만들어줘"
def double(x):
    return x + x  # 문자열 입력시 문제 발생!

double("hello")  # "hellohello" - 기대와 다른 결과

The above code works "as expected" for numbers, but behaves unpredictably with other inputs. Just like a child eating toothpaste when told to "brush your teeth." On the surface, the instruction was followed, but fundamentally, it's a flawed understanding.

The Right Approach: Clearly define specifications and write comprehensive tests that include edge cases. Verify against the requirements document, not user expectations.

Misconception 4: "LLMs only use existing libraries and APIs"

The False Belief

"Since the AI recommended this library, it must naturally exist in reality, right?"

The Truth

Absolutely not. This is known as "AI hallucination," where LLMs confidently suggest packages or APIs that do not exist.

According to recent research, for open-source AI models, 21.7% of package names were hallucinations, and even for commercial models, 5.2% were non-existent packages.

What makes this even more serious is that hackers exploit it. Attackers create and publish malicious packages using the hallucinated package names generated by LLMs. Developers then blindly trust the LLM's recommendation and end up installing those malicious packages.

// LLM이 제안한 코드
import { parseData } from 'awesome-data-parser-2024';  // 존재하지 않는 패키지!

// 실제로는
npm install awesome-data-parser-2024
// Error: Package not found

The Right Approach: Every library and API suggested by an LLM must be verifiedin the official documentation. Before installing a package, check whether it actually exists on npm, PyPI, etc., along with its download count and latest update date.

Misconception 5: "Code generated by an LLM is secure"

False Belief

"It's code generated by cutting-edge AI; there's no way it has security issues."

The Truth

It is quite the opposite. According to research by Backslash Security, when code was generated using simple prompts, all LLMs produced code that contained vulnerabilities from at least four categories in the CWE (Common Weakness Enumeration) Top 10. In particular, for the GPT-4o model, 90% of the code generated using simple prompts contained vulnerabilities.

Common Security Issues:

SQL Injection: Directly inserting user input into queries
XSS (Cross-Site Scripting): Missing input validation
Hardcoded Passwords: Writing credentials directly in the code
Path Traversal: Lack of file path validation

LLMs fundamentally tend to skip input validation. Even when explicitly asked to "write secure code," they often apply inconsistent or overly simplistic validation.

# LLM이 생성할 수 있는 취약한 코드
def get_user_data(user_id):
    query = f"SELECT * FROM users WHERE id = {user_id}"  # SQL 인젝션 위험!
    return execute_query(query)

# user_id = "1 OR 1=1" 입력시 모든 사용자 정보 노출

The Right Approach: LLM-generated code should be scanned with automated security toolsand undergo a code review by a security expert. Leverage security checklists like the OWASP Top 10.

Misconception 6: "LLMs think and reason"

False Belief

"The AI gives such intelligent answers; it must definitely be thinking."

The Truth

LLMs do not think. They don't reason either. They merely predict the statistically most plausible next word.

Borrowing the definition from philosopher Harry Frankfurt, some researchers describe LLM output as "bullshit." LLMs are indifferent to the truth of their outputs; true statements are accidentally true, and false statements are accidentally false.

Even OpenAI's latest reasoning models exhibited hallucination rates of 33% for the o3 model and 48% for the o4-mini model on questions about public figures.

This is similar to a child listening to adults' conversations and repeating them. They have simply learned word patterns without understanding the context.

The Right Approach: View LLMs as tools. Rather than thinking colleagues, they are advanced autocomplete systems that quickly find patterns. The final judgment and verification always rest with humans.

Misconception 7: "LLMs remember what was said 5 minutes ago"

False Belief

"I just said it earlier, so surely the AI remembers it too, right?"

The Truth

LLMs have no long-term memory. A conversation proceeds by rereading the entire context from the very beginning each time. It only has a limited memory space called a context window, and once that is exceeded, previous content is forgotten.

When a conversation gets long:

It starts ignoring earlier instructions
It "forgets" previously defined variables or functions
It generates code in an inconsistent style

It is just like the protagonist in the movie '50 First Dates', whose memory resets every morning.

# 대화 초반
"user_id를 정수로 처리해줘"
def get_user(user_id: int): ...

# 대화 후반 (맥락을 벗어나면)
"user_id를 문자열로 처리해줘"  # 이전 지시사항과 모순!
def update_user(user_id: str): ...

Right approach: Important requirements and constraints should be repeated with every prompt. Rather than trying to convey the entire context of a project all at once, break it down into smaller units alongside clear documentation.

Misconception 8: "LLMs work in my best interest"

False belief

"It's an AI assistant, so naturally it will do its best to help me, right?"

The truth

LLMs have no goals. They neither want to help you nor want to harm you. They simply reproduce patterns from their training data.

This carries significant implications:

If you ask a flawed question, it faithfully answers that question (without guiding you toward the right question)
Even if there are security vulnerabilities, it does not warn you (if that is the pattern in the training data)
Even if a better approach exists, it might not suggest it

Right approach: Treat an LLM like an unaccountable intern. They are talented, but without supervision, they can ruin a project. Clear instructions, continuous review, and final accountability must always remain with a human.

Misconception 9: "LLMs will warn me about ambiguous requirements"

False belief

"If my explanation is unclear, the AI will ask questions, right?"

The truth

LLMs attempt to fill in blanks plausibly. When they are not sure, instead of saying "I don't know," they make up a plausible-sounding answer.

A Senior Engineer at Microsoft explains, "AI hallucinations occur because of the probabilistic nature of AI models. They generate outputs based on statistical likelihood rather than deterministic logic."

사용자: "데이터를 처리하는 함수 만들어줘"
LLM: (어떤 데이터? 어떻게 처리? 모르지만 일단 뭔가 만들자!)
def process_data(data):  # 너무 일반적이고 실용적이지 않은 코드 생성
    return [item for item in data]

Right approach: Provide concrete and clear requirements. Explain in detail, like: "Create a function that extracts only email addresses from a user input string and returns them as a list. Ignore invalid formats."

Misconception 10: "I can ask for what I want clearly"

False belief

"Even if I don't know programming, if I just say what I want, the AI will take care of it."

The truth

This is the most fundamental yet easily overlooked issue. Most people do not know what they actually want.

The reason software development is difficult is not coding itself, but clearly defining the problem and structuring the solution. An LLM cannot do this process for you.

For example:

"Build a user-friendly interface" → What counts as friendly?
"Make a fast algorithm" → How fast is fast?
"Make it secure" → Secure from what threats?

It is just like telling a child to "play well." The child doesn't know your definition of "playing well."

The right approach: Before using an LLM, analyze requirements. First sort out what, why, and how you are going to build it. If you don't have programming experience, collaboration between domain experts and developers is still necessary.

The illusion that AI will take over developers' jobs

LLMs are certainly revolutionary tools. They speed up code writing, reduce repetitive tasks, and help in learning new technologies. However, they are not a silver bullet.

Be wary of those who claim that "you don't need developers as long as you have AI." They are selling an unverified dream. Companies that have actually introduced LLMs into production know this: AI tools do not replace developers, but rather demand a higher level of competence.

A developer in the era of LLMs is:

Not someone who writes code, but someone who reviews and validates code
Not simply someone who implements, but someone who designs architecture
Not someone who only knows technology, but someone who understands business requirements

Something I often say in lectures: Treat an LLM as if you were raising a child. Like a talented child, it has great potential, but without proper guidance and supervision, it can go astray. Don't believe in it blindly, but don't ignore it either. Using it wisely while maintaining critical thinking—that is our task living in the AI era.

Korean original ↗EOF ■