Treat LLM features as engineering, not experiments

The gap between an impressive LLM demo and a system you can put in front of customers is almost entirely engineering: grounding the model in your own data, measuring whether its answers are any good, and constraining what it is allowed to do. None of that is glamorous, and all of it is where the value is.

Retrieval beats a bigger model

A general model that can read your documents will out-perform a larger model that cannot, for any task that depends on your own knowledge. Retrieval-augmented generation is the pattern: embed your content, retrieve the relevant passages at query time, and pass them to the model as context.

def answer(question: str) -> str:
    query_vector = embed(question)
    # Retrieve the passages most relevant to THIS question.
    passages = vector_store.search(query_vector, top_k=5)

    context = "\n\n".join(p.text for p in passages)
    prompt = f"Answer using only the context below.\n\n{context}\n\nQ: {question}"
    return llm.complete(prompt)

If you cannot measure it, you cannot ship it

Before an AI feature reaches a user, you need an evaluation set: representative questions with known-good answers, scored automatically on every change. Without it, every prompt tweak is a guess and every regression ships silently. Treat the eval set the way you treat a test suite, because that is what it is.

The teams that succeed with AI are not the ones chasing the newest model. They are the ones who built the boring scaffolding — retrieval, evaluation, guardrails — and could therefore adopt each newer model in an afternoon.