CrewAI LLM Setup: Connect OpenAI, Anthropic or Ollama

Connecting a model sounds like the trivial part of a CrewAI project. It is where a surprising number of first runs die, usually with an authentication error naming a provider you never chose. The framework is not misbehaving. It has defaults, and defaults win whenever you did not state a preference.

This guide covers the three ways the documentation gives you to set a model, the provider prefix format, running a local model through Ollama, and the specific reasons a non-OpenAI project can still demand an OpenAI key.

One note before the code. Model configuration is an area where CrewAI has changed across versions, and community answers from a year ago can describe variables that no longer carry the same weight. Check what your installed version does rather than trusting any snippet, including the ones below.

How do you tell CrewAI which model to use?

There are three levels, and they layer. The environment variable sets a project-wide default, the LLM class configures one model precisely, and the llm parameter on an Agent decides what that specific agent calls. Use all three deliberately and the ambiguity disappears.

The prefix before the slash is not decoration. It tells the framework which integration to route through, and a model id without a prefix leaves that decision to a default. Get in the habit of always writing the provider prefix.

That split is the most useful pattern in this whole article. Classification and routing rarely need your most expensive model, while analysis and writing usually do. Mixing models per agent is the cheapest quality-per-dollar improvement available in a crew, and it costs you two extra lines.

Which providers work, and which need an extra install?

The documentation splits providers into two groups. Several have native SDK integrations, and everything else routes through LiteLLM as a fallback. The practical consequence is that some providers need an optional dependency installed before the prefix works.

That list is a snapshot, not a contract. Providers get promoted from the fallback path to a native integration over time, and new ones appear. Read the LLM page for your installed version before you conclude a provider is unsupported.

If a provider you want is missing from the table, the OpenAI-compatible route below is usually the answer. A large share of hosted inference endpoints speak the OpenAI wire format, which means you can point the client at a different base URL and carry on.

How do you run CrewAI against a local model with Ollama?

Pull a model with Ollama, then give the agent an LLM whose model string starts with the ollama prefix and whose base_url points at the local Ollama port. The documentation shows exactly this shape, using the default port 11434.

Local models are the reason many people pick a framework in the first place. Nothing leaves the machine, there is no per-token bill, and you can iterate on prompts as many times as you like. The trade is capability and speed, so keep the tasks you hand a local model narrower than the ones you would hand a hosted one.

Now for the error that brings most people to this page. You configured Groq, or Gemini, or a local model, and the run still stops on a missing OpenAI key. It is one of the most repeated questions in the CrewAI community forum, and there are two separate causes hiding behind one message.

Why does CrewAI ask for an OpenAI key when I set another provider?

Because something in your run did not receive your model and used a default instead. The agents documentation lists a default model for the llm field rather than nothing, and which model that is has changed between versions. Any agent you created without an explicit llm is calling that default.

The second cause is embeddings rather than generation. The crews documentation describes the embedder configuration with OpenAI as the default, so features that embed text can reach for an OpenAI key even when every agent is pointed elsewhere. Community reports about retrieval-style tools behaving this way follow the same pattern.

The manager_llm line catches people out most often. In a hierarchical crew the manager is another agent making real model calls, so a crew where every worker is configured can still fall back on the one agent you did not write down. The same logic applies to function_calling_llm and to planning_llm if you turn planning on.

One suggestion that circulates on the forum is a flag to disable the default model entirely. We could not confirm it in the current documentation, so it is deliberately left out here. If you find it in your installed version, treat that as the authority.

How do you use a provider that is not on the list?

Point the OpenAI-compatible client at a different base URL. The documentation shows this both as environment variables and as constructor arguments, and it covers most hosted inference services, gateways and self-run servers.

Prefer the explicit form in anything you will keep. Environment variables named after OpenAI while pointing at a different vendor are a reliable way to confuse the next person reading the repository, including you.

A model configuration that will not surprise you

1. Write the prefix every time — Always use provider/model-id, never a bare model name. The prefix is what selects the integration path.
2. Create one LLM object per tier — A fast, cheap model for routing and classification. A stronger model for analysis and writing. Two objects, reused.
3. Assign llm on every agent — Including the ones you think do not matter. An unassigned agent is an agent using a default you did not choose.
4. Cover the hidden agents — Set manager_llm or manager_agent for hierarchical crews, plus function_calling_llm and planning_llm if you use those features.
5. Configure embeddings separately — Memory, knowledge and retrieval tools embed text. The crew embedder setting defaults to OpenAI, so set it if you are avoiding that provider.
6. Prove it with token counts — Run once, print crew.usage_metrics, and confirm the volume matches the provider dashboard you expected to see it on.

That last step is the honest test. Provider dashboards do not lie about who received your traffic, and a five minute check there beats an afternoon of reading configuration.

Worth naming what this section really is: infrastructure plumbing that exists between you and the work you wanted done. Some readers are building a product and this plumbing is part of the product, which is a good reason to own it. Others came here to get a job done and discovered a configuration matrix on the way. That second case is what we built Sistava for, where you hire an AI Employee and the model routing, keys and fallbacks are already handled behind the hire.

If you are staying with the framework, the payoff for doing this properly is real. Once every agent has an explicit model, you can swap providers in one place, run half your crew locally, and compare cost per run between two configurations honestly.

Frequently asked questions

Model configuration is the least interesting part of building with agents and the one that wastes the most hours. Ten minutes spent making every model assignment explicit will save you the afternoon you would otherwise lose to a fallback you did not know existed.

Sources

• CrewAI documentation, LLMs
• CrewAI documentation, Connect to any LLM
• CrewAI documentation, Agents
• CrewAI documentation, Crews
• CrewAI documentation, Installation
• CrewAI community forum, why is CrewAI asking for OPENAI_API_KEY