Picking an embedding model for agent memory and retrieval

Most of the attention in agent stacks goes to the chat model. That makes sense, since it is the part that reasons, calls tools and writes the answer. But a lot of agents never get a fair shot at the right answer because of a quieter decision made much earlier: which embedding model turned the knowledge base, the past conversations and the tool docs into vectors. If retrieval hands the model the wrong five chunks, the smartest model on our leaderboard will confidently reason over the wrong five chunks.

The good news is that embedding models are cheap, boring in the best way, and easy to compare once you know which knobs matter. This post walks through the three options most agent builders are actually choosing between in November 2026, OpenAI's text-embedding-3 pair, Google's gemini-embedding-2, and a local model through Ollama, then covers the parts that bite later: dimensions and storage, query formatting, and what happens when you switch models.

Why the embedding model matters more in agents than in chat

A plain RAG chatbot embeds a question, pulls a few chunks and answers once. An agent does that many times per task. It might search memory for what the user said last week, search a docs index for an API's auth rules, and search a code index for the function it is about to edit, all inside one run. Every one of those lookups is a place where a weak embedding can quietly drop the one chunk that mattered. We covered how to structure what gets stored in designing agent memory; this post is about the model that does the matching.

The other difference is that agents generate their own queries. A human types "how do I rotate the API key." An agent might search for "credential rotation procedure for service account in staging" because that is what its plan says. Models that let you tell them a text is a search query, rather than a document, tend to handle that mismatch better, and that turns out to be one of the real differences between the options below.

OpenAI: text-embedding-3-small and text-embedding-3-large

OpenAI's embeddings guide still lists two third-generation models. By default text-embedding-3-small returns 1,536 dimensions and text-embedding-3-large returns 3,072, and both take up to 8,192 input tokens. On the pricing page, small costs $0.02 per million input tokens and large costs $0.13. The older text-embedding-ada-002 is still listed at $0.10, which means there is no reason to stay on it: small is five times cheaper and the guide's own table scores it higher on the MTEB benchmark (62.3% against 61.0%, with large at 64.6%).

The feature that matters most for agents is the dimensions parameter. Both v3 models were trained so you can shorten the vector and keep most of its meaning. The guide gives a concrete example: a text-embedding-3-large vector cut down to 256 dimensions still beats a full 1,536-dimension ada-002 vector on MTEB. If your vector store caps out at 1,024 dimensions, you can ask large for 1,024 and keep the better model. Pass the parameter at request time if you can. If you truncate vectors yourself afterward, the guide says you have to re-normalize them, and it shows the L2 normalization code to do it.

Two smaller details are worth knowing. OpenAI embeddings come back normalized to length 1, so cosine similarity and a plain dot product give identical rankings, and the dot product is a little faster. And the FAQ notes that the v3 models lack knowledge of events after September 2021. That rarely matters for retrieval, but if your corpus is full of product names coined since then, test on your own data rather than assuming.

For a big initial index, the Batch API supports /v1/embeddings at a 50% discount with a 24-hour turnaround, capped at 50,000 embedding inputs per batch. We went through when batching is worth it in batch APIs for agent workloads.

Google: Gemini Embedding 2

Google's embeddings docs now lead with gemini-embedding-2, which Google calls its first multimodal embedding model. It maps text, images, video, audio and PDFs into one shared space, with an 8,192 token input limit. gemini-embedding-001 is still available for text-only work.

Pricing on the Gemini pricing page is $0.20 per million text input tokens on the paid tier, $0.10 through batch, and free of charge on the free tier, though free-tier data is used to improve Google's products and paid-tier data is not. Images, audio and video cost more: $0.45, $6.50 and $12.00 per million tokens respectively. The older gemini-embedding-001 lists at $0.15.

The default output is 3,072 dimensions, trained with Matryoshka Representation Learning so it can be truncated, and Google recommends 768, 1,536 or 3,072. A nice change in version 2 is that truncated vectors are normalized automatically, where with 001 you had to normalize anything below 3,072 yourself.

The bigger change is how you say what a text is for. gemini-embedding-001 used a task_type field such as RETRIEVAL_QUERY and RETRIEVAL_DOCUMENT. Version 2 drops that field and wants the task written into the text. For retrieval, a query becomes task: search result | query: {content} and a document becomes title: {title} | text: {content}, with title: none when there is no title. There are matching formats for question answering, fact checking and code retrieval. For agents this is genuinely useful, because the agent's self-written queries get labeled as queries, but it is also an easy thing to forget in one code path. If your indexing job formats documents correctly and your agent's search tool sends bare strings, results get worse and nothing errors out.

Watch one more behavior change: when you pass several inputs in one request, gemini-embedding-2 returns a single aggregated embedding, not one per input. To get separate vectors you wrap each input in its own Content object or use the batch endpoint. Code that worked with 001 and assumes one vector per string will silently store the wrong thing.

Local: Ollama

If the corpus cannot leave the machine, or you just do not want a per-token bill, Ollama's embeddings docs show the local path. You run a model such as embeddinggemma and call POST /api/embed on localhost. The docs say vector length is typically 384 to 1,024 dimensions depending on the model, and that /api/embed returns L2-normalized vectors, so the same dot-product shortcut works. Their two tips are the right ones for any provider: use cosine similarity for most semantic search, and use the same embedding model for both indexing and querying.

Local embeddings pair well with the setups in our post on tool calling with local open-weight models, since an embedding model is far lighter than a chat model and can usually share the same box.

What a real index costs

Take a docs-and-memory index of 10,000 pages at roughly 800 tokens each, which is the page size OpenAI's guide uses for its pages-per-dollar table. That is 8 million tokens. At list prices, text-embedding-3-small embeds it for $0.16, text-embedding-3-large for $1.04, and gemini-embedding-2 for $1.60, or $0.80 through batch. Re-embedding the whole thing is cheaper than a single long agent run on most frontier chat models.

Storage is where the dimension choice actually costs money. A float32 vector takes 4 bytes per dimension. One million chunks at 3,072 dimensions is about 12.3 GB of raw vectors before any index overhead. At 1,536 it is about 6.1 GB, and at 768 about 3.1 GB. Most managed vector databases price on memory or storage, so cutting dimensions in half roughly halves that line item. For many agent corpora, a 1,024 or 768 dimension vector from a strong model is the sweet spot, but measure recall on your own queries before you commit.

The migration trap

The one decision that is expensive to undo is the model itself. Google's migration notes say it bluntly: the embedding spaces of gemini-embedding-001 and gemini-embedding-2 are incompatible, and upgrading means re-embedding all existing data. The same is true between any two models and even between two dimension settings of the same model. A vector made by one cannot be compared with a vector made by another.

In practice that means three habits. Store the model name, dimension count and query format version next to every vector, so you can tell what made it. Keep the raw text, so a re-embed is a batch job rather than a data recovery project. And when you switch, build the new index alongside the old one, run your agent's real queries against both, and flip the search tool over in one change. Mixing old and new vectors in the same index is the failure mode that produces strange, inconsistent retrieval for months.

A sensible default

If you are starting today with a text-only corpus and no strict privacy constraint, text-embedding-3-small is a hard default to beat on price, and you can trade up to large with a reduced dimensions value if recall is not good enough. If your agent needs to search screenshots, PDFs or audio alongside text, gemini-embedding-2 is the one built for that, as long as you are disciplined about its query and document formats. If the data has to stay local, Ollama gets you running in minutes. Whichever you pick, check it against your own queries, ideally with a small labeled set like the ones we described in evaluating agent tool calls, and compare candidates on the public MTEB leaderboard only as a starting point.

For a longer treatment of retrieval design, chunking and evaluation in production AI systems, AI Engineering by Chip Huyen covers RAG in depth and is the book we keep pointing people to.