Logo Dark
Get In Touch

Reranking in RAG: Cross-Encoders, Top-K and Top-N Explained

/
AI

Imagine building an AI chatbot for a company's internal documents. An employee asks, "How can I reset my account password?" The search system returns several passages about account settings, security, and registration. The correct password-reset instructions are available, but they appear near the bottom of the results.

If the application sends only a few passages to the language model, the information needed to answer the question may never reach it.

This is where reranking in Retrieval-Augmented Generation (RAG) becomes useful.

Reranking takes an initial set of search results and reorders them based on their relevance to the user's query. It adds another step between retrieval and response generation, helping the application select more useful information for the LLM.

In this article, we will explore how reranking works, why the position of information matters, how Top-K and Top-N selection differ, and which reranking models developers can consider for their AI applications.

What Is Reranking in RAG?

Reranking is the process of reviewing an initial list of search results and sorting them again according to their relevance to a query.

In a typical RAG application, the retrieval system searches a document collection and returns a group of candidate passages. These passages may contain useful information, but the initial ranking does not always place the most relevant passage first.

A reranker, usually a cross-encoder model, evaluates the relationship between the query and each candidate passage, assigns a relevance score, and sorts the candidates accordingly.

For example, a user asks:

"How can I cancel my subscription?"

The initial search returns these passages:

  1. How to update payment details.
  2. How to renew a subscription.
  3. How to cancel an active subscription.
  4. How to change account settings.

The third passage directly answers the question. A reranker can move it to the top so the application can prioritize it when building the context for the LLM.

Reranking does not replace the initial search. It improves the ordering of the results that search has already found.

How Reranking Works in a RAG Pipeline

Reranking usually takes place after the initial retrieval step and before the selected passages are sent to the language model.

The process looks like this:

User Question
      |
      v
Initial Retrieval
(Vector Search / Keyword Search)
      |
      v
Candidate Passages
      |
      v
Reranking Model
      |
      v
Reordered Results
      |
      v
Select Relevant Passages
      |
      v
LLM Context
      |
      v
Generated Answer

Each stage has a separate purpose.

  • Initial retrieval: Searches a large document collection and identifies candidate passages.
  • Reranking: Examines the query and candidate passages to estimate their relevance.
  • Context selection: Chooses which passages to provide to the LLM.
  • Response generation: Uses the selected information to answer the user's question.

This two-stage approach balances search efficiency with more detailed relevance scoring. The first stage narrows the candidates, while the reranker focuses on ordering that smaller set.

Why Is Reranking Important?

Initial retrieval systems can return passages that share similar words with a query without directly answering it. Reranking provides another opportunity to distinguish between related information and information that is actually useful.

Improve Search Result Relevance

A passage containing the correct answer may initially rank below a passage that shares more keywords with the query. Reranking evaluates the query-passage relationship and can improve the ordering.

Select Better Context for the LLM

Many RAG applications limit the number of passages sent to the language model. Better ranking helps the application prioritize useful information within that limited context.

Handle Similar Questions

Consider the following questions:

  • How do I cancel a subscription?
  • How do I pause a subscription?
  • How do I renew a subscription?

These questions use similar words but require different answers. A reranker can help distinguish between their corresponding passages.

Reduce Unnecessary Context

When the most relevant passages appear first, an application can select a smaller set of results instead of sending every retrieved passage to the LLM. This can help control token usage and processing costs.

However, reranking cannot recover information that the initial retrieval stage failed to find. Its effectiveness also depends on the model, the candidate passages, and the quality of the source documents.

One important challenge in RAG is known as the Lost in the Middle problem.

Research published in Transactions of the Association for Computational Linguistics found that language models can struggle to use relevant information when it appears in the middle of a long input context. In the studied tasks, performance was often stronger when relevant information appeared near the beginning or end.

Consider a RAG application that retrieves ten passages and sends all of them to an LLM. The correct answer might be contained in the sixth passage, surrounded by unrelated information.

Even if the information is present, the model may not use it as effectively as expected.

This illustrates an important distinction: retrieving relevant information and using that information effectively are not the same thing.

Reranking helps address the retrieval-ordering part of this problem by moving the most relevant passages higher in the results before context construction.

For example:

Before reranking

1. Account settings
2. Payment methods
3. Subscription renewal
4. Billing history
5. General account information
6. Subscription cancellation instructions

After reranking

1. Subscription cancellation instructions
2. Subscription renewal
3. Billing history
4. Payment methods
5. Account settings
6. General account information

The application can then select the highest-ranked passages and construct a more focused context.

Reranking does not eliminate the Lost in the Middle problem. It helps organize the retrieved information, but the final prompt, context length, and language model still influence how effectively that information is used.

Top-K vs. Top-N: How Many Results Should You Keep?

Top-K and Top-N are terms used to describe selecting a limited number of results. Their exact meaning depends on the implementation, and some systems use the terms interchangeably.

A useful way to distinguish them in a RAG pipeline is to use Top-K for initial retrieval and Top-N for the final selection after reranking.

Top-K: Retrieve Candidate Passages

Top-K is the number of results returned by the initial search.

For example, a vector database might retrieve the 20 passages most similar to a user's question.

At this stage, the goal is to collect enough candidates to give the reranker a reasonable chance of finding the relevant information.

If K is too small, the correct passage may be missed. If K is too large, reranking may require more computation and increase latency.

Top-N: Select the Final Passages

Top-N is the number of passages retained after reranking.

For example, the reranker may reorder 20 candidate passages, and the application may keep only the best three to send to the LLM.

The final number should depend on the question, passage length, context budget, and retrieval quality.

Example of Top-K and Top-N

User Question
      |
      v
Initial Retrieval: Top-K = 20
      |
      v
20 Candidate Passages
      |
      v
Reranking Model
      |
      v
Sort by Relevance
      |
      v
Final Selection: Top-N = 3
      |
      v
LLM Generates Answer

These values are illustrative, not universal recommendations.

A question requiring one specific fact may need only a few passages. A question involving several documents may require more. The best values should be determined through testing.

The main idea: Retrieve enough candidates to avoid missing relevant information, then use reranking to select a smaller, more useful set for the LLM.

Popular Reranking Models to Consider

Different reranking models can help improve search results in a RAG application. They all do the same basic job: check how well each search result matches the user's question and put the most relevant results first.

Here are three popular options.

1. Cohere Rerank

Cohere Rerank is a service that helps improve search results through an API. You send the user's question and the documents found by your search system to Cohere. It then scores the documents based on how closely they match the question and returns them in a better order.

For example, if a user asks, "How can I reset my account password?", the search system might find several documents about account settings, login problems, and password recovery. Cohere Rerank helps move the documents about password recovery to the top.

Key points:

  • Works through an API, so you don't need to manage the model yourself.
  • Can work with results from keyword search, vector search, or a combination of both.
  • Offers different models for different needs, including multilingual search.
  • You need to consider API costs, response time, and how your data is handled.

Best for: Teams that want to add reranking without managing their own AI model infrastructure.

Learn more: Cohere Rerank documentation

2. BGE Reranker v2 M3

BGE Reranker v2 M3 is an open-weight AI model that developers can run on their own servers or suitable cloud infrastructure. Unlike a hosted API service, this approach gives teams more control over where the model runs.

It is designed to compare questions and documents in multiple languages and score how relevant each document is to a question.

For example, imagine a RAG application that answers questions in English, Hindi, and other languages. BGE Reranker v2 M3 can be evaluated to see how well it ranks search results for those languages.

Key points:

  • An open-weight model that developers can download and use according to its license.
  • Supports reranking across multiple languages.
  • Can run on your own infrastructure, depending on available resources.
  • Your team is responsible for setting it up and managing its performance.

Best for: Teams that want more control over their reranking setup, especially when working with multiple languages.

Learn more: BGE Reranker v2 M3 on Hugging Face

3. MS MARCO MiniLM Cross-Encoder

MS MARCO MiniLM is a smaller reranking model that compares a question with a document and gives it a relevance score. Documents with higher scores can then be placed earlier in the search results.

For example, if a user asks, "What is Spring Boot?", the search system might find documents about Spring Boot, Spring Framework, and Java programming. MiniLM scores each document against the question to help identify which ones are the best match.

Because it is a relatively compact model, it can be useful for learning about reranking and testing it in an application. However, you should check how well it works for your specific data and questions.

Key points:

  • A smaller Cross-Encoder model for ranking search results.
  • Can be run locally with compatible model libraries.
  • Compares each question with a candidate document and produces a relevance score.
  • Its accuracy depends on the type of content and questions in your application.

Best for: Developers who want to experiment with reranking or need a smaller model to test in their own search system.

Learn more: cross-encoder/ms-marco-MiniLM-L6-v2 on Hugging Face

Which Reranking Model Should You Choose?

The right choice depends on your application's needs.

  • Cohere Rerank: Choose this if you prefer a managed API and don't want to run the model infrastructure yourself.
  • BGE Reranker v2 M3: Consider this if you want to run an open-weight model on your own infrastructure and need multilingual support.
  • MS MARCO MiniLM: Consider this if you want a compact model to experiment with and evaluate for your search use case.

Before choosing a model, test it with real questions and documents from your application. Compare the quality of the search results, response time, and cost to find the best fit.

Comparing the Three Options

FeatureCohere RerankBGE Reranker v2 M3MS MARCO MiniLM
DeploymentHosted APISelf-hosted or hostedLocal or hosted
Model accessManaged serviceOpen weightsOpen weights
Language supportDepends on selected modelMultilingualPrimarily English-focused
InfrastructureManaged inferenceYour compute or hosting providerYour compute or hosting provider
Main considerationAPI cost and data handlingCompute and operationsRanking quality and input limits

This is a high-level comparison, not a benchmark ranking. Model quality and latency depend on the hardware, document collection, language, query type, and candidate set. Hosted services also evolve, so developers should check the current documentation before choosing a model.

How to Choose a Reranking Model

The right model depends on your application rather than popularity alone.

Consider these questions before making a decision:

  • Do you need multilingual search? Evaluate the languages your users actually use.
  • Do you prefer a hosted API or local deployment? Consider infrastructure, maintenance, privacy, and operational requirements.
  • How many candidates will you rerank? Larger candidate sets can increase latency.
  • What are your cost limits? Include inference costs and the resources needed to operate the model.
  • Does it improve your search results? Test it against representative queries and known relevant passages.

A useful evaluation compares the initial retrieval results with the reranked results using metrics such as MRR@k and nDCG@k. Also measure latency and the quality of the final LLM answers.

A more complex model is not necessarily the best choice for every application. The goal is to find a model that improves relevance while meeting the application's performance and cost requirements.

Limitations of Reranking

Reranking can improve the order of search results, but it has several limitations.

Additional latency: The reranker must evaluate the candidate passages, adding another processing step.

Higher cost: Hosted inference may incur API charges, while local inference requires computing resources.

Missing candidates: If initial retrieval does not return the correct passage, reranking cannot promote it.

Input limits: Long passages may need to be shortened or split according to the model's supported input length.

Domain-specific performance: A model that works well on general search tasks may not perform as well on specialized company documents or technical terminology.

These limitations make evaluation important. Test the complete retrieval pipeline instead of assuming that adding a reranker will always improve the final answer.

Conclusion

Reranking helps RAG applications organize search results so that the most relevant information can be prioritized before it reaches the language model.

It is particularly useful when the initial search returns related passages but does not consistently place the best results first. The Lost in the Middle problem also highlights why the quality and placement of information within a long context deserve attention.

Understanding Top-K and Top-N helps developers balance candidate coverage against the amount of context passed to the LLM. Models such as Cohere Rerank, BGE Reranker v2 M3, and MS MARCO MiniLM offer different approaches to implementing this additional ranking stage.

There is no single model or configuration that fits every application. The best choice depends on the data, languages, infrastructure, latency requirements, and evaluation results.

A good RAG system does more than find relevant information. It prioritizes the information that best helps answer the user's question.

Building a RAG feature and not sure where retrieval is breaking?
Talk to our AI engineering team

FAQs

What is reranking in RAG?

Reranking is a second step after retrieval that re-sorts search results by how well each one answers the question. A reranking model reads the question and each candidate passage together, scores them, and the app keeps the top few for the LLM. Users never see it. It happens in the backend on every question.

What is the difference between a reranker and an embedding model?

An embedding model encodes questions and passages separately, so documents can be prepared ahead of time and searched in milliseconds. A reranker, usually a cross-encoder, reads the question and passage together, which is more accurate but too slow to run across a whole collection. That's why RAG uses embeddings first, then a reranker.

How many documents should I rerank?

Start by retrieving around 20 candidates (Top-K) and keeping around 3 after reranking (Top-N), then test. Sentence-Transformers' own example reranks 100. More candidates improve the chance the right passage is included but add latency, since every candidate is another model run.

Do I need a reranker for every RAG app?

No. If the right passage already ranks first for most real questions, a reranker adds latency without much gain. It earns its place when the correct answer is usually retrieved but ranks too low, especially with similar-looking documents like cancel, pause and renew policies. Measure before adding one.

Can reranking fix bad search results?

Only partly. Reranking can move a correct passage from position eight to position one, but it can't find a passage the search never returned. If the right answer is missing from the candidates entirely, fix the query with query transformation, raise Top-K, or check how the documents were ingested.

Het Shah
Software Engineer
I build software solutions using Java, Spring Boot, React.js, and modern technologies. I enjoy exploring AI, integrating AI APIs into applications, and finding better ways to solve real-world problems. I believe in continuous learning, writing clean code, and building reliable, scalable software.
India (HQ)

201, iSquare Corporate Park, Ahmedabad-380060, Gujarat, India

+91 77 97 977 977
Canada

24 Merlot Court, Timberlea, NS B3T 0C2, Canada

+1 902-333-0067

For Sales

YO@7SPAN.COM

Looking For Jobs

Apply Now
Making It Possible Light