AI Engineeringhybrid-search

Implementing Hybrid Semantic and Lexical Search for RAG with BM25 and Reciprocal Rank Fusion

P&C

Prompt & Circuit Editorial

May 26, 2026 · 12 Min Read

Implementing Hybrid Semantic and Lexical Search for RAG with BM25 and Reciprocal Rank Fusion

TL;DR

Run a lexical search (BM25) and a semantic search (dense embeddings) over the same corpus, then merge the two ranked lists with Reciprocal Rank Fusion (RRF). Documents that rank well in either list rise; documents that rank well in both rise furthest. The complete example below is about 60 lines of Python.

Most RAG prototypes start with semantic search because embeddings match meaning: a query about "connection failures" can find a document that says "could not reach". The weakness is exact tokens. Error codes, product SKUs, rare names and internal jargon carry little meaning for a general-purpose embedding model, so a query for E1047 may not put the one document containing E1047 at the top.

BM25 has the opposite profile. It scores documents by term frequency, inverse document frequency and document length, so an exact rare term is a very strong signal, but a paraphrase with no shared words scores zero.

Hybrid search runs both and fuses the results. For RAG this matters because the generator can only use what the retriever returns: a better top-k means better grounded answers, not faster ones.

Setting up

pip install rank_bm25 sentence-transformers
  • rank_bm25 (current release 0.2.2) implements BM25Okapi and variants. It does no text preprocessing, so you must tokenize documents and queries the same way yourself.
  • sentence-transformers loads pre-trained embedding models. The example uses all-MiniLM-L6-v2, which produces 384-dimensional vectors and truncates input longer than 256 word pieces, so long documents should be chunked first.

The complete example

import re

from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer

documents = [
    "Error E1047 means the backup job could not reach the storage account.",
    "To rotate an API key, create a new key, deploy it, then revoke the old one.",
    "The VPN client drops connections when the laptop goes to sleep.",
    "Invoices are generated on the first business day of each month.",
    "The storage account firewall blocks traffic from unknown networks.",
    "Reset a forgotten password from the sign-in page using the email link.",
]


def tokenize(text: str) -> list[str]:
    return re.findall(r"\w+", text.lower())


bm25 = BM25Okapi([tokenize(doc) for doc in documents])


def search_bm25(query: str) -> list[int]:
    """Return document indices, best BM25 match first."""
    scores = bm25.get_scores(tokenize(query))
    return sorted(range(len(documents)), key=lambda i: scores[i], reverse=True)


model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
doc_embeddings = model.encode(documents)  # computed once, reused per query


def search_semantic(query: str) -> list[int]:
    """Return document indices, most similar (cosine) first."""
    query_embedding = model.encode([query])
    scores = model.similarity(query_embedding, doc_embeddings)[0]
    return scores.argsort(descending=True).tolist()


def reciprocal_rank_fusion(rankings: list[list[int]], k: int = 60) -> list[tuple[int, float]]:
    """Fuse ranked lists: each document scores the sum of 1 / (k + rank)."""
    fused: dict[int, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            fused[doc_id] = fused.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(fused.items(), key=lambda item: item[1], reverse=True)


def hybrid_search(query: str, top_k: int = 3, window: int = 50) -> list[tuple[str, float]]:
    lexical = search_bm25(query)[:window]
    semantic = search_semantic(query)[:window]
    fused = reciprocal_rank_fusion([lexical, semantic])
    return [(documents[i], round(score, 4)) for i, score in fused[:top_k]]


if __name__ == "__main__":
    for query in ["E1047", "why can't my backup connect to storage?"]:
        print(query)
        for doc, score in hybrid_search(query):
            print(f"  {score:.4f}  {doc}")

Lexical retrieval

BM25Okapi takes a list of token lists. The regex tokenizer lowercases text and strips punctuation, so E1047 and e1047. match; plain str.split() would keep trailing punctuation attached to words. get_scores returns one score per document, which we turn into a ranking.

Semantic retrieval

model.encode turns text into vectors, and model.similarity compares them using the model's configured similarity function, which defaults to cosine. Document embeddings are computed once; only the query is encoded per request.

Reciprocal Rank Fusion

RRF was introduced by Cormack, Clarke and Buettcher at SIGIR 2009. Each document's fused score is the sum over the input rankings of 1 / (k + rank), with ranks starting at 1. The paper used k = 60, and Elasticsearch uses 60 as the default rank_constant for its RRF retriever.

RRF works on positions, not raw scores, which is its main advantage: BM25 scores are unbounded while cosine similarities sit between -1 and 1, so adding them directly would need normalisation and tuning. With RRF:

  • A document ranked highly in both lists receives two large contributions and usually wins.
  • A document found by only one method still receives one contribution, so an exact-match hit on a rare identifier is not lost.
  • window caps how many results from each list take part, which mirrors the rank_window_size setting in Elasticsearch. Production systems retrieve a few dozen candidates per method rather than scoring the whole corpus.

A larger k flattens the difference between rank 1 and rank 10; a smaller k rewards the very top results more strongly. Treat it as a parameter to evaluate, not a constant to copy.

Trade-offs at a glance

AspectBM25 (lexical)Embeddings (semantic)Hybrid (RRF)
StrengthExact and rare termsParaphrase and synonymsCovers both failure modes
WeaknessNo match without shared wordsWeak on codes, IDs and rare namesTwo indexes to build and keep in sync
Extra costInverted indexEmbedding model and vector indexBoth, plus a cheap fusion step

We have not included latency figures: they depend on corpus size, hardware, index type and model, so measure them on your own data.

Moving to production

  • Replace the in-memory lists with a search engine or vector database that can run hybrid queries itself. Elasticsearch, for example, offers an RRF retriever that fuses its lexical and vector results server-side; check your own engine's documentation for an equivalent before writing fusion code by hand.
  • Build an evaluation set of real queries with known relevant documents before tuning anything, and compare BM25 alone, embeddings alone and hybrid on metrics such as recall@k.
  • Chunk long documents to fit the embedding model's input limit, and use the same chunks for both indexes so the fused ranking refers to the same units.
  • Consider a cross-encoder reranker on the fused top results if quality still falls short; it is slower but scores each query and passage pair jointly.

FAQ

Do I need to keep the BM25 index in memory? rank_bm25 is an in-memory library suited to experiments and small corpora. For large or frequently updated corpora, use a search engine with a persistent inverted index, such as Elasticsearch.

Can I swap the embedding model? Yes. all-mpnet-base-v2, for example, produces 768-dimensional vectors and accepts up to 384 word pieces, at a higher compute cost than MiniLM. Re-embed the whole corpus when you change models, because vectors from different models are not comparable.

My vector database already returns ranked results. Can I still use RRF? Yes. RRF only needs the rank order from each system, so feed the returned document IDs straight into the fusion step.

How does RRF compare to a learned weighted combination? RRF needs no training data and is robust to score-scale differences. A learned combination or a trained reranker can do better when you have enough labelled relevance data, at the cost of building and maintaining that data.

Sources

Tags:hybrid-searchbm25ragreciprocal-rank-fusionpython
Share Article

Responses (0)

Sign in to join the conversation and share your thoughts.

P&C

Prompt & Circuit Editorial

Contributing Editor

Articles on AI infrastructure for cloud architects, written from official vendor sources and checked claim by claim before publishing.