Better Retrieval Using the AI Staffing Example: Late Interaction Retrieval, Cross-Encoders, and LLMs Compared

In this blog post, we explore how a document retrieval system can be improved through different approaches. We illustrate this with an internal example in which staffing involves matching project descriptions with the most suitable candidates. We look at late interaction retrieval as an alternative or complement to standard embeddings, cross-encoders for more precise scoring, and LLM-based summarization and structured information extraction, augmented by an LLM-as-a-judge mechanism for quality assurance. We round things off by examining listwise LLM-based reranking for the final selection of suitable candidates. These principles transfer well to other use cases, such as AI agents operating on top of a RAG system.

Introduction and Background

Below, we present extensions to our internal AI staffing project, which our colleague Laura Weber already introduced in her blog post on text embeddings and vector search.

In brief, the goal is to identify the most suitable consultants for a given project or job description. Depending on the chunking strategy, the search operates either at the level of entire CVs or at the level of individual projects within a CV. The improvement methods described below can also be applied to other problems involving retrieval—that is, finding the most relevant documents for a given query. One important use case is retrieval-augmented generation (RAG) systems for large language models (LLMs) in the context of generative AI (GenAI) applications.

Improvements arise not only at the model level but also in preprocessing. This includes suitable tools for handling different document types and, in many cases, an OCR (optical character recognition) step. Some CVs, for example, returned empty or near-empty content after text extraction because the text was not free text at all but was embedded in images. The main focus here, however, is model-based improvement. To that end, we first examine the limitations of single-vector embedding models, which are most commonly used in retrieval systems.

Motivation: The Limitations of Single-Vector Embeddings

In the context of retrieval, "embeddings" typically refers to single-vector embeddings. These are also commonly called dense embeddings because a text passage is encoded into a single dense numerical vector that semantically captures the content of that passage. This is done separately for the query (the project description) and the candidate documents (CVs or individual projects within CVs). Therein lies a central limitation of this approach: compressing a text into a single vector can cause relevant information to be lost. The query and the document are each encoded independently, and only afterward are the resulting embedding vectors compared using a similarity measure such as cosine similarity. For this reason, this approach is also referred to as a bi-encoder. In this use case in particular—but also in others such as medical or legal applications—individual key terms carry significant weight and can easily get lost within a single embedding vector. In the end, a single vector must represent the entire document segment.

There are, in fact, fundamental theoretical limitations to single-vector embedding methods that cannot be effectively overcome simply by increasing the embedding dimension, as demonstrated in a Google paper on theoretical limitations. These theoretical constraints are not always the decisive factor in practice. More relevant are often the separate encoding of query and document and the loss of important keywords. The keyword issue is one reason why many applications (for example, those built on Databricks: https://docs.databricks.com/aws/en/vector-search/query-vector-search) frequently rely on hybrid search that also incorporates keyword-based search. In the following section, we examine another option.

Multi-Vector Embeddings and Late Interaction Retrieval

The points raised in the previous section are key motivations for introducing multi-vector embedding methods. An important example of this class of methods is late interaction retrieval, and in particular variants of ColBERT (see https://arxiv.org/pdf/2004.12832 for the original paper; several extensions have since been developed).

The core idea is as follows: instead of aggregating token embeddings early in the neural network into a single vector, they are contextualized but retained as individual entities. This enables similarity computations at that more granular level. Late interaction retrievers propagate token embeddings from the project or job description and from the CVs through separate encoders, keeping them unaggregated all the way to the scoring step at the end of the neural network. This allows the model to compare the project description and CV information at the token embedding level—an approach similar to keyword search, but one that still incorporates semantic information.

The scoring at the end is handled by a "late interaction head" in the neural network. In ColBERT and its variants, this is done via the MaxSim operator. For each token in the query (the project or job description), the model finds the document token (such as a project entry in a CV) with the highest similarity. This similarity is computed as the dot product of the respective final token embedding vectors. The resulting similarity scores across all query tokens are then summed or averaged to produce an overall score reflecting the similarity between the job description and the respective CV (or a project within the CV). Since this computation is performed for every job description against all CVs or (CV, project-within-CV) pairs, the individuals with the highest scores can be considered the most potentially relevant candidates.

An Example from AI Staffing

Was that too abstract? A concrete example helps clarify things. For the sample job description "We are looking for a senior data scientist with extensive experience working with sensor data and robotics," the figure below shows the query broken down into individual tokens in the left column. For each query token from the job description, the second column shows the token with the highest similarity from the best-matching project description. In most cases, the most similar token from the corresponding CV project is identical to the respective token in the job description.

There are a few exceptions, however (such as "such" and "so" in the third row). The third column is not essential to the computation itself but illustrates the context surrounding the most similar token in the CV project. The final column shows the similarity score between the query token and the most similar token from the CV project. The average of these values in the last column yields the overall similarity score for the CV project relative to the job description. When this is done for all projects across all CVs and the person associated with the highest-scoring project is identified, the result is the most likely best-fit candidates.

The figure already shows the project with the highest overall score, selected from the full pool of (CV, project) pairs. The candidate associated with that CV can therefore be considered the most likely best match. In this example, tokens such as "senior," "sensor," and "robot" contribute particularly strongly to the overall score—illustrating clearly why this approach is well suited to this use case.

‍

Strengths and Weaknesses of the Approach

This example already highlights several advantages of the approach: because similarity is computed at the token embedding level, the method is more interpretable than other embedding approaches and can often achieve higher accuracy. This is partly because token-level similarity computation resembles an extended keyword search, making it less likely that important key terms are overlooked.

That said, there are also drawbacks. The memory requirements in particular are significantly higher than those of conventional single-vector methods. Extensions of ColBERT such as ColBERTv2 attempt to partially address this issue. It is also possible to use late interaction retrieval only as a refinement step on top of a prior retrieval system. An additional potential concern is a bias toward longer CVs or project descriptions with a lot of prose text: less relevant tokens can coincidentally produce a higher similarity score. This is visible in the figure, for example, in the score of the token "eine" ("a"), which carries no inherent relevance. These are reasons not to stop at late interaction retrieval alone, but to incorporate additional methods as well.

Cross-Encoders

Both standard single-vector embedding methods and multi-vector embedding methods encode all document segments in the database independently of the queries. Only after all embedding vectors for the project descriptions from the CVs and the job descriptions have been generated can they be compared via vector similarity computation. This naturally raises the question: why not score a project directly in relation to the job description? This is where cross-encoders come in.

Cross-encoders compare a job description and a CV—or a project within a CV—simultaneously by passing both through a neural network together in encoded form. On this basis, they produce a score that also reflects similarity, but one in which both texts have been part of the process from the start, rather than having their separate outputs compared after the fact. This raises the expectation that cross-encoders can achieve greater accuracy than embeddings that are first generated independently and then compared. After all, this approach allows fine-grained relationships between the job description and the CV project to be captured more effectively.

How Well Cross-Encoders Performed in the Use Case

In the AI staffing project, cross-encoders produced plausible results, but late interaction retrieval was more precise in our case. That said, this outcome depends on the specific pretrained models used and the requirements of the problem at hand. The strong reliance on keywords in this particular use case also helps explain this result. Many other use cases may favor cross-encoders, and architectures that combine both approaches are often a compelling option.

Why Cross-Encoders Are Typically Applied in a Second Step

The often higher accuracy of cross-encoders comes at a cost: they are significantly more computationally intensive. They do not scale linearly with input length but often quadratically. Additionally, the joint encoding must be repeated for every job description and CV project pair. For these reasons, cross-encoders are best suited as a refinement step—a reranker—applied after an initial candidate selection produced by a single-vector embedding search (bi-encoder) or late interaction retrieval. In this setup, the number of scoring operations required from the cross-encoder is kept manageable.

It is also advisable to limit the length of input texts (the job description and the project description from the CV) when using cross-encoders, for performance reasons. Some cross-encoders can only handle comparatively short inputs due to limited context lengths. Since shorter texts reduce runtime and help stay within length constraints, this is one of the motivations that leads us directly to the next approach.

Structured LLM-Based Text Summarization and Information Extraction

LLMs offer advantages over cross-encoders not only in terms of context length. They can also condense CVs qualitatively in ways that enable downstream models to work more precisely:

• Shorter CVs or CV project descriptions increase the effectiveness of models operating on them, both by improving efficiency (less text as input) and by reducing the associated costs.
• Accuracy can be improved because the focus stays on what matters and unnecessary filler is removed.
• A consistent language and structure improves comparability across different CVs, which also leads to higher accuracy.
• With good prompt engineering, summaries can be tailored specifically to the use case—for example, to typical requirements in data, AI, and consulting contexts.
• Alongside the short text descriptions, important keywords, ratings, and similar information can also be extracted.
• Frontier LLMs with high reasoning capacity, such as those from OpenAI, can reliably distill complex relationships, analogies, and implicit experience. Since these summaries only need to be generated at certain intervals (following CV updates) and can be cached, the costs remain manageable.

How LLM-as-a-Judge Ensures Quality

To further improve quality, a second LLM can be deployed as a judge in a subsequent step. This model also takes on the persona of HR and subject-matter experts, compares the generated summaries against the original CVs, evaluates them based on defined criteria, and refines them where necessary. This makes it easier to identify missing information, unclear phrasing, or hallucinations. During this step, for instance, it became apparent that certain keywords were missing from some project descriptions even in the original CVs (the raw data), and these were automatically added based on the free-text content.

Since the results of these summaries feed into downstream ranking LLMs as input to improve the final outputs, this process can also be viewed as a form of context engineering.

Technical Implementation with Structured Outputs

The Instructor Python package is a natural fit for this implementation. It enables high-quality summarization and extraction into a structured format—for example, JSON objects with predefined fields—which can in turn serve as the basis for building a NoSQL database such as MongoDB. Corresponding data classes can be defined for CVs and projects, with descriptions for each field (such as total years of experience in a given domain or with a specific skill) that the LLMs are instructed to populate. The package is built on Pydantic validation to ensure consistent types. If an LLM call violates a schema, automated retries can be triggered to maintain high quality.

The figure below shows an example of what such summaries and extractions can look like. This approach also provides the flexibility to apply downstream methods—such as late interaction retrievers, cross-encoders, and others—to any of the other fields; for example, keyword-based scoring based on years of experience with specific methods or technologies. It is also possible to generate summaries for the job descriptions themselves and to operate on those as well.

‍

LLM-Based Reranking of Candidate Lists

LLMs are useful not only for summarization and extraction but also for a final step in the pipeline: reranking.

Why can LLMs be particularly powerful as rerankers? While cross-encoders, as discussed, consider both the job description and a CV (or a project description from a CV, or another field from the CV summary of a given candidate), this is limited to one such CV document at a time. LLM-based rerankers that operate at the list level allow us to go one step further. Rather than passing a single CV document to the model for scoring against a job description, it is possible to pass a larger list of potential candidates. This gives the model more context and allows it to evaluate candidates not in isolation but in direct comparison with one another—which can meaningfully improve ranking quality.

In addition, supplementary contextual information can be passed to the LLM. This further improves quality because state-of-the-art LLMs, trained on vast amounts of parameters and data, are often better equipped to understand complex relationships. For example, they can assess experience with key technologies in relation to different projects that varied in duration and relevance to that technology.

Advantages and Disadvantages of Reranking

LLMs can also explain their rankings. The LLM reranker is instructed to rank according to defined criteria and is provided with a job description alongside a set of relevant CV documents, which it must rank by relevance and justify its approach. This increases transparency and makes expert review and interpretability easier. LLM-as-a-judge can also be applied here.

The disadvantages of LLM-based reranking are higher latency, greater cost, and potential context window limitations.

For these reasons, LLM-based reranking is generally best suited as a final step applied to an already narrowed-down preselection, rather than to the full pool of candidates. This keeps both runtime and costs under control.

What a Multi-Stage Retrieval Pipeline Can Look Like

In practice, such a pipeline can be structured in four steps:

1. Preprocessing and automated LLM summarization
2. Initial candidate selection using a late interaction retrieval system (optionally preceded by standard embeddings)
3. Refinement of the larger candidate pool using a cross-encoder
4. Final LLM-based reranking

The list produced in the final step identifies which candidates best match a given job description and explains why. The figure below shows an example of what such a final result can look like.

‍