diff --git a/python-recipes/vector-search/02_hybrid_search.ipynb b/python-recipes/vector-search/02_hybrid_search.ipynb index 5eaaec9..f49781f 100644 --- a/python-recipes/vector-search/02_hybrid_search.ipynb +++ b/python-recipes/vector-search/02_hybrid_search.ipynb @@ -1,43 +1,47 @@ { "cells": [ { - "cell_type": "markdown", "metadata": {}, + "cell_type": "markdown", "source": [ "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n", "# Implementing hybrid search with Redis\n", "\n", "Hybrid search is all about combining lexical search with semantic vector search to improve result relevancy. This notebook will cover 3 different hybrid search strategies with Redis:\n", "\n", - "1. Linear combination of scores from lexical search (BM25) and vector search (Cosine Distance) with the AggregateHybridQuery class\n", - "2. Client-Side Reciprocal Rank Fusion (RRF)\n", + "1. Linear combination of scores from lexical search (BM25) and vector search (Cosine Distance)\n", + "2. Reciprocal Rank Fusion (RRF)\n", "3. Client-Side Reranking with a cross encoder model\n", "\n", - ">Note: Additional work is planed within Redis Query Engine core to add more flexible hybrid search capabilities in the future.\n", + "The Redis Query Engine supports a unified interface for hybrid search with the [FT.HYBRID](https://redis.io/docs/latest/commands/ft.hybrid) command introduced in Redis Open Source 8.4.0, prior to which hybrid searches were only possible using [the aggregations API](https://redis.io/docs/latest/develop/ai/search-and-query/advanced-concepts/aggregations/). RedisVL added an interface for FT.HYBRID in 0.13.0 (via `HybridQuery`), and provided an interface for the aggregation approach for Redis prior to 8.4.0 (via `AggregateHybridQuery`). This notebook will demonstrate the usage of both approaches.\n", + "\n", + "## Requirements\n", + "- Redis 8.4.0+\n", + "- redisvl>=0.13.0\n", + "- redispy>=7.1.0\n", "\n", "## Let's Begin!\n", "\"Open\n" ] }, { - "cell_type": "markdown", "metadata": {}, - "source": [ - "### Install Packages" - ] + "cell_type": "markdown", + "source": "### Install Packages" }, { - "cell_type": "code", - "execution_count": null, "metadata": {}, + "cell_type": "code", "outputs": [], + "execution_count": null, "source": [ - "%pip install sentence-transformers pandas nltk \"redisvl>=0.11.0\"" + "# TODO: Install redisvl from PyPi when 0.13.0 is released\n", + "%pip install \"git+https://github.com/redis/redis-vl-python.git\" nltk pandas sentence-transformers" ] }, { - "cell_type": "markdown", "metadata": {}, + "cell_type": "markdown", "source": [ "### Data/Index Preparation\n", " \n", @@ -55,11 +59,9 @@ ] }, { - "cell_type": "markdown", "metadata": {}, - "source": [ - "Running remotely or in collab? Run this cell to download the necessary dataset." - ] + "cell_type": "markdown", + "source": "Running remotely or in collab? Run this cell to download the necessary dataset." }, { "metadata": {}, @@ -89,7 +91,7 @@ "metadata": {}, "cell_type": "code", "outputs": [], - "execution_count": 20, + "execution_count": null, "source": [ "# NBVAL_SKIP\n", "%%sh\n", @@ -116,6 +118,25 @@ "3. With docker: `docker run -d --name redis -p 6379:6379 redis:latest`" ] }, + { + "metadata": {}, + "cell_type": "code", + "outputs": [], + "execution_count": null, + "source": [ + "from packaging.version import Version\n", + "\n", + "from redis import __version__ as redis_version\n", + "from redisvl import __version__ as redisvl_version\n", + "\n", + "\n", + "if Version(redis_version) < Version(\"7.1.0\"):\n", + " raise RuntimeError(\"redis-py version must be >= 7.1.0\")\n", + "\n", + "if Version(redisvl_version) < Version(\"0.13.0\"):\n", + " raise RuntimeError(\"redisvl version must be >= 0.13.0\")" + ] + }, { "metadata": {}, "cell_type": "markdown", @@ -146,40 +167,35 @@ ] }, { - "cell_type": "markdown", "metadata": {}, - "source": [ - "### Create redis client, load data, generate embeddings" - ] + "cell_type": "markdown", + "source": "### Create redis client, load data, generate embeddings" }, { - "cell_type": "code", - "execution_count": 2, "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "True" - ] - }, - "execution_count": 2, - "metadata": {}, - "output_type": "execute_result" - } - ], + "cell_type": "code", + "outputs": [], + "execution_count": null, "source": [ "from redis import Redis\n", + "from redisvl.redis.connection import RedisConnectionFactory\n", "\n", "client = Redis.from_url(REDIS_URL)\n", - "client.ping()" + "client.ping()\n", + "\n", + "if Version(client.info()[\"redis_version\"]) < Version(\"8.4.0\"):\n", + " raise RuntimeError(\"Redis version must be >= 8.4.0\")\n", + "\n", + "installed_modules = RedisConnectionFactory.get_modules(client)\n", + "if \"search\" not in installed_modules:\n", + " raise RuntimeError(\"Redisearch module is not installed\")" ] }, { - "cell_type": "code", - "execution_count": 3, "metadata": {}, + "cell_type": "code", "outputs": [], + "execution_count": null, "source": [ "import json\n", "\n", @@ -188,313 +204,10 @@ ] }, { - "cell_type": "code", - "execution_count": 5, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "19:18:27 sentence_transformers.SentenceTransformer INFO Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n", - "19:18:27 sentence_transformers.SentenceTransformer INFO Use pytorch device_name: mps\n" - ] - }, - { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "6f1ad2e22cde435fa05bcfe2d4de40ae", - "version_major": 2, - "version_minor": 0 - }, - "text/plain": [ - "Batches: 0%| | 0/1 [00:00\\x17\\xbeA\\x1e\\x05\\xb9Hu\\xbfg3\\xbd$\\xcd\\xbd\\xbd\\xa1$\\xf7;\\x04\\xf5z=\\xfc\\xb4\\x8c=\\x89\\x0e\\xc6\\xbdhI\\x90\\xbd^\\x16\\xbd;z\\xe7\\x0c\\xbd\\x1b3\\xc9\\xbc\\x89\\xf8\\xbb\\xbc\\x18\\'u\\xbb>\\x8f\\xca<\\x02\\x80J=\\x0e\\xaf*=\\x8dOU\\xbd\\xcf\\xf0\\x95\\xbc \\x02\\x19=\\x19\\xf4K<\\xc5\\xc2\\t=J\\x83\\xac=\\x95\\xd7\\xb8\\xbd\\xf2\\xb5\\x9c\\xbd=\\x85\\x18=\\x94d&=03\\xf8<\\xee\\xf7\\x88<\\x80v\\xf2\\xbb9=[\\xbdG\\xac\\xee\\xbb<:A\\xbd\\xe1d\\x19\\xbd!d\\xf2\\xbb\\x1d\\xbax;\\xec;O<\\xd21,\\xbc\\xec\\xae\\xae=r\\x00-\\xbc\"\\x06\\xae\\xbdl\\xd6\\x1a=\\xc4\\xbf\\xcd=\\x19\\x150=\\xe3\\xf1\\x9d\\xbc\\xa6GK=\\xb2\\xb8 =\\xb2\\xf1I\\xbd-e\\x9e\\xbb\\xe9\\x8a\\xf7:\\x88\\xf8\\x1c=\\x7f\\xba\\xde<\\xd2n\\x16\\xbb\\xb4\\\\p\\xbb\\xd4\\xd5<<\\x89\\xa5\\xa3\\xb8\\xc79s<=4&<\\x84\\x1c\\x18<\\x18\\xd9-\\xbd\\xdf\\xe6\\x98<\\x15\\xa1N=\\xa2/\\xa5=\\x1d\\xf3\\xdd<\\x17L\\x13<\\x10\\x10\\xce\\xbac\\x9e\\xdc\\xbc\\xa68\\x05=+\\xa1\\xf5\\xbd\\x84\\x1bF\\xbd\\xa0?\\x14\\xbe\\xc4\\x8f(\\xbd\\xe6O\\x89\\xbd\\xf7\\xad\\xd4<\\xa7\\x12\\xc3=\\xaf\\x05O\\xbd\\x99\\x8ep\\xbc\\x18\\xb5\\xac\\xbc\\xc9\\x9ee\\xbdH\\x8es;$a\\xc1;\\xd9\\xfaB\\xbd\\xa8#\\xfe:\\x92\\xe6\\xf4=\\xcd\\x15*<\\x86\\xf8\\x1b=\\x01\\xfcV\\xbd\\xd3\\xd1\\r=9\\xee\\x06=\\x13u\\xba\\xbd\\xf7\\xa3\\xd6<\\x1a\\xec\\xd9;\\xb79/=\\xa4\\xc2\\x85=p\\x0b\"=\\xe1i\\xef<:\\xe8c=\\xfb2\\x08\\xbe\\xce\\x12;=OVW;V\\xa4b<\\xd0\\x9d\\xb7<\\x87r;\\xbdqz\\x91\\xbcV\\x00<\\xbd\\xfe\\x19\\xa3<\\xeaJ%\\xbc!\\xe7\\xbf\\xbb\\x7f\\x87\\x12=\\x94\\x1d\\x95=b|\\xfd\\xbc\\xf3\\xf1\\xd1\\xbd\\xf5y\\x84;\\xc9\\tu=]\\x8ai<3\\x91R\\xbd\\xec\\xf3m\\xbd\\x93\\xb83=V\\xedF=\\x1f\\xf3\\xd1\\x08yA\\xba<#\\xacO\\xbd\\x01\\x0f\\xc7;\\x7f\\xf4\\x04\\xbdP\\x82\\x92\\xbd\\x9b\\xddD=p\\xd8;\\xbc\\xd3;\\xf4\\xbc\\xb3\\x8f\\x97\\xbd1\\\\\\r\\xbd\\xea\\x8c\\xf5\\xbd\\x8c\\x13(=\\x9e\\xc8\\xc6=\\xa3\\xed\\x1a=\\x98\\xa8\\xf8=\\x84\\xc1\\xee\\xbc\\xcd-\\x18\\xbb\\xf5~;<\\xd6F\\t\\xbd\\x14\\x08\\x17=\\xa5\\xa5\\x1e=\\x14K\\xcb\\xbd.\\xf7\\x8c\\xbdyb\\xed\\xbb\\x86[\\x19\\xbc]\\x0c\\x13\\xbcgq\\x83=\\xf0wd\\xbd\\xe3\\xc7\\xd1\\xbb8lY\\xbc\\xa7|a=3\\xcf\\xfd\\xbc\\x1f\\xa5\\x83\\xbb\\x99O\\x19\\xbd6\\x02]\\xbd\\xbb\\xeaz=\\x036\\x9c=:^\\xa9\\xbd)^9\\xbcg\\xe4N\\xbcs\\x07x\\xbd\\x18{\\xa0=:\\x9f\\x96<\\xecq8\\xba\\x9e\\xbb=\\xbd\\xe4|(<\\x96\\xdf\\xb4\\xbbl\\xc9\\x0b\\xbd\\xc4\\x01\\x95\\xbd\\xf7\\xc6T=\\tp\\xd1\",\n", + " vector=,\n", + " vector_field_name=\"\",\n", + ")\n", + "```\n", + "\n", + "This defaults to using the reciprocal rank fusion (RRF) method to combine scores, and only outputs the final keys and combined scores. A more common minimal usage might be:\n", + "\n", + "```python\n", + "query = HybridQuery(\n", + " text=\"your query string here\",\n", + " text_field_name=\"\",\n", + " vector=,\n", + " vector_field_name=\"\",\n", + " combination_method=\"RRF\",\n", + " rrf_window=20,\n", + " yield_text_score_as=\"text_score\",\n", + " yield_vsim_score_as=\"vector_similarity\",\n", + " yield_combined_score_as=\"hybrid_score\",\n", + " return_fields=[\"\"],\n", + ")\n", + "```" ] }, { - "cell_type": "markdown", "metadata": {}, + "cell_type": "markdown", "source": [ - "## 1. Linear Combination using AggregateHybridQuery\n", + "## 1. Linear Combination\n", "\n", - "The goal of this technique is to calculate a weighted sum of the text similarity score for our provided text search and the cosine distance between vectors calculated via a KNN vector query. Under the hood this is possible in Redis using the [aggregations API](https://redis.io/docs/latest/develop/interact/search-and-query/advanced-concepts/aggregations/), as of `Redis 7.4.x` (search version `2.10.5`), within a single database call.\n", + "The goal of this technique is to calculate a weighted sum of the text similarity score for our provided text search and the vector similarity score for our provided vector.\n", "\n", - "As of RedisVl 0.5.0 all of this is nicely encapsulated in your `AggregateHybridQuery` class, which behaves much like our other query classes." + "The FT.HYBRID API introduced in Redis 8.4.0 supports a linear combination of text and vector scores (accessible as of RedisVL 0.13.0 in `HybridQuery`), and it is also possible with the aggregations API, as of `Redis 7.4.x` (search version `2.10.5` - accessible as of RedisVl 0.5.0 in `AggregateHybridQuery`)." ] }, { - "cell_type": "code", - "execution_count": 9, "metadata": {}, + "cell_type": "code", "outputs": [], + "execution_count": null, "source": [ "# Sample user query (can be changed for comparisons)\n", "user_query = \"action adventure movie with great fighting scenes against a dangerous criminal, crime busting, superheroes, and magic\"" ] }, { - "cell_type": "markdown", "metadata": {}, + "cell_type": "code", + "outputs": [], + "execution_count": null, "source": [ - "First, we will import our `AggregateHybridQuery` and understand its parameters.\n", - "At a minimum, the `AggregateHybridQuery` needs 4 arguments:\n", - "```python\n", - "query = AggregateHybridQuery(\n", - " text = \"your query string here\",\n", - " text_field_name = \"\",\n", - " vector = ,\n", - " vector_field_name = \"\",\n", + "import pandas as pd\n", + "\n", + "from redisvl.query.hybrid import HybridQuery\n", + "\n", + "vector = model.embed(user_query, as_buffer=True)\n", + "\n", + "query = HybridQuery(\n", + "\ttext=user_query,\n", + "\ttext_field_name=\"description\",\n", + "\tvector=vector,\n", + "\tvector_field_name=\"description_vector\",\n", + "\tcombination_method=\"LINEAR\",\n", + "\tyield_text_score_as=\"text_score\",\n", + "\tyield_vsim_score_as=\"vector_similarity\",\n", + "\tyield_combined_score_as=\"hybrid_score\",\n", + "\treturn_fields=[\"title\"],\n", ")\n", - "```" + "\n", + "results = index.query(query)\n", + "pd.DataFrame(results[:3])" ] }, { - "cell_type": "code", - "execution_count": 11, "metadata": {}, - "outputs": [ - { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "eef2a2e2bf504bbb95caea19bb8c4705", - "version_major": 2, - "version_minor": 0 - }, - "text/plain": [ - "Batches: 0%| | 0/1 [00:00[KNN 10 @description_vector $vector AS vector_distance]'" - ] - }, - "execution_count": 13, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "query._build_query_string()" - ] - }, - { "cell_type": "markdown", - "metadata": {}, "source": [ "### Choosing your stopwords for better queries\n", "You can see that the user query string has been tokenized and certain stopwords like 'and', 'for', 'with', 'but', have been removed, otherwise you would get matches on irrelevant words.\n", "RedisVL uses [NLTK](https://www.nltk.org/index.html) english stopwords as the the default. You can change which default language stopwords to use with the `stopwords` argument.\n", - "You specify a language, like 'german', 'arabic', 'greek' and many others, provide your own list of stopwords, or set it to `None` to not remove any." + "You specify a language, like 'german', 'arabic', 'greek' and many others, provide your own list of stopwords, or set it to `None` to not remove any.\n", + "\n", + "Note that both `HybridQuery` and `AggregateHybridQuery` process stopwords identically." ] }, { - "cell_type": "code", - "execution_count": 24, "metadata": {}, - "outputs": [ - { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "3e4537950607485cb399928dd7bc0c04", - "version_major": 2, - "version_minor": 0 - }, - "text/plain": [ - "Batches: 0%| | 0/1 [00:00[KNN 10 @description_vector $vector AS vector_distance]\n", - "(~@description:(action | adventure | movie | great | fighting | scenes | against | dangerous | criminal | crime | busting | superheroes | magic))=>[KNN 10 @description_vector $vector AS vector_distance]\n", - "(~@description:(action | adventure | movie | with | great | fighting | scenes | against | a | dangerous | criminal | crime | busting | superheroes | and | magic))=>[KNN 10 @description_vector $vector AS vector_distance]\n" - ] - } - ], + "cell_type": "code", + "outputs": [], + "execution_count": null, "source": [ "# translate our user query to French and use nltk french stopwords\n", "french_query_text = \"Film d'action et d'aventure avec de superbes scènes de combat, des enquêtes criminelles, des super-héros et de la magie\"\n", "\n", - "french_film_query = AggregateHybridQuery(\n", + "french_film_query = HybridQuery(\n", " text=french_query_text,\n", " text_field_name=\"description\",\n", " vector=model.embed(french_query_text, as_buffer=True),\n", @@ -838,7 +447,7 @@ " stopwords=\"french\",\n", ")\n", "\n", - "print(french_film_query._build_query_string())\n", + "print(french_film_query.query._search_query.query_string())\n", "\n", "# specify your own stopwords\n", "custom_stopwords = set([\n", @@ -847,7 +456,7 @@ " \"then\", \"there\", \"these\", \"they\", \"this\", \"to\", \"was\", \"will\", \"with\"\n", "])\n", "\n", - "stopwords_query = AggregateHybridQuery(\n", + "stopwords_query = HybridQuery(\n", " text=user_query,\n", " text_field_name=\"description\",\n", " vector=vector,\n", @@ -855,10 +464,10 @@ " stopwords=custom_stopwords,\n", ")\n", "\n", - "print(stopwords_query._build_query_string())\n", + "print(stopwords_query.query._search_query.query_string())\n", "\n", "# don't use any stopwords\n", - "no_stopwords_query = AggregateHybridQuery(\n", + "no_stopwords_query = HybridQuery(\n", " text=user_query,\n", " text_field_name=\"description\",\n", " vector=vector,\n", @@ -866,93 +475,115 @@ " stopwords=None,\n", ")\n", "\n", - "print(no_stopwords_query._build_query_string())" + "print(no_stopwords_query.query._search_query.query_string())" ] }, { - "cell_type": "markdown", "metadata": {}, + "cell_type": "markdown", "source": [ "### Choosing your text scoring function and weights\n", - "There are different ways to calculate the similarity between sets of text. Redis supports several, such as `BM25`, `TFIDF`, `DISMAX`, and others. The default is `BM25STD` and is easy to configure with the `text_scorer` parameter. Just like changing you embedding model can change your vector similarity scores, changing your text similarity measure can change your text scores.\n", + "There are different ways to calculate the similarity between sets of text. Options for text scoring functions are TFIDF, TFIDF.DOCNORM, BM25STD, BM25STD.NORM, BM25STD.TANH, DISMAX, DOCSCORE, and HAMMING; the default is BM25STD and is easy to configure with the `text_scorer` parameter. Just like changing you embedding model can change your vector similarity scores, changing your text similarity measure can change your text scores.\n", + "\n", + "> For more information about supported scoring algorithms, see [the Redis documentation on scoring](https://redis.io/docs/latest/develop/ai/search-and-query/advanced-concepts/scoring/).\n", + "\n", + "When combining text and vector scores using a linear combination (`combination_method=\"LINEAR\"` in `HybridQuery` and the only option for `AggregateHybridQuery`), you can control the relative balance of these scores with tunable parameters.\n", + "\n", + "The FT.HYBRID API calculates the combined score as:\n", + "\n", + "```python\n", + "hybrid_score = {alpha} * text_score + {beta} * vector_similarity\n", + "```\n", "\n", - "Because hybrid queries are performing a weighted average of text similarity and vector similarity you also control the relative balance of these scores with the `alpha` parameter.\n", + "Where `alpha` can be provided to `HybridQuery` via the `linear_alpha` and `beta` is calculated as `1 - alpha`. FT.HYBRID defaults to `alpha=0.3`.\n", "\n", - "The documents are ranked based on the hybrid score which is computed as:\n", + "`AggregateHybridQuery` defines the combined score in reverse as:\n", "\n", "```python\n", "hybrid_score = {1-alpha} * text_score + {alpha} * vector_similarity\n", "```\n", "\n", - "Try changing the `text_scorer` and `alpha` parameters in the query below to see how results may change.\n" + "Where the `alpha` parameter is configurable on the `AggregateHybridQuery` class. If not specified, it defaults to `0.7`.\n", + "\n", + "Try changing the `text_scorer` and `linear_alpha` parameters in the query below to see how results may change." ] }, { - "cell_type": "code", - "execution_count": 25, "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "[{'vector_distance': '0.645975351334',\n", - " 'title': 'The Incredibles',\n", - " 'description': \"A family of undercover superheroes, while trying to live the quiet suburban life, are forced into action to save the world. Bob Parr (Mr. Incredible) and his wife Helen (Elastigirl) were among the world's greatest crime fighters, but now they must assume civilian identities and retreat to the suburbs to live a 'normal' life with their three children. However, the family's desire to help the world pulls them back into action when they face a new and dangerous enemy.\",\n", - " 'vector_similarity': '0.677012324333',\n", - " 'text_score': '8',\n", - " 'hybrid_score': '6.16925308108'},\n", - " {'vector_distance': '0.653376042843',\n", - " 'title': 'The Dark Knight',\n", - " 'description': 'Batman faces off against the Joker, a criminal mastermind who threatens to plunge Gotham into chaos.',\n", - " 'vector_similarity': '0.673311978579',\n", - " 'text_score': '8',\n", - " 'hybrid_score': '6.16832799464'},\n", - " {'vector_distance': '0.608649373055',\n", - " 'title': 'Explosive Pursuit',\n", - " 'description': 'A daring cop chases a notorious criminal across the city in a high-stakes game of cat and mouse.',\n", - " 'vector_similarity': '0.695675313473',\n", - " 'text_score': '6',\n", - " 'hybrid_score': '4.67391882837'}]" - ] - }, - "execution_count": 25, - "metadata": {}, - "output_type": "execute_result" - } - ], + "cell_type": "code", + "outputs": [], + "execution_count": null, "source": [ - "tfidf_query = AggregateHybridQuery(\n", + "tfidf_query = HybridQuery(\n", " text=user_query,\n", " text_field_name=\"description\",\n", " vector=vector,\n", " vector_field_name=\"description_vector\",\n", " text_scorer=\"TFIDF\", # can be one of [TFIDF, TFIDF.DOCNORM, BM25, DISMAX, DOCSCORE, BM25STD]\n", " stopwords=None,\n", - " alpha=0.25, # weight the vector score lower\n", + "\tcombination_method=\"LINEAR\",\n", + " linear_alpha=0.75, # weight the text score higher\n", " return_fields=[\"title\", \"description\"],\n", + "\tyield_text_score_as=\"text_score\",\n", + " yield_vsim_score_as=\"vector_similarity\",\n", + " yield_combined_score_as=\"hybrid_score\",\n", ")\n", "\n", "results = index.query(tfidf_query)\n", - "\n", - "results[:3]" + "pd.DataFrame(results[:3])" ] }, { - "cell_type": "markdown", "metadata": {}, + "cell_type": "markdown", "source": [ - "## 2. Client-side fusion with RRF\n", + "## 2. Reciprocal Rank Fusion (RRF)\n", "\n", "Instead of relying on document scores like cosine similarity and BM25/TFIDF, we can fetch items and focus on their rank. This rank can be utilized to create a new ranking metric known as [Reciprocal Rank Fusion (RRF)](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf). RRF is powerful because it can handle ranked lists of different length, scores of different scales, and other complexities.\n", "\n", - "Although Redis does not currently support RRF natively, we can easily implement it on the client side." + "The FT.HYBRID API introduced in Redis 8.4.0 supports using RRF to combine results from text and vector queries (accessible as of RedisVL 0.13.0 in `HybridQuery`). Unless otherwise specified, RRF is the default combination method.\n", + "\n", + "The parameters available to customize the behaviour of RRF are `rrf_window` and `rrf_constant`. The `rrf_window` parameter controls the size of the window over which the RRF score is calculated, and the `rrf_constant` parameter controls the constant used in the RRF formula. Try changing these parameters to see how results may change." ] }, { + "metadata": {}, "cell_type": "code", - "execution_count": 26, + "outputs": [], + "execution_count": null, + "source": [ + "query = HybridQuery(\n", + "\ttext=user_query,\n", + "\ttext_field_name=\"description\",\n", + "\tvector=vector,\n", + "\tvector_field_name=\"description_vector\",\n", + "\tcombination_method=\"RRF\",\n", + "\trrf_window=20,\n", + "\trrf_constant=60,\n", + "\tyield_text_score_as=\"text_score\",\n", + "\tyield_vsim_score_as=\"vector_similarity\",\n", + "\tyield_combined_score_as=\"hybrid_score\",\n", + "\treturn_fields=[\"title\", \"description\"],\n", + ")\n", + "\n", + "results = index.query(query)\n", + "pd.DataFrame(results[:3])" + ] + }, + { "metadata": {}, + "cell_type": "markdown", + "source": [ + "### Client-side RRF for older Redis versions\n", + "\n", + "When using Redis versions prior to 8.4.0, you can still perform RRF by fetching the top-k results from both the text and vector queries, and then fusing them together on the client-side." + ] + }, + { + "metadata": {}, + "cell_type": "code", "outputs": [], + "execution_count": null, "source": [ "def fuse_rankings_rrf(*ranked_lists, weights=None, k=60):\n", " \"\"\"\n", @@ -977,45 +608,25 @@ ] }, { - "cell_type": "code", - "execution_count": 27, "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "[(2, 0.04814747488101534),\n", - " (1, 0.032266458495966696),\n", - " (6, 0.03200204813108039),\n", - " (5, 0.01639344262295082),\n", - " (4, 0.016129032258064516),\n", - " (3, 0.015873015873015872),\n", - " (7, 0.015625),\n", - " (8, 0.015384615384615385)]" - ] - }, - "execution_count": 27, - "metadata": {}, - "output_type": "execute_result" - } - ], + "cell_type": "code", + "outputs": [], + "execution_count": null, "source": [ "# Below is a simple example of RRF over a few lists of numbers\n", "fuse_rankings_rrf([1, 2, 3], [2, 4, 6, 7, 8], [5, 6, 1, 2])" ] }, { - "cell_type": "markdown", "metadata": {}, - "source": [ - "We'll want some helper functions to construct our individual text and vector queries" - ] + "cell_type": "markdown", + "source": "We'll want some helper functions to construct our individual text and vector queries" }, { - "cell_type": "code", - "execution_count": 28, "metadata": {}, + "cell_type": "code", "outputs": [], + "execution_count": null, "source": [ "# Function to create a vector query using RedisVL helpers for ease of use\n", "from redisvl.query import VectorQuery, TextQuery\n", @@ -1047,10 +658,10 @@ ] }, { - "cell_type": "code", - "execution_count": 29, "metadata": {}, + "cell_type": "code", "outputs": [], + "execution_count": null, "source": [ "from typing import List, Dict, Any\n", "\n", @@ -1081,94 +692,30 @@ ] }, { - "cell_type": "code", - "execution_count": 30, "metadata": {}, - "outputs": [ - { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "848faeb9dbfe4150917d407dfe865e92", - "version_major": 2, - "version_minor": 0 - }, - "text/plain": [ - "Batches: 0%| | 0/1 [00:00 List[Dict[str, Any]]:\n", + "from typing import Tuple\n", "\n", - " query = AggregateHybridQuery(\n", - " text,\n", - " text_field_name=\"description\",\n", - " vector=model.embed(text, as_buffer=True),\n", - " vector_field_name=\"description_vector\",\n", - " text_scorer=\"BM25\",\n", - " stopwords=\"english\",\n", - " alpha=alpha,\n", - " return_fields=[\"title\", \"hybrid_score\"],\n", + "\n", + "def hybrid_query(text, num_results: int, **kwargs) -> List[Tuple[str, float]]:\n", + "\n", + " query = HybridQuery(\n", + "\t\ttext,\n", + "\t\ttext_field_name=\"description\",\n", + "\t\tvector=model.embed(text, as_buffer=True),\n", + "\t\tvector_field_name=\"description_vector\",\n", + "\t\tstopwords=\"english\",\n", + "\t\tnum_results=num_results,\n", + "\t\treturn_fields=[\"title\"],\n", + "\t\tyield_combined_score_as=\"hybrid_score\",\n", + "\t\t**kwargs,\n", " )\n", "\n", - " results = index.query(query)\n", + " results = index.query(query)\n", "\n", " return [\n", " (\n", @@ -1358,1021 +901,59 @@ ] }, { - "cell_type": "code", - "execution_count": 36, "metadata": {}, + "cell_type": "code", "outputs": [], + "execution_count": null, "source": [ "import pandas as pd\n", "\n", "\n", "rankings = pd.DataFrame()\n", - "rankings[\"queries\"] = movie_user_queries\n", + "rankings[\"query\"] = movie_user_queries\n", "\n", "# First, add new columns to the DataFrame\n", "rankings[\"hf-cross-encoder\"] = \"\"\n", "rankings[\"rrf\"] = \"\"\n", - "rankings[\"linear-combo-bm25-cosine\"] = \"\"" + "rankings[\"linear\"] = \"\"" ] }, { - "cell_type": "code", - "execution_count": 37, "metadata": {}, - "outputs": [ - { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "c562a6abb1eb47a982891fb9d6c9fc99", - "version_major": 2, - "version_minor": 0 - }, - "text/plain": [ - "Batches: 0%| | 0/1 [00:00\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
querieshf-cross-encoderrrflinear-combo-bm25-cosine
0I'm in the mood for a high-rated action movie ...[(Mad Max: Fury Road, -11.244140625), (Toy Sto...[(The Incredibles, 0.016029143897996357), (Toy...[(The Incredibles, 0.552392188297), (Toy Story...
1What's a funny animated film about unlikely fr...[(Despicable Me, -10.441909790039062), (The In...[(Monsters, Inc., 0.015524093392945852), (Mada...[(Monsters, Inc., 0.507448260638), (Madagascar...
2Any movies featuring superheroes or extraordin...[(The Incredibles, -3.6648080348968506), (The ...[(The Incredibles, 0.01639344262295082), (The ...[(The Incredibles, 0.688644165103), (The Aveng...
3I want to watch a thrilling movie with spies o...[(Inception, -10.843631744384766), (The Incred...[(Inception, 0.015524093392945852), (Skyfall, ...[(Inception, 0.504883907887), (Skyfall, 0.4438...
4Are there any comedies set in unusual location...[(The Incredibles, -11.45376968383789), (Findi...[(Finding Nemo, 0.015524093392945852), (Madaga...[(Finding Nemo, 0.503574235889), (Madagascar, ...
\n", - "" - ], - "text/plain": [ - " queries \\\n", - "0 I'm in the mood for a high-rated action movie ... \n", - "1 What's a funny animated film about unlikely fr... \n", - "2 Any movies featuring superheroes or extraordin... \n", - "3 I want to watch a thrilling movie with spies o... \n", - "4 Are there any comedies set in unusual location... \n", - "\n", - " hf-cross-encoder \\\n", - "0 [(Mad Max: Fury Road, -11.244140625), (Toy Sto... \n", - "1 [(Despicable Me, -10.441909790039062), (The In... \n", - "2 [(The Incredibles, -3.6648080348968506), (The ... \n", - "3 [(Inception, -10.843631744384766), (The Incred... \n", - "4 [(The Incredibles, -11.45376968383789), (Findi... \n", - "\n", - " rrf \\\n", - "0 [(The Incredibles, 0.016029143897996357), (Toy... \n", - "1 [(Monsters, Inc., 0.015524093392945852), (Mada... \n", - "2 [(The Incredibles, 0.01639344262295082), (The ... \n", - "3 [(Inception, 0.015524093392945852), (Skyfall, ... \n", - "4 [(Finding Nemo, 0.015524093392945852), (Madaga... \n", - "\n", - " linear-combo-bm25-cosine \n", - "0 [(The Incredibles, 0.552392188297), (Toy Story... \n", - "1 [(Monsters, Inc., 0.507448260638), (Madagascar... \n", - "2 [(The Incredibles, 0.688644165103), (The Aveng... \n", - "3 [(Inception, 0.504883907887), (Skyfall, 0.4438... \n", - "4 [(Finding Nemo, 0.503574235889), (Madagascar, ... " - ] - }, - "execution_count": 38, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "rankings.head()" - ] + "cell_type": "code", + "outputs": [], + "execution_count": null, + "source": "rankings.head()" }, { - "cell_type": "code", - "execution_count": 39, "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "array(['Show me movies set in dystopian or post-apocalyptic worlds',\n", - " list([('Mad Max: Fury Road', -3.490626335144043), ('Despicable Me', -11.05152702331543), ('The Incredibles', -11.315656661987305), ('Finding Nemo', -10.880638122558594)]),\n", - " list([('The Incredibles', 0.01620835536753041), ('Finding Nemo', 0.013813068651778329), ('Mad Max: Fury Road', 0.011475409836065573), ('Madagascar', 0.01111111111111111)]),\n", - " list([('The Incredibles', '0.669360563015'), ('Mad Max: Fury Road', '0.452238592505'), ('Madagascar', '0.419015598297'), ('Despicable Me', '0.416218388081'), ('Skyfall', '0.411504265666'), ('The Avengers', '0.411210304499'), ('Black Widow', '0.410578405857'), ('The Lego Movie', '0.408463662863'), ('Monsters, Inc.', '0.392220947146'), ('Shrek', '0.390464794636')])],\n", - " dtype=object)" - ] - }, - "execution_count": 39, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "rankings.loc[12].values" - ] + "cell_type": "code", + "outputs": [], + "execution_count": null, + "source": "rankings.loc[12].T" }, { - "cell_type": "markdown", "metadata": {}, + "cell_type": "markdown", "source": [ "# Wrap up\n", "That's a wrap! Hopefully from this you were able to learn:\n", "- How to implement simple vector search queries in Redis\n", "- How to implement vector search queries with full-text filters\n", - "- How to implement hybrid search queries using the Redis aggregation API\n", + "- How to implement hybrid search queries using the Redis hybrid and aggregation APIs\n", "- How to perform client-side fusion and reranking techniques" ] }