{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "chBgl-RHFAKi" }, "source": [ "# Using Hypothetical Document Embeddings (HyDE) to Improve Retrieval\n", "\n", "> πŸ“š This cookbook has an accompanying article with a complete walkthrough [\"Optimizing Retrival with HyDE\"](https://haystack.deepset.ai/blog/optimizing-retrieval-with-hyde)\n", "\n", "In this coookbook, we are building Haystack components that allow us to easily incorporate HyDE into our RAG pipelines, to optimize retrieval.\n", "\n", "> To learn more about HyDE and when it's useful, check out our [guide to Hypothetical Document Embeddings (HyDE)](https://docs.haystack.deepset.ai/docs/hypothetical-document-embeddings-hyde)\n", "\n", "## Install Requirements" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "_ZIUJlvQCDvF", "outputId": "2879cabf-1d6e-49c4-ab60-17c2688f88ea" }, "outputs": [], "source": [ "!pip install haystack-ai sentence-transformers-haystack datasets" ] }, { "cell_type": "markdown", "metadata": { "id": "Kr6_UhFyFh-c" }, "source": [ "In the following sections, we will be using the `OpenAIChatGenerator`, so we need to provide our API key πŸ‘‡" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "J5l8zljUB1wc", "outputId": "228bc893-03a8-4185-f298-eedc309eccf9" }, "outputs": [], "source": [ "from getpass import getpass\n", "import os\n", "\n", "os.environ[\"OPENAI_API_KEY\"] = getpass(\"Enter your openAI key:\")" ] }, { "cell_type": "markdown", "metadata": { "id": "yiIk0Bt2GVbl" }, "source": [ "## Building a Pipeline for Hypothetical Document Embeddings\n", "\n", "We will build a Haystack pipeline that generates 'fake' documents.\n", "For this part, we are using the `OpenAIChatGenerator` with a `PromptBuilder` that instructs the model to generate paragraphs." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "id": "q7vs3ubjB7PB" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": "ChatPromptBuilder has 1 prompt variables, but `required_variables` is not set. By default, all prompt variables are treated as optional, which may lead to unintended behavior in multi-branch pipelines. To avoid unexpected execution, ensure that variables intended to be required are explicitly set in `required_variables`.\n" } ], "source": [ "from haystack.components.builders import ChatPromptBuilder\n", "from haystack.components.generators.chat import OpenAIChatGenerator\n", "from haystack.dataclasses import ChatMessage\n", "\n", "generator = OpenAIChatGenerator(\n", " model=\"gpt-4o-mini\",\n", " generation_kwargs={\"n\": 5, \"temperature\": 0.75, \"max_tokens\": 400},\n", ")\n", "\n", "template = [\n", " ChatMessage.from_user(\n", " \"\"\"Given a question, generate a paragraph of text that answers the question.\n", " Question: {{question}}\n", " Paragraph:\"\"\"\n", " )\n", "]\n", "\n", "prompt_builder = ChatPromptBuilder(template=template)" ] }, { "cell_type": "markdown", "metadata": { "id": "dJVyxytsG6Zc" }, "source": [ "Next, we use the `OutputAdapter` to transform the generated paragraphs into a List of Documents. This way, we will be able to use the `SentenceTransformersDocumentEmbedder` to create embeddings, since this component expects `List[Document]`" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "id": "K72gcW84CFtT" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": "Unsafe mode is enabled. This allows execution of arbitrary code in the Jinja template. Use this only if you trust the source of the template.\n" } ], "source": [ "from haystack import Document\n", "from haystack.components.converters import OutputAdapter\n", "from haystack_integrations.components.embedders.sentence_transformers import SentenceTransformersDocumentEmbedder\n", "from typing import List\n", "\n", "adapter = OutputAdapter(\n", " template=\"{{answers | build_doc}}\",\n", " output_type=List[Document],\n", " unsafe=True,\n", " custom_filters={\"build_doc\": lambda data: [Document(content=d.text) for d in data]}\n", ")\n", "\n", "embedder = SentenceTransformersDocumentEmbedder(model=\"sentence-transformers/all-MiniLM-L6-v2\")" ] }, { "cell_type": "markdown", "metadata": { "id": "j6L5jXO0HcQo" }, "source": [ "Finally, we create a custom component, `HypotheticalDocumentEmbedder`, that expects `documents` and can return a list of `hypotethetical_embeddings` which is the average of the embeddings from the \"hypothetical\" (fake) documents. To learn more about this technique and where it's useful, check out our [Guide to HyDE](https://docs.haystack.deepset.ai/docs/hypothetical-document-embeddings-hyde)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "id": "5A7nb9MaCXTH" }, "outputs": [], "source": [ "from numpy import array, mean\n", "from haystack import component\n", "\n", "@component\n", "class HypotheticalDocumentEmbedder:\n", "\n", " @component.output_types(hypothetical_embedding=List[float])\n", " def run(self, documents: List[Document]):\n", " stacked_embeddings = array([doc.embedding for doc in documents])\n", " avg_embeddings = mean(stacked_embeddings, axis=0)\n", " hyde_vector = avg_embeddings.reshape((1, len(avg_embeddings)))\n", " return {\"hypothetical_embedding\": hyde_vector[0].tolist()}" ] }, { "cell_type": "markdown", "metadata": { "id": "0vCg130JIW3A" }, "source": [ "We add all of our components into a pipeline to genereate a hypothetical document embedding πŸš€πŸ‘‡" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 49, "referenced_widgets": [ "e37e414744b04538a24a6ee4d9a5ba90", "3a1bc15fb77943988919ca6f504f6d4f", "66455f3ba9594dada59f669a43c85e8f", "866bb81e20e24caaa363784f4fd048ec", "d437b2869445412e8cdbbfccc1c40101", "a4be69c6d110405ab55510b59b9a6f41", "0b4c12dbf599431c987339d2861fcb78", "5d4aebc2bf35465dbb6b02f6f3512d0b", "49d16d19959c43bb9bd5cd4ce9fd3850", "625f9cd6cb934f1984e2d4108d238c43", "c519b25093884b60bf9a826ea1ef36f3" ] }, "id": "2sa91tDQCaHC", "outputId": "92359ba0-26e0-4ba0-85a1-5b9740d4ba66" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": "\rLoading weights: 0%| | 0/103 [00:00 Dict[str, Any]:\n", " data = default_to_dict(\n", " self,\n", " instruct_llm=self.instruct_llm,\n", " instruct_llm_api_key=self.instruct_llm_api_key,\n", " nr_completions=self.nr_completions,\n", " embedder_model=self.embedder_model,\n", " )\n", " data[\"pipeline\"] = self.pipeline.to_dict()\n", " return data\n", "\n", " @classmethod\n", " def from_dict(cls, data: Dict[str, Any]) -> \"HypotheticalDocumentEmbedder\":\n", " hyde_obj = default_from_dict(cls, data)\n", " hyde_obj.pipeline = Pipeline.from_dict(data[\"pipeline\"])\n", " return hyde_obj\n", "\n", " @component.output_types(hypothetical_embedding=List[float])\n", " def run(self, query: str):\n", " result = self.pipeline.run(data={\"prompt_builder\": {\"question\": query}})\n", " # return a single query vector embedding representing the average of the hypothetical document embeddings\n", " stacked_embeddings = array(\n", " [doc.embedding for doc in result[\"embedder\"][\"documents\"]]\n", " )\n", " avg_embeddings = mean(stacked_embeddings, axis=0)\n", " hyde_vector = avg_embeddings.reshape((1, len(avg_embeddings)))\n", " return {\"hypothetical_embedding\": hyde_vector[0].tolist()}" ] }, { "cell_type": "markdown", "metadata": { "id": "z3BTZF6qMtTY" }, "source": [ "## Use HyDE For Retrieval\n", "\n", "Let's see how we can use this component in a full pipeline. First, let's index some documents into an `InMemoryDocumentStore`" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 145, "referenced_widgets": [ "76720573a77442408a924d14c563d711", "d15fb403b71d4bff843c5a387a5ea36c", "9d8125468bf74c1589fcbe04b0623926", "efbcd9eaef6c40f48721cd1ef81ad4ea", "f6b5bb6382964c36999cd4c6982b664f", "897db75b6abc4cd0923d7caf2448e82b", "3afaf53201e94c7e8ba391b95fc1c3d8", "1ae296845e2142f0a78b743ecd22ace2", "fa5edd6cc3544e2b8ed9165b1536a8fe", "43f1affb1f9d4597b2ed08f4dccfa82d", "7f8a1097e0b54ac9bede391ae62a5ea0", "bfa1065e3afd4d6ab9d8b9cce855730f", "b0b397924c2f498db32d7cb57b2648db", "8c691d394cf24eb2b6eb61b2d17d59c7", "23c5ec7c16144c2e83e57ff396a9a98b", "cdbec1d537854931b737ffa577dd3fa6", "c7913c050b574b298c8a11ee6ab0fcfd", "d689fe03f55f4148b4f62445873b055c", "4ef80343c53e40a7804aae8be58bc24e", "f0e778c920f043349c44ddc66518b59b", "c3db2230ae9e40c597548a551bc5e4d6", "a561031934464b0194d648c18fa96076", "0cd8313fe64c4f62aa1235b778266137", "c7eb9ba2c7794e68a5e3b4f73e5b069a", "d3ae8fb928144977823d719cc71bf943", "1295414dbaf6409d8c78091e4003c8f6", "61119518e3104ccab011ece38978abc5", "716bec098dc240fd849be1e70c23b958", "e830d27e5dc4418588a65af93c32f7e6", "434bd85cc1694c76bd76cfd62f9a9be5", "d02c59ff64d64a619df33657727c2ac4", "df4fb8b15c8e4464b5834c6a3dbd12c9", "9df88f66f7854edaa8c37253a159a51e", "09b1b88ae086428ea2f2fcce7fec4be1", "2d6db0c1041744dd9671f0b9c7352dd5", "752a21d454cc468abb9dcff5c12ecd2d", "37e00469a4f94143a488879c6e6535bf", "3ec4e9cbfaad40d9827a4d879fded0bc", "4ce3d94567de4eb48ae6653d16a53b03", "16b211c5cad1464d9c50a7bad22a2e24", "7ea1fc96fc7f46699ce20eed96902e8f", "d9a5c93f47954f00bb4d7b2562b87e60", "efdebbeba075454c896231a39ecd9002", "c30cb96c9c9f4846ab4bf94fe2e2ad9d" ] }, "id": "9tb9zO_vMzuf", "outputId": "f059661e-6656-46b0-9e0f-344999d7de0d" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": "\rBatches: 0%| | 0/82 [00:00