Introduction<\/h2>\n\n\n\n
Have you ever struggled to find an image in a massive collection, remembering only a vague description like “sunset over a snowy mountain”<\/em> or “a cat sitting next to a laptop”<\/em>? What if you could just type a natural-language description and instantly retrieve the most relevant images? That’s exactly what we’re going to build in this article using multimodal AI and a vector database<\/strong>.<\/p>\n\n\n\n When we talk about Large Language Models (LLMs)<\/strong>, we usually mean AI models trained to understand and generate text. They’re great at answering questions, summarizing documents, or generating creative text. However, they can’t “see” or interpret images. That’s where multimodal models<\/strong> come in.<\/p>\n\n\n\n A multimodal model<\/strong> processes multiple types of input\u2014like text and images\u2014simultaneously. It can understand and describe images in natural language<\/strong>, combining vision and language capabilities in one model. One such model is LLaVA<\/strong> (Large Language and Vision Assistant), defined as a novel end-to-end trained large multimodal model that combines a vision encoder and Vicuna for general-purpose visual and language understanding<\/em>, which we’ll use in this project.<\/p>\n\n\n\n A vision encoder<\/strong> is the part of a multimodal model that processes images. It converts an image into a numerical representation (embeddings) that a language model can understand. Think of it as translating a picture into a format that an AI can “read” and describe in words.<\/p>\n\n\n\n In this guide, we’ll use LLaVA<\/strong> to automatically generate text descriptions for a repository of images<\/strong>. Instead of manually tagging each picture, we’ll let AI analyze and describe them for us. Then, we’ll store these descriptions in a vector database<\/strong> so we can perform similarity searches<\/strong>\u2014meaning, you can find images based on their textual descriptions.<\/p>\n\n\n\n We’ll write Python code<\/strong> that integrates:<\/p>\n\n\n\n If you want a more in-depth theory explanation, do not hesitate in check this link: From Messy Files to Magic Answers: How RAG Makes AI Smarter (and Life Easier)<\/a>. Also, if you want to delve deeper into RAG implementations, take a look at this: Teach Your PDFs to Chat: Build an AI-Powered Assistant That Delivers Answers<\/a>.<\/p>\n\n\n\n By the end of this article, you’ll have a working system that can catalog images automatically and make them searchable by description<\/strong>. Let’s get started!<\/p>\n\n\n\n Before diving into the examples, ensure that your development environment is set up with the necessary tools and dependencies. Here’s what you’ll need:<\/p>\n\n\n\n With these prerequisites in place, you’ll be ready to proceed to the next steps in setting up the image catalog and query it.<\/p>\n\n\n\n Before we dive into the code, here’s the plan: we’ll process a repository of images, use a multimodal AI model<\/strong> to generate text descriptions, and then store those descriptions in a vector database<\/strong> for easy retrieval.<\/p>\n\n\n\n The full code will be available at the end of the article, but here, we’ll break it down into the key steps.<\/p>\n\n\n\n We start by setting up our configurations:<\/p>\n\n\n\n This defines the multimodal model, embedding model, and the paths for storing processed data. The batch_size<\/strong> parameter is used to process image descriptions in small chunks to feed the vector database (for the sake of the example, 8 was enough\u2026 consider using a bigger number for real scenarios).<\/p>\n\n\n\n LLaVA requires images in base64 format<\/strong>, so we need to convert them before feeding them into the model. The following function reads an image, encodes it, and returns its base64 representation:<\/p>\n\n\n\n I encourage you to enhance this function to process other image types!<\/p>\n\n\n\n Now, we iterate over the images in the repository, send them to LLaVA, and capture the text descriptions:<\/p>\n\n\n\n Here’s what happens:<\/p>\n\n\n\n Once we have textual descriptions, we need to convert them into vectors<\/strong> and store them in FAISS for efficient retrieval.<\/p>\n\n\n\n This section initializes FAISS and ensures we start with a fresh<\/strong> vector database by removing any existing one.<\/p>\n\n\n\n Since we’re working with potentially large datasets<\/strong>, we process text descriptions in batches:<\/p>\n\n\n\n This approach helps to:<\/p>\n\n\n\n Finally, we store the indexed vector database for future queries:<\/p>\n\n\n\n And that’s it! Now, we have a searchable<\/strong> catalog of image descriptions stored in FAISS, making it easy to find images based on textual queries.<\/p>\n\n\n\n Now that we have a vectorized image catalog<\/strong>, it’s time to put it to use! In this section, we\u2019ll explore how to perform similarity searches<\/strong> against our FAISS vector database to retrieve the most relevant images based on a text query.<\/p>\n\n\n\n The full code will be available at the end of the article, but here, we\u2019ll break it down into the most important parts.<\/p>\n\n\n\n Instead of using traditional keyword-based<\/strong> searching, we\u2019ll leverage vector similarity search<\/strong> to find images based on meaning, even if the descriptions aren’t an exact match.<\/p>\n\n\n\n FAISS allows us to use a function called similarity_search_with_score<\/strong>, which determines how closely a stored description matches a query. The lower the score, the better the match<\/strong>\u2014a score of 0.0<\/strong> means a perfect match.<\/p>\n\n\n\n We define a function that takes a text query<\/strong> (e.g., “a cat sitting next to a laptop”<\/em>) and finds the top matching images from the vector database:<\/p>\n\n\n\n Here\u2019s how it works:<\/p>\n\n\n\n By default, the function returns three results<\/strong>, but you can adjust IMPORTANT:<\/strong> consider that the function will return the (by default) 3 closest descriptions\u2026 if you have a very small set of images, it will find the three closest descriptions, even if they do not relate with your search query.<\/p>\n\n\n\n Before running a search, we need to load the FAISS index and the text embedding model:<\/p>\n\n\n\n This ensures that:<\/p>\n\n\n\n To make the system interactive, we create a loop where users can enter text queries<\/strong>, get relevant image matches, and decide whether to search again:<\/p>\n\n\n\n This part:<\/p>\n\n\n\n This is a output sample, just for you to see the results:<\/p>\n\n\n\n And these are the images:<\/p>\n\n\n\n This is the full code of both the image catalog creation as well as the vector store queries.<\/p>\n\n\n\n In this article, we built a powerful AI-driven image search system<\/strong> using a multimodal model (LLaVA) and a vector database (FAISS)<\/strong>. Instead of manually tagging images, we let AI see, describe, and organize them<\/strong>\u2014turning an unstructured image repository into a fully searchable collection.<\/p>\n\n\n\n This workflow proves how multimodal AI and vector databases can revolutionize the way we interact with visual data<\/strong>. Whether you’re cataloging a massive photo archive, building an AI-powered image search tool, or enhancing media management workflows\u2014this technique offers a scalable, efficient solution.<\/p>\n\n\n\n You can expand on this project by:<\/p>\n\n\n\n Now it\u2019s your turn\u2014how will you use AI to make image search smarter? \ud83d\ude80<\/p>\n\n\n\n I\u2019m Gabriel, and I like computers. A lot.<\/em><\/p>\n\n\n\n For nearly 30 years, I\u2019ve explored the many facets of technology\u2014as a developer, researcher, sysadmin, security advisor, and now an AI enthusiast. Along the way, I\u2019ve tackled challenges, broken a few things (and fixed them!), and discovered the joy of turning ideas into solutions. My journey has always been guided by curiosity, a love of learning, and a passion for solving problems in creative ways.<\/p>\n\n\n\n See ya around!<\/p>","protected":false},"excerpt":{"rendered":" Have you ever struggled to find an image in a massive collection, remembering only a vague description like “sunset over a snowy mountain” or “a cat sitting next to a laptop”? What if you could just type a natural-language description and instantly retrieve the most relevant images? That’s exactly what we’re going to build in this article using multimodal AI and a vector database.<\/p>","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[54],"tags":[55,73,72,61,71,74,56,57,69,59,60,70],"_links":{"self":[{"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/posts\/14969"}],"collection":[{"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/comments?post=14969"}],"version-history":[{"count":5,"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/posts\/14969\/revisions"}],"predecessor-version":[{"id":14976,"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/posts\/14969\/revisions\/14976"}],"wp:attachment":[{"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/media?parent=14969"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/categories?post=14969"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/tags?post=14969"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}What’s a Multimodal Model, and How Is It Different from an LLM?<\/strong><\/h3>\n\n\n\n
What’s a Vision Encoder? (In Simple Terms)<\/strong><\/h4>\n\n\n\n
What Are We Building?<\/strong><\/h3>\n\n\n\n
How Are We Doing This?<\/strong><\/h3>\n\n\n\n
Prerequisites<\/h2>\n\n\n\n
pipenv<\/code> to set up the required libraries. Execute the following command in your terminal:<\/li><\/ul><\/li><\/ol>\n\n\n\npipenv install langchain langchain-community langchain-ollama faiss-cpu pillow tqdm pandas<\/code><\/pre>\n\n\n\nGenerating Image Descriptions and Storing Them for Search<\/strong><\/h2>\n\n\n\n
Step 1: Setting Up the Environment<\/strong><\/h3>\n\n\n\n
llm_model = 'llava:13b'\nembedding_model = 'nomic-embed-text'\nvectorstore_path = f'vectorstore_{embedding_model}'\nimages_repository_path = '.\/images_repository'\nimages_description_csv = '.\/images_descriptions.csv'\nbatch_size = 8<\/code><\/pre>\n\n\n\nStep 2: Preparing Image Data<\/strong><\/h3>\n\n\n\n
def get_png_as_base64(file_path):\n buffered = BytesIO()\n Image.open(file_path).save(buffered, format=\"PNG\")\n return base64.b64encode(buffered.getvalue()).decode(\"utf-8\")<\/code><\/pre>\n\n\n\nStep 3: Extracting Descriptions from Images<\/strong><\/h3>\n\n\n\n
model = OllamaLLM(model=llm_model)\nprompt = 'Describe this image in two or less sentences:'\n\ninput_files = glob.glob(f'{images_repository_path}\/*.png')\ndataframe = pd.DataFrame(columns=['image_path', 'description'])\n\nfor image_path in input_files:\n image_base64 = get_png_as_base64(image_path)\n response = model.bind(images=[image_base64]).invoke(input=prompt)\n dataframe.loc[len(dataframe)] = [image_path, response]\n\ndataframe.to_csv(images_description_csv, index=False)<\/code><\/pre>\n\n\n\n.png<\/code> files in the image repository.<\/li>Step 4: Creating the Vector Database<\/strong><\/h3>\n\n\n\n
if os.path.isdir(vectorstore_path):\n shutil.rmtree(vectorstore_path)\n\nvectorstore_embeddings = OllamaEmbeddings(model=embedding_model)\nvectorstore_index = None<\/code><\/pre>\n\n\n\nStep 5: Indexing Descriptions in FAISS<\/strong><\/h3>\n\n\n\n
for i in range(0, len(dataframe), batch_size):\n batch_df = dataframe.iloc[i:i + batch_size]\n documents = [\n Document(\n page_content=row['description'],\n metadata={\"image_path\": row[\"image_path\"], \"description\": row[\"description\"]}\n )\n for _, row in batch_df.iterrows()\n ]\n\n if vectorstore_index is None:\n vectorstore_index = FAISS.from_documents(documents, vectorstore_embeddings)\n else:\n vectorstore_index.add_documents(documents)<\/code><\/pre>\n\n\n\nStep 6: Saving the Vector Store<\/strong><\/h3>\n\n\n\n
vectorstore_index.save_local(vectorstore_path)<\/code><\/pre>\n\n\n\nSearching Your Image Catalog Using Natural Language<\/strong><\/h2>\n\n\n\n
Step 1: How Similarity Search Works<\/strong><\/h3>\n\n\n\n
Step 2: Defining the Search Function<\/strong><\/h3>\n\n\n\n
def search_images(query, top_k=3):\n results = vectorstore.similarity_search_with_score(query, k=top_k)\n return [\n (doc.metadata[\"image_path\"], doc.metadata[\"description\"], score)\n for doc, score in results\n ]<\/code><\/pre>\n\n\n\ntop_k<\/code> to show more or fewer images.<\/p>\n\n\n\nStep 3: Loading the Vector Database<\/strong><\/h3>\n\n\n\n
from langchain_ollama.embeddings import OllamaEmbeddings\nfrom langchain_community.vectorstores import FAISS\n\nembedding_model = 'nomic-embed-text'\nvectorstore_path = f'vectorstore_{embedding_model}'\n\nvectorstore_embeddings = OllamaEmbeddings(model=embedding_model)\nvectorstore = FAISS.load_local(vectorstore_path, vectorstore_embeddings, allow_dangerous_deserialization=True)<\/code><\/pre>\n\n\n\nStep 4: Running the Search Loop<\/strong><\/h3>\n\n\n\n
search_another = True\nwhile search_another:\n image_search_query = input(\"Enter an image description to search for: \")\n movie_results = search_images(image_search_query)\n\n for image_path, description, score in movie_results:\n print(f\"- Image Path: {image_path}\\n\\t- Score: {score:.4f}\\n\\t- Description: {description}\\n\")\n\n search_another = input(\"Search again? [Y,n]: \").lower() in ['y', '']<\/code><\/pre>\n\n\n\nOutput samples<\/h2>\n\n\n\n
Enter a image description to search for: a landscape\n\n---- Images related ----------------------\n\n- Image Path: .\/images_repository\/Second_Life_Landscape_champ_des_fleurs.png\n - Score: 0.4404\n - Description: This image features a digital rendering of a natural landscape, possibly from a video game or a 3D model. The scene includes wildflowers, tall grasses, and trees in the background under a clear sky. There is a blank area on the bottom part of the image where there should be more detail but it appears to have been cut off or obscured.\n\n- Image Path: .\/images_repository\/Semmering_landscape.png\n - Score: 0.6829\n - Description: This image captures a serene winter scene on a mountain slope. The landscape is covered in snow, and the forest at the base of the slope has snow-covered pine trees. There's a slight mist or fog that adds to the tranquility of the setting.<\/code><\/pre>\n\n\n\n![\"Second_Life_Landscape_champ_des_fleurs.png\"](\"https:\/\/temperies.com\/wp-content\/uploads\/2025\/01\/Second_Life_Landscape_champ_des_fleurs.png\")
![\"Semmering_landscape.png\"](\"https:\/\/temperies.com\/wp-content\/uploads\/2025\/01\/Semmering_landscape.png\")
Full code<\/h2>\n\n\n\n
index_images_description.py<\/h3>\n\n\n\n
import base64\nimport glob\nimport os\nimport pandas as pd\nimport shutil\n\nfrom io import BytesIO\nfrom langchain.schema import Document\nfrom langchain_community.vectorstores import FAISS\nfrom langchain_ollama import OllamaLLM\nfrom langchain_ollama.embeddings import OllamaEmbeddings\nfrom PIL import Image\nfrom tqdm import tqdm\n\n\n# ---- Functions definition ---------------------------------------------------\ndef get_dataframe(file_path):\n if os.path.isfile(file_path):\n os.remove(file_path)\n return pd.DataFrame(columns=['image_path', 'description'])\n\n\ndef get_png_as_base64(file_path):\n buffered = BytesIO()\n Image.open(file_path).save(buffered, format=\"PNG\") # You can change the format if needed\n return base64.b64encode(buffered.getvalue()).decode(\"utf-8\")\n\n\n# ---- Entry point ------------------------------------------------------------\nif __name__ == '__main__':\n\n # ---- Configuration ------------------------------------------------------\n llm_model = 'llava:13b'\n embedding_model = 'nomic-embed-text'\n vectorstore_path = f'vectorstore_{embedding_model}'\n images_repository_path = '.\/images_repository'\n images_description_csv = '.\/images_descriptions.csv'\n batch_size = 8\n\n # ---- Model definition ---------------------------------------------------\n model = OllamaLLM(model=llm_model)\n prompt = 'Describe this image in two or less sentences:'\n\n # ---- Grab images from repository ----------------------------------------\n print('-' * 80)\n print(f'> Grabbing PNG images from repository: {images_repository_path}')\n input_files = glob.glob(f'{images_repository_path}\/*.png')\n print(f'> Got {len(input_files)} file(s).')\n\n # ---- Get the images description dataset ---------------------------------\n print(f'> Creating images description dataset: {images_description_csv}')\n dataframe = get_dataframe(images_description_csv)\n\n # ---- Image processing ---------------------------------------------------\n print('> Updating images description dataset...')\n for i in tqdm(range(0, len(input_files), 1), desc=\"Images description generation progress\"):\n image_path = input_files[i]\n image_base64 = get_png_as_base64(image_path)\n response = model.bind(images=[image_base64]).invoke(input=prompt)\n dataframe.loc[len(dataframe)] = [image_path, response]\n\n dataframe.to_csv(images_description_csv, index=False)\n print('> ...done!')\n\n # ---- Vector Store setup -------------------------------------------------\n print('-' * 80)\n print(f'> Creating vectorstore: {vectorstore_path} ')\n if os.path.isdir(vectorstore_path):\n shutil.rmtree(vectorstore_path)\n\n # Initialize FAISS index & embedding model\n vectorstore_embeddings = OllamaEmbeddings(model=embedding_model)\n vectorstore_index = None # Placeholder for FAISS index\n\n # ---- Vector store indexing\n print(f'> Indexing {len(dataframe)} images descriptions...\\n')\n for i in tqdm(range(0, len(dataframe), batch_size), desc=\"Indexing images description progress\"):\n batch_df = dataframe.iloc[i:i + batch_size]\n documents = [\n Document(\n page_content=row['description'],\n metadata={\"image_path\": row[\"image_path\"], \"description\": row[\"description\"]}\n )\n for _, row in batch_df.iterrows()\n ]\n # Initialize or append to FAISS index\n if vectorstore_index is None:\n vectorstore_index = FAISS.from_documents(documents, vectorstore_embeddings)\n else:\n vectorstore_index.add_documents(documents)\n\n # ---- Save vector store index\n print(f'> Saving vectorstore: {vectorstore_path}')\n vectorstore_index.save_local(vectorstore_path)\n print('> Vectorstore created!')<\/code><\/pre>\n\n\n\nsearch_images_description.py<\/h3>\n\n\n\n
from langchain_ollama.embeddings import OllamaEmbeddings\nfrom langchain_community.vectorstores import FAISS\n\n# Query function\ndef search_images(query, top_k=2):\n results = vectorstore.similarity_search_with_score(query, k=top_k)\n return [\n (doc.metadata[\"image_path\"], doc.metadata[\"description\"], score)\n for doc, score in results\n ]\n\nif __name__ == '__main__':\n # ---- Configuration ------------------------------------------------------\n embedding_model = 'nomic-embed-text'\n vectorstore_path = f'vectorstore_{embedding_model}'\n\n # ---- Load FAISS index ---------------------------------------------------\n vectorstore_embeddings = OllamaEmbeddings(model=embedding_model)\n vectorstore = FAISS.load_local(vectorstore_path, vectorstore_embeddings, allow_dangerous_deserialization=True)\n\n # ---- Query vector store -------------------------------------------------\n search_another = True\n while search_another:\n print('-' * 80)\n image_search_query = input(\"Enter a image description to search for: \")\n movie_results = search_images(image_search_query)\n\n print(\"\\n---- Images related ----------------------\\n\")\n for image_path, description, score in movie_results:\n print(f\"- Image Path: {image_path}\\n\\t- Score: {score:.4f}\\n\\t- Description: {description}\\n\")\n\n search_another = input(\"Search again? [Y,n]: \").lower() in ['y', '']<\/code><\/pre>\n\n\n\nWrapping Up: From Pixels to Searchable Insights<\/strong><\/h2>\n\n\n\n
What We Accomplished<\/strong><\/h3>\n\n\n\n
Next Steps<\/strong><\/h3>\n\n\n\n
About Me<\/h2>\n\n\n\n