full script will be available at the end<\/strong> of this article, but let\u2019s go step by step and dissect the key parts.<\/p>\n\n\n\n1.1. Configuration and Setup<\/h3>\n\n\n\n
The script starts by defining important parameters:<\/p>\n\n\n\n
embedding_model = 'all-minilm' # can also try with 'nomic-embed-text'\nmovie_csv_file = \".\/movie_dataset\/animation.csv\"\nvectorstore_path = f'vectorstore_{embedding_model}'\nbatch_size = 32<\/code><\/pre>\n\n\n\n- We specify the embedding model<\/strong> (
all-minilm<\/code>) to convert text into vectors.<\/li>- The script reads data from the animation movie dataset<\/strong> (
animation.csv<\/code>).<\/li>- We define the vectorstore directory<\/strong> (
vectorstore_all-minilm<\/code>), where our FAISS index will be saved.<\/li>- The script processes 32 movies at a time<\/strong> to reduce memory usage<\/strong>.<\/li><\/ul>\n\n\n\n
1.2. Removing Old Vector Stores<\/h3>\n\n\n\n
Before creating a new vector database, we check if an existing one exists and delete it to start fresh:<\/p>\n\n\n\n
if os.path.isdir(vectorstore_path):\n shutil.rmtree(vectorstore_path)<\/code><\/pre>\n\n\n\nThis ensures that we always work with a clean FAISS index<\/strong> instead of appending to outdated data.<\/p>\n\n\n\n1.3. Loading and Validating the Dataset<\/h3>\n\n\n\n
Next, we load the CSV file into a Pandas DataFrame and ensure it contains the necessary columns<\/strong> (movie_name<\/code> and description<\/code>):<\/p>\n\n\n\ndf = pd.read_csv(movie_csv_file)\nif \"movie_name\" not in df.columns or \"description\" not in df.columns:\n raise ValueError(\"CSV file must contain 'movie_name' and 'description' columns.\")<\/code><\/pre>\n\n\n\nIf these columns are missing, the script will raise an error. This step is important because vectorization relies on movie names and descriptions<\/strong> to generate embeddings.<\/p>\n\n\n\n1.4. Initializing FAISS and the Embedding Model<\/h3>\n\n\n\nvectorstore_embeddings = OllamaEmbeddings(model=embedding_model)\nvectorstore_index = None # Placeholder for FAISS index<\/code><\/pre>\n\n\n\n- We use Ollama\u2019s embedding model<\/strong> to generate text embeddings.<\/li>
- The FAISS index<\/strong> starts as
None<\/code> because we\u2019ll create it dynamically while processing batches of movies.<\/li><\/ul>\n\n\n\n1.5. Batch Processing for Efficient Indexing<\/h3>\n\n\n\n
To handle large datasets efficiently, we process movies in batches<\/strong> rather than all at once:<\/p>\n\n\n\nfor i in tqdm(range(0, len(df), batch_size), desc=\"Indexing Progress\"):\n batch_df = df.iloc[i:i + batch_size]\n documents = [\n Document(\n page_content=f\"Title: {row['movie_name']} | Description: {row['description']}\",\n metadata={\"movie_name\": row[\"movie_name\"], \"description\": row[\"description\"]}\n )\n for _, row in batch_df.iterrows()\n ]<\/code><\/pre>\n\n\n\n- We iterate over the dataset in chunks of 32 movies per batch<\/strong>.<\/li>
- Each movie is converted into a LangChain Document<\/strong>, combining the title and description<\/strong> into a single searchable text entry.<\/li>
- Metadata (original movie name and description) is stored alongside the text for reference.<\/li><\/ul>\n\n\n\n
1.6. Creating and Updating the FAISS Index<\/h3>\n\n\n\n
Once we have a batch of movie documents, we either create a new FAISS index<\/strong> or add to an existing one<\/strong>:<\/p>\n\n\n\nif vectorstore_index is None:\n vectorstore_index = FAISS.from_documents(documents, vectorstore_embeddings)\nelse:\n vectorstore_index.add_documents(documents)<\/code><\/pre>\n\n\n\n- If this is the first batch<\/strong>, we initialize the FAISS index.<\/li>
- Otherwise, we incrementally add<\/strong> new documents to the existing index.<\/li>
- This structure makes it easy to handle datasets of any size<\/strong> without overloading memory.<\/li><\/ul>\n\n\n\n
1.7. Saving the Vector Database<\/h3>\n\n\n\n
After processing all batches, we save the FAISS index<\/strong> to disk for future use:<\/p>\n\n\n\nvectorstore_index.save_local(vectorstore_path)<\/code><\/pre>\n\n\n\nThis allows us to reuse the indexed data later<\/strong> without needing to reprocess the entire dataset.<\/p>\n\n\n\nWrapping Up<\/h3>\n\n\n\n
At this point, we have successfully:<\/p>\n\n\n\n
- Loaded an IMDB animation movie dataset<\/strong><\/li>
- Converted movie names and descriptions into vector embeddings<\/strong><\/li>
- Indexed the data in FAISS<\/strong>, a fast and scalable vector database<\/li>
- Saved the index for later searches<\/li><\/ul>\n\n\n\n
In the next section, we\u2019ll use this vectorized database<\/strong> to perform fast, fuzzy movie searches<\/strong>\u2014finding relevant results even when queries contain typos or phrasing variations.<\/p>\n\n\n\n2. Searching for Movies in the Vector Database<\/h2>\n\n\n\n
Now that we have built and stored our vectorized movie database<\/strong>, it\u2019s time to put it to work! In this section, we\u2019ll explore how to search for movies using FAISS and LangChain.<\/p>\n\n\n\nThis script allows you to enter a movie title or description<\/strong>, and it will return the most similar results<\/strong> from our FAISS index. The key concept here is vector similarity<\/strong>\u2014the closer a movie\u2019s embedding is to the search query, the better the match.<\/p>\n\n\n\nAs always, the full script will be available at the end<\/strong> of this article, but let\u2019s go through its most important parts.<\/p>\n\n\n\n2.1. Defining the Search Function<\/h3>\n\n\n\n
At the heart of this script is the search_movies<\/code><\/strong> function, which retrieves similar movies based on a given query:<\/p>\n\n\n\ndef search_movies(query, top_k=5):\n results = vectorstore.similarity_search_with_score(query, k=top_k)\n return [\n (doc.metadata[\"movie_name\"], doc.metadata[\"description\"], score)\n for doc, score in results\n ]<\/code><\/pre>\n\n\n\nHow does it work?<\/h4>\n\n\n\n- The query<\/strong> (movie name or description) is passed into FAISS.<\/li>
- FAISS performs a similarity search<\/strong> and returns the top K matches<\/strong> (default: 5).<\/li>
- Each result includes:
- The movie title<\/strong><\/li>
- The movie description<\/strong><\/li>
- A similarity score<\/strong> (lower scores mean better matches)<\/li><\/ul><\/li><\/ul>\n\n\n\n
What is the similarity score?<\/h4>\n\n\n\n
The similarity_search_with_score<\/code><\/strong> function measures how close each movie\u2019s vector is to the query.<\/p>\n\n\n\n- A score closer to 0<\/strong> means a better match<\/strong>.<\/li>
- Higher scores indicate less relevant results<\/strong>.<\/li><\/ul>\n\n\n\n
For example, if you search for \"Finding Nimo\"<\/code>, it might still return \"Finding Nemo\"<\/code> because the vectors are similar, even though the text isn\u2019t an exact match.<\/p>\n\n\n\n2.2. Loading the Vector Database<\/h3>\n\n\n\n
Before searching, we need to load our FAISS vector store<\/strong> and its embeddings:<\/p>\n\n\n\nembedding_model = 'all-minilm' # can also try with 'nomic-embed-text'\nvectorstore_path = f'vectorstore_{embedding_model}'\nvectorstore_embeddings = OllamaEmbeddings(model=embedding_model)\nvectorstore = FAISS.load_local(vectorstore_path, vectorstore_embeddings, allow_dangerous_deserialization=True)<\/code><\/pre>\n\n\n\n- We set up the same embedding model<\/strong> (
all-minilm<\/code>) that we used for indexing.<\/li>- We define the path to our stored FAISS vector database<\/strong>.<\/li>
- We reload the vector index<\/strong> using the embeddings so that we can perform searches.<\/li><\/ul>\n\n\n\n
The allow_dangerous_deserialization=True<\/code><\/strong> flag is required when loading a local FAISS index. It enables compatibility when retrieving stored vectors.<\/p>\n\n\n\n2.3. Interactive Search Loop<\/h3>\n\n\n\n
Now, we enter a loop<\/strong> that allows the user to keep searching for different movies until they decide to exit:<\/p>\n\n\n\nsearch_another = True\nwhile search_another:\n print('-' * 80)\n movie_search_query = input(\"Enter a movie\/description to search for: \")\n movie_results = search_movies(movie_search_query)\n\n print(\"\\n---- Similar Movies --------------------\\n\")\n for name, description, score in movie_results:\n print(f\"- Title: {name}\\n\\t- Score: {score:.4f}\\n\\t- Description: {description}\\n\")\n\n search_another = input(\"Search again? [Y,n]: \").lower() in ['y', '']<\/code><\/pre>\n\n\n\n- The script asks for a search query<\/strong> (a movie title or a description).<\/li>
- It retrieves the top matches<\/strong> using
search_movies(query)<\/code>.<\/li>- Results are displayed in a readable format:
- Movie Title<\/strong><\/li>
- Similarity Score<\/strong> (lower is better)<\/li>
- Movie Description<\/strong><\/li><\/ol><\/li>
- The user is asked if they want to search again<\/strong> or exit the program.<\/li><\/ol>\n\n\n\n
Example Search Output<\/h3>\n\n\n\n
For real world case, there was a movie that I’ve wanted to recommend to a friend, but did not remember the name. The movie was commedy story about a girl that went over a family trip, and at the same time, an AI rebellion occurs. Using that idea as a search argument, let\u2019s say we search for “family trip ai rebellion”<\/strong>. The output might look something like this:<\/p>\n\n\n\n----------------------------------------------------------------------\nEnter a movie\/description to search for: family trip ai rebellion\n\n---- Similar Movies --------------------\n\n- Title: The Hoovers Conquer the Universe\n - Score: 1.0515\n - Description: An animated family space adventure\n\n- Title: Untitled Animated Feature Film\n - Score: 1.1748\n - Description: A progressive sociopolitical family film.\n\n- Title: Robot Zot\n - Score: 1.1781\n - Description: The story of a wayward scout for an invading alien force, whose course goes hopelessly awry when he lands in the yard of a modern-day, suburban family with problems of their own. Based on the book by Jon Scieszka and David Shannon.\n\n- Title: The Mitchells vs the Machines\n - Score: 1.1836\n - Description: A quirky, dysfunctional family's road trip is upended when they find themselves in the middle of the robot apocalypse and suddenly become humanity's unlikeliest last hope.<\/code><\/pre>\n\n\n\nThe movie was “The Mitchells vs the Machines”<\/strong>, and it appeared in the results. This demonstrates how vector-based search handles variations<\/strong> much better than traditional SQL or regex searches!<\/p>\n\n\n\nWrapping Up<\/h3>\n\n\n\n
At this point, we have successfully:<\/p>\n\n\n\n
- Loaded the vectorized movie database<\/strong><\/li>
- Implemented a similarity search function<\/strong><\/li>
- Created an interactive search loop<\/strong><\/li>
- Returned relevant movie recommendations based on text similarity<\/strong><\/li><\/ul>\n\n\n\n
In the next section, we\u2019ll wrap everything up and provide the full code<\/strong> so you can try it yourself!<\/p>\n\n\n\nFull code<\/h2>\n\n\n\n
This is the full code of both the vector store feeder as well as the vector store queries.<\/p>\n\n\n\n
index_movie_description.py<\/h3>\n\n\n\nimport os\nimport shutil\nimport pandas as pd\nfrom tqdm import tqdm\n\nfrom langchain_community.vectorstores import FAISS\nfrom langchain_ollama.embeddings import OllamaEmbeddings\nfrom langchain.schema import Document\n\nif __name__ == '__main__':\n # ---- Configuration --------------------------------------------\n embedding_model = 'all-minilm' # can also try with 'nomic-embed-text'\n movie_csv_file = \".\/movie_dataset\/animation.csv\"\n vectorstore_path = f'vectorstore_{embedding_model}'\n batch_size = 32\n\n # ---- Vector Store setup ---------------------------------------\n print('-' * 80)\n print(f'> Creating vectorstore: {vectorstore_path} ')\n # Remove previous vectorstore\n if os.path.isdir(vectorstore_path):\n print(f' > Removing existing vectorstore... ')\n shutil.rmtree(vectorstore_path)\n\n # ---- Grab movie .csv file for the vector store\n print(f' > Processing movies from: {movie_csv_file} ')\n # Ensure column names match expected format\n df = pd.read_csv(movie_csv_file)\n if \"movie_name\" not in df.columns or \"description\" not in df.columns:\n raise ValueError(\"CSV file must contain 'movie_name' and 'description' columns.\")\n\n # Initialize FAISS index & embedding model\n vectorstore_embeddings = OllamaEmbeddings(model=embedding_model)\n vectorstore_index = None # Placeholder for FAISS index\n\n # ---- Batch Processing ----\n print(f' > Indexing {len(df)} movies in batches of {batch_size}...\\n')\n for i in tqdm(range(0, len(df), batch_size), desc=\"Indexing Progress\"):\n # Extract batch & convert movie names into Langchain Documents (concatenate movie name and description)\n batch_df = df.iloc[i:i + batch_size]\n documents = [\n Document(\n page_content=f\"Title: {row['movie_name']} | Description: {row['description']}\",\n metadata={\"movie_name\": row[\"movie_name\"], \"description\": row[\"description\"]}\n )\n for _, row in batch_df.iterrows()\n ]\n # Initialize or append to FAISS index\n if vectorstore_index is None:\n vectorstore_index = FAISS.from_documents(documents, vectorstore_embeddings)\n else:\n vectorstore_index.add_documents(documents)\n\n # ---- Save vector store index\n print(f' > Saving vectorstore: {vectorstore_path}')\n vectorstore_index.save_local(vectorstore_path)\n print('> Vectorstore created!')<\/code><\/pre>\n\n\n\nsearch_movie_description.py<\/h3>\n\n\n\nfrom langchain_ollama.embeddings import OllamaEmbeddings\nfrom langchain_community.vectorstores import FAISS\n\n# Query function\ndef search_movies(query, top_k=5):\n results = vectorstore.similarity_search_with_score(query, k=top_k)\n return [\n (doc.metadata[\"movie_name\"], doc.metadata[\"description\"], score)\n for doc, score in results\n ]\n\nif __name__ == '__main__':\n # ---- Configuration --------------------------------------------\n embedding_model = 'all-minilm' # can also try with 'nomic-embed-text'\n vectorstore_path = f'vectorstore_{embedding_model}'\n\n # ---- Load FAISS index -----------------------------------------\n vectorstore_embeddings = OllamaEmbeddings(model=embedding_model)\n vectorstore = FAISS.load_local(vectorstore_path, vectorstore_embeddings, allow_dangerous_deserialization=True)\n\n # ---- Query vector store ---------------------------------------\n search_another = True\n while search_another:\n print('-' * 80)\n movie_search_query = input(\"Enter a movie\/description to search for: \")\n movie_results = search_movies(movie_search_query)\n\n print(\"\\n---- Similar Movies --------------------\\n\")\n for name, description, score in movie_results:\n print(f\"- Title: {name}\\n\\t- Score: {score:.4f}\\n\\t- Description: {description}\\n\")\n\n search_another = input(\"Search again? [Y,n]: \").lower() in ['y', '']<\/code><\/pre>\n\n\n\nConclusion: Smarter Searches, Zero Hassle<\/h2>\n\n\n\n
We\u2019ve explored how vector databases<\/strong> can revolutionize text-based search<\/strong>, making it smarter, more flexible, and typo-resistant. Instead of relying on old-school SQL LIKE<\/code> queries or complex regex patterns, we leveraged FAISS, LangChain, and Ollama embeddings<\/strong> to perform meaning-based searches<\/strong> on an IMDB movie dataset.<\/p>\n\n\n\nBeyond Movies: What\u2019s Next?<\/h3>\n\n\n\n
This approach isn\u2019t just for movies\u2014it can be applied to product searches, document retrieval, customer support chatbots<\/strong>, and any scenario where traditional search struggles with variations in wording.<\/p>\n\n\n\nWith vector search<\/strong>, you can build Google-like search experiences<\/strong> without needing a full AI-powered chatbot. And the best part? You don\u2019t need huge infrastructure or deep learning expertise<\/strong>\u2014just smart indexing and a good embedding model.<\/p>\n\n\n\nWant to take it further? Try experimenting with different embedding models, datasets, or even multimodal (text + images) searches<\/strong>. <\/p>\n\n\n\nNow, it\u2019s your turn\u2014give it a try and let me know what you build!<\/p>\n\n\n\n
About Me<\/h2>\n\n\n\n
I\u2019m Gabriel, and I like computers. A lot.<\/em><\/p>\n\n\n\nFor nearly 30 years, I\u2019ve explored the many facets of technology\u2014as a developer, researcher, sysadmin, security advisor, and now an AI enthusiast. Along the way, I\u2019ve tackled challenges, broken a few things (and fixed them!), and discovered the joy of turning ideas into solutions. My journey has always been guided by curiosity, a love of learning, and a passion for solving problems in creative ways.<\/p>\n\n\n\n
See ya around!<\/p>","protected":false},"excerpt":{"rendered":"
Unlike SQL or regex-based searches, which rely on exact text matching, vector databases allow fuzzy, meaning-based searches. This means you can find relevant results even if the search text doesn\u2019t match exactly. Imagine searching for “The Dark Night” and still getting results for “The Dark Knight”\u2014something that would be tricky with traditional methods.<\/p>","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[54],"tags":[55,61,68,56,69,59,60],"_links":{"self":[{"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/posts\/14937"}],"collection":[{"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/comments?post=14937"}],"version-history":[{"count":6,"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/posts\/14937\/revisions"}],"predecessor-version":[{"id":14943,"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/posts\/14937\/revisions\/14943"}],"wp:attachment":[{"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/media?parent=14937"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/categories?post=14937"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/temperies.com\/es\/wp-json\/wp\/v2\/tags?post=14937"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}