Building a Watsonx.ai Chatbot RAG Server with MCP

@ruslanmv

About Building a Watsonx.ai Chatbot RAG Server with MCP

Building a Watsonx.ai Chatbot RAG Server with MCP compatible with Claude Desktop.

Config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "watsonx-rag-mcp-server": {
      "command": "python",
      "args": [
        "-m",
        "venv",
        ".venv"
      ]
    }
  }
}

Tools

No tools detected

We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.

Overview

What is Building a Watsonx.ai Chatbot RAG Server with MCP?

A Retrieval-Augmented Generation (RAG) server that uses IBM Watsonx.ai for LLM inference, ChromaDB for vector indexing, and exposes a chat tool via the Model Context Protocol (MCP) Python SDK. It processes PDF documents and answers queries based on their content, and can integrate with Claude Desktop.

How to use Building a Watsonx.ai Chatbot RAG Server with MCP?

Set up a Python 3.8+ virtual environment, install dependencies, create a .env file with Watsonx.ai credentials, place PDF files in the documents/ folder, then run server.py to start the MCP server. A client.py script is provided for testing queries, and the server can also be plugged into Claude Desktop.

Key features of Building a Watsonx.ai Chatbot RAG Server with MCP

Uses IBM Watsonx.ai for large language model inference
Indexes PDF documents in ChromaDB vector database
Exposes a single MCP tool (chat_with_manual) for RAG queries
Supports configurable chunk size and overlap for text splitting
Integrates directly with Claude Desktop via MCP

Use cases of Building a Watsonx.ai Chatbot RAG Server with MCP

Answering questions from technical manuals (e.g., drone documentation)
Building a chatbot grounded in a company’s internal PDF documents
Enabling Claude Desktop to retrieve information from local PDF files
Rapid prototyping of a RAG pipeline without custom infrastructure

FAQ from Building a Watsonx.ai Chatbot RAG Server with MCP

What are the runtime requirements?

Python 3.8+ and an IBM Cloud account with provisioned Watsonx.ai services. You need a Watsonx API key and project ID.

Where are the documents and vector data stored?

PDFs are kept in the local documents/ folder (configurable via DOCS_FOLDER). The ChromaDB vector index persists to the chroma_db_data/ directory.

How does the RAG process work?

When a user asks a question, the server retrieves relevant text chunks from ChromaDB, concatenates them with the query, and sends the combined prompt to Watsonx.ai for answer generation.

What transport does the MCP server use?

The server is built with FastMCP, which defaults to stdio transport; it is configured to work with Claude Desktop’s MCP integration.

Is authentication required?

Yes, the server reads WATSONX_APIKEY, WATSONX_URL, and PROJECT_ID from a .env file and validates them at startup.

Frequently asked questions

What are the runtime requirements?

Python 3.8+ and an IBM Cloud account with provisioned Watsonx.ai services. You need a Watsonx API key and project ID.

Where are the documents and vector data stored?

PDFs are kept in the local `documents/` folder (configurable via `DOCS_FOLDER`). The ChromaDB vector index persists to the `chroma_db_data/` directory.

How does the RAG process work?

When a user asks a question, the server retrieves relevant text chunks from ChromaDB, concatenates them with the query, and sends the combined prompt to Watsonx.ai for answer generation.

What transport does the MCP server use?

The server is built with FastMCP, which defaults to stdio transport; it is configured to work with Claude Desktop’s MCP integration.

Is authentication required?

Yes, the server reads `WATSONX_APIKEY`, `WATSONX_URL`, and `PROJECT_ID` from a `.env` file and validates them at startup.

Comments

Basic information

More Memory & Knowledge MCP servers

Notion MCP Server

suekou

A Model Context Protocol server for connecting Notion to MCP-compatible clients

910

RAG Documentation MCP Server

hannesrudolph

An MCP server implementation that provides tools for retrieving and processing documentation through vector search, enabling AI assistants to augment their responses with relevant documentation context.

267

ctxfile

Local-first MCP server that snapshots your project's working state into one context object. Open-core, privacy-first.

Tempreon

Person-owned, portable AI memory served over MCP. Your knowledge, preferences, and decisions — captured in one assistant, available in all of them.

MCP server for Obsidian

MarkusPfundstein

MCP server that interacts with Obsidian via the Obsidian rest API community plugin

4.0K