A comprehensive knowledge base builder that crawls documentation websites using deep crawling with BFS strategy to systematically explore and extract content. The application filters content to focus on meaningful text blocks while avoiding navigation elements and social media links, then transforms it into clean Markdown format optimized for RAG systems.
- Deep Web Crawling: Systematically explore websites with configurable depth using BFS strategy
- SharePoint/Active Directory Support: Crawl SharePoint sites and lists in enterprise environments
- Content Filtering: Focus on meaningful content blocks while excluding navigation, headers, etc.
- Image Support: Option to include and download images during crawling
- Knowledge Base Management: Store and organize crawled content with vector search capabilities
- Ollama Integration: Query your knowledge base using Ollama models
- Modern UI: Professional Streamlit interface with company logo support
- Export Options: Download results in JSON or Markdown formats
- Docker Support: Easy deployment with Docker and Docker Compose
- Robust Error Handling: Handle network failures, API limits, and more
The easiest way to get started is with Docker Compose, which will set up both the AI Web Crawler and Ollama in one command:
-
Clone the repository:
git clone https://github.com/urdiales/AI-Web-Crawler.git cd AI-Web-Crawler -
Start the application:
docker-compose up -d
-
Access the web interface at:
http://localhost:8501
If you prefer to run the application without Docker:
-
Clone the repository:
git clone https://github.com/urdiales/AI-Web-Crawler.git cd AI-Web-Crawler -
Create a virtual environment:
python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate
-
Install dependencies:
pip install -r requirements.txt
-
Install Playwright browsers:
python -m playwright install --with-deps chromium
-
Run the application:
streamlit run app/main.py
-
Access the web interface at:
http://localhost:8501
- Navigate to the Crawler tab
- Enter the URL you want to crawl
- (Optional) Add keywords to focus content extraction
- Configure crawl settings:
- Crawl depth
- Include images
- Stay within domain
- Maximum pages
- Click "Start Crawling"
- When crawling completes, you can download the results as JSON or Markdown
- Navigate to the Knowledge Base tab
- View existing knowledge bases
- Select a knowledge base to:
- View its content
- Export to JSON or Markdown
- Delete if no longer needed
- Navigate to the Chat tab
- Select a knowledge base to query
- Choose an Ollama model
- Ask questions about your crawled content
The included docker-compose.yml provides a complete setup with:
- AI Web Crawler application
- Ollama LLM service
- Persistent volume storage
- GPU support (if available)
To deploy with Dockge:
- Upload the
docker-compose.ymlfile to your Dockge instance - Click "Deploy Stack"
- Access the application at the published port
To deploy with Portainer:
- Go to your Portainer dashboard
- Navigate to Stacks → Add stack
- Upload the
docker-compose.ymlfile - Deploy the stack
- Access the application at the published port
The application can be configured using environment variables in the .env file or through the Docker Compose file:
| Variable | Description | Default |
|---|---|---|
| DEBUG | Enable debug logging | False |
| USE_EMBEDDINGS | Enable vector embeddings for search | True |
| USE_LOCAL_EMBEDDINGS | Use local embedding models | True |
| EMBEDDING_MODEL | Model name for embeddings | all-MiniLM-L6-v2 |
| OLLAMA_HOST | URL of Ollama API | http://localhost:11434 |
| DEFAULT_OLLAMA_MODEL | Default Ollama model | llama3 |
| SHAREPOINT_USERNAME | Username for SharePoint | |
| SHAREPOINT_PASSWORD | Password for SharePoint |
AI-Web-Crawler/
├── app/
│ ├── __init__.py
│ ├── main.py # Main Streamlit application
│ ├── crawler.py # Web crawling functionality
│ ├── processor.py # Content processing and filtering
│ ├── storage.py # Knowledge base management
│ ├── ollama_integration.py # Integration with Ollama
│ ├── utils.py # Utility functions
│ └── config.py # Configuration settings
├── data/ # Data storage (mapped volume)
├── static/ # Static assets
├── Dockerfile # Docker configuration
├── docker-compose.yml # Docker Compose configuration
├── requirements.txt # Python dependencies
└── README.md # Documentation
-
Browser Issues: If you encounter browser-related errors, try:
python -m playwright install --with-deps chromium
-
Connection to Ollama: If the application can't connect to Ollama, check:
- Ollama is running and accessible
- The OLLAMA_HOST environment variable is set correctly
- Your network allows connections to the Ollama port
-
Memory Usage: For large crawls, increase the container memory limit in the Docker Compose file:
deploy: resources: limits: memory: 4G
Contributions are welcome! Please feel free to submit a Pull Request.
This project is licensed under the MIT License - see the LICENSE file for details.