Automated AI-Driven Web Crawler Project
-
Overview & Automation Pipeline
Built a fully automated web crawler in Python to navigate, scrape, and process large volumes of web content end-to-end. The entire workflow is orchestrated via a scheduler (e.g., cron jobs or Celery beat), enabling unattended daily or hourly runs. Failures automatically trigger retry logic and notifications, ensuring a robust, zero-touch data-collection pipeline. -
Website Crawling & Scraping
- URL Discovery & Filtering:
- Starts from a configurable seed list of domains or sitemap URLs.
- Applies regex-based filters and URL normalization to avoid duplicate visits and restrict the crawl to relevant subdomains.
- Request Management:
- Uses
requestsfor fast, lightweight HTTP GET calls against static pages. - Employs
Seleniumwith headless Chrome for dynamic JavaScript-rendered content. - Implements rate limiting and randomized delays to respect
robots.txtdirectives and minimize server load.
- Uses
- Data Extraction:
- Parses HTML with
BeautifulSoupto locate and extract structured elements (e.g.,<article>,<div class="price">, metadata tags). - Extracts JSON-LD or embedded structured data when available.
- Handles paginated content via “next” link detection and auto-injection of new URLs into the crawl queue.
- Parses HTML with
- URL Discovery & Filtering:
-
AI-Powered Data Processing
- Pre-Processing:
- Cleans and normalizes raw text (HTML tag stripping, whitespace normalization, Unicode cleanup).
- Identifies and removes boilerplate (headers, footers, navigation) using heuristic rules.
- LLM Integration:
- Feeds the cleaned text snippets into OpenAI’s GPT-4 API (via the
openaiPython SDK). - Prompts the model to perform tasks such as sentiment analysis, entity extraction, summarization, or custom classification.
- Leverages Azure Cognitive Services for OCR on images or PDFs discovered during the crawl.
- Feeds the cleaned text snippets into OpenAI’s GPT-4 API (via the
- Structured Output Generation:
- Captures LLM responses as JSON objects with clearly defined schemas (e.g.,
{ "title": "...", "price": 123.45, "summary": "...", "entities": [...] }). - Validates against JSON Schema definitions and writes into a PostgreSQL database or exports to CSV/JSON files in an S3 bucket for downstream use.
- Captures LLM responses as JSON objects with clearly defined schemas (e.g.,
- Pre-Processing:
-
Reporting & Visualization
- Aggregates daily crawl results into summary reports using Python’s
pandasandmatplotlib. - Automatically generates PDF or HTML dashboards showing key metrics (e.g., new URLs discovered, average sentiment score, top entities).
- Delivers notifications via email or Slack with attached reports and a link to the live dashboard.
- Aggregates daily crawl results into summary reports using Python’s
-
Technologies & Tools
- Core: Python 3.x,
requests,BeautifulSoup, Selenium (headless Chrome) - AI/ML: OpenAI GPT-4 API, Azure Cognitive Services (OCR), custom prompt-engineering
- Data Storage & Orchestration: PostgreSQL, AWS S3, Celery, cron/Celery Beat
- Reporting: pandas, matplotlib, ReportLab (PDF generation), Flask (for simple dashboards)
- Deployment: Docker containers orchestrated with Docker Compose or Kubernetes
- Core: Python 3.x,
Impact & Use Cases
- Price Monitoring: Real-time tracking of product prices across e-commerce sites
- Content Aggregation: Automated news/article summarization for internal newsletters
- Market Intelligence: Sentiment-driven insights on customer reviews or social media posts
This AI-driven crawler not only automates large-scale data collection but enriches raw web content with intelligent analysis, delivering ready-to-use JSON datasets and actionable reports with zero manual intervention.