Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

1 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

Repository files navigation

web-crawler

Automated AI-Driven Web Crawler Project

  • Overview & Automation Pipeline
    Built a fully automated web crawler in Python to navigate, scrape, and process large volumes of web content end-to-end. The entire workflow is orchestrated via a scheduler (e.g., cron jobs or Celery beat), enabling unattended daily or hourly runs. Failures automatically trigger retry logic and notifications, ensuring a robust, zero-touch data-collection pipeline.

  • Website Crawling & Scraping

    1. URL Discovery & Filtering:
      • Starts from a configurable seed list of domains or sitemap URLs.
      • Applies regex-based filters and URL normalization to avoid duplicate visits and restrict the crawl to relevant subdomains.
    2. Request Management:
      • Uses requests for fast, lightweight HTTP GET calls against static pages.
      • Employs Selenium with headless Chrome for dynamic JavaScript-rendered content.
      • Implements rate limiting and randomized delays to respect robots.txt directives and minimize server load.
    3. Data Extraction:
      • Parses HTML with BeautifulSoup to locate and extract structured elements (e.g., <article>, <div class="price">, metadata tags).
      • Extracts JSON-LD or embedded structured data when available.
      • Handles paginated content via “next” link detection and auto-injection of new URLs into the crawl queue.
  • AI-Powered Data Processing

    1. Pre-Processing:
      • Cleans and normalizes raw text (HTML tag stripping, whitespace normalization, Unicode cleanup).
      • Identifies and removes boilerplate (headers, footers, navigation) using heuristic rules.
    2. LLM Integration:
      • Feeds the cleaned text snippets into OpenAI’s GPT-4 API (via the openai Python SDK).
      • Prompts the model to perform tasks such as sentiment analysis, entity extraction, summarization, or custom classification.
      • Leverages Azure Cognitive Services for OCR on images or PDFs discovered during the crawl.
    3. Structured Output Generation:
      • Captures LLM responses as JSON objects with clearly defined schemas (e.g., { "title": "...", "price": 123.45, "summary": "...", "entities": [...] }).
      • Validates against JSON Schema definitions and writes into a PostgreSQL database or exports to CSV/JSON files in an S3 bucket for downstream use.
  • Reporting & Visualization

    • Aggregates daily crawl results into summary reports using Python’s pandas and matplotlib.
    • Automatically generates PDF or HTML dashboards showing key metrics (e.g., new URLs discovered, average sentiment score, top entities).
    • Delivers notifications via email or Slack with attached reports and a link to the live dashboard.
  • Technologies & Tools

    • Core: Python 3.x, requests, BeautifulSoup, Selenium (headless Chrome)
    • AI/ML: OpenAI GPT-4 API, Azure Cognitive Services (OCR), custom prompt-engineering
    • Data Storage & Orchestration: PostgreSQL, AWS S3, Celery, cron/Celery Beat
    • Reporting: pandas, matplotlib, ReportLab (PDF generation), Flask (for simple dashboards)
    • Deployment: Docker containers orchestrated with Docker Compose or Kubernetes

Impact & Use Cases

  • Price Monitoring: Real-time tracking of product prices across e-commerce sites
  • Content Aggregation: Automated news/article summarization for internal newsletters
  • Market Intelligence: Sentiment-driven insights on customer reviews or social media posts

This AI-driven crawler not only automates large-scale data collection but enriches raw web content with intelligent analysis, delivering ready-to-use JSON datasets and actionable reports with zero manual intervention.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages