PulseSight — Social Media Monitoring (Twitter)

A small Flask-based social media monitoring dashboard and Twitter scraper using Selenium. It supports interactive scraping of user profiles, hashtags, and advanced search queries and stores scraped tweets in a local SQLite database.

Features

• Start scraping jobs from a web UI or via API.
• Supports username, hashtag, or arbitrary Twitter search queries (including advanced operators).
• Optional headless mode for server runs.
• Stores tweets in a local SQLite database and exposes basic analytics.

Badges

• Python 3.11+ compatible
• MIT license
• Active development

Quick start (Windows / PowerShell)

1. Create a virtual environment and activate it:

python -m venv .venv
.\.venv\Scripts\Activate.ps1

2. Install dependencies:

pip install -r requirements.txt

3. Create a copy of environment variables (recommended):

# Create .env with at least TWITTER_USERNAME and TWITTER_PASSWORD
New-Item -Path . -Name ".env" -ItemType "file" -Value "TWITTER_USERNAME=your_user`nTWITTER_PASSWORD=your_pass"

4. Run the app (development):

$env:FLASK_APP = 'app.py'
$env:FLASK_ENV = 'development'
python app.py

Open http://localhost:5000 in your browser.

Start a scraping job from the UI

• Click "New Scraping", choose Search Type, enter a query (for hashtags include #, for usernames include @), set headless if running on server, and click Start Scraping.

Start a scraping job via API (PowerShell example)

$payload = @{ search_query = 'pakistan min_replies:1000'; search_type='query'; max_tweets=50; headless=$true } | ConvertTo-Json
Invoke-RestMethod -Uri http://localhost:5000/start_scraping -Method POST -Body $payload -ContentType 'application/json'

Then poll job status:

Invoke-RestMethod -Uri http://localhost:5000/job_status/<JOB_ID> -Method GET | ConvertTo-Json -Depth 5

Environment variables

• TWITTER_USERNAME and TWITTER_PASSWORD — credentials used for interactive login
• TWITTER_SESSION_COOKIE — optional JSON array of cookies saved to resume sessions
• FLASK_APP, FLASK_ENV — Flask runtime environment

Development notes

• The app uses Selenium + webdriver-manager to manage the browser driver.
• If running headless fails to hide the browser, try toggling headless flags in twitter_scraper.py or ensure local Chrome/Chromedriver versions match.

Troubleshooting

• If you see 0 tweets returned:
  • Verify the query string is being sent correctly from the UI (check static/js/dashboard.js).
  • Confirm the scraper navigates to the expected search URL — debug prints are in twitter_scraper.py.
  • Accept or dismiss cookie banner; the scraper attempts to click refuse non-essential cookies.
• If Chrome/Chromedriver version mismatches occur, run:

pip install --upgrade webdriver-manager

Tests & linting

• Basic syntax checks can be run with:

python -m py_compile twitter_scraper.py
python -m py_compile scraper\twitter_scraper.py

Files of interest

• app.py, routes.py — Flask entrypoints and routes
• twitter_scraper.py — main Selenium-based scraper used by the app
• scraper/ — notebook-style scraper and helper modules used in experiments
• templates/, static/ — frontend assets and templates

Next steps / improvements

• Add unit/integration tests for the scraper logic (mock Selenium)
• Add retries and backoff around dynamic Twitter flows
• Add Dockerfile for reproducible server and headless runs

---

If you want, I can:

• Add badges for PyPI/GitHub Actions/coverage
• Create a minimal docker-compose.yml and Dockerfile for headless deployment
• Run a quick end-to-end scraping test here and paste logs