🧠 ETL–Machine Learning Integration: Sentiment Analysis Pipeline

🚀 Project Overview

This project automates an end-to-end ETL (Extract–Transform–Load) and Machine Learning (ML) pipeline for sentiment analysis on textual data such as product reviews.

It is designed to:

• Automatically extract and clean raw datasets
• Train a sentiment classification model (Logistic Regression + TF-IDF)
• Evaluate model performance
• Test predictions on unseen data
• Save results, models, and logs in an organized, production-ready format

The project is completely automated through a single entry point:

python run_pipeline.py

which runs the entire pipeline — from raw data to predictions.

---

🧩 Key Features

Feature	Description
ETL Integration	Automatically extracts, transforms, and loads data before training
Data Preprocessing	Cleans raw CSV data (label mapping, column renaming, text normalization)
TF-IDF Feature Engineering	Converts text to numerical features using bi-grams and stopword removal
Logistic Regression Model	Trains a fast, accurate binary sentiment classifier
Smart Automation	Skips retraining if a model already exists
Testing Pipeline	Automatically tests new datasets and generates predictions
Progress Tracking	Uses tqdm progress bars for real-time feedback
Logging	Logs all key steps and performance metrics in etl_model.log
Modular Design	Independent scripts for training, testing, and automation

---

🧠 Technologies Used

Category	Tools / Libraries
Language	Python 3.10+
ML & NLP	scikit-learn, pandas, numpy
Vectorization	TfidfVectorizer (bigrams, 5000 features, English stopwords)
Model	Logistic Regression (multithreaded, n_jobs=-1)
Automation & Logging	tqdm, logging, subprocess, os
Data Handling	CSV-based workflow (no DB dependency)

---

⚙️ Project Architecture

ETL-Machine-Learning-Integration/
├── data/
│   ├── raw/
│   │   ├── train.csv
│   │   └── test.csv
│   └── processed/
│       ├── cleaned_with_predictions.csv
│       ├── test_cleaned.csv
│       └── test_predictions.csv
├── models/
│   ├── sentiment_model.pkl
│   └── vectorizer.pkl
├── sentiment_model.py
├── test_model_full.py
├── run_pipeline.py
├── etl_model.log
├── requirements.txt
└── README.md

---

🧩 Workflow Overview

1️⃣ Data Preparation

• Reads train.csv from data/raw/
• Maps raw labels → {1: 0 (Negative), 2: 1 (Positive)}
• Removes unused columns
• Saves cleaned dataset as cleaned_with_predictions.csv

2️⃣ Model Training

• TF-IDF Vectorization (max_features=5000, ngram_range=(1,2))
• Trains Logistic Regression classifier
• Evaluates on validation split (80/20)
• Prints accuracy and classification report
• Saves sentiment_model.pkl and vectorizer.pkl in /models/

3️⃣ Testing / Inference

• Cleans and prepares test.csv
• Loads saved model & vectorizer
• Combines title + review text
• Generates predictions and saves to test_predictions.csv
• Prints accuracy, precision, recall, and F1-score (if ground truth available)

4️⃣ Automation

• Controlled by run_pipeline.py
• Handles:
  • Directory creation
  • Data verification
  • Cleaning
  • Training (auto-skips if already exists)
  • Testing
  • Logging completion timestamps

---

💻 How to Run the Project

🧰 1. Install Requirements

pip install -r requirements.txt

📦 2. Add Your Data

Place your raw CSV files in:

data/raw/train.csv
data/raw/test.csv

Expected CSV Format:

"label","title","review_text"
2,"Great CD","My lovely Pat has one of the GREAT voices..."
1,"Batteries died within a year ...","I bought this charger in Jul 2003..."

⚙️ 3. Run the Full Pipeline

python run_pipeline.py

This will:

1. Clean raw training data
2. Train model (if not already trained)
3. Prepare and test on new dataset
4. Save outputs and logs

---

📊 Example Output

Console Output:

🚀 Starting Full Sentiment Analysis Pipeline...

🧹 Preparing training data...
✅ Training data prepared -> data/processed/cleaned_with_predictions.csv

🤖 Training model (this may take a few minutes)...
✅ Model trained successfully (Accuracy: 0.869)
💾 Model and vectorizer saved to 'models/'

🧠 Testing on new dataset...
✅ Accuracy: 0.871
💾 Predictions saved to 'data/processed/test_predictions.csv'
🎯 Done! Check logs for details.

Sample of test_predictions.csv:

title	processed_review	label	predicted_label	sentiment_label
Great CD	My lovely Pat...	1	1	Positive
DVD Player crapped...	The DVD side is useless...	0	0	Negative

---

📈 Model Performance

Metric	Score
Accuracy	0.869
Precision (Positive)	0.866
Recall (Positive)	0.873
F1-Score (Weighted)	0.869

The model performs balanced on both classes and generalizes well for unseen reviews.

---

🧾 Logs and Monitoring

All runs are timestamped and saved to:

etl_model.log

Example log:

2025-10-18 18:32:44 [INFO] Model trained. Accuracy: 0.869
2025-10-18 18:35:07 [INFO] Model testing completed.
2025-10-18 18:35:07 [INFO] Pipeline run completed successfully.

---

🔒 Automation Highlights

• Automatically skips retraining if sentiment_model.pkl already exists.
• Automatically creates required directories (data/processed, models).
• Logs every event and prints progress in real-time via tqdm.
• Reproducible pipeline — run anytime to reprocess or test new data.

---

🚀 Future Improvements

Feature	Description
🧠 Multi-class Sentiment	Extend to Positive, Negative, Neutral
📊 Visualization Dashboard	Add Streamlit / Dash interface for live charts
💾 Database Storage	Replace CSVs with SQLite or PostgreSQL
☁️ Cloud Deployment	Deploy pipeline on AWS / Render / Hugging Face Spaces
🔁 Auto Retraining	Schedule periodic retraining on new data
🧮 Confidence Scores	Display probabilities instead of hard labels

---

🧑‍💻 Author

Vansh C. B.Tech (CSE), SRM University Kattankulathur (2023–2027) GitHub: VanshRajput-dev Email: [email protected]

---

🏁 Summary

This project represents a complete ML pipeline integrated with ETL automation — a professional-grade system for handling real-world sentiment analysis workflows.

• ✅ Fully modular
• ✅ Automatically executable
• ✅ Easily extensible
• ✅ Production-style logging & progress