Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

🌍 GeoFieldKit

A Python pipeline for processing, validating, and reporting on ODK geospatial field data collections.

GeoFieldKit automates the full workflow from raw ODK/KoboToolbox exports to clean reports β€” tracking worker attendance, GPS data quality, payroll calculations, and interactive maps. Built for field survey teams collecting geospatial points using mobile data collection tools.


πŸ“Œ Table of Contents


Overview

Field teams using ODK Collect or KoboToolbox generate large volumes of raw submission data β€” messy column names, duplicate GPS fields, missing values, and no easy way to track who worked, when, and how accurately. GeoFieldKit solves this with 7 focused Python scripts that take you from raw export to professional reports in minutes.

Designed for teams collecting 10 GPS points per worker per day across field surveys, land use mapping, environmental monitoring, and similar data collection programs.


✨ Features

  • 🧹 Smart ODK Cleaning β€” strips redundant system columns, renames fields, fixes types
  • πŸ“… Attendance Tracking β€” clock-in/out analysis, early/late/absent flags, attendance matrix
  • πŸ’° Payroll Calculation β€” daily pay with overtime rules, monthly summaries
  • πŸ“ GPS Quality Control β€” accuracy checks, boundary validation, duplicate detection, PASS/REVIEW/FAIL ratings
  • πŸ“Š Automated Reporting β€” executive summary with embedded charts (hours, validity, late arrivals, weekly trend)
  • πŸ—ΊοΈ Interactive Maps β€” colour-coded Leaflet maps, per-worker layers, heatmaps, GeoJSON/QGIS export
  • βš™οΈ Full Automation β€” folder watcher + email delivery when new ODK export lands

πŸ“ Project Structure

GeoFieldKit/
β”‚
β”œβ”€β”€ script_1_cleaning.py        # ODK data cleaning & standardization
β”œβ”€β”€ script_2_attendance.py      # Attendance & clock-in analysis
β”œβ”€β”€ script_3_payroll.py         # Payroll calculation with overtime
β”œβ”€β”€ script_4_gps_qc.py          # GPS & data quality control
β”œβ”€β”€ script_5_reporting.py       # Excel reports with embedded charts
β”œβ”€β”€ script_6_geospatial.py      # Interactive maps & GeoJSON export
β”œβ”€β”€ script_7_automation.py      # Full pipeline automation + email
β”‚
β”œβ”€β”€ ODK_Raw_Export.xlsx          # ← Drop your ODK export here
β”‚
β”œβ”€β”€ outputs/                     # All generated reports go here
β”‚   β”œβ”€β”€ cleaned_data.xlsx
β”‚   β”œβ”€β”€ attendance_report.xlsx
β”‚   β”œβ”€β”€ payroll_report.xlsx
β”‚   β”œβ”€β”€ quality_control_report.xlsx
β”‚   β”œβ”€β”€ full_report.xlsx
β”‚   β”œβ”€β”€ map_all_points.html
β”‚   β”œβ”€β”€ map_by_worker.html
β”‚   β”œβ”€β”€ heatmap.html
β”‚   └── points_export.geojson
β”‚
β”œβ”€β”€ incoming/                    # Drop new exports here for auto-processing
β”œβ”€β”€ pipeline.log                 # Automatic run logs
└── README.md

πŸ› οΈ Installation

Requirements: Python 3.8+

# Clone the repository
git clone https://github.com/nxrth1/GeoFieldKit.git
cd GeoFieldKit

# Install dependencies
pip install pandas openpyxl matplotlib folium watchdog

πŸš€ Quick Start

  1. Place your ODK/KoboToolbox .xlsx export in the project folder and rename it ODK_Raw_Export.xlsx

  2. Run the full pipeline at once:

python script_7_automation.py --file ODK_Raw_Export.xlsx --no-email
  1. Or run scripts individually in order:
python script_1_cleaning.py       # Always run first
python script_2_attendance.py
python script_3_payroll.py
python script_4_gps_qc.py
python script_5_reporting.py
python script_6_geospatial.py
  1. Find all outputs in the outputs/ folder.

πŸ“œ Scripts

Script 1 β€” Data Cleaning

Loads the raw ODK export and produces a clean, analysis-ready dataset.

  • Drops redundant ODK system columns (subscriberid, simserial, hidden _gps_* duplicates)
  • Renames ODK-style columns (gps_location-Latitude β†’ latitude)
  • Fixes data types β€” dates, times, numerics, categoricals
  • Flags missing GPS values
  • Adds computed hours_worked and clock_in_status columns
python script_1_cleaning.py
# Input:  ODK_Raw_Export.xlsx
# Output: cleaned_data.xlsx, cleaned_data.csv

Script 2 β€” Attendance Analysis

Builds a full attendance register from the cleaned data.

  • Clock-in status per worker per day: Early / On Time / Late (with exact minutes)
  • Detects absent workers (expected but no submission)
  • Calculates total hours, average hours, attendance rate %
  • Flags chronic latecomers (configurable threshold)
  • Outputs a calendar-grid attendance matrix
python script_2_attendance.py
# Input:  cleaned_data.csv
# Output: attendance_report.xlsx (3 sheets)

Script 3 β€” Payroll Calculation

Calculates pay for each worker based on hours worked.

  • Configurable hourly rates per worker
  • Overtime rules: hours > 8 billed at 1.5Γ— rate
  • Classifies days: Full Day / Half Day / Overtime / Absent
  • Monthly breakdown per worker
  • Grand total payroll summary
python script_3_payroll.py
# Input:  cleaned_data.csv
# Output: payroll_report.xlsx (3 sheets)

Script 4 β€” GPS Quality Control

Validates every GPS point against a set of quality rules.

  • Accuracy threshold check (default: < 10 metres)
  • Boundary box validation (configurable to your study area)
  • Duplicate coordinate detection per worker per day
  • Timestamp gap checks (flags suspiciously fast collections)
  • Distance between consecutive points
  • Per-worker QC rating: PASS / REVIEW / FAIL
python script_4_gps_qc.py
# Input:  cleaned_data.csv
# Output: quality_control_report.xlsx (3 sheets)

Script 5 β€” Reporting & Charts

Generates a polished Excel report with KPI cards and embedded charts.

  • Executive summary sheet with 8 KPI metrics
  • Bar chart: Total hours per worker
  • Bar chart: GPS validity % per worker (colour-coded PASS/REVIEW/FAIL)
  • Bar chart: Late clock-ins per worker
  • Line chart: Weekly validity trend
python script_5_reporting.py
# Input:  cleaned_data.csv
# Output: full_report.xlsx

Script 6 β€” Geospatial Maps

Produces interactive HTML maps and GIS-ready exports.

  • map_all_points.html β€” all points colour-coded (valid=green, bad accuracy=red, out of bounds=orange)
  • map_by_worker.html β€” one layer per worker, toggle on/off
  • heatmap.html β€” collection density heatmap
  • points_export.geojson β€” import directly into QGIS, ArcGIS, or Google Earth
python script_6_geospatial.py
# Input:  cleaned_data.csv
# Output: map_*.html, points_export.geojson

Script 7 β€” Automation

Orchestrates all 6 scripts and handles delivery.

  • Run the full pipeline on a single file
  • Watch a folder and auto-trigger when a new export is dropped in
  • Email the completed reports to a list of recipients
  • Logs every run to pipeline.log
# Run once on a file
python script_7_automation.py --file ODK_Raw_Export.xlsx

# Watch a folder for new exports
python script_7_automation.py --watch ./incoming

# Run without sending email
python script_7_automation.py --file ODK_Raw_Export.xlsx --no-email

βš™οΈ Configuration

Each script has a CONFIG section at the top. Key settings to update for your project:

Setting Script Default Description
HOURLY_RATES Script 3 18.00 Pay rate per worker ID
STANDARD_HOURS Script 3 8.0 Hours before overtime kicks in
OVERTIME_MULT Script 3 1.5 Overtime pay multiplier
ACCURACY_THRESHOLD Scripts 4 & 6 10.0 m Max acceptable GPS accuracy
BOUNDARY Scripts 4 & 6 Nairobi area Lat/lon bounding box for your study area
POINTS_REQUIRED Scripts 4 & 5 10 Expected points per worker per day
REQUIRED_START Scripts 1 & 2 08:00 Required clock-in time
LATE_THRESHOLD Script 2 5 min Grace period before marked Late
CHRONIC_LATE Script 2 3 days Days late before flagged chronic
EMAIL_CONFIG Script 7 Placeholder SMTP settings and recipient list

πŸ“€ Outputs

File Description
cleaned_data.xlsx/.csv Standardized ODK data ready for analysis
attendance_report.xlsx Daily register, worker summary, attendance matrix
payroll_report.xlsx Daily register, worker payroll summary, monthly totals
quality_control_report.xlsx Point-level flags, daily QC, worker PASS/REVIEW/FAIL
full_report.xlsx Executive summary with KPI cards and 4 charts
map_all_points.html Interactive map of all points (open in browser)
map_by_worker.html Per-worker toggle map
heatmap.html Collection density heatmap
points_export.geojson GIS-ready export for QGIS / ArcGIS

🎯 Use Cases

  • Land use mapping β€” validate and report on field teams collecting land cover data
  • Environmental surveys β€” track GPS collection quality across a study area
  • Agricultural assessments β€” monitor field agent attendance and productivity
  • Infrastructure surveys β€” QC GPS points for utility or road mapping
  • Any ODK/KoboToolbox project with daily point collection targets and field workers to track

🧰 Tech Stack

Library Purpose
pandas Data cleaning, analysis, grouping
openpyxl Excel file creation and formatting
matplotlib Chart generation
folium Interactive Leaflet maps
watchdog Folder monitoring for automation
smtplib Email delivery

🀝 Contributing

Contributions are welcome! To contribute:

  1. Fork the repository
  2. Create a feature branch: git checkout -b feature/your-feature-name
  3. Commit your changes: git commit -m "Add your feature"
  4. Push to your branch: git push origin feature/your-feature-name
  5. Open a Pull Request

Please keep each script self-contained and include comments for any new config options.


πŸ“„ License

This project is licensed under the MIT License β€” see the LICENSE file for details.


πŸ‘€ Author

Mark Mwari GitHub: @nxrth1


Built for field data teams working with ODK Collect and KoboToolbox.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages