Skip to content
Aegis-OutreachPublic

About

High-Performance Asynchronous Intelligence Scraping, Vectorized Scoring, & Automated Dispatch Orchestrator

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

Β 

History

4 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Aegis Outreach Banner

Aegis Outreach

High-Performance Asynchronous Intelligence Scraping, Vectorized Scoring, & Automated Dispatch Orchestrator


πŸ“œ The Myth, The Legend, The Backronym

At first glance, Aegis Outreach sounds like a high-budget, enterprise-grade cybersecurity initiative engineered by defense contractors to shield a global Fortune 500 brand's outbound communication reputation.

In reality, it is a deeply engineered, meticulously crafted backronym for:

Autonomous Engine for Gathering Intelligence & Sending Signals

Let’s be entirely transparent: the lead architect spent considerably more time wrestling with ancient Greek mythology dictionaries to transform "Aegis" (the legendary shield of Zeus and Athena) into an engineering backronym than it took to fight Selenium's anti-bot evasions and headless Chrome driver flags. While Zeus used his Aegis to inspire awe and strike fear, this Aegis ensures your event loop never blocks while harvesting high-value corporate leads and orchestrating transactional dispatches.


πŸ—οΈ Core Architecture & The Non-Blocking Concurrency Flow

Aegis Outreach is built on top of FastAPI (main.py) to serve as an ultra-fast, asynchronous API gateway. However, real-world outreach engines must handle heavy, blocking, CPU- and I/O-intensive operations: running headless browser instances via Selenium, parsing massive HTML trees with BeautifulSoup4, and processing multi-column array transformations across Pandas DataFrames and NumPy arrays.

If executed naively inside an async request handler, these synchronous tasks freeze Python's single-threaded event loop, dropping incoming concurrent HTTP connections and stalling outbound network requests.

To mitigate this, Aegis Outreach decouples thread-bound synchronous operations from the primary asynchronous loop using Python’s native asyncio.to_thread() execution pool.

Server Infrastructure

Async Execution Flowchart

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                                 INCOMING HTTP REQUEST                            β”‚
β”‚                      (POST /api/v1/orchestrate-outreach)                         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                         β”‚
                                         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                            FastAPI Async Event Loop                              β”‚
β”‚                                   (main.py)                                      β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                    β”‚                                              β”‚
        [Non-Blocking Dispatch]                               [Async Network IO]
                    β”‚                                              β”‚
                    β–Ό                                              β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”      β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚        ThreadPoolWorker Pool            β”‚      β”‚       Gmail API Client          β”‚
β”‚       (via asyncio.to_thread)           β”‚      β”‚   (httpx / Google OAuth2)       β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€      β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ 1. Selenium-Stealth DOM Scraping        β”‚      β”‚ 1. Await OAuth Token Refresh    β”‚
β”‚ 2. BeautifulSoup4 (lxml) Tree Extract   β”‚      β”‚ 2. Transmit Base64-URL Payload  β”‚
β”‚ 3. Pandas / NumPy Vectorized Scoring    β”‚      β”‚ 3. Exponential Backoff Retry    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜      β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                    β”‚                                             β”‚
                    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                           β”‚
                                           β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                             HTTP 200 OK RESPONSE                                 β”‚
β”‚                   { status: "success", leads_processed: N }                      β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

πŸ› οΈ Deep-Dive Technical Mechanics

Cyber Security Interface

1. Data Harvesting Engine (Selenium-Stealth + BeautifulSoup4)

The scraping pipeline operates under a strict rapid-exit browser model designed to minimize memory footprint and eliminate headless detection vectors.

  • Anti-Bot Evasion: Utilizes selenium-stealth to override navigator.webdriver flags, mask user-agent profiles, patch WebGL vendor strings, and randomize window dimensions.
  • Immediate Resource Freeing: Rather than navigating through multi-step browser DOM selections (which keeps heavy Chromedriver processes alive in memory), Aegis retrieves the raw driver.page_source string in a single synchronous cycle.
  • lxml Ingestion: The raw HTML string is passed directly to BeautifulSoup(page_source, 'lxml'). The browser instance is immediately torn down (driver.quit()), delegating tree traversing and CSS/XPath parsing to C-optimized CPython modules.
# Conceptual snippet from orchestrator worker
import asyncio
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium_stealth import stealth

def harvest_raw_dom(target_url: str) -> str:
    chrome_options = Options()
    chrome_options.add_argument("--headless=new")
    chrome_options.add_argument("--disable-gpu")
    chrome_options.add_argument("--no-sandbox")
    
    driver = webdriver.Chrome(options=chrome_options)
    stealth(
        driver,
        languages=["en-US", "en"],
        vendor="Google Inc.",
        platform="Win32",
        webgl_vendor="Intel Inc.",
        renderer="Intel Iris OpenGL Engine",
        fix_hairline=True,
    )
    
    try:
        driver.get(target_url)
        return driver.page_source
    finally:
        driver.quit()

async def get_parsed_html(target_url: str) -> BeautifulSoup:
    raw_html = await asyncio.to_thread(harvest_raw_dom, target_url)
    return BeautifulSoup(raw_html, "lxml")
2. Vectorized Intelligence Pipelines (Pandas + NumPy Matrix Priorities)

Raw scraped leads are converted into structured Pandas DataFrames to perform vectorized cleanups and calculate lead scores without looping over Python dictionaries.

  • Vectorized Cleaning: Replaces null structures, standardizes email text formatting, strips trailing domains, and normalizes corporate names using string vector operations (df['company'].str.strip()).
  • NumPy Matrix Prioritization: Rather than executing conditional if/else logic per record, Aegis uses np.select and matrix dot-products (np.dot) to generate scaling priority tiers (e.g., matching keywords, domain authority signals, and employee size thresholds).
import numpy as np
import pandas as pd

def compute_lead_scores(df: pd.DataFrame) -> pd.DataFrame:
    # Standardize string fields
    df['email'] = df['email'].str.lower().str.strip()
    df['company'] = df['company'].str.title().str.strip()

    # Rule-based conditions for prioritization matrix
    conditions = [
        (df['employee_count'] > 500) & (df['verified_domain'] == True),
        (df['employee_count'] > 100) & (df['verified_domain'] == True),
        (df['verified_domain'] == True)
    ]
    
    choices = [100, 75, 50]
    
    # Assign baseline scoring via vectorized selection
    df['base_score'] = np.select(conditions, choices, default=10)
    
    # Vectorized multi-factor weighting using np.dot
    weights = np.array([0.6, 0.4])
    feature_matrix = df[['base_score', 'engagement_index']].values
    df['final_priority_score'] = np.dot(feature_matrix, weights)
    
    return df.sort_values(by='final_priority_score', ascending=False)
3. Secure Delivery Engine (Google API Client + OAuth2)

Once intelligence payloads are compiled, the delivery subsystem interfaces directly with the Gmail API via official OAuth2 client libraries.

  • Base64-URL MIME Encoding: Constructs RFC 2822 compliant MIME messages using Python's email.mime modules and translates them into URL-safe Base64 encoded strings (base64.urlsafe_b64encode).
  • Resilient Delivery Cycles: Outbound REST calls are wrapped with automated exponential backoff retries using httpx or Google API's built-in execution handlers to gracefully bypass HTTP 429 (Rate Limit Exceeded) and transient 5xx server responses.
import base64
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from googleapiclient.discovery import build

def create_encoded_message(sender: str, to: str, subject: str, body_html: str) -> dict:
    message = MIMEMultipart("alternative")
    message["to"] = to
    message["from"] = sender
    message["subject"] = subject
    
    part = MIMEText(body_html, "html")
    message.attach(part)
    
    raw_bytes = message.as_bytes()
    encoded_message = base64.urlsafe_b64encode(raw_bytes).decode("utf-8")
    return {"raw": encoded_message}

async def send_outbound_email(service_credentials, sender: str, to: str, subject: str, body: str):
    service = build("gmail", "v1", credentials=service_credentials)
    payload = create_encoded_message(sender, to, subject, body)
    
    # Non-blocking thread execution for API call
    response = await asyncio.to_thread(
        service.users().messages().send(userId="me", body=payload).execute
    )
    return response

πŸš€ Local Deployment & Production Run

Follow these precise steps to deploy and run Aegis Outreach in your local environment.

1. Clone the Repository

git clone https://github.com/rayaantasnim/Aegis-Outreach.git
cd Aegis-Outreach

2. Create and Activate a Virtual Environment

On macOS/Linux:

python3 -m venv venv
source venv/bin/activate

On Windows (PowerShell):

python -m venv venv
.\venv\Scripts\Activate.ps1

3. Install Core Production Dependencies

Execute the following command to install all required libraries:

pip install fastapi uvicorn selenium selenium-stealth beautifulsoup4 lxml pandas numpy google-api-python-client google-auth-oauthlib pydantic-settings httpx

4. Launch the ASGI Production Server

Start the application with standard auto-reload flags for development:

uvicorn main:app --reload

The service will spin up on http://127.0.0.1:8000. You can inspect the automatically generated Swagger API documentation by navigating to http://127.0.0.1:8000/docs.



Authorized by Rayaan Tasnim
Β© All rights Reserved

About

High-Performance Asynchronous Intelligence Scraping, Vectorized Scoring, & Automated Dispatch Orchestrator

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages