Knowledge Hub

Slack-to-Notion knowledge base automation pipeline.

Drop a link in a Slack channel, get a structured knowledge base entry in Notion — automatically. Knowledge Hub extracts content from articles, YouTube videos, and PDFs, processes it through Gemini to generate summaries, key points, and tags, then creates a fully formatted Notion page. No manual curation required.

Why This Exists

Saving interesting articles, videos, and papers to "read later" usually means "never read again." Knowledge Hub eliminates the friction between discovery and retention. Post a URL to Slack — the system extracts the content, distills it into structured notes, and files it in Notion with proper categorization, tags, and priority. You get a searchable, organized knowledge base that builds itself.

Demo

Slack Input

# Post a URL in your configured Slack channel:
https://www.youtube.com/watch?v=dQw4w9WgXcQ

Pipeline Output (Slack Thread Reply)

Saved to Notion: How to Never Give Up — A Comprehensive Guide (Cost: $0.003)

Architecture

graph TB
    User([User]) -->|Posts URL| Slack[Slack Channel]
    Slack -->|Webhook POST /slack/events| FastAPI[FastAPI Service<br/>Cloud Run]

    FastAPI -->|HMAC Verify| SigCheck{Signature<br/>Valid?}
    SigCheck -->|No| Reject[403 Reject]
    SigCheck -->|Yes| ACK[200 ACK<br/>+ Background Task]

    ACK --> Extract[Content Extraction]

    Extract -->|Articles| Trafilatura[trafilatura]
    Extract -->|YouTube| YTApi[youtube-transcript-api]
    Extract -->|PDFs| PyPDF[pypdf]

    Extract --> LLM[LLM Processing]
    LLM -->|Structured Output| Gemini[Gemini 3 Flash]

    LLM --> Notion[Notion Service]
    Notion -->|Duplicate Check| NotionAPI[Notion API]
    Notion -->|Create Page| NotionAPI

    Notion --> Notify[Slack Notification]
    Notify -->|Thread Reply| Slack

    Scheduler([Cloud Scheduler]) -->|POST /digest| FastAPI
    Scheduler -->|POST /cost-check| FastAPI

    style FastAPI fill:#009688,color:#fff
    style Gemini fill:#4285F4,color:#fff
    style NotionAPI fill:#000,color:#fff
    style Slack fill:#4A154B,color:#fff

Detailed Pipeline Flow

sequenceDiagram
    participant S as Slack
    participant F as FastAPI
    participant E as Extraction
    participant G as Gemini
    participant N as Notion

    S->>F: POST /slack/events (webhook)
    F->>F: Verify HMAC signature
    F->>F: Dedup retries (X-Slack-Retry-Num)
    F->>F: Filter (user, bot, thread, URLs)
    F-->>S: 200 OK (immediate ACK)

    Note over F: Background task starts

    F->>F: Resolve redirects (parallel)

    loop For each URL
        F->>E: extract_with_timeout(url, 30s)
        E->>E: Detect content type
        alt Article
            E->>E: trafilatura.bare_extraction()
        else YouTube
            E->>E: YouTubeTranscriptApi.fetch()
            Note over E: Falls back to Gemini<br/>if transcript unavailable
        else PDF
            E->>E: pypdf.PdfReader()
        end
        E-->>F: ExtractedContent

        F->>G: process_content(content)
        G->>G: Build prompt (type-specific)
        G->>G: Call Gemini (4 retries, exp backoff)
        G-->>F: (NotionPage, cost_usd)

        F->>N: create_notion_page(page)
        N->>N: Normalize URL, check duplicate
        N->>N: Validate tags against schema
        N->>N: Build properties + blocks
        N-->>F: PageResult | DuplicateResult

        F->>S: Thread reply (success/duplicate/error)
    end

    F->>S: Reaction (checkmark or X, best-effort)

Features

Content Ingestion

Multi-format extraction — articles (trafilatura), YouTube videos (transcript API), and PDFs (pypdf)
Smart content detection — URL pattern matching routes to the correct extractor automatically
YouTube Gemini fallback — when transcript extraction fails (e.g., cloud IP blocking), Gemini processes the video natively via its built-in video understanding
Paywall awareness — known paywalled domains flagged; partial content still processed at lower priority
Parallel URL resolution — redirect chains resolved concurrently before extraction
30-second timeout + retry — transient network errors get one automatic retry

AI Processing

Structured output — Gemini 3 Flash generates title, summary, category, tags, priority, key points, key learnings, detailed notes, and tools mentioned via JSON schema
Content-aware prompts — video, article, thread, newsletter, and short-content prompts tailored to content type
Tag validation — LLM-suggested tags are filtered against the Notion database schema; unknown tags are silently dropped
Cost tracking — per-request cost logged and accumulated for daily/weekly reporting

Knowledge Base

10-property Notion pages — Title, URL, Source, Category, Tags, Priority, Status, Content Type, Date Added, Summary
5-section page body — Summary, Key Points (numbered), Key Learnings (what/why/how-to-apply), Detailed Notes (headings + bullets), Tools & Resources Mentioned (linked list)
Duplicate detection — normalized URL matching prevents re-processing the same content
Tag schema enforcement — only pre-approved tags from the Notion database are applied

Operations

Weekly digest — Slack DM every Monday with entry counts, category breakdown, top tags, and total Gemini cost
Daily cost alerts — notification when daily Gemini spend exceeds $5
Fire-and-forget notifications — Slack notification failures never crash the pipeline
Structured JSON logging — GCP-compatible severity levels for Cloud Run observability

Tech Stack

Category	Technology	Purpose
Runtime	Python 3.12	Application language
Framework	FastAPI	Async HTTP server, webhook handling
Server	Uvicorn	ASGI server
LLM	Gemini 3 Flash Preview	Content analysis + structured output (5-min timeout for video)
Extraction	trafilatura	Article text extraction
Extraction	youtube-transcript-api	YouTube transcript extraction
Extraction	pypdf	PDF text extraction
Database	Notion API	Knowledge base storage
Messaging	Slack SDK	Webhook verification, notifications
HTTP	httpx	Async HTTP client (redirects, PDF download)
Config	pydantic-settings	Environment variable management
Retry	tenacity	Exponential backoff with jitter
Caching	cachetools	TTL cache for Notion tag schema
Logging	python-json-logger	Structured JSON log output
Packaging	uv	Fast dependency resolution + lockfile
Deployment	Google Cloud Run	Serverless container hosting
Scheduling	Google Cloud Scheduler	Cron-triggered digest + cost check
Secrets	GCP Secret Manager	Production secret injection
Linting	Ruff	Fast Python linter (E, F, I, UP rules)
Testing	pytest + pytest-asyncio	Async-native test framework

Prerequisites

Python 3.12+ — check with python3 --version
uv — Python package manager (curl -LsSf https://astral.sh/uv/install.sh | sh)
A Slack workspace with permission to create apps
A Notion workspace with an integration and a database
A Google AI API key for Gemini access
Google Cloud SDK (for deployment only) — gcloud CLI authenticated

Required External Accounts

Service	What You Need	Where to Get It
Slack	Bot token + signing secret	api.slack.com/apps
Notion	Integration token + database ID	developers.notion.com
Google AI	Gemini API key	aistudio.google.com
GCP (deploy only)	Project with Cloud Run enabled	console.cloud.google.com

Installation & Setup

1. Clone the repository

git clone https://github.com/YOUR_USERNAME/knowledge-hub.git
cd knowledge-hub

2. Install dependencies

uv sync

This installs all runtime and dev dependencies from the lockfile (uv.lock). Expect to see packages like fastapi, slack-sdk, google-genai, etc.

3. Create the environment file

cp .env.example .env   # If .env.example exists
# Otherwise, create .env manually:

Add the following variables to .env:

# Slack
SLACK_BOT_TOKEN=xoxb-your-slack-bot-token
SLACK_SIGNING_SECRET=your-slack-signing-secret
ALLOWED_USER_ID=U0XXXXXXXXX

# Notion
NOTION_API_KEY=ntn_your-notion-api-key
NOTION_DATABASE_ID=your-notion-database-uuid

# Gemini
GEMINI_API_KEY=your-gemini-api-key

# Scheduler (used for /digest and /cost-check endpoints)
SCHEDULER_SECRET=any-random-secret-string

# App
ENVIRONMENT=development
LOG_LEVEL=INFO
PORT=8080

Secure the file:

chmod 600 .env

4. Set up the Slack app

Go to api.slack.com/apps and create a new app
Under OAuth & Permissions, add these bot token scopes:
- chat:write — send messages
- channels:history — read channel messages
- reactions:write — add reactions (optional — the bot works without this but won't add checkmark/X reactions)
Under Event Subscriptions, enable events and set the request URL to your service URL + /slack/events (e.g., https://your-service.run.app/slack/events)
Subscribe to the message.channels bot event
Install the app to your workspace
Copy the Bot User OAuth Token (xoxb-...) to SLACK_BOT_TOKEN
Copy the Signing Secret (under Basic Information) to SLACK_SIGNING_SECRET
Invite the bot to your target Slack channel — the channel must be public (message.channels does not receive events from private channels)

5. Set up the Notion database

Go to notion.so and create a new database with these properties:

Property	Type	Notes
Title	Title (default)	Cleaned-up content title
Category	Select	See categories below
Content Type	Select: `Video`, `Article`, `Newsletter`, `Podcast`, `Thread`, `LinkedIn Post`, `PDF`
Source	URL	Original link
Author/Creator	Rich text
Date Added	Date
Status	Select: `New`, `Reviewed`, `Applied`, `Archived`	Default: New
Priority	Select: `High`, `Medium`, `Low`
Tags	Multi-select	Pre-populate with relevant tags
Summary	Rich text

Categories (create all 11 as Select options): AI & Machine Learning, Marketing & Growth, Ad Tech & Media, Product & Strategy, Engineering & Development, Data & Analytics, Career & Professional Development, Productivity & Systems, Design & UX, Business & Finance, Miscellaneous

Create a Notion internal integration and connect it to the database (click "..." on the database → Connections → add your integration)
Copy the integration token to NOTION_API_KEY
Copy the database ID from the URL (the 32-char hex string) to NOTION_DATABASE_ID

6. Get a Gemini API key

Go to aistudio.google.com
Create an API key
Copy it to GEMINI_API_KEY

7. Find your Slack user ID

In Slack, click your profile picture → Profile
Click the three dots (...) → Copy member ID
Set this as ALLOWED_USER_ID in .env (only this user's messages will be processed)

Running the Project

Development Mode

uv run uvicorn knowledge_hub.app:app --reload

Expected output:

INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [12345] using StatReload

For local Slack webhook testing, use a tunnel like ngrok:

ngrok http 8000

Then set the ngrok HTTPS URL as your Slack Event Subscriptions request URL.

Production Mode (Local)

uv run uvicorn knowledge_hub.app:app --host 0.0.0.0 --port 8080

Docker

docker build -t knowledge-hub .
docker run -p 8080:8080 --env-file .env knowledge-hub

Running Tests

uv run pytest

Expected output:

================================ test session starts =================================
collected 237 items

tests/test_health.py ..                                                        [  0%]
tests/test_app.py ......                                                       [  3%]
tests/test_cost.py .....                                                       [  5%]
...
================================ 237 passed in 1.82s =================================

Run a specific test module:

uv run pytest tests/test_slack/test_pipeline.py -v

Linting

uv run ruff check src/ tests/
uv run ruff format --check src/ tests/

Deployment

Knowledge Hub is designed for Google Cloud Run with secrets managed via GCP Secret Manager.

Architecture

graph LR
    Slack([Slack API]) -->|Webhook| CR[Cloud Run<br/>knowledge-hub]
    CS([Cloud Scheduler]) -->|POST /digest<br/>POST /cost-check| CR
    CR -->|API calls| Gemini[Gemini API]
    CR -->|API calls| NotionAPI[Notion API]
    CR -->|Replies| Slack
    SM[(Secret Manager)] -->|Injected at boot| CR

    style CR fill:#4285F4,color:#fff
    style SM fill:#F4B400,color:#000

Step 1: Create secrets in GCP Secret Manager

PROJECT_ID=your-gcp-project

# Create each secret (you'll be prompted to enter the value)
for secret in slack-bot-token slack-signing-secret notion-api-key \
              notion-database-id gemini-api-key allowed-user-id scheduler-secret; do
    echo -n "Enter value for $secret: "
    read -s value
    echo
    echo -n "$value" | gcloud secrets create "$secret" \
        --project="$PROJECT_ID" \
        --data-file=-
done

Step 2: Grant the service account access

# The default compute service account needs secret access
gcloud projects add-iam-policy-binding "$PROJECT_ID" \
    --member="serviceAccount:$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')[email protected]" \
    --role="roles/secretmanager.secretAccessor"

Step 3: Deploy to Cloud Run

PROJECT_ID=your-gcp-project ./deploy.sh

What deploy.sh does

Runs gcloud run deploy --source . (builds the container remotely via Cloud Build)
Sets region to europe-west4
Injects 7 secrets from Secret Manager
Configures: --min-instances=1, --memory=512Mi, --cpu=1, --allow-unauthenticated
Outputs the service URL

Expected output:

Deploying knowledge-hub to Cloud Run...
  Project:  your-gcp-project
  Region:   europe-west4
...
Deployment complete. Service URL:
https://knowledge-hub-xxxxxxxxxx-ez.a.run.app

Step 4: Configure Slack webhook URL

Update your Slack app's Event Subscriptions request URL to:

https://knowledge-hub-xxxxxxxxxx-ez.a.run.app/slack/events

Step 5: Set up Cloud Scheduler (optional)

For weekly digest and daily cost alerts, follow the setup commands in deploy.sh comments:

# Create scheduler service account
gcloud iam service-accounts create scheduler-sa \
    --project="$PROJECT_ID" \
    --display-name="Cloud Scheduler SA"

# Grant Cloud Run invoker role
gcloud run services add-iam-policy-binding knowledge-hub \
    --region=europe-west4 \
    --project="$PROJECT_ID" \
    --member="serviceAccount:scheduler-sa@${PROJECT_ID}.iam.gserviceaccount.com" \
    --role="roles/run.invoker"

# Weekly digest — Monday 08:00 Amsterdam
# Note: Cloud Scheduler may not be available in all regions.
# Use a nearby region (e.g., europe-west3) if your Cloud Run region isn't supported.
gcloud scheduler jobs create http weekly-digest \
    --project="$PROJECT_ID" \
    --location=europe-west3 \
    --schedule="0 8 * * 1" \
    --time-zone="Europe/Amsterdam" \
    --http-method=POST \
    --uri="SERVICE_URL/digest" \
    --oidc-service-account-email="scheduler-sa@${PROJECT_ID}.iam.gserviceaccount.com" \
    --oidc-token-audience="SERVICE_URL"

# Daily cost check — 23:55 Amsterdam
gcloud scheduler jobs create http daily-cost-check \
    --project="$PROJECT_ID" \
    --location=europe-west3 \
    --schedule="55 23 * * *" \
    --time-zone="Europe/Amsterdam" \
    --http-method=POST \
    --uri="SERVICE_URL/cost-check" \
    --oidc-service-account-email="scheduler-sa@${PROJECT_ID}.iam.gserviceaccount.com" \
    --oidc-token-audience="SERVICE_URL"

Replace SERVICE_URL with the URL from Step 3.

Step 6: Verify deployment

# Health check
curl https://knowledge-hub-xxxxxxxxxx-ez.a.run.app/health
# Expected: {"status":"ok","service":"knowledge-hub","version":"0.1.0"}

Then post a URL in your Slack channel and verify a Notion page is created.

API Reference

Method	Route	Description	Auth
`GET`	`/health`	Health check	None
`POST`	`/slack/events`	Slack webhook receiver	HMAC signature
`POST`	`/digest`	Trigger weekly digest	`X-Scheduler-Secret` header
`POST`	`/cost-check`	Trigger daily cost alert	`X-Scheduler-Secret` header

`GET /health`

curl https://your-service.run.app/health

{
  "status": "ok",
  "service": "knowledge-hub",
  "version": "0.1.0"
}

`POST /slack/events`

Receives Slack Event API payloads. Handles two event types:

url_verification — responds with the challenge token (Slack setup handshake)
event_callback — processes message events from the subscribed channel

Slack retries are deduplicated via the X-Slack-Retry-Num header.

`POST /digest`

Requires X-Scheduler-Secret header matching the configured SCHEDULER_SECRET.

curl -X POST https://your-service.run.app/digest \
  -H "X-Scheduler-Secret: your-secret"

{
  "status": "sent",
  "entries_count": 12,
  "period_days": 7
}

`POST /cost-check`

Requires X-Scheduler-Secret header.

curl -X POST https://your-service.run.app/cost-check \
  -H "X-Scheduler-Secret: your-secret"

{
  "status": "ok",
  "daily_cost_usd": 0.042,
  "alert_sent": false
}

Project Structure

knowledge-hub/
├── src/knowledge_hub/
│   ├── app.py                          # FastAPI app, health + scheduled endpoints
│   ├── config.py                       # pydantic-settings configuration
│   ├── cost.py                         # Gemini cost tracking + accumulators
│   ├── digest.py                       # Weekly digest + daily cost alerts
│   ├── logging_config.py              # Structured JSON logging for GCP
│   ├── models/
│   │   ├── content.py                  # ExtractedContent, ContentType, ExtractionStatus
│   │   ├── knowledge.py               # KnowledgeEntry, Category, Priority enums
│   │   ├── notion.py                   # NotionPage (entry + 5 body sections)
│   │   └── slack.py                    # SlackEvent model
│   ├── extraction/
│   │   ├── router.py                   # URL → content type detection
│   │   ├── article.py                  # trafilatura article extraction
│   │   ├── youtube.py                  # YouTube transcript extraction
│   │   ├── pdf.py                      # PDF text extraction
│   │   ├── paywall.py                  # Paywalled domain detection
│   │   ├── paywalled_domains.yaml      # Known paywalled domains list
│   │   └── timeout.py                  # 30s timeout + retry wrapper
│   ├── llm/
│   │   ├── client.py                   # Gemini client singleton
│   │   ├── processor.py               # Content → NotionPage via Gemini
│   │   ├── prompts.py                  # System/user prompt templates
│   │   └── schemas.py                  # LLMResponse structured output schema
│   ├── notion/
│   │   ├── client.py                   # Notion client singleton
│   │   ├── service.py                  # Page creation orchestrator
│   │   ├── properties.py              # Notion property builder
│   │   ├── blocks.py                   # Notion block builder (page body)
│   │   ├── duplicates.py              # URL normalization + duplicate check
│   │   ├── tags.py                     # Tag schema cache + validation
│   │   └── models.py                   # PageResult, DuplicateResult
│   └── slack/
│       ├── router.py                   # POST /slack/events route
│       ├── handlers.py                # Event dispatch + pipeline orchestration
│       ├── client.py                   # Slack client singleton
│       ├── notifier.py                # Fire-and-forget Slack notifications
│       ├── urls.py                     # URL extraction + redirect resolution
│       └── verification.py           # HMAC signature verification
├── tests/                              # 237 tests mirroring src/ structure
├── docs/
│   ├── KB-Automation-PRD.md           # Original product requirements document
│   └── screenshots/                    # Screenshots (see Demo section)
├── Dockerfile                          # Python 3.12-slim + uv
├── deploy.sh                           # Cloud Run deployment script
├── pyproject.toml                      # Dependencies, pytest/ruff config
├── uv.lock                             # Locked dependency tree
└── .python-version                     # 3.12

Configuration

All configuration is via environment variables (or .env file for local development).

Variable	Required	Default	Description
`SLACK_BOT_TOKEN`	Yes	`""`	Slack bot OAuth token (`xoxb-...`)
`SLACK_SIGNING_SECRET`	Yes	`""`	Slack app signing secret for webhook HMAC verification
`ALLOWED_USER_ID`	Yes	`""`	Slack user ID of the single authorized user
`NOTION_API_KEY`	Yes	`""`	Notion integration token
`NOTION_DATABASE_ID`	Yes	`""`	UUID of the target Notion database
`GEMINI_API_KEY`	Yes	`""`	Google AI API key for Gemini
`SCHEDULER_SECRET`	Yes	`""`	Shared secret for `/digest` and `/cost-check` auth
`YOUTUBE_PROXY_URL`	No	`""`	HTTPS proxy for YouTube transcript requests (bypasses cloud IP blocking)
`ENVIRONMENT`	No	`development`	App environment (`development` or `production`)
`LOG_LEVEL`	No	`INFO`	Python logging level
`PORT`	No	`8080`	HTTP server port

Troubleshooting

Slack events not arriving

Verify the request URL in Slack Event Subscriptions matches your service URL + /slack/events
The channel must be public — message.channels only fires for public channels. If your channel is private, convert it to public in Slack channel settings.
Check that the bot is invited to the target channel (channel settings → Integrations → Add apps)
Confirm SLACK_SIGNING_SECRET matches the value in your Slack app's Basic Information page (not the bot token)
For local development, ensure your tunnel (ngrok) is running and the URL is updated in Slack

"Invalid scheduler secret" on `/digest` or `/cost-check`

The X-Scheduler-Secret header must exactly match the SCHEDULER_SECRET environment variable
For Cloud Scheduler, the secret is passed via the --headers flag or as a request header in the job configuration

Notion page creation fails with "validation_error"

Tag mismatch: The LLM may suggest tags not present in your Notion database schema. Tags are filtered automatically, but if the Tags property has no options at all, add some initial options to the database
Property mismatch: Ensure all 10 properties exist in the Notion database with the exact types listed in Setup Step 5
Verify the Notion integration has access to the database (click "..." on the database → "Connections" → add your integration)

YouTube transcripts not extracted

The video may have transcripts disabled by the creator
Auto-generated captions are used as fallback; if neither exists, the extraction status will be METADATA_ONLY
Cloud IP blocking: YouTube may block transcript requests from cloud provider IPs (including Cloud Run). When this happens, the pipeline automatically falls back to Gemini's native video understanding — the video URL is passed directly to Gemini for analysis. Quality is comparable but timestamps may be less precise.
Check the logs for TranscriptsDisabled, NoTranscriptFound, or will use Gemini fallback messages

High Gemini costs

Each URL costs approximately $0.001–$0.005 depending on content length
The daily cost check alerts at $5/day; adjust monitoring via the /cost-check endpoint
Cost accumulators reset on Cloud Run instance restart (since they're in-memory)

Contributing

Fork the repository
Create a feature branch: git checkout -b feature/your-feature
Install dev dependencies: uv sync
Make your changes
Run linting: uv run ruff check src/ tests/
Run tests: uv run pytest
Commit and push your branch
Open a pull request

Code Style

Linter: Ruff with rules E, F, I, UP (errors, pyflakes, isort, pyupgrade)
Line length: 100 characters
Target: Python 3.12
Async tests: asyncio_mode = "auto" — no need for @pytest.mark.asyncio

License

No license file is currently included in this repository.

Name		Name	Last commit message	Last commit date
.planning		.planning
docs/screenshots		docs/screenshots
src/knowledge_hub		src/knowledge_hub
tests		tests
.dockerignore		.dockerignore
.python-version		.python-version
Dockerfile		Dockerfile
README.md		README.md
deploy.sh		deploy.sh
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation

Knowledge Hub

Table of Contents

Why This Exists

Demo

Slack Input

Pipeline Output (Slack Thread Reply)

Architecture

Detailed Pipeline Flow

Features

Content Ingestion

AI Processing

Knowledge Base

Operations

Tech Stack

Prerequisites

Required External Accounts

Installation & Setup

1. Clone the repository

2. Install dependencies

3. Create the environment file

4. Set up the Slack app

5. Set up the Notion database

6. Get a Gemini API key

7. Find your Slack user ID

Running the Project

Development Mode

Production Mode (Local)

Docker

Running Tests

Linting

Deployment

Architecture

Step 1: Create secrets in GCP Secret Manager

Step 2: Grant the service account access

Step 3: Deploy to Cloud Run

Step 4: Configure Slack webhook URL

Step 5: Set up Cloud Scheduler (optional)

Step 6: Verify deployment

API Reference

GET /health

POST /slack/events

POST /digest

POST /cost-check

Project Structure

Configuration

Troubleshooting

Slack events not arriving

"Invalid scheduler secret" on /digest or /cost-check

Notion page creation fails with "validation_error"

YouTube transcripts not extracted

High Gemini costs

Contributing

Code Style

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`GET /health`

`POST /slack/events`

`POST /digest`

`POST /cost-check`

"Invalid scheduler secret" on `/digest` or `/cost-check`