Skip to content

About

Haystack integration for CaptureKit: fetch web pages as clean Markdown documents for RAG pipelines and agents.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

capturekit-haystack

Haystack integration for CaptureKit, a web page API that loads pages in a real browser and returns them as clean Markdown.

CaptureKitFetcher fetches web pages and returns them as Haystack Documents: the content is the page as Markdown, and the metadata holds url, title and description. No HTML converter is needed.

Installation

pip install capturekit-haystack

Get an API key at app.capturekit.dev and set it in the environment:

export CAPTUREKIT_API_KEY="your-api-key"

Usage

from haystack_integrations.components.fetchers.capturekit import CaptureKitFetcher

fetcher = CaptureKitFetcher()
docs = fetcher.run(urls=["https://example.com"])["documents"]
print(docs[0].meta["title"], docs[0].content[:200])
Parameter Default Description
api_key Secret.from_env_var("CAPTUREKIT_API_KEY") CaptureKit API key
main_content_only True Keep the article or product and drop menus, sidebars and footers
raise_on_failure True Raise when a single URL fails; with several URLs, failures are logged and skipped
timeout 120 Seconds to wait for each page

A failed request raises CaptureKitError, a Haystack ComponentError with the HTTP status. The API key never appears in the message. run_async is available too.

Index pages for RAG

from haystack import Pipeline
from haystack.components.preprocessors import DocumentSplitter
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack_integrations.components.fetchers.capturekit import CaptureKitFetcher

store = InMemoryDocumentStore()
pipeline = Pipeline()
pipeline.add_component("fetcher", CaptureKitFetcher())
pipeline.add_component("splitter", DocumentSplitter(split_by="word", split_length=200))
pipeline.add_component("writer", DocumentWriter(document_store=store))
pipeline.connect("fetcher.documents", "splitter.documents")
pipeline.connect("splitter.documents", "writer.documents")

pipeline.run({"fetcher": {"urls": ["https://docs.haystack.deepset.ai/docs/intro"]}})
print(store.count_documents())

Development

pip install -e ".[test]"
pytest

License

MIT

About

Haystack integration for CaptureKit: fetch web pages as clean Markdown documents for RAG pipelines and agents.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages