Haystack integration for CaptureKit, a web page API that loads pages in a real browser and returns them as clean Markdown.
CaptureKitFetcher fetches web pages and returns them as Haystack Documents: the content is the page as Markdown, and the metadata holds url, title and description. No HTML converter is needed.
pip install capturekit-haystackGet an API key at app.capturekit.dev and set it in the environment:
export CAPTUREKIT_API_KEY="your-api-key"from haystack_integrations.components.fetchers.capturekit import CaptureKitFetcher
fetcher = CaptureKitFetcher()
docs = fetcher.run(urls=["https://example.com"])["documents"]
print(docs[0].meta["title"], docs[0].content[:200])| Parameter | Default | Description |
|---|---|---|
api_key |
Secret.from_env_var("CAPTUREKIT_API_KEY") |
CaptureKit API key |
main_content_only |
True |
Keep the article or product and drop menus, sidebars and footers |
raise_on_failure |
True |
Raise when a single URL fails; with several URLs, failures are logged and skipped |
timeout |
120 |
Seconds to wait for each page |
A failed request raises CaptureKitError, a Haystack ComponentError with the HTTP status. The API key never appears in the message. run_async is available too.
from haystack import Pipeline
from haystack.components.preprocessors import DocumentSplitter
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack_integrations.components.fetchers.capturekit import CaptureKitFetcher
store = InMemoryDocumentStore()
pipeline = Pipeline()
pipeline.add_component("fetcher", CaptureKitFetcher())
pipeline.add_component("splitter", DocumentSplitter(split_by="word", split_length=200))
pipeline.add_component("writer", DocumentWriter(document_store=store))
pipeline.connect("fetcher.documents", "splitter.documents")
pipeline.connect("splitter.documents", "writer.documents")
pipeline.run({"fetcher": {"urls": ["https://docs.haystack.deepset.ai/docs/intro"]}})
print(store.count_documents())pip install -e ".[test]"
pytestMIT