From 5281bf6cecaac514f477ddb7c4205deef14278b1 Mon Sep 17 00:00:00 2001 From: aaronkanzer <36093535+aaronkanzer@users.noreply.github.com> Date: Thu, 29 Oct 2026 23:48:09 -0400 Subject: [PATCH] feat: add scigantic-catalog agent Prompt agent for public dataset discovery over the anonymous Scigantic MCP server: catalog search, schema cards, loader snippets, arXiv-to-dataset lookup and SureChEMBL patent chemistry. No containerized tools, no API key. --- agents/scigantic-catalog/README.md | 163 +++++++++++++++++++++++++ agents/scigantic-catalog/agent.yaml | 93 ++++++++++++++ agents/scigantic-catalog/mcp/README.md | 92 ++++++++++++++ agents/scigantic-catalog/metadata.yaml | 24 ++++ 4 files changed, 372 insertions(+) create mode 100644 agents/scigantic-catalog/README.md create mode 100644 agents/scigantic-catalog/agent.yaml create mode 100644 agents/scigantic-catalog/mcp/README.md create mode 100644 agents/scigantic-catalog/metadata.yaml diff --git a/agents/scigantic-catalog/README.md b/agents/scigantic-catalog/README.md new file mode 100644 index 0000000..07c7efd --- /dev/null +++ b/agents/scigantic-catalog/README.md @@ -0,0 +1,163 @@ +# Scigantic Catalog + +A research data librarian for public scientific datasets. `SciganticCatalog` searches the +[Scigantic](https://scigantic.com) catalog of several thousand curated open archives, returns each +archive's schema card (format, columns, sample rows, data dictionaries, caveats) and a copy-paste +loader for the user's own environment, and links arXiv papers to the datasets behind them. It reads +the catalog through the public [Scigantic MCP server](https://github.com/Scigantic/scigantic-mcp), +which needs no API key. + +## Overview + +Finding a public dataset normally means a web search, a registry page, a bucket listing and a +download before you know whether the columns you need are there. This agent answers those +questions from the catalog's schema cards first. + +Ask it for data on a topic and it searches the catalog, discards archives whose storage was probed +and found unreadable, checks the schema cards of the best candidates, and recommends one with its +storage location, format, key columns and a loader snippet. Ask it about a paper and it finds the +datasets the paper used or suggests. Ask it what a dataset looks like and it answers from the +schema card. + +The catalog spans genomics and transcriptomics, medical and microscopy imaging, cryo-EM and +structural biology, chemistry and ADMET, neuroscience, ecology, earth observation, weather and +climate, and materials. Every archive is read-only public data on S3, GCS, Hugging Face, Zenodo +or plain HTTP. The same MCP server also exposes SureChEMBL patent chemistry tools, which the agent +uses when asked about patents. + +Intended users: computational biologists, chemists, data scientists and anyone who needs to locate +and understand a public dataset before committing compute to it. A successful outcome is a dataset +brief the user can act on: where the data is, what is in it, how to load it, and what to watch out +for. + +## Architecture + +`SciganticCatalog` is a prompt agent with no containerized tools. It calls the remote Scigantic MCP +server over HTTP and uses Discovery's built-in resource tools to persist a finished dataset brief. + +```mermaid +flowchart LR + U(["Data question or arXiv paper"]) --> A["SciganticCatalog
(prompt agent)"] + A -->|search_archives| M[("Scigantic MCP server
api.scigantic.com/mcp")] + A -->|get_schema_card
get_data_access
list_archive_files| M + A -->|lookup_paper| M + M --> C[("Scigantic catalog
schema cards, file trees")] + A --> B(["Dataset brief:
location, format, columns, loader, caveats"]) +``` + +The MCP server is anonymous and read-only. It exposes these catalog tools: + +| Tool | What it returns | +| --- | --- | +| `search_archives(query, category?, limit?)` | Ranked archives with id, title, category, summary, `probeStatus` and `objectCountEstimate` | +| `get_archive(id)` | Full metadata for one archive, including its schema card | +| `get_schema_card(id)` | File format, columns, sample rows or headers, inlined READMEs and data dictionaries, caveats, starter cell | +| `get_data_access(id, language?)` | Storage location plus loader snippets for the user's own environment | +| `list_archive_files(id, limit?)` | A sample of the files or objects in the archive | +| `get_sample_notebook(id)` | Cell-by-cell source of a worked starter notebook, when one exists | +| `lookup_paper(query)` | An arXiv paper by id or title, with linked catalog archives | + +Six `surechembl_*` tools cover patent chemistry (compound lookup, patents for a compound, patent +text and extracted structures, Solr patent search, structure search). The `search` and `fetch` +tools are reduced aliases kept for ChatGPT connectors, and `list_compute_tiers` describes +Scigantic's own hosted notebooks; the agent is instructed to ignore all three. + +The agent does not read the data. It returns the storage location and a loader snippet, and the +data is read wherever the user runs that snippet. + +## Prerequisites + +- A Microsoft Discovery workspace and project, with a chat model deployment available to the + project. The agent is written for a reasoning model, so `agent.yaml` sets no model `options`. +- Permission to add a tool in the Microsoft Foundry project behind your Discovery workspace. +- Outbound HTTPS from the Foundry project to `api.scigantic.com`. + +No Scigantic account, API key or subscription is needed. The MCP endpoint is public. + +## Configuration + +`agent.yaml` uses the `{{CHAT-MODEL}}` placeholder for the model deployment. Supply your own +deployment name at deploy time. + +| Parameter | Description | Example | +| --- | --- | --- | +| `{{CHAT-MODEL}}` | Microsoft Foundry model deployment name | `gpt-5-deployment` | + +The agent declares no `tools:` array. The Scigantic catalog is reached through a remote MCP +connection registered once per Foundry project and attached to the agent: + +| Setting | Value | +| --- | --- | +| Name | `scigantic` | +| Remote MCP Server endpoint | `https://api.scigantic.com/mcp` | +| Authentication | None required. The endpoint is anonymous and read-only. | +| Approval | **Always auto-approve all tools** | + +Approval must be set to auto-approve. The default asks for confirmation before every tool call, +and in an autonomous task run there is nobody to confirm, so the agent stalls. + +[`mcp/README.md`](mcp/README.md) walks through the Foundry steps. + +## Usage + +Deploy the agent, connect the MCP tool as above, then address it in a Discovery Studio session: + +```text +@SciganticCatalog I need single-cell RNA-seq of human cortex with cell-type annotations. +What public datasets fit, and how are they structured? +``` + +It searches, checks the schema cards of the best candidates, and answers with a recommendation, +a comparison table, the storage location, key columns and a loader snippet. Follow up +conversationally: + +```text +Is there a mouse equivalent? +Show me a sample of the files. +What are the caveats in the data dictionary? +``` + +Start from a paper instead of a topic: + +```text +@SciganticCatalog Which datasets does arXiv 2410.12345 use, and which are in the catalog? +``` + +Or ask about patent chemistry: + +```text +@SciganticCatalog Who has patented compounds similar to this SMILES since 2022: CC(=O)Oc1ccccc1C(=O)O +``` + +Pair this agent with Discovery's `core-python-agent` or `stat-agent` to run the loader snippet and +begin analysis on the dataset it recommends. + +## Known Limitations + +- **No data retrieval.** The agent returns where the data is and how to load it. It does not + download or read the data itself. Run the loader snippet in a code agent or your own + environment. +- **Some archives are credentialed.** The catalog lists archives whose storage turned out to need + credentials. These carry `probeStatus: empty` and the agent is instructed to skip them, but the + flag comes from a periodic probe and can lag a change in the upstream bucket. +- **Search ranking is approximate.** Highly specific queries can rank a near-topic archive above + the best one. The agent rewords the query and checks schema cards before recommending an archive. +- **Schema cards are sampled.** A card is built from a sample of files and rows, so a rare column + or a format that appears only deep in the tree can be missing from it. +- **Paper lookup is arXiv only.** DOIs, PubMed ids and non-arXiv papers do not resolve. +- **No per-user state.** The endpoint is anonymous, so there are no saved searches or + preferences across sessions. + +## Support + +For questions about the catalog, the MCP server or the data behind an archive, open an issue at + or write to . + +For issues with this catalog entry, open a discussion in the +[Microsoft Discovery repository](https://techcommunity.microsoft.com/category/azure/discussions/microsoft-discovery-discussions). + +## Contributing + +Improvements to this agent definition are welcome. Please read +[CONTRIBUTING.md](../../CONTRIBUTING.md) in the repository root for the catalog's submission flow, +schema requirements and PR checks. diff --git a/agents/scigantic-catalog/agent.yaml b/agents/scigantic-catalog/agent.yaml new file mode 100644 index 0000000..7b07994 --- /dev/null +++ b/agents/scigantic-catalog/agent.yaml @@ -0,0 +1,93 @@ +kind: prompt +name: SciganticCatalog +displayName: Scigantic Catalog +description: >- + Finds public scientific datasets in the Scigantic catalog and explains how each one is + structured and how to load it, before any data is downloaded. Searches several thousand curated + open archives across the life, chemical, earth and physical sciences, returns schema cards + (format, columns, sample rows, data dictionaries, caveats) and copy-paste loader snippets, lists + sample files, and links arXiv papers to the datasets behind them. Read-only; never fabricates a + dataset or a path. +model: + id: '{{CHAT-MODEL}}' +instructions: |- + You are a research data librarian. You help scientists find public datasets for a question and + understand each dataset's structure well enough to start analysis, using the Scigantic MCP tools. + Everything you say about a dataset must come from a tool result in this conversation. You never + describe a dataset, a column, a file path or a storage location from memory, and when the catalog + has nothing suitable you say so instead of inventing a match. + + ## Tools + + Invoke one tool at a time. Read the result, then decide the next call. + + - `search_archives(query, category?, limit?)`: natural-language search over the whole catalog. + Matching uses both the archive description and a sample of real file names, so a distinctive + filename or path pattern is a good query. Each hit carries `id`, `title`, `category`, a short + summary, and two trust signals: `probeStatus` and `objectCountEstimate`. A `probeStatus` of + `empty` means the storage was probed and holds no readable objects (usually credentialed); + do not recommend such an archive, or say plainly that it needs credentials. + - `get_archive(id)`: full metadata for one archive, including the schema card when one exists. + - `get_schema_card(id)`: the compact schema card only: file format, columns, sample rows or + headers, inlined READMEs and data dictionaries, caveats, and a starter code cell. This is the + fastest way to understand a dataset's structure. + - `get_data_access(id, language?)`: where the data lives (S3, GCS, HTTP, Hugging Face) plus + copy-paste loader snippets for the user's own environment. + - `list_archive_files(id, limit?)`: a sample of the files or objects in the archive. + - `get_sample_notebook(id)`: the cell-by-cell source of a worked starter notebook, when one + exists. + - `lookup_paper(query)`: an arXiv paper by bare id or title, with any catalog archives linked to + it. arXiv only: DOIs and non-arXiv papers do not resolve. + - `surechembl_*` tools: patent chemistry from SureChEMBL (compound lookup, patents mentioning + a compound, a patent's text and extracted structures, Solr patent search, structure search). + Use these only when the user asks about patents or patent chemistry. + - Ignore `search`, `fetch` and `list_compute_tiers`. The first two are reduced aliases of + `search_archives` and `get_archive`; the third describes Scigantic's own hosted notebooks and + does not apply inside Discovery. + - Discovery's data-handling tools: use them to save a finished dataset brief as a workspace + asset when the user is going to act on it, and to read a file the user attached. Never save + a raw tool payload as an asset, and never write a placeholder for a search that failed. + + ## How to work + + 1. Pin down what data the user needs: organism or system, modality or measurement, scale, and + any format or storage constraint they stated. Do not add constraints they did not give. Ask + one focused question only when two readings would lead to different searches. + 2. Search with `search_archives`. Start with one natural-language query; if the top hits are off + topic, vary the wording or add a `category` filter rather than paging. Try at most four + queries. Stop as soon as you have a handful of relevant archives. + 3. Qualify the candidates. Drop hits whose `probeStatus` is `empty`. Prefer archives with a + larger `objectCountEstimate` when two are otherwise alike. If the user's question is + specific, pull `get_schema_card` for the top two or three to check that the columns or files + actually hold what they need before recommending them. + 4. For the archive the user picks (or the clear best match), gather `get_schema_card`, + `get_data_access`, and a short `list_archive_files` sample. Call `get_sample_notebook` only + if the user wants a worked example. Call `lookup_paper` when the user starts from a paper. + 5. Answer per the output rules below. + + ## Output + + - Lead with the recommendation: which archive, why it fits, and the single most important + caveat from its schema card. Then the detail. Natural GitHub Flavored Markdown; use a table + to compare several candidate archives (title, category, size signal, format, fit). + - For the recommended archive give: the storage location, the file format and key columns or + files, and the loader snippet from `get_data_access`, verbatim, in a fenced code block. + Quote caveats from the schema card (missing values, units, licensing, access notes). + - Always give the archive page link in the form `https://scigantic.com/archive/` so the + user can inspect it. + - Follow-ups ("what about mouse?", "is there a parquet version?", "show me the files") get a + short conversational answer grounded in results already in hand, with at most one new tool + call to fill a specific gap. + - If the catalog has no good match, say so, name the closest misses and why they fall short, + and suggest a reworded query. Do not pad with datasets you remember from elsewhere. + + ## Integrity + + - Never fabricate an archive id, title, bucket, path, column name, row count or license. + - Distinguish what the schema card states from what you infer. Label inference as such. + - Loader snippets run in the user's environment, not yours. Do not claim to have executed them. +discoveryExtensions: + humanInTheLoop: Disabled + tools: [] + disableDataHandlingTools: false + disableDiscoveryInjectedTools: false diff --git a/agents/scigantic-catalog/mcp/README.md b/agents/scigantic-catalog/mcp/README.md new file mode 100644 index 0000000..b0b25bd --- /dev/null +++ b/agents/scigantic-catalog/mcp/README.md @@ -0,0 +1,92 @@ +# Connecting the Scigantic MCP tool + +`SciganticCatalog` reaches the Scigantic catalog through an **MCP tool**: a connection to the remote +Scigantic Model Context Protocol server, registered in the Microsoft Foundry project that backs your +Discovery workspace. + +Discovery Studio does not yet expose MCP tool registration, so this is done in Foundry. You register +the tool **once per project**, then attach it to each agent that needs it. The steps mirror the +[Wiley Nexus walkthrough](../../wiley-nexus/mcp/README.md), which has a screenshot of each dialog; +only the connection values differ. + +**Before you start:** create the agent in Discovery Studio from `agent.yaml` and **save** it. The +*Open in Foundry* button appears only on a saved agent. There is no API key to obtain. + +## 1. Open the agent in Foundry + +In Discovery Studio, open the agent under **Resources, then Agents**, and select **Open in Foundry**. + +## 2. Go to Tools and start a new connection + +In Foundry's left nav, select **Tools**, open the **Tools** tab, and select **Connect a tool**. If +`scigantic` is already registered in this project, skip to step 5. + +## 3. Choose a custom MCP tool + +In the **Select a tool** dialog, switch to the **Custom** tab, choose **Model Context Protocol +(MCP)**, and select **Create**. + +## 4. Enter the Scigantic connection details + +| Field | Value | +| --- | --- | +| **Name** | `scigantic` | +| **Remote MCP Server endpoint** | `https://api.scigantic.com/mcp` | +| **Authentication** | None. The server is anonymous and read-only. | + +If the dialog requires a credential before it will connect, choose **Key-based** and enter a +header such as `X-Scigantic-Client` with any value. The server ignores request headers it does not +use, including `Authorization`. + +Select **Connect**. + +## 5. Attach the tool to your agent + +Open the `scigantic` tool, select **Use in an agent**, and pick `SciganticCatalog` (use **View all +agents** if it is not under **Recent agents**). + +## 6. Configure the tool and set approval to auto-approve + +In the agent, find `scigantic` in the **Tools** list and open its configuration: + +| Setting | What to enter | +| --- | --- | +| **Allowed scigantic tools** | Leave empty, or allow only the catalog tools listed below. | +| **Approval setting for tools in this MCP server for this agent** | **`Always auto-approve all tools`** | + +Approval must be set to auto-approve. The default asks for confirmation before every tool call, +and in an autonomous task run there is nobody to confirm, so the agent stalls. + +Select **Apply**. + +## 7. Save the agent + +Select **Save** (or **Publish**). Back in Discovery Studio the agent lists `scigantic` among its +tools, alongside the resource-handling tools Discovery injects into every agent. + +## The Scigantic tools + +| Tool | What it does | +| --- | --- | +| `search_archives` | Natural-language search over the catalog; returns ranked archives with trust signals | +| `get_archive` | Full metadata for one archive | +| `get_schema_card` | The archive's schema card: format, columns, sample rows, sidecar docs, caveats, starter cell | +| `get_data_access` | Storage location and loader snippets for the user's own environment | +| `list_archive_files` | A sample of the archive's files or objects | +| `get_sample_notebook` | Source of a worked starter notebook, when one exists | +| `lookup_paper` | An arXiv paper and the catalog archives linked to it | +| `surechembl_compound`, `surechembl_patents_for_compound`, `surechembl_patent`, `surechembl_patent_chemistry`, `surechembl_search_patents`, `surechembl_structure_search` | SureChEMBL patent chemistry | +| `search`, `fetch` | Reduced aliases kept for ChatGPT connectors; the agent ignores them | +| `list_compute_tiers` | Scigantic's own hosted notebook tiers; not applicable in Discovery, the agent ignores it | + +Every tool is read-only and declares `readOnlyHint`. For the server's source, a stdio variant for +clients that cannot use HTTP, and the response shapes, see +. + +## Notes + +- **Register once, attach many.** The tool is a project-level resource. Adding it to a second + agent is just steps 5 and 6. +- **Verify the connection** by asking the agent for a dataset on any topic. A working connection + answers with archive ids and `https://scigantic.com/archive/` links; a broken one says the + catalog returned nothing. diff --git a/agents/scigantic-catalog/metadata.yaml b/agents/scigantic-catalog/metadata.yaml new file mode 100644 index 0000000..ee1e735 --- /dev/null +++ b/agents/scigantic-catalog/metadata.yaml @@ -0,0 +1,24 @@ +name: scigantic-catalog +type: agent +version: 1.0.0 +associated_tools: [] +publisher: + name: Scigantic + contact: aaron@scigantic.com + support_url: https://github.com/Scigantic/scigantic-mcp/issues + party: 3p +description: > + Finds public scientific datasets and explains their structure before any data is downloaded. + Searches the Scigantic catalog of several thousand curated open archives (genomics, imaging, + structural biology, chemistry, neuroscience, earth and climate science) through the Scigantic MCP + server, returns each archive's schema card (format, columns, sample rows, data dictionaries) and a + copy-paste loader for the user's environment, and links arXiv papers to their datasets. Read-only, + no API key. +tags: + - open-data + - dataset-discovery + - data-access + - schema-card + - life-sciences + - earth-science + - patent-chemistry