Skip to content

feat(ingest): cited-by-me seed — stage the works a researcher's own papers cite (#122) - #124

Merged
jacktrnr merged 1 commit into
mainfrom
feat/122-cited-by-me
Nov 6, 2026
Merged

jacktrnr merged 1 commit into
mainfrom
feat/122-cited-by-me

Conversation

@jacktrnr

@jacktrnr jacktrnr commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Closes #122.

What this does

arjev ingest --cited-by-me <own-papers.bib> seeds the vault from what the researcher's own papers cite.

  1. Parse the researcher's own export. Any entry type is accepted (@article, @phdthesis, …). Ids are read from eprint, doi (URL forms stripped), or an arXiv:<id> printed in any field. Scholar's export puts arXiv ids in journal and rarely includes a DOI.
  2. Resolve each paper on the Semantic Scholar Graph API. The lookup uses ARXIV:<id> or DOI:<doi>. A paper with no id falls back to paper/search/match, accepted only on an exact normalized-title match; a near miss is rejected and reported.
  3. Read reference lists. paper/{ref}/references is paginated. Requests are paced at about 1/s, 429 and 5xx responses get backoff, and an optional ARJEV_S2_KEY_FILE sets x-api-key. Every network call goes through an injected http_transport.
  4. Seed.
    • Each cited work with an arXiv id or DOI becomes one staged note (seed: cited), its why: naming the citer: Cited in your paper “…” — seeded <date>.
    • A work cited by several of the researcher's papers becomes one note naming every citer.
    • The researcher's own papers become seed: authored notes.
    • Citations that carry only a DOI get doi: notes, which the fold already treats as identity-bearing.
  5. Idempotent and never-editing. A work already present in any root (matched by arXiv id or DOI) is skipped. A filename clash gets a suffix, and no existing file is ever opened for writing.
  6. Reporting. Own papers that couldn't be resolved and cited works without an identifier are listed in the run summary. arjev inspect gains fold cited-by-me: <n> (<a> authored, <c> cited), printed only when the seed has written notes, so existing exact-line tests are unchanged.

Why Semantic Scholar: its reference rows carry externalIds (ArXiv and DOI) per cited work. OpenAlex returns empty referenced_works for arXiv preprints (checked live on arXiv:2210.10982). AGENTS.md "External behavior" records this.

Live check (not in CI)

Run against my 4 Google Scholar papers (two journal articles without a DOI in the export, the Nonlinearity paper, and the arXiv preprint) into a temp vault:

cited-by-me: 172 cited works and 4 of your papers seeded (0 already in the vault, never edited)
cited-by-me: 30 cited works without an arXiv id or DOI (not seeded)
fold identity-bearing: 176 · fold cited-by-me: 176 (4 authored, 172 cited) · fold warnings: none

On the re-run, 0 were seeded and 176 were already in the vault, and the file count was unchanged.

Acceptance criteria (#122)

  • One staged note per cited work that has an arXiv id or DOI, its why naming the citing paper(s).
  • Own papers are staged as authored and distinguishable in frontmatter (seed: authored / seed: cited).
  • A work cited by several papers is one note listing every citer.
  • A work already in any root produces no duplicate, and the existing note is untouched.
  • A re-run changes nothing (rerun_new_notes == 0, tested across a date change).
  • Unresolved own papers and identifier-less works are counted and reported (unreported_unresolved == 0). API failures fail open per paper.
  • inspect reports the seed's identity-bearing count.

Tests

tests/test_cited.py has 17 tests using a recorded-shape httpx.MockTransport, with no network. The full suite and ruff check . are green, with one exception: test_init_template::test_no_lab_name_anywhere_in_the_template fails locally because my clone path contains "harmoniqs". It passes from a neutral path, and CI is unaffected.

Notes for review

…apers cite (#122)

arjev ingest --cited-by-me <own-papers.bib>: each own paper (eprint, doi,
arXiv id in any field, else an exact-title Semantic Scholar match) has its
reference list read from the S2 Graph API (paced, paginated, 429/5xx
backoff, optional ARJEV_S2_KEY_FILE). Every cited work with an arXiv id or
DOI becomes one staged note (seed: cited) whose why names the citing
paper(s); own papers stage as seed: authored. Works already in any root (by
arXiv id or DOI) are skipped; unresolved papers and identifier-less works
are reported. inspect gains fold cited-by-me.
@jacktrnr
jacktrnr force-pushed the feat/122-cited-by-me branch from 641b614 to b137532 Compare November 6, 2026 02:18
@jacktrnr
jacktrnr merged commit 3de993a into main Nov 6, 2026
1 check passed
@jacktrnr
jacktrnr deleted the feat/122-cited-by-me branch November 6, 2026 02:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(ingest): cited-by-me seed — stage the works a researcher's own papers cite

2 participants