From 6de7713ec6ba2daa3591d606371ce42dd20ad5ae Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 22 Oct 2026 15:57:57 +0000 Subject: [PATCH] Let people analyze their own files, a mix of items, or a .zip The web app only understood one dropped folder: dropping a file or several files was rejected, there was no way to choose individual files, and a zipped project (GitHub's "Download ZIP") wasn't supported. Now a drop or a picker accepts a folder, one or more source files, a mix of folders and files, or a .zip, all read inside the page: - loose files are analyzed as one project, so imports between them resolve; duplicate names stay distinct - a .zip is unpacked with fflate, skipping node_modules, build output and non-code from the archive index; a wrapper folder named like the zip (or holding the project's files) is removed, but a lone src/ is kept; a size budget and clear errors cover bad or huge archives - folder picks drop node_modules/.git before any file object is kept - the landing page has a drop area with "Choose a folder" and "Choose files or a .zip", and the app shows progress while it reads Verified with the browser's own drag-and-drop (CDP, real paths): a folder, one file, two files, a folder plus a file, and a real zip; and through both pickers. 106 tests pass (was 98). Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01ENHRK3kgqcVwqwyjswRViM --- CHANGELOG.md | 2 + README.md | 8 +-- docs/DEPLOY.md | 2 +- package-lock.json | 7 +++ packages/web/package.json | 1 + packages/web/src/App.tsx | 16 +++-- packages/web/src/sources/local.ts | 94 ++++++++++++++++++++++-------- packages/web/src/sources/zip.ts | 60 +++++++++++++++++++ packages/web/src/styles.css | 10 +++- packages/web/src/views/Landing.tsx | 61 +++++++++++-------- packages/web/test/sources.test.ts | 85 +++++++++++++++++++++++++-- 11 files changed, 282 insertions(+), 64 deletions(-) create mode 100644 packages/web/src/sources/zip.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index ad68aec..c415ac6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,8 @@ All notable changes to [`codegraph-engine`](https://www.npmjs.com/package/codegr Shareable `?repo=owner/repo` links, a "where to start" overview, search (`/`), a file tree, an inspector with the blast radius, callers, callees and source, and a layout that stays readable on large repositories (test functions and very large blast radii fold into one node per file; selecting a file shows its import neighbourhood). +- **Your own files in the web app**: drop or choose a folder, one or more source files, a mix of both, or a `.zip` + (GitHub's "Download ZIP" works; its wrapper folder is removed). Everything is read in the page and never uploaded. - `@codegraph/parser` and `@codegraph/extractors` have browser entry points; the in-memory indexing path no longer imports `node:path` or `node:fs`. diff --git a/README.md b/README.md index 1d08490..104daaf 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ ![CodeGraph's blast radius for url_for() in Flask: 40 symbols in 19 files, with the affected test files folded together](docs/screenshots/workspace.png) -Paste a GitHub repository or drop a project folder into the web app and it builds a multi-granular graph (folders → file imports → function calls → local variable dataflow). Nothing is uploaded: the parser is WebAssembly running in a Web Worker, and a GitHub repository is downloaded straight from GitHub into your tab. Then it answers: +Paste a GitHub repository into the web app, or drop a folder, some files or a `.zip` from your computer, and it builds a multi-granular graph (folders → file imports → function calls → local variable dataflow). Nothing is uploaded: the parser is WebAssembly running in a Web Worker, and a GitHub repository is downloaded straight from GitHub into your tab. Then it answers: - **Blast radius** — *"If I change this function (or this diff), what else can break, how confident are we, and which tests should I run?"* - **Slice** — *"Show me only the code relevant to this function"*: its callers and callees, with exact line ranges. @@ -34,10 +34,10 @@ Trust `resolved`; treat `probable` as a lead to check and `ambiguous` as a list ## Web app -Open the app, then paste `owner/repo` (or `owner/repo@branch`, or any `github.com` link), or drop a folder from your machine. You land on a map of the repository's folders and a short list of the code that is riskiest to change. Pick any function, class or file to see its blast radius: what could break, how sure CodeGraph is about each link, and which tests to run. Share an analysis with `?repo=owner/repo` on the app's URL. +Open the app, then paste `owner/repo` (or `owner/repo@branch`, or any `github.com` link), or drop a folder, some `.ts`/`.js`/`.py` files or a `.zip` (such as GitHub's "Download ZIP") from your machine. You land on a map of the repository's folders and a short list of the code that is riskiest to change. Pick any function, class or file to see its blast radius: what could break, how sure CodeGraph is about each link, and which tests to run. Share an analysis with `?repo=owner/repo` on the app's URL. -- **Public GitHub repositories and local folders**, in TypeScript, JavaScript and Python. Private repositories need a local folder or the CLI. -- **Anonymous GitHub API limits** apply: about 30 analyses an hour per visitor. Local folders don't use GitHub. +- **Public GitHub repositories, or anything on your own machine** (a folder, loose files, a `.zip`), in TypeScript, JavaScript and Python. Private repositories: download or clone them and use the folder or the CLI. Local files are read in the page and never uploaded. +- **Anonymous GitHub API limits** apply: about 30 analyses an hour per visitor. Local files, folders and zips don't use GitHub. Dragging a folder in is faster than choosing it, because `node_modules` and `.git` are skipped before anything is read. - **Large repositories** are trimmed to 3,000 source files, tests and examples first, and the page says so. - Host it yourself as a static site, for free on Vercel: see [`docs/DEPLOY.md`](docs/DEPLOY.md). diff --git a/docs/DEPLOY.md b/docs/DEPLOY.md index ef1b998..22f8346 100644 --- a/docs/DEPLOY.md +++ b/docs/DEPLOY.md @@ -26,7 +26,7 @@ Share a ready-made analysis with a link such as `https://.vercel.app/?r **Limits to know about** - GitHub allows 60 anonymous API requests per hour per visitor IP. One analysis uses 2 (1 when the branch is given), so a visitor can run about 30 analyses an hour. The page explains it when a visitor hits the limit. -- Only public repositories can be read from a browser. Private ones need a local folder or the CLI. +- Only public repositories can be read from GitHub by a browser. For a private one, download it (or clone it) and drop the folder or the .zip onto the page, or use the CLI. - Very large repositories are trimmed to 3,000 source files (tests and examples first), and files over 1 MB are skipped; the page says when it does this. diff --git a/package-lock.json b/package-lock.json index 637b620..641e53a 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1920,6 +1920,12 @@ } } }, + "node_modules/fflate": { + "version": "0.8.2", + "resolved": "https://registry.npmjs.org/fflate/-/fflate-0.8.2.tgz", + "integrity": "sha512-cPJU47OaAoCbg0pBvzsgpTPhmhqI5eJjh/JIu8tPj5q+T7iLvW/JAYUqmE7KOB4R1ZyEhzBaIQpQpardBF5z8A==", + "license": "MIT" + }, "node_modules/fsevents": { "version": "2.3.3", "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", @@ -2721,6 +2727,7 @@ "@codegraph/parser": "*", "@dagrejs/dagre": "^1.1.4", "@xyflow/react": "^12.3.0", + "fflate": "0.8.2", "react": "^18.3.1", "react-dom": "^18.3.1" }, diff --git a/packages/web/package.json b/packages/web/package.json index 15137f2..d986624 100644 --- a/packages/web/package.json +++ b/packages/web/package.json @@ -11,6 +11,7 @@ "@codegraph/parser": "*", "@dagrejs/dagre": "^1.1.4", "@xyflow/react": "^12.3.0", + "fflate": "0.8.2", "react": "^18.3.1", "react-dom": "^18.3.1" }, diff --git a/packages/web/src/App.tsx b/packages/web/src/App.tsx index 08ebf88..a5dae18 100644 --- a/packages/web/src/App.tsx +++ b/packages/web/src/App.tsx @@ -2,10 +2,10 @@ import { useCallback, useEffect, useRef, useState } from 'react'; import type { GraphJSON } from '@codegraph/core'; import { analyze, sessionFromGraph, type Session } from './session'; import { loadGithub } from './sources/github'; -import { loadFolder } from './sources/local'; +import { loadFolder, type Picked } from './sources/local'; import { parseRepoInput, repoLabel, type RepoRef } from './sources/parse-repo'; import { LoadError, type LoadProgress } from './sources/types'; -import { Landing, type FolderPick, type LandingError } from './views/Landing'; +import { Landing, type LandingError } from './views/Landing'; import { Loading } from './views/Loading'; import { Workspace } from './views/Workspace'; @@ -63,9 +63,13 @@ export function App() { [run], ); - const startFolder = useCallback( - (folder: FolderPick) => { - void run(folder.name, async (signal, progress) => analyze(await loadFolder(folder, { signal, onProgress: progress }), { signal, onProgress: progress })); + const startPick = useCallback( + (resolve: () => Promise) => { + void run('your files', async (signal, progress) => { + progress({ phase: 'read', done: 0, total: 0, message: 'Looking through your files…' }); + const picked = await resolve(); + return analyze(await loadFolder(picked, { signal, onProgress: progress }), { signal, onProgress: progress }); + }); }, [run], ); @@ -122,6 +126,6 @@ export function App() { case 'workspace': return ; default: - return void openGraph(f)} />; + return void openGraph(f)} />; } } diff --git a/packages/web/src/sources/local.ts b/packages/web/src/sources/local.ts index 02fcae3..15844c1 100644 --- a/packages/web/src/sources/local.ts +++ b/packages/web/src/sources/local.ts @@ -2,6 +2,7 @@ import { createFilter } from '@codegraph/extractors/browser'; import { pool } from './github'; import { LIMITS, LoadError, type LoadProgress, type LoadedSource } from './types'; +import { entriesFromZip, isZip } from './zip'; /** A file with its path relative to the chosen folder. */ export interface FolderFile { @@ -9,17 +10,50 @@ export interface FolderFile { file: File; } -/** From ``: every file's `webkitRelativePath` starts with the folder's own name. */ -export function entriesFromFileList(list: FileList | File[]): { name: string; files: FolderFile[] } { +/** Everything a person can hand over: a folder, some files, or a zip, as a list of files with paths. */ +export interface Picked { + name: string; + files: FolderFile[]; + /** Second line for the header (`local folder`, `from a .zip`). */ + detail?: string; + notes?: string[]; +} + +const NOTHING = () => new LoadError('invalid-input', 'There is nothing to analyze there.', 'Pick a project folder, some .ts, .js or .py files, or a .zip.'); + +/** Distinct paths for loose files (two picked files may share a name when they came from different folders). */ +function flat(files: File[]): FolderFile[] { + const seen = new Map(); + return files.map((file) => { + const n = (seen.get(file.name) ?? 0) + 1; + seen.set(file.name, n); + const dot = file.name.lastIndexOf('.'); + return { path: n === 1 ? file.name : dot > 0 ? `${file.name.slice(0, dot)}-${n}${file.name.slice(dot)}` : `${file.name}-${n}`, file }; + }); +} + +const looseLabel = (files: File[]) => (files.length === 1 ? files[0].name : `${files.length} files`); + +/** + * What an `` returned. From a folder picker (`webkitdirectory`) every file's `webkitRelativePath` + * starts with the folder's own name; from a plain picker the files are loose. A single .zip is unpacked. + */ +export async function entriesFromFileList(list: FileList | File[]): Promise { const all = Array.from(list); + if (!all.length) throw NOTHING(); + const rel = (f: File) => (f as File & { webkitRelativePath?: string }).webkitRelativePath || ''; + if (all.length === 1 && isZip(all[0]) && !rel(all[0])) return entriesFromZip(all[0]); + if (!all.some(rel)) return { name: looseLabel(all), files: flat(all), detail: all.length === 1 ? 'one file' : 'local files' }; + const prune = createFilter(); const files: FolderFile[] = []; let name = ''; for (const file of all) { - const rel = (file as File & { webkitRelativePath?: string }).webkitRelativePath || file.name; - const slash = rel.indexOf('/'); + const r = rel(file); + const slash = r.indexOf('/'); if (slash < 0) continue; - name ||= rel.slice(0, slash); - files.push({ path: rel.slice(slash + 1), file }); + name ||= r.slice(0, slash); + const path = r.slice(slash + 1); + if (!prune.ignoresFile(path)) files.push({ path, file }); // node_modules and friends are dropped before anything is read } return { name: name || 'folder', files }; } @@ -33,20 +67,23 @@ interface FsEntry { } /** - * From a drop. The entries must be taken from the event synchronously (they are invalid after the first await), and - * directories are pruned while walking: `node_modules` or `.git` are never entered, which matters when a project has - * a hundred thousand files in them. + * From a drop: one folder, loose files, several folders and files together, or a .zip. The entries must be taken from + * the event synchronously (they are invalid after the first await), so call this directly from the drop handler. + * Folders are pruned while walking: `node_modules` or `.git` are never entered, which matters when a project has a + * hundred thousand files in them. */ -export async function entriesFromDrop(dt: DataTransfer): Promise<{ name: string; files: FolderFile[] } | null> { +export async function entriesFromDrop(dt: DataTransfer): Promise { const roots: FsEntry[] = []; for (const item of Array.from(dt.items)) { + if (item.kind !== 'file') continue; const entry = (item as DataTransferItem & { webkitGetAsEntry?: () => FsEntry | null }).webkitGetAsEntry?.(); if (entry) roots.push(entry); } - const dirs = roots.filter((r) => r.isDirectory); - if (dirs.length !== 1 || roots.length !== 1) return null; // exactly one folder + const loose = Array.from(dt.files); // for browsers that give no entries + if (!roots.length) return entriesFromFileList(loose); + const prune = createFilter(); - const files: FolderFile[] = []; + const fileOf = (e: FsEntry) => new Promise((ok, err) => e.file!(ok, err)); const readAll = async (dir: FsEntry) => { const reader = dir.createReader!(); const out: FsEntry[] = []; @@ -56,24 +93,35 @@ export async function entriesFromDrop(dt: DataTransfer): Promise<{ name: string; out.push(...batch); } }; + const files: FolderFile[] = []; const walk = async (dir: FsEntry, path: string) => { for (const e of await readAll(dir)) { const p = path ? `${path}/${e.name}` : e.name; if (e.isDirectory) { if (!prune.ignoresDir(p)) await walk(e, p); - } else if (e.isFile && !prune.ignoresFile(p)) { - const file = await new Promise((ok, err) => e.file!(ok, err)); - files.push({ path: p, file }); - } + } else if (e.isFile && !prune.ignoresFile(p)) files.push({ path: p, file: await fileOf(e) }); } }; - await walk(dirs[0], ''); - return { name: dirs[0].name, files }; + + if (roots.length === 1 && roots[0].isDirectory) { + await walk(roots[0], ''); + return { name: roots[0].name, files }; + } + if (roots.length === 1) return entriesFromFileList([await fileOf(roots[0])]); // one file, or one .zip + // Several things at once: folders keep their own name as a path prefix, files sit at the top. + const looseFiles: File[] = []; + for (const r of roots) { + if (r.isDirectory) await walk(r, r.name); + else if (r.isFile) looseFiles.push(await fileOf(r)); + } + files.push(...flat(looseFiles)); + const onlyFiles = looseFiles.length === roots.length; + return { name: onlyFiles ? looseLabel(looseFiles) : `${roots.length} items`, files, detail: onlyFiles ? 'local files' : 'dropped items' }; } /** Applies the same selection rules as the GitHub loader and reads the surviving files. */ export async function loadFolder( - folder: { name: string; files: FolderFile[] }, + folder: Picked, opts: { signal?: AbortSignal; onProgress?: (p: LoadProgress) => void } = {}, ): Promise { const byPath = new Map(folder.files.map((f) => [f.path, f.file])); @@ -92,7 +140,7 @@ export async function loadFolder( else sources.push(f); } if (!sources.length) { - throw new LoadError('empty', `No TypeScript, JavaScript or Python files found in ${folder.name}.`, 'CodeGraph reads .ts, .tsx, .js, .jsx, .mjs, .cjs and .py files.'); + throw new LoadError('empty', `No TypeScript, JavaScript or Python files found in ${folder.name}.`, 'CodeGraph reads .ts, .tsx, .js, .jsx, .mjs, .cjs and .py files, from a folder, loose files or a .zip.'); } sources.sort((a, b) => (a.path < b.path ? -1 : 1)); @@ -117,8 +165,8 @@ export async function loadFolder( ); if (opts.signal?.aborted) throw new LoadError('cancelled', 'Cancelled.'); - const notes: string[] = []; + const notes: string[] = [...(folder.notes ?? [])]; if (tooLarge) notes.push(`${tooLarge} file${tooLarge === 1 ? ' was' : 's were'} skipped for being over 1 MB.`); if (failed) notes.push(`${failed} file${failed === 1 ? '' : 's'} could not be read and ${failed === 1 ? 'was' : 'were'} skipped.`); - return { label: folder.name, detail: 'local folder', origin: { kind: 'folder' }, files, configs: configMap, notes }; + return { label: folder.name, detail: folder.detail ?? 'local folder', origin: { kind: 'folder' }, files, configs: configMap, notes }; } diff --git a/packages/web/src/sources/zip.ts b/packages/web/src/sources/zip.ts new file mode 100644 index 0000000..fedecd7 --- /dev/null +++ b/packages/web/src/sources/zip.ts @@ -0,0 +1,60 @@ +// Reads a .zip (for example GitHub's "Download ZIP") in the page. Only the entries worth analyzing are decompressed: +// dependencies, build output and anything that isn't source or config are skipped from the archive's index, so a zip +// with a node_modules folder costs nothing extra. +import { unzip, type UnzipFileInfo } from 'fflate'; +import { createFilter } from '@codegraph/extractors/browser'; +import type { FolderFile } from './local'; +import { LIMITS, LoadError } from './types'; + +const MAX_ZIP_BYTES = 250_000_000; +const KEEP_NAMES = new Set(['.gitignore', '.codegraphignore']); +/** Files that say "this folder is a project root". */ +const PROJECT_MARKERS = new Set(['package.json', 'pyproject.toml', 'setup.py', 'setup.cfg', 'requirements.txt', 'tsconfig.json']); +const base = (p: string) => p.slice(p.lastIndexOf('/') + 1); + +export const isZip = (f: File) => /\.zip$/i.test(f.name) || f.type === 'application/zip' || f.type === 'application/x-zip-compressed'; + +export async function entriesFromZip(file: File): Promise<{ name: string; files: FolderFile[]; detail: string; notes: string[] }> { + if (file.size > MAX_ZIP_BYTES) { + throw new LoadError('empty', `${file.name} is larger than ${Math.round(MAX_ZIP_BYTES / 1e6)} MB.`, 'Zip just the source folders, or choose the project folder itself.'); + } + const data = new Uint8Array(await file.arrayBuffer()); + const prune = createFilter(); + const tops = new Set(); // first path segment of every entry, kept or not + const markerTops = new Set(); // top-level folders that hold a project file directly + let anyFile = false; + let bytes = 0; + let overBudget = 0; + const filter = (info: UnzipFileInfo) => { + const name = info.name; + if (name.endsWith('/') || name.startsWith('__MACOSX/')) return false; + anyFile = true; + tops.add(name.includes('/') ? name.slice(0, name.indexOf('/')) : ''); + if (name.split('/').length === 2 && PROJECT_MARKERS.has(base(name))) markerTops.add(name.slice(0, name.indexOf('/'))); + if (prune.ignoresFile(name) || info.originalSize > LIMITS.maxFileBytes) return false; + const b = base(name); + if (!(prune.isSource(b) || prune.isConfig(b) || KEEP_NAMES.has(b))) return false; + if (bytes + info.originalSize > LIMITS.maxTotalBytes) { + overBudget++; + return false; + } + bytes += info.originalSize; + return true; + }; + const out = await new Promise>((resolve, reject) => + unzip(data, { filter }, (err, res) => (err ? reject(new LoadError('invalid-input', `Couldn't open ${file.name} as a zip file.`, 'Check that it is a valid .zip archive.')) : resolve(res))), + ); + if (!anyFile) throw new LoadError('empty', `${file.name} is empty.`); + // GitHub's "Download ZIP" (and "Compress" in a file manager) wrap everything in one folder, named like the zip or + // holding the project's files; paths should start inside it. A zip that just happens to hold one folder (a `src/` + // on its own) is left alone. + const zipName = file.name.replace(/\.zip$/i, ''); + const only = tops.size === 1 && !tops.has('') ? [...tops][0] : undefined; + const strip = only && (only.toLowerCase() === zipName.toLowerCase() || markerTops.has(only)) ? only + '/' : ''; + const files: FolderFile[] = Object.entries(out).map(([path, bytes]) => { + const rel = strip && path.startsWith(strip) ? path.slice(strip.length) : path; + return { path: rel, file: new File([new Uint8Array(bytes)], base(rel)) }; + }); + const notes = overBudget ? [`The zip holds more source than fits in a browser tab, so ${overBudget} file${overBudget === 1 ? ' was' : 's were'} left out.`] : []; + return { name: zipName, files, detail: 'from a .zip', notes }; +} diff --git a/packages/web/src/styles.css b/packages/web/src/styles.css index a8ca9a3..96b0606 100644 --- a/packages/web/src/styles.css +++ b/packages/web/src/styles.css @@ -369,8 +369,9 @@ button.link.more { .drop { width: 100%; display: flex; + flex-wrap: wrap; align-items: center; - gap: 16px; + gap: 14px 16px; text-align: left; padding: 20px 22px; background: rgba(17, 22, 31, 0.6); @@ -379,6 +380,12 @@ button.link.more { color: var(--dim); transition: border-color 0.15s, background 0.15s; } +.drop-actions { + display: flex; + flex-wrap: wrap; + gap: 8px; + margin-left: auto; +} .drop:hover { border-color: var(--accent); background: var(--accent-soft); @@ -388,6 +395,7 @@ button.link.more { color: var(--accent-2); } .drop-text { + flex: 1 1 280px; display: flex; flex-direction: column; gap: 2px; diff --git a/packages/web/src/views/Landing.tsx b/packages/web/src/views/Landing.tsx index 1b1ee71..a81fcbd 100644 --- a/packages/web/src/views/Landing.tsx +++ b/packages/web/src/views/Landing.tsx @@ -1,17 +1,12 @@ import { useEffect, useRef, useState, type FormEvent } from 'react'; import { IconArrowRight, IconGithub, IconLogo, IconUpload } from '../ui/icons'; -import { entriesFromDrop, entriesFromFileList, type FolderFile } from '../sources/local'; +import { entriesFromDrop, entriesFromFileList, type Picked } from '../sources/local'; export interface LandingError { message: string; hint?: string; } -export interface FolderPick { - name: string; - files: FolderFile[]; -} - const EXAMPLES: { repo: string; lang: string; note?: string }[] = [ { repo: 'pallets/flask', lang: 'Python' }, { repo: 'psf/requests', lang: 'Python' }, @@ -27,19 +22,20 @@ export function Landing({ error, initialValue = '', onRepo, - onFolder, + onPick, onGraphFile, }: { error?: LandingError | null; initialValue?: string; onRepo: (input: string) => void; - onFolder: (folder: FolderPick) => void; + /** A folder, some files or a .zip from this machine. `resolve` reads them; the app shows progress while it runs. */ + onPick: (resolve: () => Promise) => void; onGraphFile: (file: File) => void; }) { const [value, setValue] = useState(initialValue); const [dragging, setDragging] = useState(false); - const [dropError, setDropError] = useState(null); const folderInput = useRef(null); + const filesInput = useRef(null); const graphInput = useRef(null); const depth = useRef(0); @@ -67,11 +63,8 @@ export function Landing({ setDragging(false); const dt = e.dataTransfer; if (!dt) return; - setDropError(null); - void entriesFromDrop(dt).then((folder) => { - if (folder) onFolder(folder); - else setDropError('Drop a single project folder, not individual files.'); - }); + const picked = entriesFromDrop(dt); // must start inside the event: the dropped entries expire after it + onPick(() => picked); }; window.addEventListener('dragenter', enter); window.addEventListener('dragover', over); @@ -83,14 +76,14 @@ export function Landing({ window.removeEventListener('dragleave', leave); window.removeEventListener('drop', drop); }; - }, [onFolder]); + }, [onPick]); const submit = (e: FormEvent) => { e.preventDefault(); onRepo(value); }; - const shown = error ?? (dropError ? { message: dropError } : null); + const shown = error ?? null; return (
@@ -114,7 +107,7 @@ export function Landing({ See what breaks before you change it

- Paste a GitHub repository or drop a project folder. CodeGraph maps the imports and calls across TypeScript, JavaScript and Python, then shows exactly which functions, files and tests a change could reach. + Paste a GitHub repository, or drop a folder, some files or a .zip from your computer. CodeGraph maps the imports and calls across TypeScript, JavaScript and Python, then shows exactly which functions, files and tests a change could reach.

@@ -155,22 +148,42 @@ export function Landing({ or
- + + + + + )} onChange={(e) => { - const list = e.target.files; - if (list?.length) onFolder(entriesFromFileList(list)); + const files = Array.from(e.target.files ?? []); // copy: clearing the input below empties the live list + e.target.value = ''; + if (files.length) onPick(() => entriesFromFileList(files)); + }} + /> + { + const files = Array.from(e.target.files ?? []); e.target.value = ''; + if (files.length) onPick(() => entriesFromFileList(files)); }} /> - Drop your project folder + Drop a folder, files or a .zip It is read in this tab and never uploaded )} diff --git a/packages/web/test/sources.test.ts b/packages/web/test/sources.test.ts index 6ce9a21..0f7d64c 100644 --- a/packages/web/test/sources.test.ts +++ b/packages/web/test/sources.test.ts @@ -1,6 +1,7 @@ import { promises as fs } from 'node:fs'; import { join, resolve } from 'node:path'; import { fileURLToPath } from 'node:url'; +import { strToU8, zipSync } from 'fflate'; import { describe, expect, it } from 'vitest'; import { indexFiles } from '@codegraph/extractors/browser'; import { indexRepo } from '@codegraph/extractors'; @@ -200,12 +201,13 @@ describe('loadGithub', () => { }); }); -describe('local folders', () => { +describe('local folders, files and zips', () => { const file = (rel: string, text: string) => { const f = new File([text], rel.slice(rel.lastIndexOf('/') + 1)); Object.defineProperty(f, 'webkitRelativePath', { value: 'myproj/' + rel }); return f; }; + const loose = (name: string, text: string) => new File([text], name); // no webkitRelativePath: picked as a plain file const list = [ file('src/a.ts', "import { b } from './b'\nexport function a() { return b() }\n"), file('src/b.ts', 'export function b() { return 1 }\n'), @@ -216,9 +218,10 @@ describe('local folders', () => { file('notes.md', '#'), ]; - it('strips the folder name and keeps the same files the GitHub loader would', async () => { - const folder = entriesFromFileList(list); + it('a folder: strips the folder name and keeps the same files the GitHub loader would', async () => { + const folder = await entriesFromFileList(list); expect(folder.name).toBe('myproj'); + expect(folder.files.some((f) => f.path.startsWith('node_modules'))).toBe(false); // dropped before anything is read const src = await loadFolder(folder); expect([...src.files.keys()]).toEqual(['src/a.ts', 'src/b.ts']); expect([...src.configs.keys()]).toEqual(['package.json']); @@ -226,8 +229,80 @@ describe('local folders', () => { expect(src.origin.kind).toBe('folder'); }); - it('reports an empty folder', async () => { - const err = await loadFolder(entriesFromFileList([file('README.md', '#')])).catch((e) => e); + it('a folder with nothing to analyze is reported', async () => { + const err = await loadFolder(await entriesFromFileList([file('README.md', '#')])).catch((e) => e); expect(err.code).toBe('empty'); }); + + it('loose files: analyzed together as one project, so imports between them resolve', async () => { + const picked = await entriesFromFileList([loose('main.ts', "import { helper } from './util'\nexport function main() { return helper() }\n"), loose('util.ts', 'export function helper() { return 1 }\n')]); + expect(picked.name).toBe('2 files'); + expect(picked.detail).toBe('local files'); + const src = await loadFolder(picked); + expect([...src.files.keys()]).toEqual(['main.ts', 'util.ts']); + const { graph } = await indexFiles(src.files, src.configs); + expect(graph.edges('calls').some((e) => graph.node(e.from)?.name === 'main' && graph.node(e.to)?.name === 'helper' && e.confidence === 'resolved')).toBe(true); + }); + + it('one file is labelled with its name, and two files that share a name stay distinct', async () => { + const one = await entriesFromFileList([loose('solo.py', 'def f():\n return 1\n')]); + expect(one.name).toBe('solo.py'); + expect(one.detail).toBe('one file'); + const two = await entriesFromFileList([loose('index.js', 'a'), loose('index.js', 'b')]); + expect(two.files.map((f) => f.path)).toEqual(['index.js', 'index-2.js']); + }); + + it('nothing picked, or only non-code files, is reported rather than ignored', async () => { + await expect(entriesFromFileList([])).rejects.toMatchObject({ code: 'invalid-input' }); + const err = await loadFolder(await entriesFromFileList([loose('notes.txt', 'hi'), loose('photo.png', 'x')])).catch((e) => e); + expect(err.code).toBe('empty'); + }); + + const zipOf = (entries: Record, name: string) => { + const bytes = zipSync(Object.fromEntries(Object.entries(entries).map(([k, v]) => [k, strToU8(v)]))); + return new File([new Uint8Array(bytes)], name, { type: 'application/zip' }); + }; + + it('a GitHub-style zip: unpacks, strips the wrapper folder, and skips dependencies and non-code', async () => { + const zip = zipOf( + { + 'flask-main/src/app.py': 'from .util import helper\n\ndef main():\n return helper()\n', + 'flask-main/src/util.py': 'def helper():\n return 1\n', + 'flask-main/pyproject.toml': '[project]\n', + 'flask-main/package.json': '{}', + 'flask-main/node_modules/x/index.js': 'module.exports = 1\n', + 'flask-main/README.md': '# flask\n', + 'flask-main/.gitignore': 'build_out/\n', + 'flask-main/build_out/gen.py': 'x = 1\n', + '__MACOSX/flask-main/._app.py': 'junk', + }, + 'flask-main.zip', + ); + const picked = await entriesFromFileList([zip]); + expect(picked.name).toBe('flask-main'); + expect(picked.detail).toBe('from a .zip'); + const src = await loadFolder(picked); + expect([...src.files.keys()]).toEqual(['src/app.py', 'src/util.py']); // .gitignore removed build_out/ + expect([...src.configs.keys()]).toEqual(['package.json']); + }); + + it("a zip whose only folder is part of the project (a lone src/) keeps that folder", async () => { + const picked = await entriesFromFileList([zipOf({ 'src/a.ts': 'export const a = 1\n', 'src/b.ts': 'export const b = 1\n' }, 'code.zip')]); + expect(picked.files.map((f) => f.path).sort()).toEqual(['src/a.ts', 'src/b.ts']); + }); + + it('a zip with a wrapper folder that holds project files is stripped even when named differently', async () => { + const picked = await entriesFromFileList([zipOf({ 'my-app/package.json': '{}', 'my-app/index.ts': 'export const x = 1\n' }, 'download.zip')]); + expect(picked.files.map((f) => f.path).sort()).toEqual(['index.ts', 'package.json']); + }); + + it('a zip with several top-level folders keeps them all', async () => { + const picked = await entriesFromFileList([zipOf({ 'api/a.py': 'x = 1\n', 'web/b.ts': 'export const b = 1\n' }, 'both.zip')]); + expect(picked.files.map((f) => f.path).sort()).toEqual(['api/a.py', 'web/b.ts']); + }); + + it('a file that is not a valid zip says so', async () => { + const bad = new File([new Uint8Array([1, 2, 3, 4])], 'broken.zip', { type: 'application/zip' }); + await expect(entriesFromFileList([bad])).rejects.toMatchObject({ code: 'invalid-input' }); + }); });