Let people analyze their own files, a mix of items, or a .zip in the web app

CHANGELOG.md

@@ -10,6 +10,8 @@ All notable changes to [`codegraph-engine`](https://www.npmjs.com/package/codegr
   Shareable `?repo=owner/repo` links, a "where to start" overview, search (`/`), a file tree, an inspector with the
   blast radius, callers, callees and source, and a layout that stays readable on large repositories (test functions and
   very large blast radii fold into one node per file; selecting a file shows its import neighbourhood).
+- **Your own files in the web app**: drop or choose a folder, one or more source files, a mix of both, or a `.zip`
+  (GitHub's "Download ZIP" works; its wrapper folder is removed). Everything is read in the page and never uploaded.
 - `@codegraph/parser` and `@codegraph/extractors` have browser entry points; the in-memory indexing path no longer
   imports `node:path` or `node:fs`.
 

README.md

@@ -8,7 +8,7 @@
 
 ![CodeGraph's blast radius for url_for() in Flask: 40 symbols in 19 files, with the affected test files folded together](docs/screenshots/workspace.png)
 
-Paste a GitHub repository or drop a project folder into the web app and it builds a multi-granular graph (folders → file imports → function calls → local variable dataflow). Nothing is uploaded: the parser is WebAssembly running in a Web Worker, and a GitHub repository is downloaded straight from GitHub into your tab. Then it answers:
+Paste a GitHub repository into the web app, or drop a folder, some files or a `.zip` from your computer, and it builds a multi-granular graph (folders → file imports → function calls → local variable dataflow). Nothing is uploaded: the parser is WebAssembly running in a Web Worker, and a GitHub repository is downloaded straight from GitHub into your tab. Then it answers:
 
 - **Blast radius** — *"If I change this function (or this diff), what else can break, how confident are we, and which tests should I run?"*
 - **Slice** — *"Show me only the code relevant to this function"*: its callers and callees, with exact line ranges.
@@ -34,10 +34,10 @@ Trust `resolved`; treat `probable` as a lead to check and `ambiguous` as a list
 
 ## Web app
 
-Open the app, then paste `owner/repo` (or `owner/repo@branch`, or any `github.com` link), or drop a folder from your machine. You land on a map of the repository's folders and a short list of the code that is riskiest to change. Pick any function, class or file to see its blast radius: what could break, how sure CodeGraph is about each link, and which tests to run. Share an analysis with `?repo=owner/repo` on the app's URL.
+Open the app, then paste `owner/repo` (or `owner/repo@branch`, or any `github.com` link), or drop a folder, some `.ts`/`.js`/`.py` files or a `.zip` (such as GitHub's "Download ZIP") from your machine. You land on a map of the repository's folders and a short list of the code that is riskiest to change. Pick any function, class or file to see its blast radius: what could break, how sure CodeGraph is about each link, and which tests to run. Share an analysis with `?repo=owner/repo` on the app's URL.
 
-- **Public GitHub repositories and local folders**, in TypeScript, JavaScript and Python. Private repositories need a local folder or the CLI.
-- **Anonymous GitHub API limits** apply: about 30 analyses an hour per visitor. Local folders don't use GitHub.
+- **Public GitHub repositories, or anything on your own machine** (a folder, loose files, a `.zip`), in TypeScript, JavaScript and Python. Private repositories: download or clone them and use the folder or the CLI. Local files are read in the page and never uploaded.
+- **Anonymous GitHub API limits** apply: about 30 analyses an hour per visitor. Local files, folders and zips don't use GitHub. Dragging a folder in is faster than choosing it, because `node_modules` and `.git` are skipped before anything is read.
 - **Large repositories** are trimmed to 3,000 source files, tests and examples first, and the page says so.
 - Host it yourself as a static site, for free on Vercel: see [`docs/DEPLOY.md`](docs/DEPLOY.md).
 

docs/DEPLOY.md

@@ -26,7 +26,7 @@ Share a ready-made analysis with a link such as `https://<project>.vercel.app/?r
 **Limits to know about**
 - GitHub allows 60 anonymous API requests per hour per visitor IP. One analysis uses 2 (1 when the branch is given),
   so a visitor can run about 30 analyses an hour. The page explains it when a visitor hits the limit.
-- Only public repositories can be read from a browser. Private ones need a local folder or the CLI.
+- Only public repositories can be read from GitHub by a browser. For a private one, download it (or clone it) and drop the folder or the .zip onto the page, or use the CLI.
 - Very large repositories are trimmed to 3,000 source files (tests and examples first), and files over 1 MB are
   skipped; the page says when it does this.
 

packages/web/package.json

@@ -11,6 +11,7 @@
     "@codegraph/parser": "*",
     "@dagrejs/dagre": "^1.1.4",
     "@xyflow/react": "^12.3.0",
+    "fflate": "0.8.2",
     "react": "^18.3.1",
     "react-dom": "^18.3.1"
   },

packages/web/src/App.tsx

@@ -2,10 +2,10 @@ import { useCallback, useEffect, useRef, useState } from 'react';
 import type { GraphJSON } from '@codegraph/core';
 import { analyze, sessionFromGraph, type Session } from './session';
 import { loadGithub } from './sources/github';
-import { loadFolder } from './sources/local';
+import { loadFolder, type Picked } from './sources/local';
 import { parseRepoInput, repoLabel, type RepoRef } from './sources/parse-repo';
 import { LoadError, type LoadProgress } from './sources/types';
-import { Landing, type FolderPick, type LandingError } from './views/Landing';
+import { Landing, type LandingError } from './views/Landing';
 import { Loading } from './views/Loading';
 import { Workspace } from './views/Workspace';
 
@@ -63,9 +63,13 @@ export function App() {
     [run],
   );
 
-  const startFolder = useCallback(
-    (folder: FolderPick) => {
-      void run(folder.name, async (signal, progress) => analyze(await loadFolder(folder, { signal, onProgress: progress }), { signal, onProgress: progress }));
+  const startPick = useCallback(
+    (resolve: () => Promise<Picked>) => {
+      void run('your files', async (signal, progress) => {
+        progress({ phase: 'read', done: 0, total: 0, message: 'Looking through your files…' });
+        const picked = await resolve();
+        return analyze(await loadFolder(picked, { signal, onProgress: progress }), { signal, onProgress: progress });
+      });
     },
     [run],
   );
@@ -122,6 +126,6 @@ export function App() {
     case 'workspace':
       return <Workspace key={screen.session.label + (screen.session.ms ?? '')} session={screen.session} shareUrl={screen.shareUrl} onHome={home} />;
     default:
-      return <Landing error={screen.error} initialValue={screen.prefill} onRepo={startRepo} onFolder={startFolder} onGraphFile={(f) => void openGraph(f)} />;
+      return <Landing error={screen.error} initialValue={screen.prefill} onRepo={startRepo} onPick={startPick} onGraphFile={(f) => void openGraph(f)} />;
   }
 }

packages/web/src/sources/local.ts

@@ -2,24 +2,58 @@
 import { createFilter } from '@codegraph/extractors/browser';
 import { pool } from './github';
 import { LIMITS, LoadError, type LoadProgress, type LoadedSource } from './types';
+import { entriesFromZip, isZip } from './zip';
 
 /** A file with its path relative to the chosen folder. */
 export interface FolderFile {
   path: string;
   file: File;
 }
 
-/** From `<input type="file" webkitdirectory>`: every file's `webkitRelativePath` starts with the folder's own name. */
-export function entriesFromFileList(list: FileList | File[]): { name: string; files: FolderFile[] } {
+/** Everything a person can hand over: a folder, some files, or a zip, as a list of files with paths. */
+export interface Picked {
+  name: string;
+  files: FolderFile[];
+  /** Second line for the header (`local folder`, `from a .zip`). */
+  detail?: string;
+  notes?: string[];
+}
+
+const NOTHING = () => new LoadError('invalid-input', 'There is nothing to analyze there.', 'Pick a project folder, some .ts, .js or .py files, or a .zip.');
+
+/** Distinct paths for loose files (two picked files may share a name when they came from different folders). */
+function flat(files: File[]): FolderFile[] {
+  const seen = new Map<string, number>();
+  return files.map((file) => {
+    const n = (seen.get(file.name) ?? 0) + 1;
+    seen.set(file.name, n);
+    const dot = file.name.lastIndexOf('.');
+    return { path: n === 1 ? file.name : dot > 0 ? `${file.name.slice(0, dot)}-${n}${file.name.slice(dot)}` : `${file.name}-${n}`, file };
+  });
+}
+
+const looseLabel = (files: File[]) => (files.length === 1 ? files[0].name : `${files.length} files`);
+
+/**
+ * What an `<input type="file">` returned. From a folder picker (`webkitdirectory`) every file's `webkitRelativePath`
+ * starts with the folder's own name; from a plain picker the files are loose. A single .zip is unpacked.
+ */
+export async function entriesFromFileList(list: FileList | File[]): Promise<Picked> {
   const all = Array.from(list);
+  if (!all.length) throw NOTHING();
+  const rel = (f: File) => (f as File & { webkitRelativePath?: string }).webkitRelativePath || '';
+  if (all.length === 1 && isZip(all[0]) && !rel(all[0])) return entriesFromZip(all[0]);
+  if (!all.some(rel)) return { name: looseLabel(all), files: flat(all), detail: all.length === 1 ? 'one file' : 'local files' };
+  const prune = createFilter();
   const files: FolderFile[] = [];
   let name = '';
   for (const file of all) {
-    const rel = (file as File & { webkitRelativePath?: string }).webkitRelativePath || file.name;
-    const slash = rel.indexOf('/');
+    const r = rel(file);
+    const slash = r.indexOf('/');
     if (slash < 0) continue;
-    name ||= rel.slice(0, slash);
-    files.push({ path: rel.slice(slash + 1), file });
+    name ||= r.slice(0, slash);
+    const path = r.slice(slash + 1);
+    if (!prune.ignoresFile(path)) files.push({ path, file }); // node_modules and friends are dropped before anything is read
   }
   return { name: name || 'folder', files };
 }
@@ -33,20 +67,23 @@ interface FsEntry {
 }
 
 /**
- * From a drop. The entries must be taken from the event synchronously (they are invalid after the first await), and
- * directories are pruned while walking: `node_modules` or `.git` are never entered, which matters when a project has
- * a hundred thousand files in them.
+ * From a drop: one folder, loose files, several folders and files together, or a .zip. The entries must be taken from
+ * the event synchronously (they are invalid after the first await), so call this directly from the drop handler.
+ * Folders are pruned while walking: `node_modules` or `.git` are never entered, which matters when a project has a
+ * hundred thousand files in them.
  */
-export async function entriesFromDrop(dt: DataTransfer): Promise<{ name: string; files: FolderFile[] } | null> {
+export async function entriesFromDrop(dt: DataTransfer): Promise<Picked> {
   const roots: FsEntry[] = [];
   for (const item of Array.from(dt.items)) {
+    if (item.kind !== 'file') continue;
     const entry = (item as DataTransferItem & { webkitGetAsEntry?: () => FsEntry | null }).webkitGetAsEntry?.();
     if (entry) roots.push(entry);
   }
-  const dirs = roots.filter((r) => r.isDirectory);
-  if (dirs.length !== 1 || roots.length !== 1) return null; // exactly one folder
+  const loose = Array.from(dt.files); // for browsers that give no entries
+  if (!roots.length) return entriesFromFileList(loose);
+
   const prune = createFilter();
-  const files: FolderFile[] = [];
+  const fileOf = (e: FsEntry) => new Promise<File>((ok, err) => e.file!(ok, err));
   const readAll = async (dir: FsEntry) => {
     const reader = dir.createReader!();
     const out: FsEntry[] = [];
@@ -56,24 +93,35 @@ export async function entriesFromDrop(dt: DataTransfer): Promise<{ name: string;
       out.push(...batch);
     }
   };
+  const files: FolderFile[] = [];
   const walk = async (dir: FsEntry, path: string) => {
     for (const e of await readAll(dir)) {
       const p = path ? `${path}/${e.name}` : e.name;
       if (e.isDirectory) {
         if (!prune.ignoresDir(p)) await walk(e, p);
-      } else if (e.isFile && !prune.ignoresFile(p)) {
-        const file = await new Promise<File>((ok, err) => e.file!(ok, err));
-        files.push({ path: p, file });
-      }
+      } else if (e.isFile && !prune.ignoresFile(p)) files.push({ path: p, file: await fileOf(e) });
     }
   };
-  await walk(dirs[0], '');
-  return { name: dirs[0].name, files };
+
+  if (roots.length === 1 && roots[0].isDirectory) {
+    await walk(roots[0], '');
+    return { name: roots[0].name, files };
+  }
+  if (roots.length === 1) return entriesFromFileList([await fileOf(roots[0])]); // one file, or one .zip
+  // Several things at once: folders keep their own name as a path prefix, files sit at the top.
+  const looseFiles: File[] = [];
+  for (const r of roots) {
+    if (r.isDirectory) await walk(r, r.name);
+    else if (r.isFile) looseFiles.push(await fileOf(r));
+  }
+  files.push(...flat(looseFiles));
+  const onlyFiles = looseFiles.length === roots.length;
+  return { name: onlyFiles ? looseLabel(looseFiles) : `${roots.length} items`, files, detail: onlyFiles ? 'local files' : 'dropped items' };
 }
 
 /** Applies the same selection rules as the GitHub loader and reads the surviving files. */
 export async function loadFolder(
-  folder: { name: string; files: FolderFile[] },
+  folder: Picked,
   opts: { signal?: AbortSignal; onProgress?: (p: LoadProgress) => void } = {},
 ): Promise<LoadedSource> {
   const byPath = new Map(folder.files.map((f) => [f.path, f.file]));
@@ -92,7 +140,7 @@ export async function loadFolder(
     else sources.push(f);
   }
   if (!sources.length) {
-    throw new LoadError('empty', `No TypeScript, JavaScript or Python files found in ${folder.name}.`, 'CodeGraph reads .ts, .tsx, .js, .jsx, .mjs, .cjs and .py files.');
+    throw new LoadError('empty', `No TypeScript, JavaScript or Python files found in ${folder.name}.`, 'CodeGraph reads .ts, .tsx, .js, .jsx, .mjs, .cjs and .py files, from a folder, loose files or a .zip.');
   }
   sources.sort((a, b) => (a.path < b.path ? -1 : 1));
 
@@ -117,8 +165,8 @@ export async function loadFolder(
   );
   if (opts.signal?.aborted) throw new LoadError('cancelled', 'Cancelled.');
 
-  const notes: string[] = [];
+  const notes: string[] = [...(folder.notes ?? [])];
   if (tooLarge) notes.push(`${tooLarge} file${tooLarge === 1 ? ' was' : 's were'} skipped for being over 1 MB.`);
   if (failed) notes.push(`${failed} file${failed === 1 ? '' : 's'} could not be read and ${failed === 1 ? 'was' : 'were'} skipped.`);
-  return { label: folder.name, detail: 'local folder', origin: { kind: 'folder' }, files, configs: configMap, notes };
+  return { label: folder.name, detail: folder.detail ?? 'local folder', origin: { kind: 'folder' }, files, configs: configMap, notes };
 }

packages/web/src/sources/zip.ts

@@ -0,0 +1,60 @@
+// Reads a .zip (for example GitHub's "Download ZIP") in the page. Only the entries worth analyzing are decompressed:
+// dependencies, build output and anything that isn't source or config are skipped from the archive's index, so a zip
+// with a node_modules folder costs nothing extra.
+import { unzip, type UnzipFileInfo } from 'fflate';
+import { createFilter } from '@codegraph/extractors/browser';
+import type { FolderFile } from './local';
+import { LIMITS, LoadError } from './types';
+
+const MAX_ZIP_BYTES = 250_000_000;
+const KEEP_NAMES = new Set(['.gitignore', '.codegraphignore']);
+/** Files that say "this folder is a project root". */
+const PROJECT_MARKERS = new Set(['package.json', 'pyproject.toml', 'setup.py', 'setup.cfg', 'requirements.txt', 'tsconfig.json']);
+const base = (p: string) => p.slice(p.lastIndexOf('/') + 1);
+
+export const isZip = (f: File) => /\.zip$/i.test(f.name) || f.type === 'application/zip' || f.type === 'application/x-zip-compressed';
+
+export async function entriesFromZip(file: File): Promise<{ name: string; files: FolderFile[]; detail: string; notes: string[] }> {
+  if (file.size > MAX_ZIP_BYTES) {
+    throw new LoadError('empty', `${file.name} is larger than ${Math.round(MAX_ZIP_BYTES / 1e6)} MB.`, 'Zip just the source folders, or choose the project folder itself.');
+  }
+  const data = new Uint8Array(await file.arrayBuffer());
+  const prune = createFilter();
+  const tops = new Set<string>(); // first path segment of every entry, kept or not
+  const markerTops = new Set<string>(); // top-level folders that hold a project file directly
+  let anyFile = false;
+  let bytes = 0;
+  let overBudget = 0;
+  const filter = (info: UnzipFileInfo) => {
+    const name = info.name;
+    if (name.endsWith('/') || name.startsWith('__MACOSX/')) return false;
+    anyFile = true;
+    tops.add(name.includes('/') ? name.slice(0, name.indexOf('/')) : '');
+    if (name.split('/').length === 2 && PROJECT_MARKERS.has(base(name))) markerTops.add(name.slice(0, name.indexOf('/')));
+    if (prune.ignoresFile(name) || info.originalSize > LIMITS.maxFileBytes) return false;
+    const b = base(name);
+    if (!(prune.isSource(b) || prune.isConfig(b) || KEEP_NAMES.has(b))) return false;
+    if (bytes + info.originalSize > LIMITS.maxTotalBytes) {
+      overBudget++;
+      return false;
+    }
+    bytes += info.originalSize;
+    return true;
+  };
+  const out = await new Promise<Record<string, Uint8Array>>((resolve, reject) =>
+    unzip(data, { filter }, (err, res) => (err ? reject(new LoadError('invalid-input', `Couldn't open ${file.name} as a zip file.`, 'Check that it is a valid .zip archive.')) : resolve(res))),
+  );
+  if (!anyFile) throw new LoadError('empty', `${file.name} is empty.`);
+  // GitHub's "Download ZIP" (and "Compress" in a file manager) wrap everything in one folder, named like the zip or
+  // holding the project's files; paths should start inside it. A zip that just happens to hold one folder (a `src/`
+  // on its own) is left alone.
+  const zipName = file.name.replace(/\.zip$/i, '');
+  const only = tops.size === 1 && !tops.has('') ? [...tops][0] : undefined;
+  const strip = only && (only.toLowerCase() === zipName.toLowerCase() || markerTops.has(only)) ? only + '/' : '';
+  const files: FolderFile[] = Object.entries(out).map(([path, bytes]) => {
+    const rel = strip && path.startsWith(strip) ? path.slice(strip.length) : path;
+    return { path: rel, file: new File([new Uint8Array(bytes)], base(rel)) };
+  });
+  const notes = overBudget ? [`The zip holds more source than fits in a browser tab, so ${overBudget} file${overBudget === 1 ? ' was' : 's were'} left out.`] : [];
+  return { name: zipName, files, detail: 'from a .zip', notes };
+}

packages/web/src/styles.css

@@ -369,8 +369,9 @@ button.link.more {
 .drop {
   width: 100%;
   display: flex;
+  flex-wrap: wrap;
   align-items: center;
-  gap: 16px;
+  gap: 14px 16px;
   text-align: left;
   padding: 20px 22px;
   background: rgba(17, 22, 31, 0.6);
@@ -379,6 +380,12 @@ button.link.more {
   color: var(--dim);
   transition: border-color 0.15s, background 0.15s;
 }
+.drop-actions {
+  display: flex;
+  flex-wrap: wrap;
+  gap: 8px;
+  margin-left: auto;
+}
 .drop:hover {
   border-color: var(--accent);
   background: var(--accent-soft);
@@ -388,6 +395,7 @@ button.link.more {
   color: var(--accent-2);
 }
 .drop-text {
+  flex: 1 1 280px;
   display: flex;
   flex-direction: column;
   gap: 2px;