+ 摘要：本文报告记录了使用 export_hf_model_tree.py 和
+ evaluate_dna_probe_sweep.py 的实际运行，所用的模型关系文件位于
+ out/hf_model_tree/model_relations.jsonl，实验输出位于
+ out/dna_probe_sweep_modeltree。
out/dna_probe_sweep_modeltree。test_run_name）作为保留的测试运行。out/hf_model_tree/model_relations.jsonl 中的
+ is_same_or_related(query, base) 判定。sampled_negatives = round(negative_ratio * #positives) 随机采样（默认
+ negative_ratio=1.0），脚本使用 RNG 种子 42 保证可复现性。x = -distance（取负以使较大值倾向正类），
+ 经 StandardScaler 变换后喂入带 class_weight='balanced' 的 LinearSVC。test_run_name）构建的所有配对示例。--run 规范）构建的所有配对示例组成。balance_examples() 随机采样负样本至配置的
+ negative_ratio（默认 1.0）；测试路径也采用相同的采样（测试负样本同样采样为 1.0 倍正样本）。GroupKFold 或显式的组保留策略。negative_ratio=1.0）共同决定。每个集合在采样后包含 正样本 + 采样负样本
+ （正负样本数近似相等），因此正样本率由设计决定约为 50%。在生成的摘要
+ （temp_top_p_modeltree_summary.json）中，每个运行的测试占比大约在 7.9%–13.5%
+ （平均约 12.5%），差异来源于不同被保留运行生成的配对数量不同。负样本采样的 RNG 种子为 42，
+ 保证了采样的确定性与可复现性。脚本生成了 temp_top_p_modeltree_summary.json，包含每个运行的聚合指标。表格展示了部分运行的关键指标。
| 运行 | 温度 | Top-p | 模型数 | Top1 | MRR | 线性 AUC |
|---|---|---|---|---|---|---|
| temp00_p08_r1 | 0.0 | 0.8 | 66 | 0.924 | 0.940 | 0.936 |
| temp02_p08_r1 | 0.2 | 0.8 | 96 | 0.406 | 0.496 | 0.818 |
| temp02_p09_r1 | 0.2 | 0.9 | 97 | 0.361 | 0.449 | 0.801 |
| temp02_p10_r1 | 0.2 | 1.0 | 93 | 0.387 | 0.484 | 0.804 |
| temp03_p08_r1 | 0.3 | 0.8 | 98 | 0.327 | 0.428 | 0.773 |
| temp03_p09_r1 | 0.3 | 0.9 | 98 | 0.296 | 0.400 | 0.759 |
| temp03_p10_r1 | 0.3 | 1.0 | 97 | 0.227 | 0.312 | 0.723 |
| temp05_p08_r1 | 0.5 | 0.8 | 88 | 0.239 | 0.311 | 0.706 |
生成的诊断图（来自实验输出目录）：
+
+ 
+ 
+ evaluate_dna_probe_sweep.py 修改为使用 GroupKFold
+ 或按连通分量 id 显式保留并重新生成指标。temp_top_p_modeltree_summary.json
+ / temp_top_p_modeltree_summary.csv 作为表和图的来源。（生成于机器上，图片与 JSON 输出路径均为相对工作区路径。）
+ + diff --git a/docs/svm_modeltree_experiment_zh.tex b/docs/svm_modeltree_experiment_zh.tex new file mode 100644 index 0000000..0000be7 --- /dev/null +++ b/docs/svm_modeltree_experiment_zh.tex @@ -0,0 +1,131 @@ +\documentclass[11pt,a4paper]{article} +\usepackage{graphicx} +\usepackage{booktabs} +\usepackage{amsmath} +\usepackage[utf8]{inputenc} +\usepackage{geometry} +\geometry{margin=1in} +\title{DNA 探针扫描 — 模型树 SVM 实验（实际结果）} +\author{} +\date{\today} + +\begin{document} +\maketitle + +\begin{abstract} +本文报告记录了使用 \texttt{export\_hf\_model\_tree.py} 和 +\texttt{evaluate\_dna\_probe\_sweep.py} 的实际运行过程，所用的模型关系文件位于 +\texttt{out/hf\_model\_tree/model\_relations.jsonl}，实验输出位于 +\texttt{out/dna\_probe\_sweep\_modeltree}。 +\end{abstract} + +\section{概述} +本次流水线执行步骤如下： +\begin{enumerate} + \item 使用流式并查集（union--find）从模型列表导出 Hugging Face 模型关系（连通分量）。 + \item 在多种生成设置（runs）上，使用最近邻和成对线性 SVM 探针评估 DNA 可识别性。 + \item 将汇总指标、预测 CSV 和诊断图写入 \texttt{out/dna\_probe\_sweep\_modeltree}。 +\end{enumerate} + +\section{模型树导出摘要} +导出器写入了 \texttt{out/hf\_model\_tree/model\_relations.jsonl} 及摘要文件 +\texttt{out/hf\_model\_tree/model\_relations\_summary.json}。摘要中的关键计数： +\begin{itemize} + \item 输入模型数量：11812 + \item 解析后节点数：13330 + \item 提取的 base 边数：6282 + \item 大小 $\ge2$ 的连通分量（组）数：790 + \item 跳过的模型（获取元数据时出错）：618 +\end{itemize} + +\section{SVM 探针的实际实现} +脚本的实际做法概述： +\begin{itemize} + \item 对每个运行（生成设置）加载预计算的模型 DNA 特征向量。 + \item 构建查询模型与候选基模型之间的所有成对示例，每对记录两者间的距离（默认为余弦距离）。 + \item 如果查询与基模型为同一模型或在导出的模型关系中属于同一关系组，则标记为正样本，否则为负样本。 + \item 具体划分过程： + \begin{enumerate} + \item 选择一个运行（\texttt{test\_run\_name}）作为保留的测试运行。 + \item 通过遍历其余所有运行构建训练示例，生成这些运行中的所有（query, base）配对。 + \item 测试示例仅来自被保留的测试运行（该运行中查询与基模型的所有配对）。 + \item 训练和测试中的正样本依据 \texttt{out/hf\_model\_tree/model\_relations.jsonl} 中的 + \texttt{is\_same\_or\_related(query, base)} 判定。 + \item 负样本按 \texttt{sampled\_negatives = round(negative\_ratio * \#positives)} 随机采样（默认 + \texttt{negative\_ratio=1.0}），脚本使用 RNG 种子 42 保证可复现性。 + \end{enumerate} + \item 特征与分类器：使用单一特征 $x = -\text{distance}$（取负以使较大值倾向正类）， + 经 \texttt{StandardScaler} 变换后喂入带 \texttt{class\_weight='balanced'} 的 \texttt{LinearSVC}。 +\end{itemize} + +\section{训练/测试划分（具体）} +本次运行采用跨运行（leave‑one‑run‑out）划分： +\begin{itemize} + \item 测试集：仅包含来自单个被保留运行（\texttt{test\_run\_name}）构建的所有配对示例。 + \item 训练集：由其他所有运行（除被保留运行外的 \texttt{--run} 规范）构建的所有配对示例组成。 + \item 负样本采样：配对构建后，按代码中的 \texttt{balance\_examples()} 随机采样负样本至配置的 + \texttt{negative\_ratio}（默认 1.0）；测试路径也采用相同的采样（测试负样本同样采样为 1.0 倍正样本）。 + \item 后果：该划分衡量跨生成设置（temperature/top‑p）的泛化能力，但不保证严格的组级排除—— + 如果相关模型在不同运行中都被生成，则可能同时出现在训练和测试中。若要做组级隔离评估 + （确保测试组的相关模型不出现在训练中），请使用基于连通分量 id 的 \texttt{GroupKFold} 或显式的组保留策略。 + \item 观测到的测试/训练比率：该比率并非事先固定选择，而是由 leave‑one‑run‑out 设计、每运行配对构建 + 以及平衡负样本（\texttt{negative\_ratio=1.0}）共同决定。每个集合在采样后包含 正样本 + 采样负样本 + （正负样本数近似相等），因此正样本率由设计决定约为 50\%。在生成的摘要 + （\texttt{temp\_top\_p\_modeltree\_summary.json}）中，每个运行的测试占比大约在 7.9\%–13.5\% + （平均约 12.5\%），差异来源于不同被保留运行生成的配对数量不同。负样本采样的 RNG 种子为 42， + 保证了采样的确定性与可复现性。 +\end{itemize} + +\section{结果（示例运行）} +脚本生成了 \texttt{temp\_top\_p\_modeltree\_summary.json}，包含每个运行的聚合指标。 +表\ref{tab:summary} 展示了本次扫描中部分运行的关键指标。 + +\begin{table}[ht] +\centering +\caption{每运行汇总（最近邻 + 线性 SVM 指标）}\label{tab:summary} +\begin{tabular}{llrrrrr} +\toprule +运行 & 温度 & Top‑p & 模型数 & Top1 & MRR & 线性 AUC \\ +\midrule +temp00\_p08\_r1 & 0.0 & 0.8 & 66 & 0.924 & 0.940 & 0.936 \\ +temp02\_p08\_r1 & 0.2 & 0.8 & 96 & 0.406 & 0.496 & 0.818 \\ +temp02\_p09\_r1 & 0.2 & 0.9 & 97 & 0.361 & 0.449 & 0.801 \\ +temp02\_p10\_r1 & 0.2 & 1.0 & 93 & 0.387 & 0.484 & 0.804 \\ +temp03\_p08\_r1 & 0.3 & 0.8 & 98 & 0.327 & 0.428 & 0.773 \\ +temp03\_p09\_r1 & 0.3 & 0.9 & 98 & 0.296 & 0.400 & 0.759 \\ +temp03\_p10\_r1 & 0.3 & 1.0 & 97 & 0.227 & 0.312 & 0.723 \\ +temp05\_p08\_r1 & 0.5 & 0.8 & 88 & 0.239 & 0.311 & 0.706 \\ +\bottomrule +\end{tabular} +\end{table} + +\section{图} +以下为生成的诊断图（来自实验输出目录），如需展示可替换为更高分辨率导出图像。 + +\begin{figure}[ht] + \centering + \includegraphics[width=0.8\\linewidth]{out/dna_probe_sweep_modeltree/temp_top_p_modeltree_curves.png} + \caption{按设置组织的热图网格。每个面板显示 top-1/top-3/top-5/MRR 在 temperature 与 top-p 上的分布，并标出最佳单元。} +\end{figure} + +\begin{figure}[ht] + \centering + \begin{tabular}{cc} + \includegraphics[width=0.45\\linewidth]{out/dna_probe_sweep_modeltree/temp_top_p_modeltree_linear_accuracy_setting_heatmap.png} & + \includegraphics[width=0.45\\linewidth]{out/dna_probe_sweep_modeltree/temp_top_p_modeltree_top1_setting_heatmap.png} \\ + (a) 线性准确率热图 & (b) Top‑1 热图 \\ + \end{tabular} + \caption{按设置（temperature × top-p）划分的单指标热图。} +\end{figure} + +\section{说明与下一步} +\begin{itemize} + \item 该 SVM 探针故意保持简单（单特征线性分类器），在低温度运行（例如 \texttt{temp00\_p08\_r1}）中表现良好， + 因为最近邻效果已很强；随着温度升高，自我距离增大，性能下降。 + \item 若需要组感知评估（在未见的相关组上测试），我可以将 \texttt{evaluate\_dna\_probe\_sweep.py} + 修改为使用 \texttt{GroupKFold} 或按连通分量 id 做显式保留并重新生成指标。 + \item 演示或展示时可直接使用文中 PNG 以及 \texttt{temp\_top\_p\_modeltree\_summary.json} + / \texttt{temp\_top\_p\_modeltree\_summary.csv} 作为表和图的来源。 +\end{itemize} + +\end{document} diff --git a/docs/weekly_phd_update_2026-08-05.tex b/docs/weekly_phd_update_2026-08-05.tex new file mode 100644 index 0000000..d0f0fef --- /dev/null +++ b/docs/weekly_phd_update_2026-08-05.tex @@ -0,0 +1,120 @@ +\documentclass[10pt,a4paper]{article} +\usepackage[margin=0.68in]{geometry} +\usepackage{booktabs,tabularx,array,xcolor,enumitem} +\usepackage[hidelinks]{hyperref} +\setlength{\parindent}{0pt} +\setlength{\parskip}{4pt} +\setlist{nosep,leftmargin=*} +\renewcommand{\arraystretch}{1.08} +\newcommand{\good}[1]{\textcolor{green!45!black}{#1}} +\newcommand{\warn}[1]{\textcolor{orange!75!black}{#1}} + +\title{Weekly Update} +\author{} +\date{5 August 2026} + +\begin{document} +\maketitle +\vspace{-2.2em} + +\section*{What changed this week} + +\textbf{Grid generation finished its scheduled pass.} We now have +\good{2,528/2,600 successful artifacts (97.2\%)} across 100 validated models, +13 decoding settings, and two rounds. There are no ordinary pending cells and +no experiment worker is currently active. The 72 terminal failures comprise +52 missing-file failures and 20 empty responses; they require targeted repair, +not another generic restart. + +\begin{table}[h] +\centering +\caption{Successful artifacts in the two-round grid (maximum 200 per cell).} +\begin{tabular}{crrr} +\toprule +$T$ & $p=0.8$ & $p=0.9$ & $p=1.0$ \\ +\midrule +0.0 & -- & -- & 200/200 \\ +0.2 & 199/200 & 200/200 & 198/200 \\ +0.3 & 199/200 & 189/200 & 185/200 \\ +0.5 & 198/200 & 196/200 & 176/200 \\ +0.7 & 196/200 & 197/200 & 195/200 \\ +\bottomrule +\end{tabular} +\end{table} + +\textbf{Why the strict cohort is 59 despite 97.2\% completion.} The 72 failed +slots are spread across 41 different models, so only $100-41=59$ models have +all 26 required artifacts. This does \emph{not} mean that only 59 models ran. +Per-setting paired cohorts are much larger (77--100), and bounded common +cohorts are 97, 86, 63, and 59 for $\tau=0.2,0.3,0.5,0.7$, respectively. + +\textbf{Method comparison is now controlled.} For cosine, linear SVM, and RBF +SVM, we will use the same model IDs, prompts, response files, 128-D DNA, +decoding cells, repeat-1 training/reference split, repeat-2 held-out queries, +candidate set, and metrics. Only the decision rule changes. Hyperparameters are +selected using repeat 1 only; differences are paired on identical queries and +reported with 95\% model-cluster bootstrap intervals. + +\textbf{Preliminary finding (exploratory 84-model cohort).} Linear SVM was best +on average over the available cells, but temperature/top-$p$ effects were not +monotone. At $T=0.3$, for example, all methods fell sharply at $p=1.0$ relative +to $p=0.8$ or $0.9$. This supports a limited interaction finding, not a broad +causal conclusion, because the final cohort and seed audit are unfinished. + +\clearpage +\section*{New method and next steps} + +\textbf{Can the proposed distributional method be compared fairly?} Yes, but +the claim must match the available sampling budget. At $K=1$, RBF-MMD reduces +to a kernel point-distance check and cannot demonstrate a multi-sample +advantage. A fair $K=2$ comparison needs two independent reference responses +and two independent query responses for every prompt, with the same encoder, +prompts, tokens, and samples used by the mean-cosine baseline. The kernel +bandwidth must be calibrated on reference data only and then frozen. The +existing two rounds support the cosine/SVM comparison but not this $K=2$ +design without additional generation. + +\begin{table}[h] +\centering +\small +\caption{Controlled comparison and current readiness.} +\begin{tabularx}{\linewidth}{>{\raggedright\arraybackslash}p{0.27\linewidth} + >{\raggedright\arraybackslash}p{0.25\linewidth}X} +\toprule +Comparison & Variable changed & Status \\ +\midrule +Cosine vs linear/RBF SVM & Decision rule only & Ready after cohort recovery and round audit \\ +Mean cosine vs RBF-MMD, $K=1$ & Distance only & Feasible as a point check; not multi-sample evidence \\ +Mean cosine vs RBF-MMD, $K=2$ & Representation/distance only & Requires additional independent responses \\ +\bottomrule +\end{tabularx} +\end{table} + +\textbf{Recovery priority.} Twenty-four affected models each lack only one +slot. Repairing any 21 of them is the minimum theoretical route from 59 to the +preregistered strict common cohort of at least 80. We will first diagnose the +missing-file compatibility/source issue and the empty-response issue, then run +only the affected model--cell--round keys. + +\textbf{Expected completion from the 5 August snapshot:} +\begin{itemize} + \item \textbf{Failure diagnosis --- 1 working day:} produce a + failure-specific repair list. + \item \textbf{Targeted recovery and audit --- 2--3 working days:} reach a + strict common $N\geq80$ and confirm independent rounds. + \item \textbf{Frozen cosine/SVM evaluation --- 1 working day:} produce paired + results and 95\% confidence intervals. + \item \textbf{Report integration --- 1 working day:} prepare the primary + report for supervisor review. + \item \textbf{Optional fair $K=2$ MMD study --- 3--5 additional working + days:} obtain two independent samples on both sides. +\end{itemize} + +\textbf{This week's deliverables.} The short update now foregrounds the new +progress, controlled comparison, limited findings, and completion dates. The +full robustness report has also been refreshed with the latest grid counts, +paired/common cohort sizes, failure accounting, controlled-comparison matrix, +and revised execution schedule. Exploratory results remain explicitly starred +until rerun on the recovered primary cohort. + +\end{document} diff --git a/pyproject.toml b/pyproject.toml index 16a1cca..d202f2e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -32,11 +32,36 @@ dependencies = [ "wonderwords>=2.2.0", "openai>=1.0.0", "tiktoken>=0.7.0", + "python-dotenv>=1.0.0", ] [project.optional-dependencies] model_scraping = ["requests>=2.31.0"] +apple = [ + "mlx>=0.10.0; sys_platform == 'darwin' and platform_machine == 'arm64'", + "mlx-lm>=0.10.0; sys_platform == 'darwin' and platform_machine == 'arm64'", +] + +quantization = [ + "bitsandbytes>=0.46.1", + "autoawq>=0.2.0", + "auto-gptq>=0.5.0", + "optimum>=1.16.0", + "compressed-tensors>=0.1.0", +] + +model_families = [ + "mamba-ssm>=1.0.0; sys_platform == 'linux'", + "timm>=0.9.0", +] + +full = [ + "llm-dna[apple]", + "llm-dna[quantization]", + "llm-dna[model_families]", +] + vllm = ["vllm>=0.4.0"] dev = [ diff --git a/reports/primary_robustness_run_grid.csv b/reports/primary_robustness_run_grid.csv new file mode 100644 index 0000000..2a0b27f --- /dev/null +++ b/reports/primary_robustness_run_grid.csv @@ -0,0 +1,27 @@ +temperature,top_p,repeat,proposed_backfill_seed,recorded_seeds,success,failed,skipped,pending_status,no_status,missing_success,started,manifest_models,cohort_overlap,sources +0.0,1.0,r1,42001,not explicitly set or recorded,100,0,0,0,0,0,True,87,87,out/rand_chinese_temp_top_p_sweep_t00_p10_r1 +0.0,1.0,r2,42002,42002,100,0,0,0,0,0,True,100,100,out/primary_robustness_validated100_20260730/primary_grid_backfill_20260730_t00_p10_r2 +0.2,0.8,r1,42001,not explicitly set or recorded,100,0,0,0,0,0,True,87,87,out/rand_chinese_temp_top_p_sweep_t02_p08_r1 +0.2,0.8,r2,42002,not explicitly set or recorded,99,1,0,0,0,1,True,87,87,out/rand_chinese_temp_top_p_sweep_t02_p08_r2 +0.2,0.9,r1,42001,not explicitly set or recorded,100,0,0,0,0,0,True,87,87,out/rand_chinese_temp_top_p_sweep_t02_p09_r1 +0.2,0.9,r2,42002,not explicitly set or recorded,100,0,0,0,0,0,True,87,87,out/rand_chinese_temp_top_p_sweep_t02_p09_r2 +0.2,1.0,r1,42001,not explicitly set or recorded,99,1,0,0,0,1,True,87,87,out/rand_chinese_temp_top_p_sweep_t02_p10_r1 +0.2,1.0,r2,42002,not explicitly set or recorded,99,1,0,0,0,1,True,87,87,out/rand_chinese_temp_top_p_sweep_t02_p10_r2 +0.3,0.8,r1,42001,not explicitly set or recorded,100,0,0,0,0,0,True,87,87,out/rand_chinese_temp_top_p_sweep_t03_p08_r1 +0.3,0.8,r2,42002,not explicitly set or recorded,99,1,0,0,0,1,True,87,87,out/rand_chinese_temp_top_p_sweep_t03_p08_r2 +0.3,0.9,r1,42001,not explicitly set or recorded,96,4,0,0,0,4,True,87,87,out/rand_chinese_temp_top_p_sweep_t03_p09_r1 +0.3,0.9,r2,42002,not explicitly set or recorded,93,7,0,0,0,7,True,87,87,out/rand_chinese_temp_top_p_sweep_t03_p09_r2 +0.3,1.0,r1,42001,not explicitly set or recorded,94,6,0,0,0,6,True,87,87,out/rand_chinese_temp_top_p_sweep_t03_p10_r1 +0.3,1.0,r2,42002,not explicitly set or recorded,91,9,0,0,0,9,True,87,87,out/rand_chinese_temp_top_p_sweep_resume_t03p10r2_t03_p10_r2 +0.5,0.8,r1,42,not explicitly set or recorded,100,0,0,0,0,0,True,87,87,out/rand_chinese_temp_top_p_sweep_resume_t05_t07_t05_p08_r1 +0.5,0.8,r2,42,42,98,2,0,0,0,2,True,100,100,out/primary_robustness_validated100_20260730/primary_grid_backfill_20260730_t05_p08_r2 +0.5,0.9,r1,42,not explicitly set or recorded,98,2,0,0,0,2,True,87,87,out/rand_chinese_temp_top_p_sweep_t05_p09_r1 +0.5,0.9,r2,42,not explicitly set or recorded,98,2,0,0,0,2,True,87,87,out/rand_chinese_temp_top_p_sweep_t05_p09_r2 +0.5,1.0,r1,42,not explicitly set or recorded,98,2,0,0,0,2,True,87,87,out/rand_chinese_temp_top_p_sweep_t05_p10_r1 +0.5,1.0,r2,42,42,78,22,0,0,0,22,True,100,100,out/primary_robustness_validated100_20260730/primary_grid_backfill_20260730_t05_p10_r2 +0.7,0.8,r1,42001,not explicitly set or recorded,97,3,0,0,0,3,True,87,87,out/rand_chinese_temp_top_p_sweep_t07_p08_r1 +0.7,0.8,r2,42002,42002,99,1,0,0,0,1,True,100,100,out/primary_robustness_validated100_20260730/primary_grid_backfill_20260730_t07_p08_r2 +0.7,0.9,r1,42001,not explicitly set or recorded,99,1,0,0,0,1,True,87,87,out/rand_chinese_0p3b_7b_temp07 +0.7,0.9,r2,42002,42002,98,2,0,0,0,2,True,100,100,out/primary_robustness_validated100_20260730/primary_grid_backfill_20260730_t07_p09_r2 +0.7,1.0,r1,42001,not explicitly set or recorded,96,4,0,0,0,4,True,87,87,out/rand_chinese_temp_top_p_sweep_t07_p10_r1 +0.7,1.0,r2,42002,42002,99,1,0,0,0,1,True,100,100,out/primary_robustness_validated100_20260730/primary_grid_backfill_20260730_t07_p10_r2 diff --git a/reports/primary_robustness_run_grid.md b/reports/primary_robustness_run_grid.md new file mode 100644 index 0000000..930f88a --- /dev/null +++ b/reports/primary_robustness_run_grid.md @@ -0,0 +1,65 @@ +# Primary robustness experiment — run grid + +Updated: 2026-08-05T18:55:39+08:00 + +Revised plan: + +- Fixed cohort: **the latest validated100 model list** from `top_p_control_t05_t05_p06_r2/manifest.json`. +- Historical artifacts are reused only when their model ID and decoding cell match this cohort/grid. +- Two repeats per condition. The shared `temperature=0.5` row uses **seed 42 for both rounds** so its `top_p=0.8/0.9/1.0` cells can also serve the validated top-p control. Other proposed grid backfills remain **r1 / 42001** and **r2 / 42002**. +- The historical sweep did not explicitly set or record a response-generation seed. Its recorded `random_seed=42` controlled probe selection and DNA reduction, not the generation sampler. +- Stochastic grid: temperature `{0.2, 0.3, 0.5, 0.7}` × top-p `{0.8, 0.9, 1.0}`. +- Deterministic control: `temperature=0`, `top_p=1.0` (top-p inactive). +- Total: **13 settings × 2 repeats × 100 models = 2,600 successful model artifacts required**. + +Important cohort note: the two validated100 variants overlap by 97 models, but the older temperature/top-p sweep used here overlaps the latest validated100 cohort by **87 models**. Therefore those grid cells require 13 cohort backfills plus retries for any failed/pending overlapping models. + +Legend: `✅` both repeats have 100 successful models; `♻️` historical results are reusable but the cell is incomplete; `⬜` no matching run was found; `—` is not a planned condition. + +| Temperature \\ Top-p | 0.8 | 0.9 | 1.0 | +|---:|---:|---:|---:| +| 0 | — | — | ✅ 200/200 | +| 0.2 | ♻️ 199/200；还需 1 | ✅ 200/200 | ♻️ 198/200；还需 2 | +| 0.3 | ♻️ 199/200；还需 1 | ♻️ 189/200；还需 11 | ♻️ 185/200；还需 15 | +| 0.5 | ♻️ 198/200；还需 2 | ♻️ 196/200；还需 4 | ♻️ 176/200；还需 24 | +| 0.7 | ♻️ 196/200；还需 4 | ♻️ 197/200；还需 3 | ♻️ 195/200；还需 5 | + +## Overall progress + +- Successful artifacts: **2528/2600**. +- Settings started: **13/13**. +- Settings fully complete: **2/13**. +- Remaining successful artifacts required: **72**. + +## Repeat-level detail + +| Temperature | Top-p | Round | Historical generation seed | Proposed backfill seed (not run) | Reusable success | Failed | Pending/no status | Still needed | State | +|---:|---:|---:|---|---:|---:|---:|---:|---:|---| +| 0 | 1 | r1 | not explicitly set or recorded | 42001 | 100 | 0 | 0 | 0 | ✅ complete | +| 0 | 1 | r2 | 42002 | 42002 | 100 | 0 | 0 | 0 | ✅ complete | +| 0.2 | 0.8 | r1 | not explicitly set or recorded | 42001 | 100 | 0 | 0 | 0 | ✅ complete | +| 0.2 | 0.8 | r2 | not explicitly set or recorded | 42002 | 99 | 1 | 0 | 1 | ♻️ reusable/incomplete | +| 0.2 | 0.9 | r1 | not explicitly set or recorded | 42001 | 100 | 0 | 0 | 0 | ✅ complete | +| 0.2 | 0.9 | r2 | not explicitly set or recorded | 42002 | 100 | 0 | 0 | 0 | ✅ complete | +| 0.2 | 1 | r1 | not explicitly set or recorded | 42001 | 99 | 1 | 0 | 1 | ♻️ reusable/incomplete | +| 0.2 | 1 | r2 | not explicitly set or recorded | 42002 | 99 | 1 | 0 | 1 | ♻️ reusable/incomplete | +| 0.3 | 0.8 | r1 | not explicitly set or recorded | 42001 | 100 | 0 | 0 | 0 | ✅ complete | +| 0.3 | 0.8 | r2 | not explicitly set or recorded | 42002 | 99 | 1 | 0 | 1 | ♻️ reusable/incomplete | +| 0.3 | 0.9 | r1 | not explicitly set or recorded | 42001 | 96 | 4 | 0 | 4 | ♻️ reusable/incomplete | +| 0.3 | 0.9 | r2 | not explicitly set or recorded | 42002 | 93 | 7 | 0 | 7 | ♻️ reusable/incomplete | +| 0.3 | 1 | r1 | not explicitly set or recorded | 42001 | 94 | 6 | 0 | 6 | ♻️ reusable/incomplete | +| 0.3 | 1 | r2 | not explicitly set or recorded | 42002 | 91 | 9 | 0 | 9 | ♻️ reusable/incomplete | +| 0.5 | 0.8 | r1 | not explicitly set or recorded | 42 | 100 | 0 | 0 | 0 | ✅ complete | +| 0.5 | 0.8 | r2 | 42 | 42 | 98 | 2 | 0 | 2 | ♻️ reusable/incomplete | +| 0.5 | 0.9 | r1 | not explicitly set or recorded | 42 | 98 | 2 | 0 | 2 | ♻️ reusable/incomplete | +| 0.5 | 0.9 | r2 | not explicitly set or recorded | 42 | 98 | 2 | 0 | 2 | ♻️ reusable/incomplete | +| 0.5 | 1 | r1 | not explicitly set or recorded | 42 | 98 | 2 | 0 | 2 | ♻️ reusable/incomplete | +| 0.5 | 1 | r2 | 42 | 42 | 78 | 22 | 0 | 22 | ♻️ reusable/incomplete | +| 0.7 | 0.8 | r1 | not explicitly set or recorded | 42001 | 97 | 3 | 0 | 3 | ♻️ reusable/incomplete | +| 0.7 | 0.8 | r2 | 42002 | 42002 | 99 | 1 | 0 | 1 | ♻️ reusable/incomplete | +| 0.7 | 0.9 | r1 | not explicitly set or recorded | 42001 | 99 | 1 | 0 | 1 | ♻️ reusable/incomplete | +| 0.7 | 0.9 | r2 | 42002 | 42002 | 98 | 2 | 0 | 2 | ♻️ reusable/incomplete | +| 0.7 | 1 | r1 | not explicitly set or recorded | 42001 | 96 | 4 | 0 | 4 | ♻️ reusable/incomplete | +| 0.7 | 1 | r2 | 42002 | 42002 | 99 | 1 | 0 | 1 | ♻️ reusable/incomplete | + +Ignored 43 manifest(s) outside the revised primary grid; see the CSV/source tree for audit. diff --git a/scripts/analyze_openrouter_dna_stability.py b/scripts/analyze_openrouter_dna_stability.py new file mode 100644 index 0000000..3137668 --- /dev/null +++ b/scripts/analyze_openrouter_dna_stability.py @@ -0,0 +1,255 @@ +#!/usr/bin/env python3 +"""Generate sliding 50-response subset DNAs from cached responses.""" + +from __future__ import annotations + +import json +import logging +import os +import sys +from dataclasses import dataclass, replace +from datetime import datetime +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +SRC = ROOT / "src" +if str(SRC) not in sys.path: + sys.path.insert(0, str(SRC)) + +from llm_dna.api import ( + DNAExtractionConfig, + _default_model_metadata, + _extract_signature_from_text_responses, + _save_signature_outputs, +) + +DATA_DIR = ROOT / "dna-out-chat" / "rand_chinese" +OUTPUT_ROOT = ROOT / "dna-out-chat" / "rand_chinese_subsets" +ENCODER_NAME = "all-mpnet-base-v2" +WINDOW_SIZE = 50 +WINDOW_STEP = 5 + + +@dataclass(frozen=True) +class ValidModel: + key: str + model_name: str + responses: list[str] + payload: dict + + +def load_valid_models(data_dir: Path) -> list[ValidModel]: + """Load complete artifacts with at least one non-empty cached response.""" + records: list[ValidModel] = [] + if not data_dir.exists(): + return records + for model_dir in sorted(path for path in data_dir.iterdir() if path.is_dir()): + response_path = model_dir / "responses.json" + summary_files = list(model_dir.glob("*_summary.json")) + dna_files = list(model_dir.glob("*_dna.json")) + if not response_path.exists() or not summary_files or not dna_files: + continue + try: + payload = json.loads(response_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + continue + responses = [ + str(item.get("response", "")).strip() + for item in payload.get("items", []) + if isinstance(item, dict) and str(item.get("response", "")).strip() + ] + if responses: + records.append( + ValidModel( + key=model_dir.name, + model_name=str(payload.get("model") or model_dir.name), + responses=responses, + payload=payload, + ) + ) + return records + + +def timestamp() -> str: + return datetime.now().strftime("%Y-%m-%d %H:%M:%S") + + +def default_run_dir() -> Path: + stamp = datetime.now().strftime("%Y%m%d_%H%M%S") + return OUTPUT_ROOT / f"run_{stamp}" + + +def log(message: str, log_file: Path) -> None: + line = f"[{timestamp()}] {message}" + print(line, flush=True) + with log_file.open("a", encoding="utf-8") as handle: + handle.write(line + "\n") + + +def load_models(limit: int | None) -> list[tuple[str, dict]]: + models: list[tuple[str, dict]] = [] + for record in load_valid_models(DATA_DIR): + payload = dict(record.payload) + items = [item for item in payload.get("items", []) if isinstance(item, dict) and str(item.get("response", "")).strip()] + if len(items) < WINDOW_SIZE: + continue + payload["items"] = items + models.append((record.key, payload)) + if limit is not None and len(models) >= limit: + break + return models + + +def build_windows(item_count: int) -> list[tuple[str, int, int]]: + windows = [] + index = 1 + for start in range(0, item_count - WINDOW_SIZE + 1, WINDOW_STEP): + end = start + WINDOW_SIZE + windows.append((f"50_{index:03d}", start, end)) + index += 1 + return windows + + +def subset_payload(base_payload: dict, subset_name: str, start: int, end: int) -> dict: + subset_items = base_payload["items"][start:end] + return { + "model": base_payload.get("model", ""), + "dataset": base_payload.get("dataset", "rand_chinese"), + "count": len(subset_items), + "subset": subset_name, + "subset_start": start, + "subset_end": end, + "items": subset_items, + } + + +def save_subset_responses(path: Path, payload: dict) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8") + + +def build_config(model_name: str, output_dir: Path, output_path: Path, sample_count: int) -> DNAExtractionConfig: + return DNAExtractionConfig( + model_name=model_name, + model_type="openrouter", + dataset="rand_chinese", + probe_set="rand", + max_samples=sample_count, + extractor_type="embedding", + dna_dim=128, + reduction_method="random_projection", + embedding_merge="concat", + output_dir=output_dir, + output_path=output_path, + save=True, + device="cpu", + random_seed=42, + max_length=1024, + ) + + +def main() -> int: + limit = None if len(sys.argv) <= 1 or sys.argv[1] == "all" else int(sys.argv[1]) + run_dir = Path(sys.argv[2]) if len(sys.argv) > 2 else default_run_dir() + run_dir.mkdir(parents=True, exist_ok=True) + log_file = run_dir / "run.log" + distance_file = run_dir / "distance_summary.json" + log_file.write_text("", encoding="utf-8") + + os.environ.setdefault("HF_HUB_OFFLINE", "1") + os.environ.setdefault("TRANSFORMERS_OFFLINE", "1") + logging.getLogger().setLevel(logging.WARNING) + + models = load_models(limit) + if not models: + raise SystemExit(f"no models with at least {WINDOW_SIZE} non-empty responses found in {DATA_DIR}") + + log(f"loaded {len(models)} model(s)", log_file) + all_distances: dict[str, dict[str, object]] = {} + + for index, (model_key, payload) in enumerate(models, 1): + log(f"processing {index}/{len(models)}: {model_key}", log_file) + signatures = {} + subset_dirs = {} + windows = build_windows(len(payload["items"])) + log(f" windows={len(windows)}", log_file) + + for subset_name, start, end in windows: + tagged_key = f"{model_key}__subset_{subset_name}" + subset_dir = run_dir / tagged_key + subset_dirs[subset_name] = subset_dir + subset_data = subset_payload(payload, subset_name, start, end) + save_subset_responses(subset_dir / "responses.json", subset_data) + + output_path = subset_dir / f"{tagged_key}_dna.json" + config = build_config( + model_name=payload.get("model", model_key), + output_dir=run_dir, + output_path=output_path, + sample_count=len(subset_data["items"]), + ) + signature, _, elapsed = _extract_signature_from_text_responses( + model_name=payload.get("model", model_key), + responses=[item["response"] for item in subset_data["items"]], + config=config, + model_meta=_default_model_metadata(payload.get("model", model_key)), + generation_device="cpu", + sentence_encoder=ENCODER_NAME, + encoder_device="cpu", + ) + _save_signature_outputs( + signature=signature, + config=config, + output_path=output_path, + summary_path=output_path.with_name(f"{output_path.stem}_summary.json"), + elapsed_seconds=elapsed, + ) + signatures[subset_name] = signature + + pairwise = {} + subset_names = list(signatures) + cosine_values = [] + euclidean_values = [] + for i in range(len(subset_names)): + for j in range(i + 1, len(subset_names)): + a = subset_names[i] + b = subset_names[j] + cosine = signatures[a].distance_to(signatures[b], metric="cosine") + euclidean = signatures[a].distance_to(signatures[b], metric="euclidean") + pairwise[f"{a}__{b}"] = {"cosine": cosine, "euclidean": euclidean} + cosine_values.append(cosine) + euclidean_values.append(euclidean) + + all_distances[model_key] = { + "subset_dirs": {name: str(path) for name, path in subset_dirs.items()}, + "windows": [ + {"name": subset_name, "start": start, "end": end} + for subset_name, start, end in windows + ], + "pairwise": pairwise, + "summary": { + "window_size": WINDOW_SIZE, + "window_step": WINDOW_STEP, + "window_count": len(windows), + "mean_cosine": sum(cosine_values) / len(cosine_values) if cosine_values else 0.0, + "max_cosine": max(cosine_values) if cosine_values else 0.0, + "mean_euclidean": sum(euclidean_values) / len(euclidean_values) if euclidean_values else 0.0, + "max_euclidean": max(euclidean_values) if euclidean_values else 0.0, + }, + } + log( + f"finished {model_key}: " + f"window_count={len(windows)}, " + f"mean_cos={all_distances[model_key]['summary']['mean_cosine']:.4f}, " + f"max_cos={all_distances[model_key]['summary']['max_cosine']:.4f}", + log_file, + ) + + distance_file.write_text(json.dumps(all_distances, ensure_ascii=False, indent=2), encoding="utf-8") + log(f"saved distance summary to {distance_file}", log_file) + log(f"saved run log to {log_file}", log_file) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/analyze_primary_decoding_surface.py b/scripts/analyze_primary_decoding_surface.py new file mode 100644 index 0000000..5ad1bc7 --- /dev/null +++ b/scripts/analyze_primary_decoding_surface.py @@ -0,0 +1,458 @@ +#!/usr/bin/env python3 +"""Analyze and plot the live temperature/top-p retrieval surface. + +The analysis uses one fixed strict cohort across all 26 planned run keys. It +reports (1) queries against a deterministic T=0 reference gallery, (2) +same-setting r1-to-r2 repeat stability, and (3) equal-cell mean and worst-cell +accuracy under joint bounds T <= tau and top-p <= pi. +""" + +from __future__ import annotations + +import csv +import importlib.util +import json +import re +import sys +from collections import defaultdict +from datetime import datetime +from pathlib import Path +from typing import Any + +import numpy as np +from PIL import Image, ImageDraw, ImageFont + + +ROOT = Path(__file__).resolve().parents[1] +OUT_DIR = ROOT / "docs" / "figures" / "primary_robustness_20260813" +GRID_SCRIPT = ROOT / "scripts" / "build_primary_run_grid.py" +TEMPERATURES = (0.0, 0.2, 0.3, 0.5, 0.7) +STOCHASTIC_TEMPERATURES = (0.2, 0.3, 0.5, 0.7) +TOP_PS = (0.8, 0.9, 1.0) +REPEATS = (1, 2) +COLORS = {0.8: "#2563eb", 0.9: "#d97706", 1.0: "#059669"} + + +def load_grid_module(): + spec = importlib.util.spec_from_file_location("primary_grid_surface", GRID_SCRIPT) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +def font(size: int, bold: bool = False) -> ImageFont.FreeTypeFont: + name = "DejaVuSans-Bold.ttf" if bold else "DejaVuSans.ttf" + return ImageFont.truetype(f"/usr/share/fonts/truetype/dejavu/{name}", size) + + +def status_rows(path: Path) -> dict[str, dict[str, Any]]: + final: dict[str, dict[str, Any]] = {} + if not path.exists(): + return final + for line in path.read_text(encoding="utf-8", errors="ignore").splitlines(): + try: + row = json.loads(line) + except json.JSONDecodeError: + continue + model = row.get("model_id") + if model: + final[str(model)] = row + return final + + +def run_key(manifest_path: Path, manifest: dict[str, Any]) -> tuple[float, float, int] | None: + try: + temperature = float(manifest["temperature"]) + top_p = float(manifest["top_p"]) + except (KeyError, TypeError, ValueError): + return None + suffix = str(manifest.get("output_suffix", "")) + match = re.search(r"_r([12])$", suffix) + repeat = int(match.group(1)) if match else None + if ( + repeat is None + and manifest_path.parent.name == "rand_chinese_0p3b_7b_temp07" + and temperature == 0.7 + and top_p == 0.9 + ): + repeat = 1 + if ( + repeat not in REPEATS + or temperature not in TEMPERATURES + or top_p not in TOP_PS + or (temperature == 0.0 and top_p != 1.0) + ): + return None + return temperature, top_p, int(repeat) + + +def candidate_priority(manifest: dict[str, Any], row: dict[str, Any]) -> tuple[int, str]: + seed = manifest.get("generation_seed") + seed_priority = 2 if seed in (42001, 42002) else 1 if seed is not None else 0 + return seed_priority, str(row.get("recorded_at", "")) + + +def collect_artifacts(cohort: set[str]) -> dict[tuple[str, float, float, int], Path]: + selected: dict[tuple[str, float, float, int], tuple[tuple[int, str], Path]] = {} + for manifest_path in sorted((ROOT / "out").rglob("manifest.json")): + try: + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + continue + key = run_key(manifest_path, manifest) + if key is None: + continue + temperature, top_p, repeat = key + for model, row in status_rows(manifest_path.parent / "status.jsonl").items(): + if model not in cohort or row.get("status") != "success": + continue + raw_path = row.get("output_path") + if not raw_path: + continue + dna_path = Path(str(raw_path)) + if not dna_path.is_absolute(): + dna_path = ROOT / dna_path + if not dna_path.exists(): + continue + artifact_key = model, temperature, top_p, repeat + candidate = candidate_priority(manifest, row), dna_path + if artifact_key not in selected or candidate[0] > selected[artifact_key][0]: + selected[artifact_key] = candidate + return {key: value[1] for key, value in selected.items()} + + +def planned_run_keys() -> list[tuple[float, float, int]]: + return [ + (temperature, top_p, repeat) + for temperature in TEMPERATURES + for top_p in TOP_PS + if temperature > 0 or top_p == 1.0 + for repeat in REPEATS + ] + + +def load_vector(path: Path) -> np.ndarray: + payload = json.loads(path.read_text(encoding="utf-8")) + vector = np.asarray(payload["signature"], dtype=np.float64) + if vector.ndim != 1 or not np.all(np.isfinite(vector)): + raise ValueError(f"Invalid DNA vector: {path}") + return vector + + +def matrix( + artifacts: dict[tuple[str, float, float, int], Path], + cohort: list[str], + temperature: float, + top_p: float, + repeat: int, +) -> np.ndarray: + return np.stack( + [load_vector(artifacts[(model, temperature, top_p, repeat)]) for model in cohort] + ) + + +def normalize(values: np.ndarray) -> np.ndarray: + return values / np.maximum(np.linalg.norm(values, axis=1, keepdims=True), 1e-12) + + +def retrieval_metrics(gallery: np.ndarray, query: np.ndarray) -> dict[str, float]: + scores = normalize(query) @ normalize(gallery).T + order = np.argsort(-scores, axis=1, kind="stable") + truth = np.arange(len(query)) + ranks = np.asarray([int(np.flatnonzero(order[i] == i)[0]) + 1 for i in truth]) + return { + "top1": float(np.mean(ranks == 1)), + "top3": float(np.mean(ranks <= 3)), + "top5": float(np.mean(ranks <= 5)), + "mrr": float(np.mean(1.0 / ranks)), + } + + +def setting_rows( + artifacts: dict[tuple[str, float, float, int], Path], cohort: list[str] +) -> list[dict[str, Any]]: + fixed_gallery = matrix(artifacts, cohort, 0.0, 1.0, 1) + rows: list[dict[str, Any]] = [] + for temperature in TEMPERATURES: + for top_p in TOP_PS: + if temperature == 0.0 and top_p != 1.0: + continue + query = matrix(artifacts, cohort, temperature, top_p, 2) + same_gallery = matrix(artifacts, cohort, temperature, top_p, 1) + for comparison, gallery in ( + ("fixed_t0_gallery", fixed_gallery), + ("same_setting_repeat", same_gallery), + ): + rows.append( + { + "comparison": comparison, + "temperature": temperature, + "top_p": top_p, + "model_count": len(cohort), + **retrieval_metrics(gallery, query), + } + ) + return rows + + +def bounded_rows( + artifacts: dict[tuple[str, float, float, int], Path], cohort: list[str] +) -> list[dict[str, Any]]: + rows: list[dict[str, Any]] = [] + for tau in STOCHASTIC_TEMPERATURES: + for p_bound in TOP_PS: + settings = [(0.0, 1.0)] + [ + (temperature, top_p) + for temperature in STOCHASTIC_TEMPERATURES + for top_p in TOP_PS + if temperature <= tau and top_p <= p_bound + ] + galleries = [matrix(artifacts, cohort, t, p, 1) for t, p in settings] + centroid = np.mean(np.stack(galleries), axis=0) + cell_rows = [] + for temperature, top_p in settings: + query = matrix(artifacts, cohort, temperature, top_p, 2) + cell_rows.append(retrieval_metrics(centroid, query)) + for metric in ("top1", "mrr"): + values = [row[metric] for row in cell_rows] + rows.append( + { + "temperature_bound": tau, + "top_p_bound": p_bound, + "metric": metric, + "cell_count": len(settings), + "model_count": len(cohort), + "equal_cell_mean": float(np.mean(values)), + "worst_setting": float(np.min(values)), + "best_setting": float(np.max(values)), + } + ) + return rows + + +def write_csv(path: Path, rows: list[dict[str, Any]]) -> None: + fields = list(rows[0]) + with path.open("w", encoding="utf-8", newline="") as handle: + writer = csv.DictWriter(handle, fieldnames=fields, lineterminator="\n") + writer.writeheader() + writer.writerows(rows) + + +def panel(draw: ImageDraw.ImageDraw, box: tuple[int, int, int, int], title: str) -> None: + draw.rounded_rectangle(box, radius=18, fill="#f8fafc", outline="#cbd5e1", width=3) + draw.text((box[0] + 25, box[1] + 20), title, font=font(27, True), fill="#0f172a") + + +def axes( + draw: ImageDraw.ImageDraw, + box: tuple[int, int, int, int], + xs: list[float], + xlabel: str, + ylabel: str, +) -> tuple[callable, callable]: + left, top, right, bottom = box + draw.line((left, bottom, right, bottom), fill="#475569", width=3) + draw.line((left, top, left, bottom), fill="#475569", width=3) + for value in np.linspace(0, 1, 6): + y = int(bottom - value * (bottom - top)) + draw.line((left, y, right, y), fill="#e2e8f0", width=2) + draw.text((left - 62, y - 11), f"{value:.1f}", font=font(18), fill="#475569") + for index, value in enumerate(xs): + x = int(left + index * (right - left) / max(1, len(xs) - 1)) + draw.line((x, bottom, x, bottom + 8), fill="#475569", width=2) + draw.text((x - 17, bottom + 12), f"{value:g}", font=font(18), fill="#475569") + draw.text(((left + right) // 2 - 85, bottom + 48), xlabel, font=font(20, True), fill="#334155") + draw.text((left - 78, top - 30), ylabel, font=font(18, True), fill="#334155") + return ( + lambda index: int(left + index * (right - left) / max(1, len(xs) - 1)), + lambda value: int(bottom - value * (bottom - top)), + ) + + +def draw_line_panel( + draw: ImageDraw.ImageDraw, + box: tuple[int, int, int, int], + rows: list[dict[str, Any]], +) -> None: + panel(draw, box, "A. Fixed top-p: exact Top-1 vs temperature") + plot_box = (box[0] + 95, box[1] + 105, box[2] - 35, box[3] - 85) + xmap, ymap = axes(draw, plot_box, list(TEMPERATURES), "Temperature", "Top-1") + lookup = { + (float(row["temperature"]), float(row["top_p"])): float(row["top1"]) + for row in rows + if row["comparison"] == "fixed_t0_gallery" + } + for top_p in TOP_PS: + points = [] + for index, temperature in enumerate(TEMPERATURES): + key = (temperature, top_p) + if key in lookup: + points.append((xmap(index), ymap(lookup[key]), lookup[key])) + if len(points) > 1: + draw.line([(x, y) for x, y, _ in points], fill=COLORS[top_p], width=6) + for x, y, value in points: + draw.ellipse((x - 7, y - 7, x + 7, y + 7), fill=COLORS[top_p], outline="white", width=2) + draw.text((x - 23, y - 31), f"{value:.2f}", font=font(16, True), fill=COLORS[top_p]) + legend_x = box[0] + 125 + for top_p in TOP_PS: + draw.line((legend_x, box[1] + 68, legend_x + 38, box[1] + 68), fill=COLORS[top_p], width=6) + draw.text((legend_x + 46, box[1] + 55), f"top-p={top_p:g}", font=font(18), fill="#334155") + legend_x += 180 + + +def heat_color(value: float) -> str: + low = (254, 226, 226) + high = (22, 163, 74) + ratio = min(1.0, max(0.0, value)) + rgb = [round(a + (b - a) * ratio) for a, b in zip(low, high)] + return "#" + "".join(f"{part:02x}" for part in rgb) + + +def draw_heatmap( + draw: ImageDraw.ImageDraw, + box: tuple[int, int, int, int], + title: str, + rows: list[dict[str, Any]], + comparison: str, +) -> None: + panel(draw, box, title) + lookup = { + (float(row["temperature"]), float(row["top_p"])): float(row["top1"]) + for row in rows + if row["comparison"] == comparison + } + left, top = box[0] + 105, box[1] + 115 + cell_w, cell_h = 145, 80 + for column, top_p in enumerate(TOP_PS): + draw.text((left + column * cell_w + 35, top - 42), f"p={top_p:g}", font=font(19, True), fill="#334155") + for row_index, temperature in enumerate(TEMPERATURES): + draw.text((box[0] + 28, top + row_index * cell_h + 25), f"T={temperature:g}", font=font(19, True), fill="#334155") + for column, top_p in enumerate(TOP_PS): + coords = ( + left + column * cell_w, + top + row_index * cell_h, + left + (column + 1) * cell_w - 10, + top + (row_index + 1) * cell_h - 10, + ) + key = (temperature, top_p) + if key not in lookup: + draw.rounded_rectangle(coords, radius=9, fill="#e2e8f0") + draw.text((coords[0] + 48, coords[1] + 20), "N/A", font=font(18, True), fill="#64748b") + continue + value = lookup[key] + draw.rounded_rectangle(coords, radius=9, fill=heat_color(value), outline="#94a3b8", width=2) + draw.text((coords[0] + 35, coords[1] + 13), f"{value:.3f}", font=font(21, True), fill="#0f172a") + draw.text((coords[0] + 43, coords[1] + 42), "Top-1", font=font(15), fill="#334155") + + +def plot_surface(rows: list[dict[str, Any]], cohort_size: int, generated_at: str, output: Path) -> None: + image = Image.new("RGB", (2000, 1120), "white") + draw = ImageDraw.Draw(image) + draw.text((65, 38), "Temperature and top-p relationship — live primary cohort", font=font(40, True), fill="#0f172a") + draw.text((65, 92), f"Fixed strict cohort N={cohort_size}; r1 gallery vs held-out r2 query; generated {generated_at}", font=font(22), fill="#475569") + draw_line_panel(draw, (45, 145, 860, 1050), rows) + draw_heatmap(draw, (895, 145, 1430, 1050), "B. Fixed T=0 gallery", rows, "fixed_t0_gallery") + draw_heatmap(draw, (1465, 145, 1990, 1050), "C. Same-setting repeat", rows, "same_setting_repeat") + output.parent.mkdir(parents=True, exist_ok=True) + image.save(output, optimize=True) + + +def plot_bounded(rows: list[dict[str, Any]], cohort_size: int, generated_at: str, output: Path) -> None: + image = Image.new("RGB", (1900, 1050), "white") + draw = ImageDraw.Draw(image) + draw.text((65, 38), "Joint decoding bounds — cosine centroid retrieval", font=font(40, True), fill="#0f172a") + draw.text((65, 92), f"T≤tau and top-p≤pi; deterministic control always included; strict cohort N={cohort_size}; {generated_at}", font=font(22), fill="#475569") + selected = [row for row in rows if row["metric"] == "top1"] + for panel_index, (value_key, title) in enumerate( + (("equal_cell_mean", "A. Equal-cell mean Top-1"), ("worst_setting", "B. Worst-setting Top-1")) + ): + x0 = 45 + panel_index * 735 + box = (x0, 145, x0 + 700, 980) + panel(draw, box, title) + plot_box = (box[0] + 95, box[1] + 110, box[2] - 35, box[3] - 85) + xmap, ymap = axes(draw, plot_box, list(STOCHASTIC_TEMPERATURES), "Temperature bound tau", "Top-1") + lookup = { + (float(row["temperature_bound"]), float(row["top_p_bound"])): float(row[value_key]) + for row in selected + } + legend_x = box[0] + 110 + for p_bound in TOP_PS: + points = [(xmap(i), ymap(lookup[(tau, p_bound)]), lookup[(tau, p_bound)]) for i, tau in enumerate(STOCHASTIC_TEMPERATURES)] + draw.line([(x, y) for x, y, _ in points], fill=COLORS[p_bound], width=6) + for x, y, _ in points: + draw.ellipse((x - 7, y - 7, x + 7, y + 7), fill=COLORS[p_bound], outline="white", width=2) + draw.line((legend_x, box[1] + 72, legend_x + 32, box[1] + 72), fill=COLORS[p_bound], width=6) + draw.text((legend_x + 38, box[1] + 59), f"p≤{p_bound:g}", font=font(17), fill="#334155") + legend_x += 145 + + box = (1510, 145, 1885, 980) + panel(draw, box, "C. Mean Top-1 grid") + lookup = { + (float(row["temperature_bound"]), float(row["top_p_bound"])): float(row["equal_cell_mean"]) + for row in selected + } + left, top, cell_w, cell_h = box[0] + 90, box[1] + 145, 90, 135 + for column, p_bound in enumerate(TOP_PS): + draw.text((left + column * cell_w + 8, top - 48), f"p≤{p_bound:g}", font=font(16, True), fill="#334155") + for row_index, tau in enumerate(STOCHASTIC_TEMPERATURES): + draw.text((box[0] + 18, top + row_index * cell_h + 43), f"T≤{tau:g}", font=font(17, True), fill="#334155") + for column, p_bound in enumerate(TOP_PS): + value = lookup[(tau, p_bound)] + coords = (left + column * cell_w, top + row_index * cell_h, left + (column + 1) * cell_w - 8, top + (row_index + 1) * cell_h - 10) + draw.rounded_rectangle(coords, radius=9, fill=heat_color(value), outline="#94a3b8", width=2) + draw.text((coords[0] + 10, coords[1] + 42), f"{value:.3f}", font=font(17, True), fill="#0f172a") + output.parent.mkdir(parents=True, exist_ok=True) + image.save(output, optimize=True) + + +def main() -> int: + grid = load_grid_module() + fixed_cohort = grid.load_cohort() + artifacts = collect_artifacts(fixed_cohort) + run_keys = planned_run_keys() + strict = sorted( + model + for model in fixed_cohort + if all((model, temperature, top_p, repeat) in artifacts for temperature, top_p, repeat in run_keys) + ) + if not strict: + raise SystemExit("No strict common cohort is available.") + generated_at = datetime.now().astimezone().isoformat(timespec="seconds") + per_setting = setting_rows(artifacts, strict) + bounded = bounded_rows(artifacts, strict) + OUT_DIR.mkdir(parents=True, exist_ok=True) + write_csv(OUT_DIR / "primary_decoding_surface.csv", per_setting) + write_csv(OUT_DIR / "primary_bounded_surface.csv", bounded) + payload = { + "generated_at": generated_at, + "strict_cohort_size": len(strict), + "strict_cohort": strict, + "artifact_count": len(artifacts), + "selection_priority": "explicit 42001/42002 success, then other explicit seed, then historical unknown seed", + "per_setting": per_setting, + "bounded": bounded, + } + (OUT_DIR / "primary_decoding_analysis.json").write_text( + json.dumps(payload, indent=2, ensure_ascii=False), encoding="utf-8" + ) + stamp = generated_at.replace("T", " ") + plot_surface(per_setting, len(strict), stamp, OUT_DIR / "temperature_top_p_relationship.png") + plot_bounded(bounded, len(strict), stamp, OUT_DIR / "bounded_decoding_relationship.png") + print(json.dumps({ + "generated_at": generated_at, + "artifact_count": len(artifacts), + "strict_cohort_size": len(strict), + "outputs": [ + str(OUT_DIR / "temperature_top_p_relationship.png"), + str(OUT_DIR / "bounded_decoding_relationship.png"), + str(OUT_DIR / "primary_decoding_surface.csv"), + str(OUT_DIR / "primary_bounded_surface.csv"), + ], + }, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/analyze_tree_aware_errors.py b/scripts/analyze_tree_aware_errors.py new file mode 100644 index 0000000..8fa2a9f --- /dev/null +++ b/scripts/analyze_tree_aware_errors.py @@ -0,0 +1,216 @@ +#!/usr/bin/env python3 +"""Analyze exact-classification errors and DNA distance by model-tree distance.""" + +from __future__ import annotations + +import argparse +import csv +import json +from collections import Counter, deque +from pathlib import Path + +import numpy as np + +from evaluate_exact_dna_classification import ( + cosine_scores, + load_records_for_suffix, + parse_run_spec, + select_common_cohort, +) + + +def load_components(path: Path | None) -> dict[str, int]: + lookup: dict[str, int] = {} + if path is None: + return lookup + for index, line in enumerate(path.read_text(encoding="utf-8").splitlines()): + if not line.strip(): + continue + row = json.loads(line) + for model in row.get("models", []): + lookup[str(model)] = index + return lookup + + +def load_graph(path: Path | None) -> dict[str, set[str]]: + graph: dict[str, set[str]] = {} + if path is None or not path.exists(): + return graph + for line in path.read_text(encoding="utf-8").splitlines(): + if not line.strip(): + continue + row = json.loads(line) + left = str(row.get("source", "")).strip() + right = str(row.get("target", "")).strip() + if left and right: + graph.setdefault(left, set()).add(right) + graph.setdefault(right, set()).add(left) + return graph + + +def shortest_paths(graph: dict[str, set[str]], models: list[str]) -> dict[tuple[str, str], int]: + allowed = set(models) + distances: dict[tuple[str, str], int] = {} + for source in models: + queue = deque([(source, 0)]) + seen = {source} + while queue: + node, distance = queue.popleft() + if node in allowed: + distances[(source, node)] = distance + for neighbor in graph.get(node, set()): + if neighbor not in seen: + seen.add(neighbor) + queue.append((neighbor, distance + 1)) + return distances + + +def relation_bucket( + left: str, + right: str, + paths: dict[tuple[str, str], int], + components: dict[str, int], +) -> str: + if left == right: + return "exact" + distance = paths.get((left, right)) + if distance == 1: + return "direct_1hop" + if distance == 2: + return "indirect_2hop" + if distance is not None: + return "indirect_3plus" + if left in components and components.get(left) == components.get(right): + return "same_component_unknown_hops" + return "disconnected" + + +def analyze_run( + labels: list[str], + reference_matrix: np.ndarray, + query_matrix: np.ndarray, + paths: dict[tuple[str, str], int], + components: dict[str, int], +) -> tuple[dict[str, object], list[dict[str, object]], list[dict[str, object]]]: + scores = cosine_scores(query_matrix, reference_matrix) + distances = 1.0 - scores + error_counts: Counter[str] = Counter() + predictions: list[dict[str, object]] = [] + pair_values: dict[str, list[float]] = {} + for row_index, truth in enumerate(labels): + prediction = labels[int(np.argmax(scores[row_index]))] + bucket = relation_bucket(truth, prediction, paths, components) + error_counts[bucket] += 1 + predictions.append({"model": truth, "predicted_model": prediction, "relation_bucket": bucket}) + for column_index, candidate in enumerate(labels): + pair_bucket = relation_bucket(truth, candidate, paths, components) + pair_values.setdefault(pair_bucket, []).append(float(distances[row_index, column_index])) + + count = len(labels) + exact = error_counts["exact"] + direct = error_counts["direct_1hop"] + two_hop = error_counts["indirect_2hop"] + component = error_counts["indirect_3plus"] + error_counts["same_component_unknown_hops"] + summary: dict[str, object] = { + "model_count": count, + "exact_top1": exact / count, + "within_1hop_top1": (exact + direct) / count, + "within_2hop_top1": (exact + direct + two_hop) / count, + "same_component_top1": (exact + direct + two_hop + component) / count, + **{f"top1_error_{key}": value for key, value in sorted(error_counts.items())}, + } + distance_rows = [ + { + "relation_bucket": bucket, + "ordered_pair_count": len(values), + "mean_cosine_distance": float(np.mean(values)), + "median_cosine_distance": float(np.median(values)), + "q25_cosine_distance": float(np.quantile(values, 0.25)), + "q75_cosine_distance": float(np.quantile(values, 0.75)), + } + for bucket, values in sorted(pair_values.items()) + ] + return summary, predictions, distance_rows + + +def write_csv(path: Path, rows: list[dict[str, object]]) -> None: + if not rows: + path.write_text("", encoding="utf-8") + return + fields: list[str] = [] + for row in rows: + fields.extend(key for key in row if key not in fields) + with path.open("w", encoding="utf-8", newline="") as handle: + writer = csv.DictWriter(handle, fieldnames=fields) + writer.writeheader() + writer.writerows(rows) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description="Tree-distance and hierarchical error analysis.") + parser.add_argument("--dataset-dir", type=Path, default=Path("out/rand_chinese")) + parser.add_argument("--reference-suffix", required=True) + parser.add_argument("--run", action="append", default=[], help="name:suffix:temperature:top_p:repeat") + parser.add_argument("--components-file", type=Path, default=Path("out/hf_model_tree/model_relations.jsonl")) + parser.add_argument("--edges-file", type=Path, default=Path("out/hf_model_tree/model_direct_edges.jsonl")) + parser.add_argument("--minimum-models", type=int, default=100) + parser.add_argument("--output-dir", type=Path, default=Path("out/tree_aware_errors")) + return parser.parse_args() + + +def main() -> int: + args = parse_args() + runs = [parse_run_spec(value) for value in args.run] + if not runs: + raise SystemExit("At least one --run is required.") + reference = load_records_for_suffix(args.dataset_dir, args.reference_suffix) + records = {run.name: load_records_for_suffix(args.dataset_dir, run.suffix) for run in runs} + all_records = {"reference": reference, **records} + labels = select_common_cohort(all_records, all_records) + if len(labels) < args.minimum_models: + raise SystemExit( + f"Only {len(labels)} common models are available; minimum is {args.minimum_models}." + ) + components = load_components(args.components_file) + graph = load_graph(args.edges_file) + paths = shortest_paths(graph, labels) + reference_matrix = np.stack([reference[label].vector for label in labels]) + + summaries: list[dict[str, object]] = [] + predictions: list[dict[str, object]] = [] + distance_rows: list[dict[str, object]] = [] + for run in runs: + query = np.stack([records[run.name][label].vector for label in labels]) + summary, run_predictions, run_distances = analyze_run( + labels, reference_matrix, query, paths, components + ) + identity = { + "run": run.name, + "temperature": run.temperature, + "top_p": run.top_p, + "repeat": run.repeat, + } + summaries.append({**identity, **summary}) + predictions.extend({**identity, **row} for row in run_predictions) + distance_rows.extend({**identity, **row} for row in run_distances) + + args.output_dir.mkdir(parents=True, exist_ok=True) + write_csv(args.output_dir / "tree_aware_summary.csv", summaries) + write_csv(args.output_dir / "tree_aware_predictions.csv", predictions) + write_csv(args.output_dir / "tree_distance_distributions.csv", distance_rows) + payload = { + "cohort_size": len(labels), + "direct_edges_available": bool(graph), + "warning": None if graph else "Direct-edge export unavailable; same-component hop distances are unknown.", + "summaries": summaries, + "distance_distributions": distance_rows, + } + (args.output_dir / "tree_aware_summary.json").write_text( + json.dumps(payload, indent=2, ensure_ascii=False), encoding="utf-8" + ) + print(json.dumps({"output_dir": str(args.output_dir), "cohort_size": len(labels)}, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/build_distdna_pilot_seed_queue.py b/scripts/build_distdna_pilot_seed_queue.py new file mode 100755 index 0000000..a893cd8 --- /dev/null +++ b/scripts/build_distdna_pilot_seed_queue.py @@ -0,0 +1,141 @@ +#!/usr/bin/env python3 +"""Build the missing-model queue for one explicit DistDNA pilot seed. + +An artifact is reusable only when its run manifest explicitly records the +requested generation seed and decoding setting, its latest per-model status is +successful, and both the DNA and summary files still exist. Historical runs +whose seed is absent from the manifest are deliberately ignored. +""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path +from typing import Any + + +ROOT = Path(__file__).resolve().parents[1] + + +def read_json(path: Path) -> Any: + return json.loads(path.read_text(encoding="utf-8")) + + +def resolved_artifact(path_value: object) -> Path | None: + if not isinstance(path_value, str) or not path_value.strip(): + return None + path = Path(path_value) + return path if path.is_absolute() else ROOT / path + + +def explicit_successes( + output_root: Path, + model_ids: set[str], + seed: int, + temperature: float, + top_p: float, +) -> dict[str, dict[str, str]]: + successes: dict[str, dict[str, str]] = {} + for manifest_path in output_root.glob("**/manifest.json"): + try: + manifest = read_json(manifest_path) + if int(manifest.get("generation_seed")) != seed: + continue + if float(manifest.get("temperature")) != temperature: + continue + if float(manifest.get("top_p")) != top_p: + continue + except (OSError, ValueError, TypeError, json.JSONDecodeError): + continue + + status_path = manifest_path.with_name("status.jsonl") + if not status_path.is_file(): + continue + + latest: dict[str, dict[str, Any]] = {} + try: + for line in status_path.read_text(encoding="utf-8").splitlines(): + if not line.strip(): + continue + record = json.loads(line) + model_id = record.get("model_id") + if model_id in model_ids: + latest[model_id] = record + except (OSError, json.JSONDecodeError): + continue + + for model_id, record in latest.items(): + if record.get("status") != "success": + continue + dna_path = resolved_artifact(record.get("output_path")) + summary_path = resolved_artifact(record.get("summary_path")) + if not dna_path or not summary_path: + continue + if not dna_path.is_file() or dna_path.stat().st_size == 0: + continue + if not summary_path.is_file() or summary_path.stat().st_size == 0: + continue + successes[model_id] = { + "manifest": str(manifest_path.relative_to(ROOT)), + "status": str(status_path.relative_to(ROOT)), + "dna": str(dna_path.relative_to(ROOT)), + "summary": str(summary_path.relative_to(ROOT)), + } + return successes + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--cohort", type=Path, required=True) + parser.add_argument("--seed", type=int, required=True) + parser.add_argument("--temperature", type=float, default=0.7) + parser.add_argument("--top-p", type=float, default=1.0) + parser.add_argument("--search-root", type=Path, default=ROOT / "out") + parser.add_argument("--missing-jsonl", type=Path, required=True) + parser.add_argument("--inventory-json", type=Path, required=True) + args = parser.parse_args() + + cohort_path = args.cohort if args.cohort.is_absolute() else ROOT / args.cohort + rows = [json.loads(line) for line in cohort_path.read_text(encoding="utf-8").splitlines() if line.strip()] + model_ids = {str(row["model_id"]) for row in rows} + if len(rows) != 10 or len(model_ids) != 10: + raise SystemExit(f"pilot cohort must contain exactly 10 unique models: {cohort_path}") + + search_root = args.search_root if args.search_root.is_absolute() else ROOT / args.search_root + successes = explicit_successes(search_root, model_ids, args.seed, args.temperature, args.top_p) + missing = [row for row in rows if row["model_id"] not in successes] + + args.missing_jsonl.parent.mkdir(parents=True, exist_ok=True) + args.inventory_json.parent.mkdir(parents=True, exist_ok=True) + args.missing_jsonl.write_text( + "".join(json.dumps(row, ensure_ascii=False) + "\n" for row in missing), + encoding="utf-8", + ) + args.inventory_json.write_text( + json.dumps( + { + "temperature": args.temperature, + "top_p": args.top_p, + "generation_seed": args.seed, + "cohort_size": len(rows), + "reusable_explicit_success_count": len(successes), + "missing_count": len(missing), + "reusable_explicit_successes": successes, + "missing_models": [row["model_id"] for row in missing], + }, + ensure_ascii=False, + indent=2, + ) + + "\n", + encoding="utf-8", + ) + print( + f"seed={args.seed} reusable={len(successes)}/10 " + f"missing={len(missing)}/10 queue={args.missing_jsonl}" + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/build_experiment_cohort.py b/scripts/build_experiment_cohort.py new file mode 100644 index 0000000..460053b --- /dev/null +++ b/scripts/build_experiment_cohort.py @@ -0,0 +1,205 @@ +#!/usr/bin/env python3 +"""Build a larger, reproducible model cohort from existing reference artifacts. + +Selection never uses evaluation accuracy. Eligible models must have a valid +unsuffixed reference DNA and recorded parameter count, then deterministic +round-robin sampling balances architecture and logarithmic size strata. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +import re +from collections import defaultdict +from pathlib import Path + +import numpy as np + + +DEFAULT_RELIABILITY_REPLACEMENTS = { + "tiiuae/Falcon3-3B-Instruct": "princeton-nlp/Sheared-LLaMA-2.7B", + "sarvamai/sarvam-30b": "tiiuae/Falcon3-10B-Base", + "allenai/Olmo-3-7B-Instruct-DPO": "allenai/Llama-3.1-Tulu-3-8B", +} + + +def safe_model_name(model_name: str) -> str: + return re.sub(r"[^A-Za-z0-9._-]+", "_", model_name.strip("/")) + + +def load_catalog(path: Path) -> dict[str, dict[str, object]]: + output: dict[str, dict[str, object]] = {} + for line in path.read_text(encoding="utf-8").splitlines(): + if not line.strip(): + continue + row = json.loads(line) + model = str(row.get("model_id", "")).strip() + if model: + output[model] = row + return output + + +def load_eligible_references( + dataset_dir: Path, minimum_billions: float, maximum_billions: float +) -> tuple[dict[str, dict[str, object]], dict[str, str]]: + eligible: dict[str, dict[str, object]] = {} + excluded: dict[str, str] = {} + for path in sorted(dataset_dir.glob("*/*_dna.json")): + try: + payload = json.loads(path.read_text(encoding="utf-8")) + metadata = payload.get("metadata", {}) + model = str(metadata.get("model_name", "")).strip() + vector = np.asarray(payload.get("signature", []), dtype=np.float32) + except (OSError, ValueError, TypeError, json.JSONDecodeError): + continue + if not model or path.parent.name != safe_model_name(model): + continue + size = metadata.get("model_metadata", {}).get("size", {}).get("parameter_count_billions") + if not isinstance(size, (int, float)): + excluded[model] = "missing_parameter_count" + continue + if not minimum_billions <= float(size) <= maximum_billions: + excluded[model] = "outside_size_range" + continue + if vector.ndim != 1 or vector.size == 0 or not np.all(np.isfinite(vector)): + excluded[model] = "invalid_reference_dna" + continue + architecture = str( + metadata.get("model_metadata", {}).get("architecture", {}).get("model_type") or "unknown" + ) + eligible[model] = { + "parameter_count_billions": float(size), + "architecture": architecture, + "reference_path": str(path), + } + return eligible, excluded + + +def size_bucket(size_billions: float) -> str: + lower = 2 ** math.floor(math.log2(max(size_billions, 0.125))) + upper = lower * 2 + return f"{lower:g}-{upper:g}B" + + +def select_stratified( + eligible: dict[str, dict[str, object]], target: int, seed: int +) -> list[str]: + if target > len(eligible): + raise ValueError(f"Requested {target} models but only {len(eligible)} are eligible.") + strata: dict[tuple[str, str], list[str]] = defaultdict(list) + for model, metadata in eligible.items(): + key = (str(metadata["architecture"]), size_bucket(float(metadata["parameter_count_billions"]))) + strata[key].append(model) + rng = np.random.default_rng(seed) + for values in strata.values(): + rng.shuffle(values) + ordered_keys = sorted(strata, key=lambda key: (key[0], key[1])) + selected: list[str] = [] + while len(selected) < target: + made_progress = False + for key in ordered_keys: + if strata[key] and len(selected) < target: + selected.append(strata[key].pop()) + made_progress = True + if not made_progress: + break + return selected + + +def apply_reliability_replacements( + selected: list[str], + eligible: dict[str, dict[str, object]], + replacements: dict[str, str], +) -> tuple[list[str], list[dict[str, object]]]: + """Replace known generation failures without using evaluation accuracy.""" + output = list(selected) + audit_rows: list[dict[str, object]] = [] + for removed, replacement in replacements.items(): + if removed not in output: + continue + if replacement not in eligible: + raise ValueError(f"Reliability replacement is not eligible: {replacement}") + if replacement in output: + raise ValueError(f"Reliability replacement is already selected: {replacement}") + index = output.index(removed) + output[index] = replacement + audit_rows.append( + { + "removed_model": removed, + "removed_size_billions": eligible[removed]["parameter_count_billions"], + "replacement_model": replacement, + "replacement_size_billions": eligible[replacement]["parameter_count_billions"], + "reason": "repeated_failed_empty_response", + "validation": "cached baseline contains 100 non-empty responses", + } + ) + return output, audit_rows + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description="Create a stratified DNA experiment cohort.") + parser.add_argument("--dataset-dir", type=Path, default=Path("out/rand_chinese")) + parser.add_argument("--catalog", type=Path, default=Path("configs/huggingface_llm_list.jsonl")) + parser.add_argument("--target", type=int, default=300) + parser.add_argument("--minimum-billions", type=float, default=0.3) + parser.add_argument("--maximum-billions", type=float, default=7.5) + parser.add_argument("--seed", type=int, default=42) + parser.add_argument("--output", type=Path, default=Path("configs/rand_chinese_stratified_300.jsonl")) + parser.add_argument("--audit-output", type=Path, default=Path("configs/rand_chinese_stratified_300_audit.json")) + return parser.parse_args() + + +def main() -> int: + args = parse_args() + catalog = load_catalog(args.catalog) + eligible, excluded = load_eligible_references( + args.dataset_dir, args.minimum_billions, args.maximum_billions + ) + try: + selected = select_stratified(eligible, args.target, args.seed) + selected, reliability_replacements = apply_reliability_replacements( + selected, eligible, DEFAULT_RELIABILITY_REPLACEMENTS + ) + except ValueError as exc: + raise SystemExit(str(exc)) from exc + args.output.parent.mkdir(parents=True, exist_ok=True) + with args.output.open("w", encoding="utf-8") as handle: + for model in selected: + source = catalog.get(model, {}) + row = { + "model_id": model, + "provider": source.get("provider", "huggingface"), + "task": source.get("task", "text-generation"), + "downloads": int(source.get("downloads", 0) or 0), + "parameter_count_billions": eligible[model]["parameter_count_billions"], + "architecture": eligible[model]["architecture"], + } + handle.write(json.dumps(row, ensure_ascii=False) + "\n") + digest = hashlib.sha256(args.output.read_bytes()).hexdigest() + audit = { + "selection_uses_performance": False, + "selection_method": "seeded round-robin over architecture x logarithmic parameter-size strata, followed by documented reliability replacements that do not use evaluation accuracy", + "seed": args.seed, + "target": args.target, + "eligible_count": len(eligible), + "selected_count": len(selected), + "minimum_billions": args.minimum_billions, + "maximum_billions": args.maximum_billions, + "output_sha256": digest, + "post_selection_replacements": reliability_replacements, + "selected_models": selected, + "selected_metadata": {model: eligible[model] for model in selected}, + "exclusion_reason_counts": { + reason: sum(value == reason for value in excluded.values()) for reason in sorted(set(excluded.values())) + }, + } + args.audit_output.write_text(json.dumps(audit, indent=2, ensure_ascii=False), encoding="utf-8") + print(json.dumps({"output": str(args.output), "audit": str(args.audit_output), **{key: audit[key] for key in ["eligible_count", "selected_count", "output_sha256"]}}, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/build_experiment_inventory.py b/scripts/build_experiment_inventory.py new file mode 100644 index 0000000..d205940 --- /dev/null +++ b/scripts/build_experiment_inventory.py @@ -0,0 +1,384 @@ +#!/usr/bin/env python3 +"""Build a reproducible inventory of experiment manifests and final statuses.""" + +from __future__ import annotations + +import csv +import json +from collections import Counter +from pathlib import Path +from typing import Any + + +ROOT = Path(__file__).resolve().parents[1] +REPORT_DIR = ROOT / "reports" +MANIFEST_NAME = "manifest.json" +STATUS_NAME = "status.jsonl" + +PARAMETER_FIELDS = [ + "dataset", + "max_samples", + "providers", + "max_model_size_b", + "resume_mode", + "temperature", + "top_p", + "do_sample", + "generation_seed", + "ignore_response_cache", + "output_suffix", + "model_timeout_seconds", + "max_retries", + "oom_retries", + "retry_delay_seconds", + "gpu_memory_headroom_gb", + "gpu_memory_per_billion_gb", +] + + +def relative(path: Path) -> str: + return path.resolve().relative_to(ROOT).as_posix() + + +def load_json(path: Path) -> dict[str, Any]: + with path.open(encoding="utf-8") as handle: + return json.load(handle) + + +def load_statuses(path: Path) -> list[dict[str, Any]]: + records: list[dict[str, Any]] = [] + if not path.exists(): + return records + with path.open(encoding="utf-8") as handle: + for line in handle: + line = line.strip() + if not line: + continue + try: + record = json.loads(line) + except json.JSONDecodeError: + continue + if isinstance(record, dict) and record.get("model_id"): + records.append(record) + return records + + +def task_identity(task: dict[str, Any]) -> str: + provider = task.get("provider") + model_id = task.get("model_id", "") + return f"{provider}:{model_id}" if provider else str(model_id) + + +def model_label(identity: str) -> str: + return identity.split(":", 1)[1] if ":" in identity else identity + + +def final_statuses( + records: list[dict[str, Any]], planned: set[str] +) -> dict[str, dict[str, Any]]: + """Keep the last record per model, matching provider-less legacy statuses.""" + by_model_id = {model_label(identity): identity for identity in planned} + final: dict[str, dict[str, Any]] = {} + for record in records: + identity = task_identity(record) + if identity not in planned: + identity = by_model_id.get(str(record.get("model_id")), identity) + final[identity] = record + return final + + +def status_bucket(status: str) -> str: + if status == "success": + return "success" + if status.startswith("failed"): + return "fail" + if status.startswith("skipped"): + return "skipped" + return "other" + + +def compact_parameters(manifest: dict[str, Any]) -> str: + parts: list[str] = [] + aliases = { + "max_samples": "n", + "providers": "provider", + "max_model_size_b": "max_B", + "resume_mode": "resume", + "temperature": "temp", + "generation_seed": "seed", + "ignore_response_cache": "no_cache", + "model_timeout_seconds": "timeout_s", + "retry_delay_seconds": "retry_delay_s", + "gpu_memory_headroom_gb": "gpu_headroom_GB", + "gpu_memory_per_billion_gb": "gpu_GB_per_B", + } + for field in PARAMETER_FIELDS: + if field in manifest: + value = manifest[field] + if isinstance(value, bool): + value = str(value).lower() + parts.append(f"{aliases.get(field, field)}={value}") + return "; ".join(parts) + + +def diff_summary(added: list[str], removed: list[str]) -> str: + if not added and not removed: + return "BASE (same 100)" + count = len(added) + len(removed) + if count <= 4: + pieces = [f"+{model_label(item)}" for item in added] + pieces += [f"−{model_label(item)}" for item in removed] + return "; ".join(pieces) + return f"+{len(added)} / −{len(removed)} (see model diffs CSV)" + + +def write_csv(path: Path, rows: list[dict[str, Any]], fields: list[str]) -> None: + with path.open("w", newline="", encoding="utf-8") as handle: + writer = csv.DictWriter(handle, fieldnames=fields, extrasaction="ignore") + writer.writeheader() + writer.writerows(rows) + + +def main() -> None: + manifests: list[tuple[Path, dict[str, Any]]] = [] + for path in sorted(ROOT.rglob(MANIFEST_NAME)): + if ".git" in path.parts: + continue + try: + manifests.append((path, load_json(path))) + except (OSError, json.JSONDecodeError): + continue + + model_sets = [ + frozenset(task_identity(task) for task in data.get("tasks", [])) + for _, data in manifests + ] + frequencies = Counter(model_sets) + baseline = set(frequencies.most_common(1)[0][0]) + baseline_frequency = frequencies.most_common(1)[0][1] + + rows: list[dict[str, Any]] = [] + diff_rows: list[dict[str, Any]] = [] + manifest_dirs: set[Path] = set() + + for manifest_path, manifest in manifests: + run_dir = manifest_path.parent + manifest_dirs.add(run_dir.resolve()) + planned = { + task_identity(task) + for task in manifest.get("tasks", []) + if task.get("model_id") + } + records = load_statuses(run_dir / STATUS_NAME) + final = final_statuses(records, planned) + buckets = Counter( + status_bucket(str(record.get("status", ""))) for record in final.values() + ) + observed_planned = len(set(final) & planned) + observed_total = len(final) + extra_observed = len(set(final) - planned) + unobserved = max(0, len(planned) - observed_planned) + added = sorted(planned - baseline) + removed = sorted(baseline - planned) + + known_fields = set(PARAMETER_FIELDS) | { + "created_at", + "tasks", + "task_count", + } + extra_parameters = { + key: value for key, value in manifest.items() if key not in known_fields + } + run_name = relative(run_dir) + row = { + "created_at": manifest.get("created_at", ""), + "run": run_name, + "models": len(planned), + "observed": observed_total, + "extra_observed": extra_observed, + "success": buckets["success"], + "fail": buckets["fail"], + "skipped": buckets["skipped"], + "other_final_status": buckets["other"], + "unobserved": unobserved, + "model_diff": diff_summary(added, removed), + "parameters": compact_parameters(manifest), + "manifest": relative(manifest_path), + "status_log": ( + relative(run_dir / STATUS_NAME) + if (run_dir / STATUS_NAME).exists() + else "" + ), + "extra_parameters_json": json.dumps( + extra_parameters, ensure_ascii=False, sort_keys=True + ), + } + for field in PARAMETER_FIELDS: + row[field] = manifest.get(field, "") + rows.append(row) + diff_rows.append( + { + "run": run_name, + "is_baseline": not added and not removed, + "model_count": len(planned), + "added_count": len(added), + "removed_count": len(removed), + "added_models": ";".join(added), + "removed_models": ";".join(removed), + } + ) + + # Include legacy status-only runs so "all runs" is not limited to manifests. + for status_path in sorted(ROOT.rglob(STATUS_NAME)): + if ".git" in status_path.parts or status_path.parent.resolve() in manifest_dirs: + continue + records = load_statuses(status_path) + final = final_statuses(records, set()) + models = set(final) + buckets = Counter( + status_bucket(str(record.get("status", ""))) for record in final.values() + ) + summary_path = status_path.parent / "summary.json" + summary = load_json(summary_path) if summary_path.exists() else {} + added = sorted(models - baseline) + removed = sorted(baseline - models) + row = { + "created_at": summary.get("timestamp", ""), + "run": relative(status_path.parent), + "models": len(models), + "observed": len(models), + "extra_observed": 0, + "success": buckets["success"], + "fail": buckets["fail"], + "skipped": buckets["skipped"], + "other_final_status": buckets["other"], + "unobserved": 0, + "model_diff": diff_summary(added, removed), + "parameters": "; ".join( + f"{key}={summary[key]}" + for key in ("dataset", "samples", "gpus") + if key in summary + ), + "manifest": "", + "status_log": relative(status_path), + "extra_parameters_json": json.dumps( + summary, ensure_ascii=False, sort_keys=True + ), + } + for field in PARAMETER_FIELDS: + row[field] = "" + rows.append(row) + diff_rows.append( + { + "run": row["run"], + "is_baseline": False, + "model_count": len(models), + "added_count": len(added), + "removed_count": len(removed), + "added_models": ";".join(added), + "removed_models": ";".join(removed), + } + ) + + rows.sort(key=lambda row: (str(row["created_at"]), str(row["run"]))) + diff_rows.sort(key=lambda row: str(row["run"])) + REPORT_DIR.mkdir(exist_ok=True) + + csv_fields = [ + "created_at", + "run", + *PARAMETER_FIELDS, + "models", + "observed", + "extra_observed", + "success", + "fail", + "skipped", + "other_final_status", + "unobserved", + "model_diff", + "parameters", + "manifest", + "status_log", + "extra_parameters_json", + ] + write_csv(REPORT_DIR / "experiment_inventory.csv", rows, csv_fields) + write_csv( + REPORT_DIR / "experiment_model_diffs.csv", + diff_rows, + [ + "run", + "is_baseline", + "model_count", + "added_count", + "removed_count", + "added_models", + "removed_models", + ], + ) + with (REPORT_DIR / "experiment_baseline_models.txt").open( + "w", encoding="utf-8" + ) as handle: + handle.write("\n".join(sorted(baseline)) + "\n") + + markdown = [ + "# Experiment inventory", + "", + ( + f"Generated from {len(manifests)} manifests plus " + f"{len(rows) - len(manifests)} status-only run(s). " + f"The baseline is the modal exact model cohort: {len(baseline)} models, " + f"used by {baseline_frequency} manifests." + ), + "", + ( + "`success`, `fail`, `skipped`, and `other` count each model's final " + "status record (retry records are collapsed to the last record). " + "`unobserved` means a model was planned in the manifest but has no " + "status record; `extra` means a status model was not in the manifest. " + "A missing status log therefore does not become a failure." + ), + "", + "| Date | Run | Parameters | Planned models | Status models | Success | Fail | Skipped | Other | Unobserved | Extra | Model diff vs baseline |", + "|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---|", + ] + for row in rows: + created = str(row["created_at"])[:19].replace("T", " ") + markdown.append( + "| {date} | `{run}` | {params} | {models} | {observed} | {success} | " + "{fail} | {skipped} | {other} | {unobserved} | {extra} | {diff} |".format( + date=created or "unknown", + run=row["run"], + params=str(row["parameters"]).replace("|", "\\|"), + models=row["models"], + observed=row["observed"], + success=row["success"], + fail=row["fail"], + skipped=row["skipped"], + other=row["other_final_status"], + unobserved=row["unobserved"], + extra=row["extra_observed"], + diff=str(row["model_diff"]).replace("|", "\\|"), + ) + ) + markdown += [ + "", + "Companion files:", + "", + "- `experiment_inventory.csv`: all manifest parameter columns and source paths.", + "- `experiment_model_diffs.csv`: full added/removed model lists for every run.", + "- `experiment_baseline_models.txt`: the exact 100-model baseline cohort.", + "", + ] + (REPORT_DIR / "experiment_inventory.md").write_text( + "\n".join(markdown), encoding="utf-8" + ) + + print( + f"Wrote {len(rows)} runs; baseline={len(baseline)} models " + f"across {baseline_frequency} manifests" + ) + + +if __name__ == "__main__": + main() diff --git a/scripts/build_primary_recovery_configs.py b/scripts/build_primary_recovery_configs.py new file mode 100755 index 0000000..f15fe6d --- /dev/null +++ b/scripts/build_primary_recovery_configs.py @@ -0,0 +1,148 @@ +#!/usr/bin/env python3 +"""Validate a failed-slot plan and build one Hugging Face list per grid cell.""" + +from __future__ import annotations + +import argparse +import json +from collections import Counter, defaultdict +from pathlib import Path +from typing import Any + + +ALLOWED_FAILURES = {"failed_empty_response", "failed_missing_file"} + + +def cell_id(row: dict[str, Any]) -> str: + temperature = int(round(float(row["temperature"]) * 10)) + top_p = int(round(float(row["top_p"]) * 10)) + return f"t{temperature:02d}_p{top_p:02d}_r{int(row['repeat'])}" + + +def load_plan(path: Path) -> list[dict[str, Any]]: + rows: list[dict[str, Any]] = [] + with path.open(encoding="utf-8") as handle: + for line_number, line in enumerate(handle, start=1): + line = line.strip() + if not line: + continue + try: + row = json.loads(line) + except json.JSONDecodeError as exc: + raise ValueError(f"Invalid JSON on {path}:{line_number}") from exc + required = { + "model_id", + "provider", + "temperature", + "top_p", + "repeat", + "generation_seed", + "prior_status", + } + missing = required - set(row) + if missing: + raise ValueError( + f"Missing fields on {path}:{line_number}: {sorted(missing)}" + ) + if row["provider"] != "huggingface": + raise ValueError(f"Unsupported provider on line {line_number}") + if row["prior_status"] not in ALLOWED_FAILURES: + raise ValueError(f"Unexpected prior status on line {line_number}") + rows.append(row) + return rows + + +def validate(rows: list[dict[str, Any]], expected_slots: int) -> None: + if len(rows) != expected_slots: + raise ValueError(f"Expected {expected_slots} failed slots, found {len(rows)}") + + keys = [ + ( + str(row["model_id"]), + float(row["temperature"]), + float(row["top_p"]), + int(row["repeat"]), + ) + for row in rows + ] + if len(set(keys)) != len(keys): + duplicates = [key for key, count in Counter(keys).items() if count > 1] + raise ValueError(f"Duplicate failed-slot keys: {duplicates}") + + for row in rows: + temperature = float(row["temperature"]) + repeat = int(row["repeat"]) + expected_seed = 42 if temperature == 0.5 else 42000 + repeat + if int(row["generation_seed"]) != expected_seed: + raise ValueError( + f"Wrong generation seed for {row['model_id']} {cell_id(row)}: " + f"expected {expected_seed}, found {row['generation_seed']}" + ) + + +def atomic_write(path: Path, content: str) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + temporary = path.with_suffix(path.suffix + ".tmp") + temporary.write_text(content, encoding="utf-8") + temporary.replace(path) + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--plan", type=Path, required=True) + parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument("--expected-slots", type=int, default=72) + args = parser.parse_args() + + rows = load_plan(args.plan) + validate(rows, args.expected_slots) + + grouped: dict[tuple[float, float, int, int], list[dict[str, Any]]] = defaultdict(list) + for row in rows: + key = ( + float(row["temperature"]), + float(row["top_p"]), + int(row["repeat"]), + int(row["generation_seed"]), + ) + grouped[key].append(row) + + index_lines = ["cell_id\ttemperature\ttop_p\trepeat\tgeneration_seed\ttask_count\tconfig"] + for key in sorted(grouped): + temperature, top_p, repeat, generation_seed = key + cell_rows = sorted(grouped[key], key=lambda row: str(row["model_id"])) + identifier = cell_id(cell_rows[0]) + config_path = args.output_dir / f"{identifier}.jsonl" + content = "".join( + json.dumps( + {"model_id": row["model_id"], "provider": "huggingface"}, + ensure_ascii=False, + ) + + "\n" + for row in cell_rows + ) + atomic_write(config_path, content) + index_lines.append( + "\t".join( + [ + identifier, + str(temperature), + str(top_p), + str(repeat), + str(generation_seed), + str(len(cell_rows)), + str(config_path), + ] + ) + ) + + atomic_write(args.output_dir / "cells.tsv", "\n".join(index_lines) + "\n") + failures = Counter(str(row["prior_status"]) for row in rows) + print( + f"Prepared {len(rows)} failed slots across {len(grouped)} cells: " + + ", ".join(f"{status}={count}" for status, count in sorted(failures.items())) + ) + + +if __name__ == "__main__": + main() diff --git a/scripts/build_primary_robustness_100_shards.py b/scripts/build_primary_robustness_100_shards.py new file mode 100644 index 0000000..b797aac --- /dev/null +++ b/scripts/build_primary_robustness_100_shards.py @@ -0,0 +1,116 @@ +#!/usr/bin/env python3 +"""Split the fixed 100-model primary cohort across RTX 3090 and A100 workers.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +from pathlib import Path + + +def load_jsonl(path: Path) -> list[dict[str, object]]: + rows = [ + json.loads(line) + for line in path.read_text(encoding="utf-8").splitlines() + if line.strip() + ] + model_ids = [str(row.get("model_id", "")).strip() for row in rows] + if len(rows) != 100: + raise ValueError(f"Expected exactly 100 cohort rows, found {len(rows)}.") + if any(not model_id for model_id in model_ids): + raise ValueError("Every cohort row must contain a non-empty model_id.") + if len(set(model_ids)) != len(model_ids): + raise ValueError("The cohort contains duplicate model IDs.") + return rows + + +def write_jsonl(path: Path, rows: list[dict[str, object]]) -> str: + path.parent.mkdir(parents=True, exist_ok=True) + content = "".join(json.dumps(row, ensure_ascii=False) + "\n" for row in rows) + path.write_text(content, encoding="utf-8") + return hashlib.sha256(content.encode("utf-8")).hexdigest() + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description=( + "Assign the largest primary models to A100s and the remainder to " + "RTX 3090s." + ) + ) + parser.add_argument( + "--cohort", + type=Path, + default=Path("configs/rand_chinese_stratified_100.jsonl"), + ) + parser.add_argument("--a100-models", type=int, default=23) + parser.add_argument( + "--rtx3090-output", + type=Path, + default=Path("configs/rand_chinese_stratified_100_rtx3090.jsonl"), + ) + parser.add_argument( + "--a100-output", + type=Path, + default=Path("configs/rand_chinese_stratified_100_a100.jsonl"), + ) + parser.add_argument( + "--audit-output", + type=Path, + default=Path("configs/rand_chinese_stratified_100_shards_audit.json"), + ) + return parser.parse_args() + + +def main() -> int: + args = parse_args() + rows = load_jsonl(args.cohort) + if not 0 < args.a100_models < len(rows): + raise ValueError("--a100-models must be between 1 and 99.") + + ranked = sorted( + rows, + key=lambda row: ( + -float(row.get("parameter_count_billions", 0.0) or 0.0), + str(row["model_id"]), + ), + ) + a100_ids = {str(row["model_id"]) for row in ranked[: args.a100_models]} + a100_rows = [row for row in rows if str(row["model_id"]) in a100_ids] + rtx3090_rows = [row for row in rows if str(row["model_id"]) not in a100_ids] + + rtx3090_sha = write_jsonl(args.rtx3090_output, rtx3090_rows) + a100_sha = write_jsonl(args.a100_output, a100_rows) + cohort_sha = hashlib.sha256(args.cohort.read_bytes()).hexdigest() + audit = { + "cohort": str(args.cohort), + "cohort_sha256": cohort_sha, + "assignment_rule": ( + f"the {args.a100_models} largest models by recorded " + "parameter_count_billions go to the two A100 GPUs; all remaining " + "models go to the four-GPU RTX 3090 fleet" + ), + "rtx3090": { + "path": str(args.rtx3090_output), + "count": len(rtx3090_rows), + "sha256": rtx3090_sha, + "models": [row["model_id"] for row in rtx3090_rows], + }, + "a100": { + "path": str(args.a100_output), + "count": len(a100_rows), + "sha256": a100_sha, + "models": [row["model_id"] for row in a100_rows], + }, + } + args.audit_output.write_text( + json.dumps(audit, indent=2, ensure_ascii=False) + "\n", + encoding="utf-8", + ) + print(json.dumps(audit, indent=2, ensure_ascii=False)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/build_primary_run_grid.py b/scripts/build_primary_run_grid.py new file mode 100644 index 0000000..3311f54 --- /dev/null +++ b/scripts/build_primary_run_grid.py @@ -0,0 +1,350 @@ +#!/usr/bin/env python3 +"""Build the live completion grid using the latest validated-100 cohort.""" + +from __future__ import annotations + +import csv +import json +from collections import defaultdict +from datetime import datetime +from pathlib import Path +from typing import Any + + +ROOT = Path(__file__).resolve().parents[1] +OUTPUT_ROOT = ROOT / "out" +COHORT_MANIFEST = ( + OUTPUT_ROOT + / "top_p_control_t05_validated100_20260719" + / "top_p_control_t05_t05_p06_r2" + / "manifest.json" +) +COHORT_CONFIG = ROOT / "configs" / "rand_chinese_validated100_latest.jsonl" +REPORT_DIR = ROOT / "reports" + +TEMPERATURES = (0.0, 0.2, 0.3, 0.5, 0.7) +TOP_PS = (0.8, 0.9, 1.0) +REPEATS = (1, 2) +GENERATION_SEEDS = {1: 42001, 2: 42002} + + +def load_cohort() -> set[str]: + if COHORT_MANIFEST.exists(): + manifest = json.loads(COHORT_MANIFEST.read_text(encoding="utf-8")) + models = { + str(task["model_id"]) + for task in manifest.get("tasks", []) + if task.get("model_id") + } + else: + models = { + str(json.loads(line)["model_id"]) + for line in COHORT_CONFIG.read_text(encoding="utf-8").splitlines() + if line.strip() + } + if len(models) != 100: + raise ValueError(f"Expected 100 fixed models, found {len(models)}") + return models + + +def planned_condition(temperature: float, top_p: float) -> bool: + # At T=0 sampling is disabled, so top-p is inactive and recorded once. + return temperature > 0 or top_p == 1.0 + + +def load_statuses(path: Path) -> dict[str, str]: + final: dict[str, str] = {} + if not path.exists(): + return final + with path.open(encoding="utf-8") as handle: + for line in handle: + line = line.strip() + if not line: + continue + try: + record = json.loads(line) + except json.JSONDecodeError: + continue + model_id = record.get("model_id") + if model_id: + final[str(model_id)] = str(record.get("status", "")) + return final + + +def classify(status: str) -> str: + if status == "success": + return "success" + if status.startswith("failed") or status == "incompatible": + return "failed" + if status.startswith("skipped"): + return "skipped" + return "pending" + + +def collect_runs( + cohort: set[str], +) -> tuple[dict[tuple[float, float, int], dict[str, Any]], list[str]]: + runs: dict[tuple[float, float, int], dict[str, Any]] = defaultdict( + lambda: { + "manifest_models": set(), + "final": {}, + "sources": [], + "manifest_count": 0, + "generation_seeds": set(), + "cohort_overlaps": [], + } + ) + ignored: list[str] = [] + if not OUTPUT_ROOT.exists(): + return runs, ignored + + for manifest_path in sorted(OUTPUT_ROOT.rglob("manifest.json")): + try: + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + temperature = float(manifest["temperature"]) + top_p = float(manifest["top_p"]) + except (OSError, ValueError, KeyError, TypeError, json.JSONDecodeError): + ignored.append(manifest_path.relative_to(ROOT).as_posix()) + continue + + suffix = str(manifest.get("output_suffix", "")) + repeat = next( + (repeat for repeat in REPEATS if suffix.endswith(f"_r{repeat}")), None + ) + # The original T=0.7, top-p=0.9 execution was labelled "prelim" + # rather than r1. It is still a reusable first execution of this cell. + if ( + repeat is None + and manifest_path.parent.name == "rand_chinese_0p3b_7b_temp07" + and temperature == 0.7 + and top_p == 0.9 + ): + repeat = 1 + if ( + repeat is None + or temperature not in TEMPERATURES + or top_p not in TOP_PS + or not planned_condition(temperature, top_p) + ): + ignored.append(manifest_path.relative_to(ROOT).as_posix()) + continue + + key = (temperature, top_p, repeat) + run = runs[key] + run["manifest_count"] += 1 + run["sources"].append(manifest_path.parent.relative_to(ROOT).as_posix()) + manifest_models = { + str(task["model_id"]) + for task in manifest.get("tasks", []) + if task.get("model_id") + } + overlap = manifest_models & cohort + run["manifest_models"].update(overlap) + run["cohort_overlaps"].append(len(overlap)) + seed = manifest.get("generation_seed") + run["generation_seeds"].add( + str(seed) + if seed is not None + else "not explicitly set or recorded" + ) + for model_id, status in load_statuses( + manifest_path.parent / "status.jsonl" + ).items(): + if model_id in cohort: + # A valid success from a targeted recovery satisfies the slot + # even if another matching historical run ended in failure. + previous = run["final"].get(model_id) + if previous != "success" or status == "success": + run["final"][model_id] = status + return runs, ignored + + +def summarize_run(run: dict[str, Any], cohort: set[str]) -> dict[str, Any]: + counts = defaultdict(int) + for model_id, status in run["final"].items(): + if model_id in cohort: + counts[classify(status)] += 1 + observed = sum(counts.values()) + no_status = len(cohort) - observed + missing_success = len(cohort) - counts["success"] + return { + "success": counts["success"], + "failed": counts["failed"], + "skipped": counts["skipped"], + "pending_status": counts["pending"], + "no_status": no_status, + "missing_success": missing_success, + "started": bool(run["manifest_count"] or observed), + "manifest_models": len(run["manifest_models"]), + "cohort_overlap": max(run["cohort_overlaps"], default=0), + "recorded_seeds": ",".join(sorted(run["generation_seeds"])), + "sources": ";".join(run["sources"]), + } + + +def cell_text(repeat_rows: list[dict[str, Any]]) -> str: + success = sum(int(row["success"]) for row in repeat_rows) + target = 100 * len(REPEATS) + if success == target: + return f"✅ {success}/{target}" + if any(bool(row["started"]) for row in repeat_rows): + return f"♻️ {success}/{target}；还需 {target - success}" + return f"⬜ 0/{target}；待跑 r1/r2" + + +def main() -> None: + cohort = load_cohort() + runs, ignored = collect_runs(cohort) + detail_rows: list[dict[str, Any]] = [] + setting_rows: dict[tuple[float, float], list[dict[str, Any]]] = defaultdict(list) + + for temperature in TEMPERATURES: + for top_p in TOP_PS: + if not planned_condition(temperature, top_p): + continue + for repeat in REPEATS: + summary = summarize_run( + runs[(temperature, top_p, repeat)], cohort + ) + row = { + "temperature": temperature, + "top_p": top_p, + "repeat": f"r{repeat}", + # T=0.5 is shared with the validated top-p control, whose + # explicit-seed runs use 42 in both rounds. + "proposed_backfill_seed": ( + 42 + if temperature == 0.5 + else GENERATION_SEEDS[repeat] + ), + **summary, + } + detail_rows.append(row) + setting_rows[(temperature, top_p)].append(row) + + REPORT_DIR.mkdir(exist_ok=True) + with (REPORT_DIR / "primary_robustness_run_grid.csv").open( + "w", newline="", encoding="utf-8" + ) as handle: + fields = [ + "temperature", + "top_p", + "repeat", + "proposed_backfill_seed", + "recorded_seeds", + "success", + "failed", + "skipped", + "pending_status", + "no_status", + "missing_success", + "started", + "manifest_models", + "cohort_overlap", + "sources", + ] + writer = csv.DictWriter(handle, fieldnames=fields, lineterminator="\n") + writer.writeheader() + writer.writerows(detail_rows) + + total_success = sum(int(row["success"]) for row in detail_rows) + target_tasks = len(detail_rows) * 100 + completed_settings = sum( + sum(int(row["success"]) for row in rows) == 200 + for rows in setting_rows.values() + ) + started_settings = sum( + any(bool(row["started"]) for row in rows) + for rows in setting_rows.values() + ) + + markdown = [ + "# Primary robustness experiment — run grid", + "", + f"Updated: {datetime.now().astimezone().isoformat(timespec='seconds')}", + "", + "Revised plan:", + "", + "- Fixed cohort: **the latest validated100 model list** from " + "`top_p_control_t05_t05_p06_r2/manifest.json`.", + "- Historical artifacts are reused only when their model ID and decoding " + "cell match this cohort/grid.", + "- Two repeats per condition. The shared `temperature=0.5` row uses " + "**seed 42 for both rounds** so its `top_p=0.8/0.9/1.0` cells can also " + "serve the validated top-p control. Other proposed grid backfills remain " + "**r1 / 42001** and **r2 / 42002**.", + "- The historical sweep did not explicitly set or record a response-generation " + "seed. Its recorded `random_seed=42` controlled probe selection and DNA " + "reduction, not the generation sampler.", + "- Stochastic grid: temperature `{0.2, 0.3, 0.5, 0.7}` × top-p `{0.8, 0.9, 1.0}`.", + "- Deterministic control: `temperature=0`, `top_p=1.0` (top-p inactive).", + f"- Total: **13 settings × 2 repeats × 100 models = {target_tasks:,} successful model artifacts required**.", + "", + "Important cohort note: the two validated100 variants overlap by 97 models, " + "but the older temperature/top-p sweep used here overlaps the latest " + "validated100 cohort by **87 models**. Therefore those grid cells require " + "13 cohort backfills plus retries for any failed/pending overlapping models.", + "", + "Legend: `✅` both repeats have 100 successful models; `♻️` historical " + "results are reusable but the cell is incomplete; `⬜` no matching run was " + "found; `—` is not a planned condition.", + "", + "| Temperature \\\\ Top-p | 0.8 | 0.9 | 1.0 |", + "|---:|---:|---:|---:|", + ] + for temperature in TEMPERATURES: + cells = [] + for top_p in TOP_PS: + if not planned_condition(temperature, top_p): + cells.append("—") + else: + cells.append(cell_text(setting_rows[(temperature, top_p)])) + markdown.append( + f"| {temperature:g} | " + " | ".join(cells) + " |" + ) + + markdown += [ + "", + "## Overall progress", + "", + f"- Successful artifacts: **{total_success}/{target_tasks}**.", + f"- Settings started: **{started_settings}/13**.", + f"- Settings fully complete: **{completed_settings}/13**.", + f"- Remaining successful artifacts required: **{target_tasks - total_success}**.", + "", + "## Repeat-level detail", + "", + "| Temperature | Top-p | Round | Historical generation seed | Proposed backfill seed (not run) | Reusable success | Failed | Pending/no status | Still needed | State |", + "|---:|---:|---:|---|---:|---:|---:|---:|---:|---|", + ] + for row in detail_rows: + state = ( + "✅ complete" + if row["success"] == 100 + else ("♻️ reusable/incomplete" if row["started"] else "⬜ not run") + ) + historical_seed = row["recorded_seeds"] or "—" + markdown.append( + f"| {row['temperature']:g} | {row['top_p']:g} | {row['repeat']} | " + f"{historical_seed} | {row['proposed_backfill_seed']} | {row['success']} | " + f"{row['failed']} | " + f"{row['pending_status'] + row['skipped'] + row['no_status']} | " + f"{row['missing_success']} | {state} |" + ) + if ignored: + markdown += [ + "", + f"Ignored {len(ignored)} manifest(s) outside the revised primary grid; see the CSV/source tree for audit.", + ] + markdown.append("") + (REPORT_DIR / "primary_robustness_run_grid.md").write_text( + "\n".join(markdown), encoding="utf-8" + ) + print( + f"Wrote primary grid: {total_success}/{target_tasks} successful artifacts, " + f"{started_settings}/13 settings started" + ) + + +if __name__ == "__main__": + main() diff --git a/scripts/build_sampling_settings_summary.py b/scripts/build_sampling_settings_summary.py new file mode 100644 index 0000000..cbbc8d1 --- /dev/null +++ b/scripts/build_sampling_settings_summary.py @@ -0,0 +1,285 @@ +#!/usr/bin/env python3 +"""Summarize temperature/top-p experiments by setting rather than run.""" + +from __future__ import annotations + +import csv +import json +import re +from collections import defaultdict +from pathlib import Path +from typing import Any + + +ROOT = Path(__file__).resolve().parents[1] +REPORT_DIR = ROOT / "reports" + + +def load_json(path: Path) -> dict[str, Any]: + with path.open(encoding="utf-8") as handle: + return json.load(handle) + + +def cohort_for(path: Path) -> tuple[int, str]: + text = path.as_posix() + if "top_p_control_t05_validated100_20260719" in text: + return 3, "Top-p control — validated 100" + if "top_p_control_t05_20260717" in text: + return 2, "Top-p control — pilot" + return 1, "Temperature/top-p sweep" + + +def repeat_for(manifest: dict[str, Any], path: Path) -> str: + suffix = str(manifest.get("output_suffix", "")) + match = re.search(r"_r(\d+)$", suffix) + if match: + return f"r{match.group(1)}" + if path.parent.name == "rand_chinese_0p3b_7b_temp07": + return "prelim" + return "run" + + +def read_final_statuses(path: Path) -> dict[tuple[str, str], str]: + final: dict[tuple[str, str], str] = {} + if not path.exists(): + return final + with path.open(encoding="utf-8") as handle: + for line in handle: + line = line.strip() + if not line: + continue + try: + record = json.loads(line) + except json.JSONDecodeError: + continue + model_id = record.get("model_id") + if model_id: + final[(str(record.get("provider", "")), str(model_id))] = str( + record.get("status", "") + ) + return final + + +def classify(status: str) -> str: + if status == "success": + return "success" + if status.startswith("failed") or status == "incompatible": + return "fail" + if status.startswith("skipped"): + return "skipped" + return "pending" + + +def outcome_for(path: Path, manifest: dict[str, Any]) -> dict[str, Any]: + planned = { + (str(task.get("provider", "")), str(task["model_id"])) + for task in manifest.get("tasks", []) + if task.get("model_id") + } + final = read_final_statuses(path.parent / "status.jsonl") + # Only count the cohort declared by the manifest. Stale status records from + # models outside the cohort are tracked separately. + current = {identity: status for identity, status in final.items() if identity in planned} + counts = defaultdict(int) + for status in current.values(): + counts[classify(status)] += 1 + unobserved = len(planned - set(current)) + pending = counts["pending"] + unobserved + terminal = counts["success"] + counts["fail"] + counts["skipped"] + complete = terminal == len(planned) and pending == 0 + fully_successful = complete and counts["success"] == len(planned) + return { + "repeat": repeat_for(manifest, path), + "planned": len(planned), + "success": counts["success"], + "fail": counts["fail"], + "skipped": counts["skipped"], + "pending": pending, + "unobserved": unobserved, + "extra_status_models": len(set(final) - planned), + "complete": complete, + "fully_successful": fully_successful, + "seed": manifest.get("generation_seed", ""), + "created_at": manifest.get("created_at", ""), + "source": path.parent.resolve().relative_to(ROOT).as_posix(), + } + + +def format_outcomes(outcomes: list[dict[str, Any]], field: str) -> str: + return "; ".join( + f"{outcome['repeat']} {outcome[field]}/{outcome['planned']}" + for outcome in outcomes + ) + + +def setting_state(outcomes: list[dict[str, Any]]) -> str: + if all(outcome["fully_successful"] for outcome in outcomes): + return "✅ 100% successful" + if all(outcome["complete"] for outcome in outcomes): + return "✅ Complete" + complete = sum(bool(outcome["complete"]) for outcome in outcomes) + return f"⏳ Incomplete ({complete}/{len(outcomes)} repeats complete)" + + +def grid_cell(row: dict[str, Any]) -> str: + outcomes = json.loads(str(row["details_json"])) + values = "/".join(str(outcome["success"]) for outcome in outcomes) + return values if all(outcome["complete"] for outcome in outcomes) else f"{values}*" + + +def append_grid( + markdown: list[str], title: str, rows: list[dict[str, Any]] +) -> None: + temperatures = sorted({float(row["temperature"]) for row in rows}) + top_ps = sorted({float(row["top_p"]) for row in rows}) + lookup = { + (float(row["temperature"]), float(row["top_p"])): grid_cell(row) + for row in rows + } + markdown += [ + f"## {title}", + "", + "| Temperature \\\\ Top-p | " + + " | ".join(f"{top_p:g}" for top_p in top_ps) + + " |", + "|---:|" + "|".join("---:" for _ in top_ps) + "|", + ] + for temperature in temperatures: + cells = [ + lookup.get((temperature, top_p), "—") + for top_p in top_ps + ] + markdown.append( + f"| {temperature:g} | " + " | ".join(cells) + " |" + ) + markdown.append("") + + +def main() -> None: + grouped: dict[ + tuple[int, str, float, float], list[dict[str, Any]] + ] = defaultdict(list) + for path in sorted(ROOT.rglob("manifest.json")): + if ".git" in path.parts: + continue + try: + manifest = load_json(path) + except (OSError, json.JSONDecodeError): + continue + if "temperature" not in manifest or "top_p" not in manifest: + continue + order, cohort = cohort_for(path) + temperature = float(manifest["temperature"]) + top_p = float(manifest["top_p"]) + grouped[(order, cohort, temperature, top_p)].append( + outcome_for(path, manifest) + ) + + rows: list[dict[str, Any]] = [] + for (order, cohort, temperature, top_p), outcomes in sorted(grouped.items()): + outcomes.sort(key=lambda item: (item["repeat"], item["created_at"])) + rows.append( + { + "order": order, + "experiment": cohort, + "temperature": temperature, + "top_p": top_p, + "repeats": len(outcomes), + "successful_models": format_outcomes(outcomes, "success"), + "failed_models": format_outcomes(outcomes, "fail"), + "pending_models": format_outcomes(outcomes, "pending"), + "status": setting_state(outcomes), + "seeds": "; ".join( + f"{item['repeat']} {item['seed'] if item['seed'] != '' else 'not recorded'}" + for item in outcomes + ), + "sources": ";".join(item["source"] for item in outcomes), + "details_json": json.dumps(outcomes, ensure_ascii=False), + } + ) + + REPORT_DIR.mkdir(exist_ok=True) + csv_path = REPORT_DIR / "sampling_settings_summary.csv" + fields = [ + "experiment", + "temperature", + "top_p", + "repeats", + "successful_models", + "failed_models", + "pending_models", + "status", + "seeds", + "sources", + "details_json", + ] + with csv_path.open("w", newline="", encoding="utf-8") as handle: + writer = csv.DictWriter(handle, fieldnames=fields, extrasaction="ignore") + writer.writeheader() + writer.writerows(rows) + + markdown = [ + "# Temperature and top-p settings", + "", + ( + f"{len(rows)} distinct settings are shown. Repeated executions of the " + "same setting are combined in one row. Counts use each planned model's " + "latest status; status records outside the manifest cohort are excluded." + ), + "", + ( + "A setting is **Complete** when every planned model in every repeat has " + "a terminal success/failure result. **Incomplete** means at least one " + "model is pending, waiting to retry, or has no status record." + ), + "", + "## Success-count grids", + "", + ( + "Each cell is the number of successful models out of 100. Two values " + "mean `r1/r2`; `*` marks an incomplete setting; `—` means not tested." + ), + "", + ] + for cohort in ( + "Temperature/top-p sweep", + "Top-p control — pilot", + "Top-p control — validated 100", + ): + append_grid( + markdown, + cohort, + [row for row in rows if row["experiment"] == cohort], + ) + markdown += ["## Per-setting details", ""] + last_cohort = "" + for row in rows: + if row["experiment"] != last_cohort: + if last_cohort: + markdown.append("") + last_cohort = str(row["experiment"]) + markdown += [ + f"### {last_cohort}", + "", + "| Temperature | Top-p | Repeats | Successful models | Failed models | Pending models | Current status |", + "|---:|---:|---:|---|---|---|---|", + ] + markdown.append( + f"| {row['temperature']:g} | {row['top_p']:g} | {row['repeats']} | " + f"{row['successful_models']} | {row['failed_models']} | " + f"{row['pending_models']} | {row['status']} |" + ) + markdown += [ + "", + "The companion CSV includes seeds, source directories, unobserved counts, " + "and the complete per-repeat details.", + "", + ] + (REPORT_DIR / "sampling_settings_summary.md").write_text( + "\n".join(markdown), encoding="utf-8" + ) + print(f"Wrote {len(rows)} distinct sampling settings from {sum(r['repeats'] for r in rows)} repeats") + + +if __name__ == "__main__": + main() diff --git a/scripts/build_validated100_config.py b/scripts/build_validated100_config.py new file mode 100755 index 0000000..07149ac --- /dev/null +++ b/scripts/build_validated100_config.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python3 +"""Reconstruct the latest validated-100 JSONL from its canonical manifest.""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--manifest", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + args = parser.parse_args() + + manifest = json.loads(args.manifest.read_text(encoding="utf-8")) + rows = [ + { + "model_id": str(task["model_id"]), + "provider": str(task.get("provider", "huggingface")), + } + for task in manifest.get("tasks", []) + if task.get("model_id") + ] + model_ids = [row["model_id"] for row in rows] + if len(rows) != 100 or len(set(model_ids)) != 100: + raise ValueError( + f"Expected 100 unique manifest models, found " + f"{len(rows)} rows / {len(set(model_ids))} unique IDs" + ) + + args.output.parent.mkdir(parents=True, exist_ok=True) + temporary = args.output.with_suffix(args.output.suffix + ".tmp") + temporary.write_text( + "".join( + json.dumps(row, ensure_ascii=False) + "\n" + for row in rows + ), + encoding="utf-8", + ) + temporary.replace(args.output) + print(f"Wrote {len(rows)} models to {args.output}") + + +if __name__ == "__main__": + main() diff --git a/scripts/calc_dna.py b/scripts/calc_dna.py index 99d3a46..5de1c3f 100644 --- a/scripts/calc_dna.py +++ b/scripts/calc_dna.py @@ -68,11 +68,51 @@ def main(): action="store_true", help="Continue processing remaining models if one fails (batch mode only)" ) + parser.add_argument( + "--try-vllm", + action="store_true", + help="Try vLLM first for decoder-only HuggingFace models" + ) + parser.add_argument( + "--temperature", + type=float, + default=0.0, + help="Generation temperature. Defaults to 0.0 to preserve deterministic historical runs.", + ) + parser.add_argument( + "--top-p", + type=float, + default=1.0, + help="Top-p nucleus sampling value used when sampling is enabled.", + ) + parser.add_argument( + "--do-sample", + action="store_true", + default=False, + help="Enable stochastic sampling during generation.", + ) + parser.add_argument( + "--no-do-sample", + dest="do_sample", + action="store_false", + help="Disable stochastic sampling during generation.", + ) parser.add_argument( "--no-save", action="store_true", help="Don't save results to disk" ) + parser.add_argument( + "--output-suffix", + type=str, + default="", + help="Suffix appended to the safe model output directory name, e.g. _2.", + ) + parser.add_argument( + "--ignore-response-cache", + action="store_true", + help="Regenerate responses even if responses.json already exists; new responses are still saved.", + ) args = parser.parse_args() # Set paths relative to project root @@ -94,8 +134,14 @@ def main(): data_root=data_root, metadata_file=metadata_file if metadata_file.exists() else None, output_dir=output_dir, + output_suffix=args.output_suffix, save=not args.no_save, trust_remote_code=True, + try_vllm=args.try_vllm, + temperature=args.temperature, + do_sample=args.do_sample, + top_p=args.top_p, + use_response_cache=not args.ignore_response_cache, ) # Batch mode: process multiple models from file @@ -140,4 +186,3 @@ def main(): if __name__ == "__main__": sys.exit(main()) - diff --git a/scripts/debug_openrouter_max_tokens.py b/scripts/debug_openrouter_max_tokens.py new file mode 100644 index 0000000..9ad0d67 --- /dev/null +++ b/scripts/debug_openrouter_max_tokens.py @@ -0,0 +1,204 @@ +#!/usr/bin/env python3 +"""Minimal OpenRouter max_tokens debugger.""" + +from __future__ import annotations + +import argparse +import json +import os +import sys +from typing import Any + +from dotenv import load_dotenv +from openai import OpenAI + + +def _resolve_api_key_with_source() -> tuple[str, str]: + for key_name in ("OPENROUTER_API_KEY", "APIKEY_OPENROUTER", "OPENROUTER_KEY"): + value = os.getenv(key_name, "").strip() + if value: + return value, key_name + raise RuntimeError( + "OpenRouter API key not found. Set OPENROUTER_API_KEY (or APIKEY_OPENROUTER / OPENROUTER_KEY)." + ) + + +def _mask_secret(value: str) -> str: + if not value: + return "p\d{2})_r(?P \d{2})_r(?P