diff --git a/blog/2026-07-23-vibe-coding/index.md b/blog/2026-07-23-vibe-coding/index.md new file mode 100644 index 0000000..518e4ad --- /dev/null +++ b/blog/2026-07-23-vibe-coding/index.md @@ -0,0 +1,127 @@ +--- +slug: vibe-coding +title: "PG as the Perfect Partner for Vibe Coding: AI Agent Development the 'Simplicity-First' Way" +authors: [萧少聪] +category: IvorySQL +image: img/blog/covers/vibe-coding-en.png +tags: [PostgreSQL, AI, Agent, VibeCoding, HOW2026] +--- + +> Based on Xiao Shaocong's presentation at HOW 2026. Xiao is the former PostgreSQL Association President, Chinese Community Chair, and IvorySQL Expert Advisory Committee member. + +Over the past two years, I've held one firm conviction: in the era of AI-driven development with LLMs, PostgreSQL will inevitably become the **default database** for any AI project. Not that PG can rule the world forever — at some point, certain workloads and business scenarios may indeed require migrating specific data to specialized databases. But as a **starting point**, PG is undisputedly the best choice. + +Today I'll cover four topics: first, our headache — Token Anxiety; second, how "One SQL" lets you achieve more with less; third, how a unified data plane gives AI "blind-spot-free" operation; and fourth, the boundaries of PG as the AI-first database. + +## 1. The Vibe Coding Headache: System Complexity as a "Token Incinerator" + +When building a system, you typically start with one database — business isn't that complex at first. But as you develop — whether AI applications or others — you find you need JSON, search, AI capabilities. Each addition is a new data model, each decision perfectly reasonable at the time. The result? Your system grows from 1 database to 5, or more. You bring in MongoDB for documents, Elasticsearch for search, Milvus for vectors — you've responsibly chosen the industry's best. But here's the question: **has your business taken off?** The project just started, and you're already saddled with the most complex architecture. That's not ideal. + +Worse, in Vibe Coding, multi-database architectures inccur a cost that manifests as **Token inflation**: + +- Different databases have different syntax — AI must learn multiple query languages +- Data must sync across systems — AI must write and maintain ETL logic +- Cross-system queries get split into multiple steps — each consuming tokens +- Context windows get flooded, models become "dumb" + +This isn't Vibe Coding — it's a **Token Incinerator**. + +**The solution?** Within one boundary, use one database to solve all problems. That database is PostgreSQL. + +## 2. One SQL, Double the Output: 10 Days, 50K Lines Validated + +Last November, I ran an experiment: pure AI-assisted development to build a project called OntologyAlpha in 10 days — ~54,000 lines generated, ~11,300 effective lines deployed. + +This is an "ontology" system, requiring four data types at the storage layer: + +- **JSON**: AI input/output, frontend-backend communication +- **Vectors**: Semantic representation of text for similarity search +- **Graph**: Knowledge point relationship tracking and hierarchical management +- **Time-series**: Contextual communication sequence recording + +The architecture: **PostgreSQL native multi-modal storage** at the bottom, Async Workers and Python sandbox above, Next.js visualization on top — all code AI-generated, zero lines written by me. + +**Development approach**: I used Google Gemini as "Chief Data Officer/CTO" for architecture planning and task breakdown; Cursor (free tier) for code implementation. Total: 82 work hours, ~10 person-days. + +**Results**: + +- CPU/memory monitoring with precise search and fuzzy semantic search +- Elementary math textbook knowledge vectored into a knowledge graph (using two relational tables, no dedicated graph DB) +- PDF documents (e.g., Singapore talent policy) auto-extracting keywords and business relationships into upstream/downstream chains +- Python sandbox integration: CPU overheating automatically triggers upstream chain state changes + +**The key SQL looks like this**: + +```sql +-- One SQL: JSON extraction + relational graph traversal + vector similarity +SELECT ... +FROM ... +WHERE name->>'xxx' = '...' -- JSON field extraction + AND relation_type = '...' -- Relational graph logic + AND embedding <-> '...' -- Vector similarity +``` + +One SQL, three data models. In PG, transactions, permissions, and backups are one unified system — no cross-system consistency headaches. + +Throughout development, I used the most basic AI packages — Google Gemini at $20/month, Cursor free tier. Token consumption was very manageable. + +## 3. Unified Data Plane, One Step Ahead: AI's Full-Coverage Operation + +Many complain PG can't do vectors at scale — memory-hungry, hard to reach hundred-million scale. + +Here I recommend a project: **pgvectorscale**, by the TimescaleDB team. It places vector indexes on disk via DiskANN + quantization, breaking the memory barrier for massive-scale vector retrieval at reduced cost. + +But PG's advantage isn't being #1 in any single category. If your system must handle 100M+ vectors at extreme QPS, definitely choose a specialized vector database. But when your business needs to manage relational, vector, JSON, time-series, and graph data simultaneously, **PG is an all-rounder scoring ~80 in everything**. + +The combined advantage? **Dramatically reduced system fragmentation, significantly lower development and operations complexity.** + +In multi-database architectures, your application must manage: + +- Relational DB uses SQL, the vector DB might not support SQL at all +- Do you need transactions between systems? How to guarantee consistency? +- With ETL, the app must track latency — which data is trustworthy, which is stale? +- How to unify permissions? How to unify backups? + +Managing all this degrades Vibe Coding quality — consumption, accuracy, output all suffer. + +In PG's unified data plane: + +- **1 Network Hop**: Application accesses only one database +- **1 Transaction**: All operations in a single transaction +- **1 Permission Model**: Unified access control +- **1 Backup System**: Unified disaster recovery + +**System complexity overhead often outweighs pure single-point performance gains. Most of the time, you're paying the tax for multi-system architecture.** + +## 4. PG as AI's "Safe Bet": Boundaries Give Confidence + +I love PG. So when should you consider introducing specialized databases? + +My recommendation: **Use PG to solve 80% of problems first. Build business capability fast. Once you earn your first dollar, then consider whether to migrate.** + +Specifically: + +| Data Type | PG's Limit | When to Migrate | +|-----------|-----------|-----------------| +| **Vector** | Sub-100M scale, moderate QPS | Billion-scale vectors + extreme QPS | +| **Time-series** | Regular logs, metrics, monitoring | Massive volume + special compression needs | +| **JSON** | Most scenarios | Ultra-large JSON (thousands of lines) or high-frequency updates | +| **Graph** | 3-4 level shallow relationships | Graph depth and complexity exceeding PG's capability | + +**Clear boundaries actually make you more confident to use it.** + +PG 19 will natively support better graph queries. Today, use the AGE extension, or like me, "fake" graph structures with a few relational tables — sufficient for most AI applications. + +## Conclusion: Fewer Systems, Not Stronger Systems + +In the AI era, I believe everyone should think like an architect. But from an architectural mindset, **the goal should be fewer systems, not stronger systems.** + +There was once this idea: the more complex my architecture, the harder it is for the boss to fire me. But today, with AI here, if your boss lets you go, it's not because of AI — it's business reality. We don't need that burden. + +What truly matters: + +> **PostgreSQL is the default starting point, not the destination.** + +Start with PG, validate your business model fast, save tokens, save management time, focus on monetization. Once your business is stable and profitable, and you hit clear technical bottlenecks, then carefully evaluate whether to introduce specialized databases. + +A clean, simple architecture is the most resilient architecture for the future. diff --git a/blog/2026-07-28-incremental-checkpoint/index.md b/blog/2026-07-28-incremental-checkpoint/index.md new file mode 100644 index 0000000..bf0138c --- /dev/null +++ b/blog/2026-07-28-incremental-checkpoint/index.md @@ -0,0 +1,95 @@ +--- +slug: incremental-checkpoint +title: "The Struggles of Incremental Checkpoints: Lock Storms and Full Page Writes" +authors: [吕海波] +category: IvorySQL +image: img/blog/covers/incremental-ckpt.png +tags: [PostgreSQL, Checkpoint, Kernel, Performance, FPW, HOW2026] +--- + +> Based on Lyu Haibo's presentation at HOW 2026. Lyu is Chief Researcher at Yijing Technology, PG ACED, and Enterprise Mentor at Peking University. + +## 1. Why Incremental Checkpoints? + +While building a shared-storage cluster architecture (similar to Oracle RAC) based on PostgreSQL, a practical problem emerged: when using PG's original full checkpoint mechanism, dirty pages continuously accumulated across nodes, capping stress test performance. To solve this, we introduced incremental checkpoints. + +The core idea isn't complex: add a checkpoint queue (ckptq) in shared memory, ordering all dirty blocks by their "dirtied" time, then flush them along the queue in high-frequency, small-batch increments. Compared to full checkpoints traversing all dirty pages at once, this theoretically smooths I/O load. + +But actual implementation revealed two problems trickier than expected: ckptq shared memory lock management, and the coupling between incremental checkpoints and FPW (Full Page Writes). + +## 2. ckptq Shared Memory Lock Management: The Hidden Cost of Spinlock Contention + +Placing ckptq in shared memory means multi-process concurrent dirty block access inevitably involves lock management. We initially used PG's built-in SpinLock, but severe performance issues emerged under high contention. + +### 2.1 What is a Spinlock? + +A spinlock is essentially a memory variable — 1, 2, 4, or 8 bytes. Process A holding the lock changes the value from 0 to 1; Process B, finding the value non-zero, keeps looping until it returns to 0. This "busy waiting" avoids yielding the CPU, preventing context switches and cache pollution. + +The problem: when multiple processes compete for the same spinlock, the consequences go far beyond CPU spinning. + +### 2.2 Inter-Core Communication Storm + +With 16 cores, suppose Core 0 holds the lock and 15 cores are spinning. When Core 0 releases the lock (changing 1 to 0): + +1. Core 0 must broadcast **Invalidate** messages to all 15 cores, notifying them that their L1/L2 cache copies are stale +2. After all cores acknowledge, Core 0 modifies the variable to 0 +3. The 15 waiting cores immediately send **Write Update** messages to Core 0 requesting the latest value +4. After CPU arbitration, one core (say Core 9) gains modification rights, broadcasting **Write Invalidate** to 15 others +5. After all confirm, Core 9 sets the variable to 1, acquiring the lock + +One lock release-reacquire cycle involves dozens of inter-core message broadcasts. At 16 cores this is already significant; modern CPUs with tens or hundreds of cores amplify this enormously. **Round after round of message synchronization can degrade i9 performance to 386 levels.** This is the "lock storm" — hotspot contention compounded by inter-core communication latency. + +This isn't unique to incremental checkpoints. Any spinlock in PG experiencing contention can trigger the same inter-core communication storm, causing performance jitter. + +### 2.3 Improvement Approach + +The solution's inspiration comes from CPU cache coherence protocols and RAC's cache fusion. The core idea: assign each core its own independent lock variable. When spinning, each core only polls its own variable — no broadcast messages needed. + +To release the lock, the holder sends a single modification message to the target core's private variable, completing ownership transfer. This reduces inter-core communication from O(n²) to O(1). See the paper "Non-scalable locks are dangerous" — traditional spinlock scalability issues in many-core systems have long been established, just easily overlooked in practice. + +## 3. Incremental Checkpoints & FPW: Page Split Impact Analysis + +In PG, full checkpoints and FPW are tightly coupled. Introducing incremental checkpoints dramatically extends full checkpoint intervals — what does this mean for FPW's ability to protect against page splits? + +### 3.1 What is a Page Split (Partial Write)? + +A database page (e.g., PG's 8KB) typically consists of multiple OS pages (e.g., 4KB) at the OS level. When the database initiates an 8KB write, it's actually two 4KB writes at the storage layer. If power fails mid-write, you might get the first 4KB written but not the second — the database page becomes "half new, half old" corrupted state. This is a page split. + +### 3.2 Simulating Page Splits + +Page splits have long been hard to verify because outside of pulling the power cable, they're nearly impossible to reproduce. But using kernel dynamic tracing tools like eBPF/systemtap, you can intercept `pwrite` syscalls and tamper with the write length from 8KB to 4KB — the OS dutifully writes only half. This perfectly simulates page splits while excluding all other interfering factors. + +We tested Oracle, PostgreSQL, and MySQL under the same conditions. + +### 3.3 Oracle: No Software-Level Solution + +Intercepting `pwrite` during checkpoint flushing, Oracle detects I/O errors and crashes. On restart, instance recovery begins — it locates the checkpoint position, identifies dirty blocks needing recovery — then fails. + +The test conclusion is clear: Oracle doesn't solve page splits at the software level. It doesn't rely on filesystem atomic writes, nor does it special-handle the code. Oracle's strategy: detect corruption, rely on backups for media recovery, and provide BlockRecover for single-block recovery. **Pushing the problem to operations is itself a choice.** + +### 3.4 PostgreSQL: Completely Solved + +Under the same procedure, PG didn't crash on I/O errors — it only reported them. We used `kill -9` to kill all processes simulating an unexpected crash. On restart, PG read the checkpoint position from the control file, applied corresponding WAL logs — data fully recovered, zero loss. + +**Through the FPW mechanism, PG writes the entire page to WAL on first modification, ensuring that even if a page split occurs, the log can completely redo the page.** The cost: obvious I/O amplification. The benefit: deterministic data consistency. + +### 3.5 MySQL (InnoDB): Double Write Limitations + +MySQL InnoDB uses a double-write mechanism: write pages to the double-write buffer first, then to the actual data file. Tests revealed: + +- If only target table file writes are intercepted, double-write can recover +- But if system tablespace writes (e.g., undo tablespace) are intercepted, **the database fails to start, unrecoverable** + +Conclusion: **Double-write solves page splits in some scenarios but fails when system tablespace is damaged.** In a real "power loss + system tablespace write truncation" scenario, double-write cannot guarantee database recovery. + +### 3.6 Three-DB Comparison + +| Database | Solution | Truly Solves Page Splits? | +|----------|---------|--------------------------| +| Oracle | Backup + block recovery | Not at software level | +| MySQL | Double Write | Partial; fails on system tablespace damage | +| PostgreSQL | Full Page Write | Complete, at performance cost | + +Among the three mainstream databases, **only PG sacrifices performance to truly solve page splits at the software level.** Oracle pushes it to hardware/operations; MySQL's double-write has blind spots on the critical path. + +Back to the TC architecture: the underlying custom shared storage supports atomic writes, so FPW can be disabled in TC. But for users without atomic write storage — is FPW truly optional? No standard answer. Interested readers should follow this presentation's steps, actually simulate page splits, experience the fundamentals firsthand, then decide. diff --git a/blog/authors.yml b/blog/authors.yml index 4019552..5de07f4 100644 --- a/blog/authors.yml +++ b/blog/authors.yml @@ -22,6 +22,10 @@ ZhangChen: name: ZhangChen 陶郑: name: 陶郑 +萧少聪: + name: 萧少聪 +吕海波: + name: 吕海波 Oreo Yang: name: Oreo Yang 矫顺田: diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/1.png b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/1.png new file mode 100644 index 0000000..1cf9565 Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/1.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/2.png b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/2.png new file mode 100644 index 0000000..7c558fe Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/2.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/3.png b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/3.png new file mode 100644 index 0000000..b84c511 Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/3.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/4.png b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/4.png new file mode 100644 index 0000000..8692b92 Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/4.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/5.png b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/5.png new file mode 100644 index 0000000..ab58f05 Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/5.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/6.png b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/6.png new file mode 100644 index 0000000..992ea94 Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/6.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/index.md b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/index.md new file mode 100644 index 0000000..049887f --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-23-vibe-coding/index.md @@ -0,0 +1,147 @@ +--- +slug: vibe-coding +title: "PG 成为 Vibe Coding 的首选搭配:AI Agent 开发的「大道至简」" +authors: [萧少聪] +category: IvorySQL +image: img/blog/covers/vibe-coding-zh.png +tags: [PostgreSQL, AI, Agent, VibeCoding, HOW2026] +--- + + + +# PG 成为 Vibe Coding 的首选搭配:AI Agent 开发的"大道至简" + +> 本文整理于 HOW 2026 演讲内容,演讲者:萧少聪,前 PostgreSQL 分会会长及中文社区主席、IvorySQL 专家顾问委员。 + +过去的两年里,我一直坚信一个判断:在我们现在用 AI 方法、用大语言模型进行开发的模式下,PostgreSQL 一定会成为任何一个 AI 项目的**首选搭配**。当然,我不是说 PG 可以包打天下、做永远的底座——到了某个时间点,某些工作负载、某些业务场景,确实可能需要把特定形态的数据迁移到专用的数据库上。但作为**起点**,PG 毋庸置疑是最佳选择。 + +今天我会分四个话题来展开:第一是我们的烦恼——Token 焦虑;第二是"One SQL"如何让你事半功倍;第三是统一数据面如何让 AI"无死角"地工作;第四是讨论一下 PG 作为 AI 首选的边界在哪里。 + +## 一、Vibe Coding 的烦恼:系统复杂度带来的"Token 焚化炉" + +我们通常开发一个系统的时候,一开始一个数据库就足够了,业务不会那么复杂。但在应用过程中——不管你是开发 AI 应用还是其他应用——你会发现,我需要 JSON 功能,我需要搜索功能,我需要 AI 相关的各种能力。每一次增加都是一种新的业务形态,每一个决策在当下看起来都是非常正确和合理的。 + +但结果是什么?你的系统会从 1 个数据库变成 5 个,甚至更多。 + +你引入了 MongoDB 处理文档,引入了 Elasticsearch 处理搜索,引入了 Milvus 处理向量——看上去你很负责任,为团队选择了业界最好的产品。但大家要想一个问题:**你的业务上去了吗?** 项目可能才刚刚开始,你一下子就拿了一个最复杂的架构,这其实并不合适。 + +更重要的是,在 Vibe Coding 的过程中,多数据库带来的代价是**Token 的急剧膨胀**: + +- 不同数据库有不同的语法,AI 要学习多套查询语言 +- 数据需要在多个系统间同步,AI 要编写和维护 ETL 逻辑 +- 跨系统的查询被拆分成多个步骤,每一步都要消耗 Token +- 上下文窗口被撑爆,模型直接"降智"变傻 + +这哪里是 Vibe Coding?这简直是**Token 焚化炉**。 + +**解决方案是什么?** 在一个边界之内,用同一个数据库去解决所有问题。而这个数据库,就是 PostgreSQL。 + +![img](1.png) + +## 二、One SQL,事半功倍:10 天 5 万行的实战验证 + +去年 11 月,我做了一个实验:用纯 AI 辅助开发的方式,在 10 天内完成了一个名为 OntologyAlpha 的项目,总共生成了约 5.4 万行代码,最终上线有效代码约 1.13 万行。 + +这个项目是个"本体论"(Ontology)相关的系统,底层需要处理四类数据: + +![img](2.png) + +- **JSON**:AI 的输入输出,前后端交互信息 +- **向量**:文本的语义化表达,用于相似搜索 +- **图**:知识点上下游的关联追踪和分层管理 +- **时序**:上下文沟通的前后顺序记录 + +![img](3.png) + +项目架构上,存储层就是**PostgreSQL 原生多模态存储**,上面是 Async Workers 和 Python 安全沙箱,再上面是 Next.js 可视化工作台——全部代码由 AI 生成,我自己没有写一行。 + +**开发方式**:我用 Google AI Studio 的 Gemini 模型担任"首席数据官/CTO",负责架构规划和任务拆分;然后用 Cursor(免费默认模型)进行具体代码实现。总投入 82 个工时,大约 10 个人天。 + +**实现效果**: + +- CPU/内存设备监控,支持精准搜索和模糊语义搜索 +- 小学数学课本知识向量化,形成知识图谱(没有用专用图数据库,而是用两张关系表模拟图结构) +- PDF 文档(如新加坡人才政策)自动抽取关键词和业务关联关系,形成上下游链路 +- Python 沙箱联动:CPU 过热时自动触发上游链路状态变更 + +**关键的 SQL 长这样**: + +```sql +-- 一条SQL同时完成:JSON提取 + 关系型图遍历 + 向量相似度搜索 +SELECT ... +FROM ... +WHERE name->>'xxx' = '...' -- JSON字段提取 + AND relation_type = '...' -- 关系型图逻辑 + AND embedding <-> '...' -- 向量相似度 +``` + +一条 SQL,搞定三种数据模型的联合查询。在 PG 里面,事务、权限、备份全是一套体系,不用操心跨系统的数据一致性问题。 + +整个开发过程,我用的都是最基础的 AI 套餐——Google Gemini 20 美元/月,Cursor 免费套餐。Token 消耗非常可控。 + +## 三、统一数据面,快人一步:AI 的"无死角"覆盖 + +很多人吐槽说 PostgreSQL 做向量不行——内存消耗太大,难以扩展到亿级数据。 + +这里我要推荐一个项目:**pgvectorscale**,由 TimescaleDB 团队开发。它把向量索引放到磁盘上,通过 DiskANN + 量化技术,突破了内存限制,支持超大规模向量检索的同时大幅降低成本。 + +但我想强调的是:**PG 的优势不在于单项做到最强**。你的系统如果必须处理超过 1 亿级别的向量、超高 QPS,那确实要选专用向量数据库。但当你的业务需要对关系型、向量、JSON、时序、图等多种数据模型进行组合管理时,**PG 是一个各项能力都在 80 分左右的全能选手**。 + +![img](4.png) + +组合后的优势是什么?**极大地减少系统拆分,显著降低开发与运维复杂度。** + +在多数据库混合架构下,你的应用程序要自己管理: + +- 关系库用 SQL,向量库可能根本写不了 SQL +- 两个系统之间要不要事务?怎么保证一致性? +- 引入 ETL 后,应用程序要关注 ETL 延迟,哪些数据可信、哪些已过期? +- 权限怎么统一?备份怎么统一? + +这套东西全部管起来,Vibe Coding 的消耗量、准确度、产出效果都会变得非常糟糕。 + +![img](5.png) + +而在 PG 里面,统一数据面的价值在于: + +- **1 次 Network Hop**:应用只访问一个数据库 +- **1 套事务**:所有操作在同一事务中完成 +- **1 套权限**:统一的数据访问控制 +- **1 套备份**:统一的容灾体系 + +**系统复杂度带来的损耗,往往大于纯粹的单点性能收益。很多时候,你是在为多系统架构买单。** + +## 四、PG 是 AI 的"爱"的首选:有边界,更放心 + +我这么喜欢 PG,那什么时候应该考虑引入专用数据库? + +我自己的建议是这样的——**先用 PG 解决 80%的问题,快速形成业务能力。等赚到第一块钱了,再考虑要不要迁移。** + +具体来说: + +| 数据类型 | PG 能撑到什么程度 | 何时考虑迁移 | +| -------- | -------------------- | ---------------------------------- | +| **向量** | 亿级以下、中等 QPS | 向量规模爆发至亿级且 QPS 极高时 | +| **时序** | 常规的日志/埋点/监控 | 数据量巨大且有特殊压缩需求时 | +| **JSON** | 绝大多数场景 | 遇到数千行的超大 JSON 或高频更新时 | +| **图** | 3~4 层深度的浅层关联 | 图的深度和复杂度超出 PG 处理能力时 | + +**明确边界,反而更敢用。** + +![img](6.png) + +PG 19 会原生支持更好的图查询。现阶段你也可以用 AGE 插件,或者像我一样用几张关系表把图结构"糊"出来——对于大多数 AI 应用场景,这完全够用。 + +## 结语:更少的系统,而不是更强的系统 + +在 AI 时代,我觉得每个人都应该是一个架构师。但从架构的思维来看,**目标应该是更少的系统,而不是更强的系统。** + +以前有一种想法:我把架构做得越复杂,老板越不敢开掉我。但今天 AI 来了,老板想开你也不是因为 AI,而是因为经营不善。 + +我们不要有这种负担。真正有价值的是: + +> **PostgreSQL 是默认的起点,而不是终点。** + +用 PG 起步,快速验证商业模式,节省 Token,节省管理时间,把精力放在业务变现上。等到业务稳定盈利了、遇到明确的技术瓶颈了,再谨慎地评估是否引入专用数据库。 + +干净简洁的架构,才是未来弹性最大的架构。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-28-incremental-checkpoint/1.png b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-28-incremental-checkpoint/1.png new file mode 100644 index 0000000..dfbc7c5 Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-28-incremental-checkpoint/1.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-28-incremental-checkpoint/2.png b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-28-incremental-checkpoint/2.png new file mode 100644 index 0000000..19d4112 Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-28-incremental-checkpoint/2.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-28-incremental-checkpoint/index.md b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-28-incremental-checkpoint/index.md new file mode 100644 index 0000000..d71f9c3 --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-28-incremental-checkpoint/index.md @@ -0,0 +1,106 @@ +--- +slug: incremental-checkpoint +title: "增量检查点的「困」与「难」" +authors: [吕海波] +category: IvorySQL +image: img/blog/covers/incremental-ckpt.png +tags: [PostgreSQL, Checkpoint, Kernel, Performance, HOW2026] +--- + + + +# 增量检查点的"困"与"难" + +> 本文整理于 HOW 2026 演讲内容,演讲者:吕海波,易景科技首席研究员,PG ACED,北京大学企业导师。 + +## 一、增量检查点的引入背景 + +在基于PostgreSQL开发共享存储集群架构(类似Oracle RAC)的过程中,一个现实问题浮出水面:完全沿用PG原有的全量检查点机制时,各节点间脏页会持续累积,导致压力测试下的性能表现始终上不去。为了解决这个问题,我们引入了增量检查点。 + +![img](1.png) + +增量检查点的核心思想并不复杂:增加一个位于共享内存中的检查点队列(ckptq),按脏块变脏的顺序排列所有脏块,然后以高频次、小批量的方式沿队列定期刷新脏页。相比全量检查点一次性遍历所有脏页,这种方式理论上能更平滑地控制I/O负载。 + +但真正落地实现时,有两个关键问题比预想中棘手:一是ckptq共享内存锁的管理机制,二是增量检查点与FPW(全页写)之间的耦合关系。下文逐一展开。 + + +## 二、ckptq共享内存锁管理:自旋锁竞争的隐藏成本 + +ckptq放在共享内存中,多进程并发连接脏块必然涉及锁管理。我们最初采用的是PG自带的SpinLock自旋锁,但在高竞争场景下暴露出严重的性能问题。 + +### 2.1 自旋锁的本质 + +自旋锁本质上就是一个内存变量——1字节、2字节、4字节或8字节。进程A持有锁时把值从0改成1;进程B发现值不为0,就不停地循环检查,直到值变回0。这种"忙等待"的目的是不让出CPU,避免上下文切换和Cache污染。 + +问题在于:当多个进程同时竞争同一把自旋锁时,后果远不止是CPU空转。 + +### 2.2 CPU核间通信风暴 + +假设16个核,Core 0持有锁,另外15个核在同时自旋等待。当Core 0要释放锁(将1改为0)时,会发生以下连锁反应: + +1. Core 0需要向其他15个核广播**Invalidate消息**,通知它们自己L1/L2 Cache中该锁变量的副本已失效 +2. 等待所有核确认后,Core 0才能将变量修改为0 +3. 15个等待的核立即向Core 0发送**Write Update消息**,请求获取最新的变量值 +4. CPU内部仲裁后,某个核(如Core 9)获得修改权限,再向其他15个核广播**Write Invalidate消息** +5. 待全部确认后,Core 9将变量改为1,持有锁 + +一轮锁释放与重新获取,涉及数十次核间消息广播。在16核规模下尚且如此,当代CPU动辄几十乃至上百核,这种开销会急剧放大。**一轮轮消息同步,足以让i9的性能"退化"到386水平。** 这就是所谓的"锁风暴"——热点竞争造成的阻塞被核间通信延迟进一步加剧。 + +这个问题并非增量检查点独有。PG中所有使用自旋锁的地方,一旦产生竞争,都可能触发同样的核间通信风暴,造成性能抖动。 + +### 2.3 改进思路 + +解决方案的灵感其实来自于CPU自身的缓存一致性协议和RAC的缓存融合机制。核心思路很简单:为每个核分配独立的锁变量,自旋时每个核只轮询自己的变量,互不干扰,因此无需广播消息。 + +释放锁时,持有者只需要向获得锁的目标核的私有变量发送一次修改消息,完成所有权转移。这从O(n²)量级的核间通信降到了O(1)。相关学术研究可参考论文《Non-scalable locks are dangerous》——传统自旋锁在大规模多核系统中的可扩展性问题早有定论,只是实践中容易被忽视。 + + +## 三、增量检查点与FPW:页分裂问题的实测对比 + +在PG中,完全检查点和FPW是强绑定的。引入增量检查点后,完全检查点的频率被大幅拉长,这会对FPW保护页分裂的能力产生什么影响?要回答这个问题,得先搞清楚FPW到底在解决什么问题,以及其他数据库是如何处理的。 + +![img](2.png) + +### 3.1 什么是页分裂(Partial Write) + +数据库的页(如PG的8KB)在操作系统层面通常由多个OS页(如4KB)组成。当数据库发起一个8KB的写操作时,在存储层实际上是两次4KB写入。如果写入中途断电或系统崩溃,可能出现前4KB写入成功、后4KB未写入的情况——这个数据库页就变成了"一半新一半旧"的损坏状态,这就是**页分裂**。 + +### 3.2 如何模拟页分裂 + +长期以来,页分裂问题难以验证,因为真实场景下除了拔电源几乎无法复现。但通过eBPF/systemtap等内核动态跟踪工具,可以拦截`pwrite`系统调用,将写入长度参数从8KB篡改为4KB,操作系统就会乖乖地只写一半——完美模拟页分裂,且完全排除其他干扰因素。 + +我们分别对Oracle、PostgreSQL和MySQL执行了相同的测试。 + +### 3.3 Oracle:软件层面不处理 + +拦截`pwrite`后,Oracle在检查点刷脏时检测到I/O错误,直接崩溃。重启后开始实例恢复,定位到检查点位置,识别出需要恢复的脏块——然后恢复失败。 + +测试结论很明确:Oracle在软件层面并不解决页分裂问题。它既不依赖文件系统的原子写,也不在代码中做特殊处理。Oracle的策略是检测到损坏后依赖备份进行介质恢复,并提供了BlockRecover工具用于单块恢复。**将问题转移给运维,本身就是一种选择。** + +### 3.4 PostgreSQL:彻底解决 + +同样流程下,PG在I/O错误后并没有崩溃,仅报告错误。我们用`kill -9`杀掉所有进程模拟意外宕机,重启后PG从控制文件中读取检查点位置,应用对应的WAL日志——数据完整恢复,无一丢失。 + +**PG通过FPW机制,在第一次修改脏页时将整个页写入WAL,确保即使发生页分裂,也能通过日志完整重做。** 代价是明显的I/O放大,但换来了数据一致性的确定性。 + +### 3.5 MySQL(InnoDB):双写的局限 + +MySQL InnoDB使用双写机制:先将页写入双写缓冲区,再写入实际数据文件。测试发现: + +- 如果只拦截目标表文件的写入,双写可以恢复 +- 但如果拦截系统表空间(如undo表空间)的写入,**数据库启动报错,无法恢复** + +结论是:**双写解决了部分场景下的页分裂问题,但在系统表空间受损时无能为力。** 在真实的"断电+系统表空间写入被截断"场景下,双写并不能保证数据库一定能起来。 + +### 3.6 三库对比小结 + +| 数据库 | 方案 | 是否真正解决页分裂 | +|--------|------|-------------------| +| Oracle | 依赖备份与块恢复 | 软件层不解决 | +| MySQL | Double Write | 部分解决,系统表空间损坏时失效 | +| PostgreSQL | Full Page Write | 彻底解决,有性能代价 | + +三个主流数据库中,**只有PG以性能为代价,真正从软件层面解决了页分裂问题。** Oracle将问题推给硬件/运维,MySQL的双写在关键路径上存在盲区。 + +回到TC架构的实际情况:底层自研共享存储支持原子写,因此在TC中FPW可以关闭。但如果用户没有原子写存储,FPW真的是可有可无的吗?这个问题没有一个标准答案,有兴趣的同学,不妨按照本次分享中的步骤,实际模拟下页分裂,深入体会下底层原理,再做定论。 + diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/authors.yml b/i18n/zh-CN/docusaurus-plugin-content-blog/authors.yml index 4019552..5de07f4 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-blog/authors.yml +++ b/i18n/zh-CN/docusaurus-plugin-content-blog/authors.yml @@ -22,6 +22,10 @@ ZhangChen: name: ZhangChen 陶郑: name: 陶郑 +萧少聪: + name: 萧少聪 +吕海波: + name: 吕海波 Oreo Yang: name: Oreo Yang 矫顺田: diff --git a/static/img/blog/covers/incremental-ckpt.png b/static/img/blog/covers/incremental-ckpt.png new file mode 100644 index 0000000..0764015 Binary files /dev/null and b/static/img/blog/covers/incremental-ckpt.png differ diff --git a/static/img/blog/covers/vibe-coding-en.png b/static/img/blog/covers/vibe-coding-en.png new file mode 100644 index 0000000..d361f4f Binary files /dev/null and b/static/img/blog/covers/vibe-coding-en.png differ diff --git a/static/img/blog/covers/vibe-coding-zh.png b/static/img/blog/covers/vibe-coding-zh.png new file mode 100644 index 0000000..6adcaf7 Binary files /dev/null and b/static/img/blog/covers/vibe-coding-zh.png differ