Debug directives

AGENTS.md

@@ -74,6 +74,7 @@ exactly where the mutable state is supposed to be defined and used.
 
 Never use default arguments in core business logic.
 Default arguments should be reserved for user-facing APIs.
+Do NOT encode default arguments through overloading to skirt this rule.
 
 Keep it DRY: if you find yourself copying and pasting code,
 consider refactoring it into a reusable function or class.
@@ -94,6 +95,10 @@ avoid vague claims about where a "lifecycle" or "responsibility" belongs. If his
 or a rejected alternative matters, include enough context to explain the tradeoff
 after commits are squashed.
 
+Do NOT include comments that simply describe what the code is doing
+without adding any additional insight, or comments that talk about
+parts of the system that are not directly related to the code being commented on.
+
 
 ## User-Facing Diagnostics
 

doc/reference.md

@@ -27,6 +27,7 @@ which is an evolving language that has not stabilized just yet.
 17. [Indentation and Block Syntax](#17-indentation-and-block-syntax)
 18. [Miscellaneous](#18-miscellaneous)
 19. [Built-in Types and Prelude](#19-built-in-types-and-prelude)
+20. [Compiler Debugging](#20-compiler-debugging)
 
 ---
 
@@ -2035,6 +2036,128 @@ parenthesizedIf(x, cond)     // wrap in parens if cond
 
 ---
 
+## 20. Compiler Debugging
+
+The compiler can report what it does in each of its phases, and show the intermediate programs it produces.
+This is configured in the source code, either for a whole file with the `#dbg` directive,
+or for individual definitions with the `@dbg` annotation.
+The output is meant for compiler developers: its format is not stable and may change at any time.
+
+### Debugging Aspects
+
+Each argument of `#dbg` or `@dbg` enables or disables one *aspect*.
+The following aspects trace a phase of the compiler:
+
+| Aspect | Traces |
+|---|---|
+| `elaboration` | elaboration of syntax trees into terms |
+| `lowering` | lowering of terms into the intermediate representation (IR) |
+| `optimizations` | the optimization passes run on the IR |
+
+With the new name resolution (`#lang(0.3.x)`), names are resolved during elaboration,
+so `elaboration` traces name resolution as well.
+
+Parser tracing is available through the diff-test command `:dp`.
+Source directives are read after parsing and cannot enable parser tracing.
+
+The following aspects display a program representation, without tracing:
+
+| Aspect | Displays |
+|---|---|
+| `showParsedTree` | the syntax tree produced by the parser |
+| `showElaboratedTree` | the elaborated term, as a tree |
+| `showElaboratedTerm` | the elaborated term, in readable form |
+| `showLoweredTree` | the IR produced by lowering, as a tree |
+| `showIR` | the IR produced by lowering, in readable form |
+| `showOptimizedIR` | the IR after optimizations, in readable form |
+
+Finally, `showUids` (enabled by default) controls whether traces show the unique identifiers of symbols.
+Disabling it makes traces easier to compare across compiler runs.
+UIDs are local to a compilation unit, so imported symbols use disambiguated names without numeric UIDs.
+
+### Syntax
+
+An aspect is enabled by writing its name as a quoted identifier,
+or enabled or disabled by giving it a Boolean value:
+
+```mlscript
+#dbg('lowering)                         // enable an aspect
+#dbg(elaboration: true, showIR: true)   // enable aspects explicitly
+#dbg(showUids: false)                   // disable an aspect
+```
+
+Unknown aspects and non-Boolean values are reported as errors.
+
+### Output Destination
+
+Debugging output is written to the standard output by default, which `out: StdIO` selects explicitly.
+The `out: File(...)` argument writes it to a file instead:
+
+```mlscript
+#dbg('lowering, out: File("lowering.log"))
+```
+
+As for imports, a relative `File` path is resolved against the directory of the file
+in which the `#dbg` directive or `@dbg` annotation is written, and an absolute path is used as is.
+The first time the compiler writes to a file during a compilation session, it replaces the file's previous contents;
+it then appends all later output, including the output of other files compiled in the same session,
+such as imported files.
+Output from concurrent compilations to the same file may be interleaved.
+
+Any other destination is reported as an error.
+
+### The `#dbg` Directive
+
+The `#dbg` directive configures debugging for the file in which it appears, and must appear at its top level:
+
+```mlscript
+#dbg('elaboration, 'showOptimizedIR, out: File("debug.log"))
+
+fun f(x) = x + 1
+```
+
+Several `#dbg` directives may set different aspects.
+Currently, debugging settings apply to the whole file regardless of where the directive appears,
+so if several directives set the same aspect, the last one takes effect.
+
+### The `@dbg` Annotation
+
+The `@dbg` annotation takes the same arguments as `#dbg`,
+but only applies them to the definition it annotates.
+It can annotate functions, methods, values, classes, modules, and objects:
+
+```mlscript
+@dbg('elaboration, 'showIR, showUids: false)
+fun selected(x) = x + 1
+
+fun unrelated(x) = x   // not traced, not displayed
+```
+
+Annotations combine with the file's settings:
+for example, `@dbg(lowering: false)` stops a lowering trace enabled by `#dbg` for the annotated definition.
+They also apply to the definitions nested in the annotated definition,
+which can override them with their own `@dbg` annotations:
+
+```mlscript
+@dbg('lowering, out: File("lowering.log"))
+fun traced() =
+  fun helper() = 1                        // traced to lowering.log
+  @dbg(lowering: false) fun muted() = 2   // not traced
+  helper() + muted()
+```
+
+The statements that follow an annotated definition are not affected by its annotation.
+The lowering trace of an annotated class also covers the evaluation of its superclass, if any,
+and its companion module, if any, which is compiled together with the class.
+
+The `optimizations` aspect cannot be selected per definition, as optimization passes process whole files.
+Setting it in `@dbg` produces a warning and has no effect; use `#dbg` instead.
+
+Currently, the destination selected by an enclosing `@dbg` annotation only applies to traces:
+the displays requested by the annotation of a nested definition use the file's destination.
+
+---
+
 ## Appendix: Grammar Summary
 
 ```

hkmc2/js/src/main/scala/hkmc2/io/InMemoryFileSystem.scala

@@ -7,17 +7,22 @@ import scala.scalajs.js.annotation.JSExportTopLevel
 
 /**
  * In-memory file system for testing and web compiler. Stores files as a map
- * from normalized path strings to content strings.
+ * from normalized path strings to buffers, so repeated appends do not copy the whole file.
  */
 class InMemoryFileSystem(initialFiles: Map[String, String]) extends FileSystem:
-  private val files: MutMap[String, (Int, String)] = MutMap.from:
-    initialFiles.map { case (k, v) => (Path(k).toString, (0, v)) }
+  private val files: MutMap[String, (Int, StringBuilder)] = MutMap.from:
+    initialFiles.map { case (k, v) => (Path(k).toString, (0, new StringBuilder(v))) }
 
   def read(path: Path): String = read(path.toString)
 
   def write(path: Path, content: String): Unit =
     write(path.toString, content)
 
+  def append(path: Path, content: String): Unit =
+    files.updateWith(path.toString):
+      case Some((ts, buffer)) => Some((ts + 1, buffer.append(content)))
+      case None => Some((0, new StringBuilder(content)))
+
   def exists(path: Path): Bool = files.contains(path.toString)
 
   def getLastChangedTimestamp(path: Path): Long =
@@ -28,20 +33,20 @@ class InMemoryFileSystem(initialFiles: Map[String, String]) extends FileSystem:
   @JSExport("write")
   def write(path: Str, content: Str): Unit =
     files.updateWith(Path(path).toString):
-      case Some((ts, _)) => Some((ts + 1, content))
-      case None => Some((0, content))
+      case Some((ts, _)) => Some((ts + 1, new StringBuilder(content)))
+      case None => Some((0, new StringBuilder(content)))
 
   @JSExport("read")
   def read(path: Str): Str =
     val normalizedPath = Path(path)
     files.getOrElse(normalizedPath.toString,
-      throw new FileSystem.FileNotFoundException(normalizedPath))._2
+      throw new FileSystem.FileNotFoundException(normalizedPath))._2.result()
 
   @JSExport("list")
-  def list: js.Array[Str] = allFiles.keys.toJSArray
+  def list: js.Array[Str] = files.keys.toJSArray
 
   /** Get all files (for debugging) */
-  def allFiles: Map[String, String] = files.view.mapValues(_._2).toMap
+  def allFiles: Map[String, String] = files.view.mapValues(_._2.result()).toMap
 
 object InMemoryFileSystem:
   /** Create an empty in-memory file system. */

hkmc2/js/src/main/scala/hkmc2/io/node/NodeFileSystem.scala

@@ -17,6 +17,9 @@ private[io] class NodeFileSystem extends FileSystem:
   def write(path: Path, content: String): Unit =
     fs.writeFileSync(path.toString, content)
 
+  def append(path: Path, content: String): Unit =
+    fs.appendFileSync(path.toString, content)
+
   def exists(path: Path): Bool =
     fs.existsSync(path.toString)
 

hkmc2/js/src/main/scala/hkmc2/io/node/package.scala

@@ -19,6 +19,7 @@ package object node:
   object fs extends js.Object:
     def readFileSync(path: Str, encoding: Str): Str = js.native
     def writeFileSync(path: Str, data: Str): Unit = js.native
+    def appendFileSync(path: Str, data: Str): Unit = js.native
     def readdirSync(path: Str): js.Array[Str] = js.native
     def existsSync(path: Str): Bool = js.native
     def lstatSync(path: Str): Stats = js.native

hkmc2/js/src/test/scala/hkmc2/CompilerTest.scala

@@ -252,8 +252,7 @@ class CompilerTest extends AnyFunSuite:
     compiler.compile("/B.mls")
 
     given DebugPrinter = new DebugPrinter
-    given TL = new TraceLogger:
-      override def doTrace = false
+    given TL = TraceLogger.silent
     given Raise = diagnostic => fail(diagnostic.toString)
     val prelude = cctx.getPrelude(paths.preludeFile).ctx
     val artifact = cctx.getElaboratedBlock(Path("/A.mls"), prelude)
@@ -275,8 +274,7 @@ class CompilerTest extends AnyFunSuite:
     val fs = new InMemoryFileSystem(loadStandardLibrary())
     given cctx: CompilerCtx = CompilerCtx.fresh(fs, paths, Config.default(io.Path("/")))
     given DebugPrinter = new DebugPrinter
-    given TL = new TraceLogger:
-      override def doTrace = false
+    given TL = TraceLogger.silent
     given Raise = diagnostic => fail(diagnostic.toString)
     val prelude = cctx.getPrelude(paths.preludeFile).ctx
     val map = prelude.builtins.Array.defn.get.body.members("map").asTrm.get.defn.get
@@ -316,8 +314,7 @@ class CompilerTest extends AnyFunSuite:
     val fs = new InMemoryFileSystem(loadStandardLibrary())
     given cctx: CompilerCtx = CompilerCtx.fresh(fs, paths, Config.default(io.Path("/")))
     given DebugPrinter = new DebugPrinter
-    given TL = new TraceLogger:
-      override def doTrace = false
+    given TL = TraceLogger.silent
     given Raise = diagnostic => fail(diagnostic.toString)
     fs.write("/Generic.mls", """#lang(0.3.x, strictResolution: true)
                                  |module Generic with
@@ -353,8 +350,7 @@ class CompilerTest extends AnyFunSuite:
     val fs = new InMemoryFileSystem(loadStandardLibrary())
     given cctx: CompilerCtx = CompilerCtx.fresh(fs, paths, Config.default(io.Path("/")))
     given DebugPrinter = new DebugPrinter
-    given TL = new TraceLogger:
-      override def doTrace = false
+    given TL = TraceLogger.silent
     given Raise = diagnostic => fail(diagnostic.toString)
     fs.write("/Types.mls", """#lang(0.3.x, strictResolution: true)
                                |module Types with
@@ -391,8 +387,7 @@ class CompilerTest extends AnyFunSuite:
     val fs = new InMemoryFileSystem(loadStandardLibrary())
     given cctx: CompilerCtx = CompilerCtx.fresh(fs, paths, Config.default(io.Path("/")))
     given DebugPrinter = new DebugPrinter
-    given TL = new TraceLogger:
-      override def doTrace = false
+    given TL = TraceLogger.silent
     given Raise = diagnostic => fail(diagnostic.toString)
     val prelude = cctx.getPrelude(paths.preludeFile).ctx
     val parameter = prelude.builtins.Array.defn.get.tparams.head.sym
@@ -425,8 +420,7 @@ class CompilerTest extends AnyFunSuite:
     val fs = new InMemoryFileSystem(loadStandardLibrary())
     given cctx: CompilerCtx = CompilerCtx.fresh(fs, paths, Config.default(io.Path("/")))
     given DebugPrinter = new DebugPrinter
-    given TL = new TraceLogger:
-      override def doTrace = false
+    given TL = TraceLogger.silent
     val errors = scala.collection.mutable.ArrayBuffer.empty[Diagnostic]
     given Raise = errors += _
     fs.write("/Unsafe.mls", """#lang(0.3.x, strictResolution: false)

hkmc2/js/src/test/scala/hkmc2/io/VirtualPathTests.scala

@@ -170,3 +170,16 @@ class VirtualPathTests extends AnyFunSuite:
     assert(fs.read("/a/b.mls") == "initial")
     fs.write("/a/c/../b.mls", "updated")
     assert(fs.read("/a/./b.mls") == "updated")
+
+  test("in-memory append preserves snapshots and updates normalized paths"):
+    val fs = new InMemoryFileSystem(Map.empty)
+    val path = Path("/a/b.log")
+    fs.append(path, "first")
+    val snapshot = fs.allFiles
+    val timestamp = fs.getLastChangedTimestamp(path)
+    fs.append(Path("/a/c/../b.log"), " second")
+    assert(fs.read("/a/b.log") == "first second")
+    assert(fs.getLastChangedTimestamp(path) == timestamp + 1)
+    assert(snapshot("/a/b.log") == "first", "Previously read contents must remain immutable")
+    fs.write(path, "replacement")
+    assert(fs.read(path) == "replacement")

hkmc2/jvm/src/main/scala/hkmc2/io/PlatformFileSystem.scala

@@ -8,6 +8,8 @@ private[io] class JavaFileSystem extends FileSystem:
 
   def write(path: Path, content: String): Unit = os.write.over(unwrap(path), content)
 
+  def append(path: Path, content: String): Unit = os.write.append(unwrap(path), content)
+
   def exists(path: Path): Bool = os.exists(unwrap(path))
 
   def getLastChangedTimestamp(path: Path): Long = os.mtime(unwrap(path))

hkmc2/jvm/src/test/scala/hkmc2/CompilerCacheTest.scala

@@ -138,6 +138,7 @@ class CompilerCacheTest extends AnyFunSuite:
         underlyingFs.read(path)
 
       def write(path: io.Path, content: String): Unit = underlyingFs.write(path, content)
+      def append(path: io.Path, content: String): Unit = underlyingFs.append(path, content)
       def exists(path: io.Path): Boolean = underlyingFs.exists(path)
       def getLastChangedTimestamp(path: io.Path): Long =
         underlyingFs.getLastChangedTimestamp(path)

hkmc2/jvm/src/test/scala/hkmc2/MarksTest.scala

@@ -20,8 +20,7 @@ class MarksTest extends AnyFunSuite:
     given ResolutionContext = owner.newResolverState.context
     given Raise = diagnostic => fail(diagnostic.theMsg)
     given DebugPrinter = new DebugPrinter
-    given TraceLogger = new TraceLogger:
-      override def doTrace: Boolean = false
+    given TraceLogger = TraceLogger.silent
     // Path normalization needs a value that retains scope exits. Dynamic values
     // omit those exits because they carry no references into their source scope.
     val resolution = new TypeResolution(Term.UnitVal(), _ => fail("Unexpected type error"))

hkmc2/jvm/src/test/scala/hkmc2/SubstitutionGrowthTest.scala

@@ -47,9 +47,8 @@ class SubstitutionGrowthTest extends AnyFunSuite:
   private final case class Measurement(views: Int, largestSubstitution: Int, activations: Int, typeInstances: Int)
 
   private val cctx = TestFolders.compilerCtx(os.pwd)
-  private given DebugPrinter = new DebugPrinter
-  private val tl = new TraceLogger:
-    override def doTrace: Bool = false
+  private given DebugPrinter = new DebugPrinter()
+  private val tl = TraceLogger.silent
 
   /** Resolves a generated unit through ordinary elaboration, in a fresh state. */
   private def measure(name: Str, source: Str): Measurement =
@@ -129,8 +128,7 @@ class SubstitutionGrowthTest extends AnyFunSuite:
     given owner: Elaborator.State = new Elaborator.State
     given state: ResolutionContext = owner.newResolverState.context
     given Raise = diagnostic => fail(diagnostic.theMsg)
-    given trace: TraceLogger = new TraceLogger:
-      override def doTrace: Boolean = false
+    given trace: TraceLogger = TraceLogger.silent
     given Config = Config.default(TestFolders.mainTestDir(os.pwd))
     given CompilerCtx = CompilerCtx.fresh(io.FileSystem.default, TestFolders.compilerPaths(os.pwd), summon[Config])
     val resolver = Elaborator(trace, TestFolders.mainTestDir(os.pwd), Elaborator.Ctx.empty)

hkmc2/jvm/src/test/scala/hkmc2/TypeRelationTest.scala

@@ -13,9 +13,8 @@ class TypeRelationTest extends AnyFunSuite:
     given owner: Elaborator.State = new Elaborator.State
     given state: ResolutionContext = owner.newResolverState.context
     given Raise = diagnostic => fail(diagnostic.theMsg)
-    given DebugPrinter = new DebugPrinter
-    given trace: TraceLogger = new TraceLogger:
-      override def doTrace: Boolean = false
+    given DebugPrinter = new DebugPrinter()
+    given trace: TraceLogger = TraceLogger.silent
     given Config = Config.default(TestFolders.mainTestDir(os.pwd))
     given CompilerCtx = CompilerCtx.fresh(io.FileSystem.default, TestFolders.compilerPaths(os.pwd), summon[Config])
     val resolver = Elaborator(trace, TestFolders.mainTestDir(os.pwd), Elaborator.Ctx.empty)