Design proposal: opt-in support for maps larger than 256×256

[mrallanodreman]

Design proposal: opt-in support for maps larger than 256×256

Hi @sven-n, and thank you — both for OpenMU itself and for the unusually
detailed review on #867. You were right on every count, and I want to start by
owning the mistake: I opened that PR far too early, as one big lump, before I
understood how a change like this should enter the project. Your feedback gave me
the map I was missing. So this is me stepping back and doing it the right way —
opening the design conversation first, before any more code.

I'm closing/leaving #867 aside as you suggested and splitting the useful parts
out (see "Immediate small PRs" below).

And honestly — I'm genuinely excited to share where this got to. We got large maps
actually rendering in the client, terrain height sculpting working, and it
looks great in-game. That's the part I want to put in front of the community: not a
promise, but something that already runs.

The goal

Let servers opt in to maps larger than the current fixed 256×256 grid, so
map makers can finally build spaces the classic client never could — bigger
continents, raid zones, event arenas, sculpted terrain with real elevation —
without changing anything for the 99% of servers that stay on 256×256. This opens
a whole new canvas for map makers, and the exciting part is we've already seen it
work end to end: server → protocol → client → rendered on screen.

Why this is worth a design discussion rather than just a feature flag: coordinates
are 8-bit across the whole stack, so "bigger maps" touches the data model, the
terrain loader, the wire protocol, the pathfinding, and the tools. It has to be
done as a sequence of reviewable, backward-compatible steps, not one PR.

The client story (the part #867 was missing)

You correctly pointed out that no stock client can use any of this, which is what
makes an out-of-tree protocol dialect unmaintainable for OpenMU. That gap is real,
and here's how I'd like to close it so the server side has a legitimate consumer:

• An open client — and it's already public. We maintain
  LinuxMu, a fork of the Season 6
  client (based on the MuMain sources) that runs
  against OpenMU on Linux and Windows, and large maps render in it today. It's out
  in the open, not a private binary only we can build — it lives on
  main, and you can read the entire
  client-side change in this
  full diff.
  That's the "documented consumer" the protocol was missing — inspectable,
  buildable, and real.

• An open map editor (work in progress) — terrain height editing already works.
  The editor itself isn't finished, so I don't want to oversell it. But one part is
  already working end to end, and it's the part that matters most for "new maps":
  editing terrain height. Concretely, what works today:

  • Sculpting the heightmap with a brush — raising/lowering elevation over a radius,
    writing into the client's per-vertex height buffer (PrimaryTerrainHeight,
    via an AddTerrainHeight(x, y, height, range)-style operation).
  • The client renders the resulting relief: it loads the per-vertex height data and
    RequestTerrainHeight returns the new elevation, so the change is visible in-game.
  • Exporting a validated map bundle (terrain + attributes) and applying it to the
    server as a terrain-only update.

  That's the difference between "a bigger grid" and "actually new maps": not just
  more flat tiles, but terrain a map maker can shape. The rest of the editor (object
  placement, textures, polish) is still in progress; I'm only claiming height editing
  as done.

• Opening up reserved space. Larger maps let servers use map/terrain ranges
  that the fixed 256 grid keeps locked away, which is where a lot of the new
  gameplay potential comes from.

I'm not asking OpenMU to adopt the client or the editor — only flagging that a
maintained, open client + editor exist and will be published, so the protocol
work has a documented consumer instead of being a dialect for one hidden client.

Proposed sequencing

Each step is its own PR, backward-compatible, and reviewable in isolation. Nothing
after step 1 lands until the one before it is merged and agreed.

1. Data model + migration, no behaviour change. Introduce a per-map size
   concept and the persistence migration, with robust terrain-size handling
   and zero behaviour change for 256×256 maps. This is the step you named as
   the reviewable starting point, and I agree it's the right one. Explicitly in
   scope here: an unexpected terrain blob length becomes a loud, specific
   error, never a silent "pad the tail and guess" (addressing the terrain-loader
   concern from the review directly).
2. Coordinate type. Widen the coordinate representation behind the data model,
   keeping 256×256 paths byte-identical on the wire.
3. Protocol. Only once 1–2 are agreed: a properly versioned set of packets
   — not sub-codes that collide with the legacy count byte, and batched like the
   packets they extend. Opcode allocation to be agreed with you up front.
4. Server plumbing + tools. Pathfinding, view plug-ins (no hard-throwing
   checked casts on un-migrated paths — graceful fallback), and the admin panel
   renderer, each migrated deliberately.

What this issue is (and isn't)

To be clear up front: I'm not asking OpenMU to adopt this. I'm sharing the effort
openly, in the spirit of the project — because I think part of the community may
see the potential in larger, sculptable maps: new terrain to build on, new kinds
of gameplay, and reserved map space finally usable.

You rightly asked for an issue describing the goal and the client story before any
code, so here it is. If larger maps are ever a direction OpenMU would want, I'd be
glad to align on the step-1 shape (per-map size + migration, no behaviour change
for 256×256) and do it properly this time. And if it's not a direction the project
wants, that's completely fine — the work stays open and available for whoever in
the community wants to pick it up or build on it.

Immediate small PRs (independent of this proposal)

The two pieces you said you'd take, cleanly on top of current master, each on its
own — already opened:

• Prevent duplicate map entry on repeated F3 12 packet #873 — Duplicate-map-entry guard on repeated F3 12. Confirmed on master:
  ClientReadyAfterMapChangeAsync has no re-entry guard, so a repeated F3 12
  packet re-runs it and calls CurrentMap.AddAsync(this) (and the summon's
  AddAsync + OnSpawn) a second time. Since CurrentMap is set to null during
  a map change and only reassigned in that handler, a non-null value on entry is a
  reliable "already ran" signal.
• Bounds-check client-supplied walk steps against the terrain grid #874 — Bounds check before WalkMap[target.X, target.Y]. Noted honestly in
  the PR: on master Point is (byte, byte) and WalkMap is [256,256], so the
  index can't go out of range yet — it's defensive hardening that becomes
  load-bearing once coordinates widen. Kept separate from this proposal so you can
  take or leave it on its own merit.

Thanks again for taking the time to explain the why and not just the what.

[sven-n]

Thanks for taking the reset seriously — this is the right way to start, and the
demo you describe is genuinely interesting. I'm open to supporting larger maps as
an opt-in feature, but I want to be upfront about the conditions, because the
previous attempt showed where this gets hard.

1. It has to be complete

The thing I noticed immediately when looking at #876: coordinates are not limited
to the handful of packets needed to walk around. Counting the packet definitions
in our XMLs and leaving the old 075/095 variants aside, roughly 30
server→client and 10 client→server packets carry x/y fields — #876 covered 8 of
them. The rest are not exotic:

• drops: ItemsDropped, MoneyDropped / MoneyDroppedExtended
• skills: AreaSkill, AreaSkillHit, AreaSkillAnimation
• ChangeTerrainAttributes (Start/End), PartyList, TeleportTarget,
  EnterGateRequest, InstantMoveRequest, DropItemRequest
• mini games and events: ChaosCastlePositionSet, IllusionTempleState,
  the Castle Siege mini map/NPC/machine packets, fireworks and fanfare

To be clear about what doesn't need to change: the 075 and 095 packet
variants stay exactly as they are. Those serve old clients which will never see a
large map, so widening them would be pure churn.

Not every one of the remaining packets necessarily has to be widened on day one —
but the rule for which ones do has to be explicit, and for the rest it has to
be provable that they can never be reached on a large map. "It works when I walk
around" is not the bar.

The same applies outside the protocol. A large map that can't be edited or
inspected is not a feature we can ship:

• Map editor (src/Web/Shared/Components/MapEditor): MapCoordinateService
  has a hard MapSize = 256, and the drag/zoom/percentage math, the object
  resizer and the export/import JSON schema all build on it.
• Live map (src/Web/Map): World.ts has sideLength = 256, MapApp.ts has
  MAP_SIZE = 256, and TerrainShader.ts does its texel math against 256.
• Terrain rendering for the admin panel (GameMapTerrainExtensions) and the
  Network Analyzer views.
• Generated docs: the ~500 files under docs/Packets come out of the XML via
  the generator. They must be regenerated, never hand-written.

And the data/terrain side: GameMapTerrain does terrainData.AsSpan(3) and
allocates three fixed [256,256] grids; gates, MonsterSpawnArea, Rectangle,
IMapArea, Character.PositionX/Y, the chat commands, the bots, Chaos Castle's
shrinking and the Castle Siege NPC controller all assume the byte range. Your
point about a loud, specific error on an unexpected terrain blob length is
exactly right and I'd like that to be non-negotiable.

So: "done" for me means a map maker can create a large map, edit it in the map
editor, watch it on the live map, walk it, and have monsters spawn, path, aggro,
drop items and cast area skills on it. Otherwise it's a demo, not a feature.

2. It has to be consistent

• Endianness. feat: global map coordinates for large maps (512x512+) #876 mixed ShortBigEndian (WalkRequestGlobal) with
  ShortLittleEndian (everything server→client), and that mix has crept into
  previous PRs too. For new fields I'd prefer LittleEndian throughout. The reason
  is simply that it's the native memory layout on x86 (and ARM as we use it), on
  both the server and the client: a little endian field can be read and written
  as a plain memory access, while big endian needs a byte swap on every read and
  write, and it's one more thing for a client implementer to get wrong. The
  legacy packets are what they are — but where we get to choose, let's choose the
  layout that parses for free, and let's choose it consistently.
• Naming. "Global" as a suffix is odd — nothing about these coordinates is
  global; they're still per-map coordinates that simply don't fit in a byte. It
  also doesn't match our existing convention, where a suffix denotes the client
  dialect (075, 095, Extended). Same for GlobalPoint / LocalPoint /
  ChunkId in the game logic: that vocabulary promises a chunked world design
  that isn't what's actually being built. Let's agree on the names before the
  first packet is written.
• Opcodes. Allocation has to be agreed up front and registered in
  PacketHeaders.xml, must not collide with the legacy count byte of the C2
  packets, and should ideally be a reserved contiguous range so future official
  packets don't run into it.

3. No downsides for the default byte case

This is my hardest condition. I don't want a ushort Point in the default
build. Point is a packed 2-byte struct, FullGridNetwork allocates
Node[0x10000] and indexes with (Y << 8) + X, BucketMap sizes itself from
0x100 / bucketSideLength, and every map holds three 256×256 grids. Widening
unconditionally doubles the cache pressure on the hottest paths we have.

And yes — I think there is a clean way, and it's the type alias rather than
generics. SharedGlobalUsings.cs is linked into every project by
Directory.Build.props, so:

#if LARGE_COORDINATES
global using Coordinate = System.UInt16;
#else
global using Coordinate = System.Byte;
#endif

with record struct Point(Coordinate X, Coordinate Y) and (Coordinate) casts /
Coordinate.MaxValue in the arithmetic. Both alias targets support that, so the
same source compiles either way, and in the default build the emitted code is
identical to today — that's the acceptance criterion, and it's verifiable.

A nice side effect: the widened build simply won't compile until every site has
been dealt with, so the compiler enumerates the remaining work for us instead of
checked casts exploding at runtime.

I'd deliberately not use generics (Point<T> where T : IBinaryInteger<T>).
That infects ILocateable, IAreaOfInterestManager, every view plugin interface
and the plugin points, it can't be used in the EF entity model, and
devirtualization across those interface boundaries isn't guaranteed. The alias
buys the same thing without any of it.

Two more consequences of that approach:

• Persistence: widen the stored coordinate columns to int
  unconditionally. One migration, one schema for both builds, zero hot-path
  cost — and the default build then validates on load and refuses a map that
  exceeds its range, loudly, instead of truncating.
• View plugins: no per-value branching (if (x > 255) send the other packet). We already select view plugins per client dialect via
  MinimumClient — the wide packets belong to a dialect, so the plugin set is
  picked once at connection time. In the default build the wide plugins aren't
  compiled or registered at all, which removes both the branch and most of the
  duplication you were worried about. Where a wide variant is genuinely needed,
  it should derive from the legacy plugin and override only the packet writing.

For the grid code, prefer passing per-map dimensions into the constructors over
adding conditionals — BucketMap already has a ctor that takes a side length.

4. Other things to consider

• The client should be MuMain, not a fork. LinuxMu is a fork of MuMain, which
  I maintain and which already contains MuEditor. If the client story is supposed
  to be durable, the changes belong upstream there — otherwise the "documented
  consumer" is free to drift away from what the server expects. Concretely it's
  the same exercise: TERRAIN_SIZE is a compile-time 256 with
  TERRAIN_SIZE_MASK used in ~130 places, the terrain arrays are fixed-size
  globals, and OpenTerrainAttribute validates Width == 255 && Height == 255.
• File formats. .att, the OZB height map and EncTerrain.map all encode
  256 structurally (the .att header stores width/height in single bytes). They
  need a version bump with real width/height fields on both sides, and our
  TerrainData blob needs the same treatment.
• Map size and map number are two different features. feat: global map coordinates for large maps (512x512+) #876 mixed
  "coordinate > 255" with "map id > 255". Please keep map ids out of scope here.
• Object density and scope. View range, aggro and scope are expressed in
  coordinates. A 1024×1024 map at the usual monster density means far more
  objects per map than the client's object list was built for. I'd like a
  paragraph on this in the design before step 1 — per-map configurable ranges
  are probably needed.
• Memory. Three grids per map is ~192 KB today and ~3 MB at 1024×1024, plus
  the precomputed spawn point array and a pathfinder node array growing from 65k
  to 1M entries. That's per map, so it's worth a thought about how many large
  maps a server can realistically host.
• Don't bother with Pathfinding/PreCalculation. That code was an experiment;
  the path finder became fast enough that pre-calculated paths aren't needed
  anymore and they're effectively unused. Its serializers encode a coordinate in
  a single byte, so they'd otherwise look like blockers — they aren't. Leave them
  alone (or we drop them separately), don't spend effort widening them.
• CI. The default build stays the gate, but add a job that builds and tests
  the LARGE_COORDINATES configuration so it doesn't rot — plus round-trip
  packet tests proving the default build's bytes are unchanged.
• PR hygiene. Small PRs, and descriptions that claim only what was actually
  tested. Bounds-check client-supplied walk steps against the terrain grid #874 was honest that the bounds check can't trigger on master today —
  that's the standard I'd like everywhere. Generated files must come from a
  generator run, not be edited by hand.

Where I'd start

If you're fine with all of that, let's nail down step 1 concretely, and it
contains no protocol and no client at all:

1. the Coordinate alias plus the Point change, with a byte-identical
   default build;
2. per-map size in the data model, int columns, one migration, and strict
   terrain-blob validation with a specific error;
3. the tools (map editor, live map, terrain rendering) reading the map size from
   the definition instead of a constant.

Once that's merged and stable, we talk about opcode allocation and the wire
format.

---

Generated by Claude Code