Planned changes to Mindcode

[cardillan]

Preamble

This describes currently contemplated upcoming changes to Mindcode. Comments and suggestions are welcome.

The document is updated occasionally. Already implemented changes are removed from this description during updates.

Change in default language target

The default language target will be bumped to 8 as soon as regular Mindustry 8 release is made.

Roadmap

A broad plan for the next few releases. Everything may change, though.

See changelog in the development branch for a list of features ready for the upcoming release.

Upcoming release

Changes intended for the next regular release.

• Full support for parallel execution in the emulator
  • In the web app, all four modules will support running the code
  • In the tool app, two new additional actions: run mlog (rm) and run schematics (rs).
• break in code blocks
• If expression telescoping optimization

Next releases

These are topics that I'm spending time thinking about. One of these should appear in the next release after the upcoming one.

• Partial loop unrolling optimization
• Instruction reordering optimization
• Rethinking of initialization code and the implicit loop. Some initialization code really shouldn't be part of the implicit loop; even the end() funcion should avoid them (and should, therefore, be implemented as a jump and not as end). Candidates for initialization code:
  • Compiler initialization (processor ID, target guard, setrate)
  • Guards for linked variables
    • This is problematic because these may be intertwined with normal global declarations
  • Global variable initialization??
  • Code blocks in modules
  • Code blocks in the main program??
    • For consistency with modules, we might introduce a main function after all.
  • Strict-compliant code compiled as relaxed must produce the same results as strict code. This complicates global initializations a lot.
• Modules/namespaces (see Modules #149)
  • Implement public/private modifiers for variables, functions, and constants.
  • Refactoring of Mindustry Logic functions
• Variable types
  • "Natural" types
  • Structures/records, including arrays of structures
  • Enums
  • Local arrays/structures
  • Function pointers/array pointers

Active

These are topics that I'm currently planning to implement.

• Redesign of optimization options
• AST interpretation for much more powerful compile-time evaluation.
• Optimization engine rewrite
  • An overhaul of all existing optimizations (similar in principle to the recently completed compiler rewrite) would be very useful. Possibly written from scratch, with no links to the code structure generated by Mindcode (it could optimize any mlog code). Potentially a base for a decompiler as well. No commitment yet, as it would be a lot of work.
• Terminology: intrinsic functions
• Further optimizations (maybe in the new optimization engine)
  • Array access fusing
  • Array operation injection
  • Linear transformation of loop control variables and conditions (induction variables)

Small tasks backlog

These are relatively small tasks that might be slipped in when I feel like it.

• Detection of unexpected processor restarts
• Verify (using unit tests or extended tests) that the dynamic optimizations estimate optimization costs well (i.e., the resulting code must not be larger than the initial estimate).
• Storing precompiled AST trees of system functions
• Allow specifying URL, not just port, for watcher (offline)
• Output variable mapping (source code names to mlog names) to a log file
• Extended testing:
  • Avoid repeated compilation
  • Add support for running tests for a predetermined time instead of a specific number of test cases.
• Customizable variable naming schemes

Shelf

These are topics that I've spent some time thinking about and still want them, but they're in a distant future at best.

• Implement some way to parametrize schematic creation:
  • Alternative 1: add parametrization or processing to Schemacode. E.g., a loop that will create a column or row of blocks. Needs way to express loops and basic expressions evaluation.
  • Alternative 2: provide some means to generate schematics from Mindcode. E.g., a Mindcode program, run by the emulator, creates a schematic, populates it with blocks, sets block configurations, and writes the result into a file.
• Autolinked variables
• fallthrough and yield in case expressions
• Memory model

Issue details

Autolinked variables

Support for declaring a variable as a block of a certain type, e.g.

linked @message msg, @switch button, @memory-bank storage;

Mindcode would generate code to iterate through linked blocks and automatically assign them to variables:

• msg would be assigned the first block of type @message
• button would

Refactoring of Mindustry Logic functions

Implementing namespaces would allow better organization of Mindcode functions into related groups (e.g., separate namespace for world-processor functions).

The current system would be accessible in one namespace for backwards compatibility, new systems would come with their own namespaces.

Redesign of optimization options

Instead of configuring optimization by individual optimizer, allow each optimization to be configured independently. The options would be structured hierarchically, e.g.:

• opt.if-expression.value-propagation
• opt.if-expression.instruction-propagation
• opt.if-expression.forward-assignment
• opt.data-flow.preserve-globals
• opt.data-flow.preserve-main
• opt.data-flow.constant-folding
• opt.data-flow.constant-folding
• opt.print-merging.use-format
• etc.

Possible values in the set directive would be on, off and force.
force would force the optimization even when it would be deemed dangerous by the optimizer (e.g., print merging using the
format instruction in presence of {0} string literals).

#set directives will set all values in given subtree, e.g.

#set opt = off;                             // Turn off all
#set opt.data-flow = off;                   // Turn off just the data flow
#set opt.data-flow.constant-folding = off;  // Turn off just the constant folding

Furthermore, the general optimization levels won't be used to choose the optimization 'intrusiveness' or 'riskiness,' but to choose how much effort is put into optimizations. On lower levels, more expensive optimizations (in terms of compiler/optimizer effort) will be skipped.

fallthrough and yield in case expressions

The fallthrough keyword can be used in a when branch in
case expression to transfer the control to the next branch:

case number
    when 1 then 
        print "one";
        fallthrough;
    when 2 then
        print "two";
end;

When number is 1, the code will print "onetwo". When number is 2, the output will be just "two".
fallthrough can be used anywhere in the when branch, for example, inside an if statement.

The
yield keyword can be used in a when branch in case expression to terminate the evaluation of the case expression and provide the resulting value (the code will output "even"):

number = 4;
output = case number
    when 1 .. 10 then
        if number % 2 == 0 then
            yield "even";
        end;
        print("not even\n");
        "odd"; 
    else
        "too big";
end;
print(output);

The value after yield will be optional; when not given, the resulting value of the case expression will be null.

Memory model

Optimizations of external variables are planned for the future. These should prevent unnecessary reads and writes from/to memory blocks. To help the compiler with these optimizations, it will be possible to assign a memory block or a variable one of three possible memory models:

• volatile: this means that the external memory may be modified by another processor, and therefore reads and writes from/to this memory must not be eliminated from the code.
• aliased: this means that another processor will not modify the external memory, but the memory block might be accessed using two different variables.
• restricted: this means the memory block will never be accessed using two different variables.

A memory model can be assigned to a memory block, or to a concrete slot inside a memory block. Model assigned to a slot has precedence over a model assigned to the whole block. Conflicting assignments on a block or slot level cause compilation error. Declaring a variable as restricted when it can be inferred from a code it is actually aliased causes a compilation error.

Example (the syntax is outdated and wonky):

param MEMORY1 = bank1;
param MEMORY2 = bank2;

external MEMORY1[0] $state;

declare MEMORY1 restricted;
declare MEMORY2[0 ... 256] volatile;
declare MEMORY2[256 ... 512] aliased;

declare $state volatile;

external MEMORY1[0] $state2;
declare $state2 restricted;             // Error: we know $state2 is an alias to $state

Enums

It will be possible to declare enums using the enum keyword:

enum enum_name is element1, element2, element3 end;

Mindcode will assign values to the enums as it sees fit. There are no guarantees on the numbers whatsoever, except preserving the declaration order. They could be instruction addresses inside a case expression, for example, if there's just one case expression.

Mindcode will provide functions to access enum properties (e.g. element.name, element.next,
element.previous). They might be a bit costly to use.

Support for enums in list iteration Loops: for i in enum_name.

Function pointers

Note

This text is outdated and doesn't incorporate the upcoming adoption of types in Mindcode and the possibility of implementing lambdas/function pointers through interfaces.

Function pointers would allow storing a function address in a variable and calling the function through that variable. The internal mechanism is briefly described in the road map.

Assigning a function address to a variable could happen in several ways:

def foo()
    print("foo");
end;

fptr1 = foo;                // Note: no parentheses

fptr2 = def bar() 
    print("bar");
end;

// Lambda syntax
fptr3 = () -> print("baz");
fptr4 = n -> print("n");    // No parentheses required for lambdas with single parameter 
fptr5 = (x, y) -> x + y;

// Code block needs to be used if the lambda is more than one expression
fptr6 = n -> begin
    sum = 0;
    while n > 0 do
        sum += n--;
    end;
    return n;               // Lambda is just another way of writing functions, return is allowed
end;

// Code block is not needed if the lambda is just one expression, even a complex one
fptr7 = n-> while n > 0
    println(n--);
end;

// Lambdas with output parameters
fptr8 = (x, out y) -> begin y = x * x; return y + x + 1; end;

// Calls via the function pointers
fptr3();
fptr4("Hello");
print(fptr5(5, 7));

// Will print numbers 10 to 1 in descending order and then null, as loops always returns null.  
print(fptr7(10));

// Mindcode lambdas will probably have read-only access to the outside variables:
status1 = () -> printf("Running: $count reactors active\n");
status2 = () -> printf("Shutdown: cryofluid level $level too low\n");

print_status(running ? status1 : status2);

def print_status(fptr_status)
    ...
    println("Power plant status:");
    printf("Energy consumption: $consumption\n");
    fptr_status();
    printf("Battery level: $battery\n");
end;

[cardillan]

Two more brief ideas:

• Support for combining source files. Probably by including files at the source level (e.g. C-like #include). Implementing it with ANTLR won't be straightforward, unfortunately.
• Support for Ruby-like condition modifier, e.g print("debug") if debug;. Ifs with just one statement are hard to write otherwise. Or maybe we could abuse the ternary operator by making the else branch optional: debug ? print("debug"); I may like the latter a bit more.

[JeanJPNM]

I think that having an approach similar to python modules can be more advantageous, because #include usually relies on other compiler directives like #ifdef to not generate the same code twice, which can happen when there are cyclic dependencies

graph BT;
    A-->B;
    A-->C;
    B-->A;

Loading

or when two files share a dependency.

graph BT;
    A-->B;
    A-->C;
    B-->D;
    C-->D;

Loading

With modules, the reasoning is simpler because you can guarantee that each module is only included once,
with the order being the same as a depth first search on the module graph.

In practice, you can do the same with #include, but it brings the previously mentioned connotations from
its most famous implementation in C and C++. So people may expected it to behave in the same way because of its name.

[cardillan]

That's a good suggestion! I've already read about your plans to add modules to mlogjs some time ago and my solution is now posted at the bottom of this discussion. Your project has inspired me a lot during the development of Mindcode - thanks :)

[tom-leys]

Wow! You have a lot of ideas for improvement.

Should global variables be explicitly declared (e.g. declare STATUS global) instead of relying on the upper-case convention for global variable names?

I dislike the current system. I'd prefer that all variables declared outside a method (in the top level) are considered global.

There should be a main function (def main) (case insensitive) which is the program's entry point. Rather than putting all code at the top (global) level.

If you don't have main, then you might make every line of code at global scope automatially "in main"- this is a copy of what C# does recently.

So if you have a variable in main, it's local. If it is outside of main, it's global. Done.

The $ prefix of external variables is the same as the $ formatting character in the printf function. Should one of them be changed, and how?

Never used external variables (don't know what these are) but sounds like you want a declare statement for these (based on later ideas you mention)

I'd like to add a variant of printf which would output a newline character in addition to the normal output. What's a good name for such a function: printfln, printlnf or something else entirely?

Yeah, I think printlnf makes sense here. Or just make all print statements printf (support $). An alternative - Python lets you make any string formatted with f"blah blah" (note the f prefix) so you could go that way - i.e str = f"hello $var"; println(str) is valid.

I plan defining processor-variables backed arrays by just the array keyword. Should arrays be declared using the allocate or declare keywords instead?

Dislike the word allocate here because two CPUs might be sharing memory and it sounds like this CPU is authorative when it is not.

Declare makes sense. Though we might have a lot of declares... You'll need declare memory and other similar combos.

Mindcode currently has a list of variable names that may represent blocks linked to the processor, such as cell1 or switch5. This list needs to be kept in sync with list of existing blocks in Mindustry, and adding new types of blocks may require releasing a new version of Mindcode. Block types defined by mods are entirely unsupported. Would it be better to require explicitly declaring such variables (declare cell1 linked) and treat other variable names as normal?

I think it is good form to copy these linked components to named variables in the code anyway. You can achieve both with the right declare statements.

There is a gotcha here though.

My prefered code structure is

<CONSTS>

loop{
  <COPY LINKS>
  code
}

If you don't put the copy of linked components in the loop, those links will break if the relevant structures are built after the CPU starts looping (or if they are destroyed and replaced).

[tom-leys]

Enums

Might be useful to have a function say int() which outputs 0,1,2,3 for different enum labels. And one say str() which outputs the relevant tag as a string for debugging.

(Pointers to arrays might, perhaps, come one day).

You could achieve this (probably) with a "pointer" being a pair of - a link to the relevant array function (jump table) and the length of the array. From there you can dynamically compute the jumps required. Length can also be used for range checks.

Structs would indeed be nice to have.

[cardillan]

@tom-leys Thanks for your feedback! Having these conversations helps really a lot.

I think it is time I flesh out my objectives explicitly here:

• Using variables without declaration unless they're special in some way.
  • Global variables and external variables (i.e. variables in memory cells/banks) may be considered special.
• Using linked blocks without declaration, if possible.
• Supporting simple programs with no boilerplate at all.
  • By simple I mean a program which is just a sequence of expressions/statements. No functions, no external variables. Global variables wouldn't be needed in such a program.
• Not mangling variable names when compiling to mlog when possible.

There should be a main function (def main) (case insensitive) which is the program's entry point. Rather than putting all code at the top (global) level.

If you don't have main, then you might make every line of code at global scope automatially "in main"- this is a copy of what C# does recently.

This makes sense, and having a main function would automatically enforce a nice structure of declarations, but I don't want to have to support two different ways of doing this. Given the choice I'll opt for global scope main code. A small plus is that variable names in functions are mangled, while variable names outside functions are not, which is consistent and feels intuitive.

Code blocks will allow you to enclose the main code in begin ... end; incidentally this is how Borland Pascal did it back in the day (feels old :D). It won't add any functionality at all, it just might look nicer.

So if you have a variable in main, it's local. If it is outside of main, it's global. Done.

As there would be no code outside of main, global variables would have to be declared anyway - there would be no other way to put them there. So having to declare globals explicitly seems natural now.

Never used external variables (don't know what these are) but sounds like you want a declare statement for these (based on later ideas you mention)

External variables represent memory cell/bank slots. More about them is here: Variables and constants.

In Mindcode as designed by @francois, no variables were declared, even the external ones. So they needed the prefix to be identifiable. I still kinda like it - they're such a different beasts that having them stand out in the code seems useful to me.

I'm now tending towards declaring them explicitly, in which case the prefix can be dropped. Again, users can choose whatever convention they want for them.

An alternative - Python lets you make any string formatted with f"blah blah" (note the f prefix) so you could go that way - i.e str = f"hello $var"; println(str) is valid.

I originally wanted to do this, but as there isn't any string manipulation in mlog whatsoever, everything needs to be resolved at the compile time. I couldn't handle the f"..." strings being assigned to a variable or a function parameter, for example. There's an exception for inline functions, I think, where a string passed as an argument can be later passed to printf.

However, maybe I could keep just print and println, and if the first argument is a formattable string, it would trigger the formatting variant. I might like it more if the prefix of such string was $, as this would correspond with the formatting directive inside the string. So, for example println($"Position: $x, $y");. Passing the formattable strings to an inline function might be more problematic, though.

A global replace of printf(" with print($" would handle this transition. And I could still keep printf around and just make it deprecated.

Dislike the word allocate here because two CPUs might be sharing memory and it sounds like this CPU is authorative when it is not.

Declare makes sense. Though we might have a lot of declares... You'll need declare memory and other similar combos.

I'll keep allocate for allocating heap and stack in external memory. I'll use declare for everything else - globals, external variables, external/internal arrays. var would be shorter, but declare feels better, as not everything declared like this will be a variable (arrays are more like global static structures than variables).

I think it is good form to copy these linked components to named variables in the code anyway. You can achieve both with the right declare statements.

There's a particular problem, though: say I use wave1 as a variable. However, there's a block which can be linked to a processor under that name. When that happens, even by mistake, your wave1 variable becomes a linked block - and the program will probably break down. This is why I prefer to use an automatic mechanism for identifying linked blocks. I'll probably use another mechanism for adding a new word to the list of reserved words that may represent a linked block. It would be either declare balloon linked, or maybe #linked balloon, both implicitly making balloon1 a linked variable. I prefer the latter, as the command basically extends Mindcode's metadata, so it is more like a compiler directive.

I might allow to declare linked blocks explicitly, say declare switch1 linked, but it might be confusing, as it would do precisely nothing. Perhaps a strict syntax option could be added which would require all linked blocks to be explicitly declared, and that would prevent using variable names reserved for linked blocks, but undeclared, from being used an any way.

If you don't put the copy of linked components in the loop, those links will break if the relevant structures are built after the CPU starts looping (or if they are destroyed and replaced).

Yes, I've run into this as well. I don't think this can be resolved at the language level, but I'll add a chapter to tips and tricks.

I usually load blocks dynamically to solve both delayed building and dynamic block linking (where the name isn't fixed anyway) issues:

while MEMORY == null || DISPLAY == null
    print("Item/Liquid/Energy Level Display")
    n = @links
    while n > 0
        n = n - 1
        block = getlink(n)
        print("\nFound: ", block)
        case block.type
            when @large-logic-display       DISPLAY = block
            when @message                   MESSAGE = block
            when @switch                    SWITCH = block
            when @memory-bank, @memory-cell MEMORY = block
        end
    end
    
    printf("\nMessage: $MESSAGE\nSwitch: $SWITCH\nMemory: $MEMORY\nDisplay: $DISPLAY")
    if MEMORY == null || DISPLAY == null
        print("\nNo memory or display.")
    end
    printflush(MESSAGE)
end

[gecgel]

If a main function feels like too much boilerplate, how about an entrypoint scope like in HLML. Putting globals and locals of main in the same scope feels like a step backwards.

// globals
entrypoint
  // locals
  // code
end

It is writing only 2 extra keywords to gain better structure.

[cardillan]

This could work - I'd only prefer the begin keyword, making it a code block:

declare x;   // global
begin
    declare y; // local (optional)
    // main code
end;

I strongly prefer to let users just type in a few statements into the web app and see them compiled. However, this might perhaps be helped a bit if clicking the "start with a new script" link would load something like this into the editor:

begin
    // Type your code here:
    <cursor here>
end;

It would be very important to produce a clear error message if the begin/end were missing. I'm not sure ANTLR will produce that on its own, maybe it will need some help.

Now, since today, we have remarks. And parameters would be declared in the global scope. So we'd have to allow specifying remarks in the global scope too:

allocate heap in cell1;             // doesn't produce any mlog code
remark("Configurable options:");    // generates the remark
param MIN = 10;                     // generates set instruction
param MAX = 100;                    // this too

declare x;                          // doesn't produce any mlog code

remark("Don't modify anything below this line.");
begin
    for i in MIN .. MAX
        print("Here is some actual code");
    end;
end;

It shouldn't be a problem, I guess.

Without the main function/code block, declaring global variables would have to be explicit (declare x global;). That's not very nice.

[cardillan]

If you don't put the copy of linked components in the loop, those links will break if the relevant structures are built after the CPU starts looping (or if they are destroyed and replaced).

There actually might be a way to help this at the language level. I currently envision that linked blocks might be declared in the language either as parameters (and therefore changeable in the mlog code), or as directly used linked blocks. The syntax might look like this:

param memory = cell1;
param display = display1;
linked switch1;

begin 
    ...
end;

So the compiler knows which external blocks are expected, and can automatically generate a loop that would wait for them to materialize, essentially something like this:

do
    memory = cell1;
    display = display1;
loop while memory == null or display == null or switch1 == null;

// main code block

I plan to suggest a feature to Anuken where blocks could have tags settable by a processor which would appear on mouseover. If this gets accepted, such code could include setting the processor's tag to something like "Waiting for cell1, display1 and switch1 to be connected."

[cardillan]

Mindcode modules: moved to #149.

[cardillan]

I'll create issues for individual topic where some concrete idea has already emerged. Relevant passages in the original post will be replaced with a reference to the specific issue.

[cardillan]

Updated the initial post to reflect current developments.

[cardillan]

I've updated the priorities a bit. I want to introduce processor-variables backed arrays as soon as possible, since I believe that currently it is a feature which will bring the greatest benefit.

While pondering the implementation I realized that arrays of records/structures would be very useful performance-wise. Storing structure members in several separate variables would mean that retrieving each member at given index would require separate function call, while accessing a structure in an array would require just one function call. There will still be difficulties and optimization challenges, but structures will have additional benefits, e.g. for function return values.

[cardillan]

Updated the initial after the pre-release of Mindcode 3.0.