Meet NLT — true — generator

The disadvantage of any success is it wears off quicker than you worked for it. I finished NLT true generator yesterday, and while working today on some tutorial-example I already forgot how good it felt to see it finally in action.

It brings more speed (it is about x2 faster) but it also provides detection of used arguments in parser actions. Thanks to that the generated code is more compact — no dummy proxies. Another feature added in this version is altogether groups (I didn’t know how to name those better). Consider parser rule with set group:

[ TYPE , ASSIGN EXPR ]

This will create two productions — one with “TYPE”, and the other with “ASSIGN EXPR” symbols. Altogether group works just like set group with one extra production — all elements in given order:

< TYPE , ASSIGN EXPR >

You will get “TYPE”, “ASSIGN EXPR” and “TYPE ASSIGN EXPR” productions — I use exactly this form when parsing variable declaration in Skila.

One feature that didn’t make its way to this release was shuffle mode (think of “public static” / “static public” expressed as one rule) — it is a nice thing to have, but it can wait a bit more.

And you have it — big change, short news…

Advertisement

Polishing the parser generator

The best way to get a reason for improvement is using your own program — currently I constantly switch between Skila and NLT parser generator, because every feature I add makes my work on Skila much easier.

I added types definitions, so no longer I have to cast the symbol objects in the code blocks:

types
  expression Expression;
  IDENTIFIER IdentifierSymbol;
end

With such definitions I can use symbol object “expression” and C# code will know it is of type “Expression”.

With sets and sequences I can reduce amount of rules significantly — sets:

fparam -> [ r:REF s:SINK e:EXCL ]? id:IDENTIFIER 
          { new FunctionParameter(...) };

Here we see an optional set (very similar notion to regular expressions) — identifier can be preceded by “PARAM_REF” or “SINK” or “EXCLAMATION” or (set above is optional) nothing.

Once I implemented set, sequence was an obvious addition:

field -> id:IDENTIFIER t:type (EQ ex:expression)?
         { new ClassField(...) };

Type can be followed by “ASSIGN” and “expression”. Unlike set it is everything (entire sequence in given order) or — in case of optional sequence, like here — nothing.

On Skila front I made some progress working on reference types (class) but finishing it will take some more time.

Grammar generator is coming to NLT

For now not very powerful, but already capable of more than existing builder. OK, it is not really generator, it just rewrites the rules into builder method calls but since I focus solely on grammar syntax it is fine with me.

Let’s take a look at the core — production rules:

data_type ->
             STRUCT
             { DataTypeEnum.ValueType }
           | CLASS
             { DataTypeEnum.ReferenceType }
           ;

The syntax is way more compact than written in pure C#, on the other hand the file format is not supported by any IDE so you cannot refactor the grammar or the code (enclosed in braces).

Of course this “generator” is used to build parser for Skila — and now I can write productions with optional symbols:

deco_identifier -> sink:SINK? name:IDENTIFIER
    { new IdentifierDecorated((IdentifierSymbol)name)
      { WithSink = (sink!=null) } };

Probably not the best example (I am writing this entry right after testing and uploading NLT), but it shows you can squeeze two productions into one — it is up to user which form is more readable.

This ability is the main reason why I created parser generator for NLT — one option in production is no brainer, but take two and you will have four productions. The number of rules quickly explodes as you have more and more options inside production.

Behind the scene single rule as shown above is rewritten into two regular rules — the difference is it is done by computer, not human. And with generator it is easy to implement because all it does is writing text — there are no boundaries for it. Handling such case with builder (just C#) is doable but leads to huge amounts of code — each permutation of optional symbols has to be created in advance by hand and put inside builder.

Since further enhancement of the builder would be a really fruitless effort I decided to go with “generator”. Next I will try to add value constraints, groups, then integrate lexer patterns and as last step I will make it true generator.