After an ugly start — a bug which was introduced with bugfix for another bug — I managed to go straight to the last missing piece in NLT, table driven lexer. It does not mean NLT is finished, but it is complete — all structural features are there and the table for lexer was the last one.
User can specify to compile all regex and string patterns while building a lexer into one big table. When the lexer runs there are no longer repetitions, classes, ranges, just a table with transitions. I was afraid of handling Unicode ranges, but after reading “Regular Expression Matching in the Wild” by Russ Cox I knew what to do, especially I was not building general purpose regex engine, but simplified multi regex one (no back references for example).
Without any optimization at all it is twice as fast as previous approach — checking patterns one by one during runtime. And I didn’t notice any difference in time when building the lexer, which is rather good news.
From now on, NLT is more for maintenance — fixing bugs, optimization, better error reporting, and so on.
Skila 