Into the valley of nulls

Working on the new design of null brought so many challenges that I didn’t implement any special support for them in generic types — the concept is pretty alien to me and I don’t know what to fix because nothing is broken yet.

Skila features stackable null values with no performance penalty compared to regular values. It means indexOf for String does not use magic value -1 any longer, the pair first/firstOrDefault of the collection is the pattern of the past. All you need to return is optional value.

You can work with optional values safely:

let i ?Int = null;
test(i); // compile error

def test(i Int) Void ...

Can you guess why you will get compile error? Type mismatch? Close, but not enough — types do match here but conditionally (there is precisely one level of Option<T> difference) and this means the entire call is executed conditionally as well. Statement became expression with the outcome failure or success which has to be read or explicitly dropped. Like here:

let i ?Int = null;
_ = test(i); 

def test(i Int) Void ...

For the record there is no magic here, just regular error:

let i ??Int = null;
test(i); // classic type mismatch

def test(i Int) Void ...

There was no need to introduce special syntax for optional chaining (besides there would be asymmetry between optional caller and optional argument), so instead of clunky “?.” you write:

var s ?String = null;
s = s.toLower(); 

which is equivalent of:

var s ?String = null;
s = s is null ? null : s'value.toLower(); 

Unlike pre-6.0 C# no exception is thrown, unlike C# 6.0 the syntax is clear, and unlike Icon you cannot simply ignore the optional execution. Time will tell whether such approach is indeed a winner.

Advertisement

Leaving the trenches of the Option<T> type

Please note currently Skila has only reference types (not that it is well thought over decision, I simply don’t have enough time to deal with value types).

The problem how to return data from the function efficiently while fitting in special “no result” value, haunts me for some time. I am aware of several approaches:

1. magic values, like -1 for indices (it is still used, see Go),
2. thin Nullable<T> type to mark there is no result (you can see this in Kotlin),
3. rich Option<T> type for wrapping the actual value (Swift, Scala approach),
4. Try-Parse pattern (for sure it is used in C# libraries).

The first approach is dead simple and fast, yet it is disaster when it comes to serious programming — each time you have to check what magic value you can get. The second approach is much better but it does not scale very well — you can use it for a substring lookup, but it fails with dictionary getter or such functions as first on collection. Yet, the performance is on par with magic values.

For me the only realistic answers are the last two — with Option<T> type you can work with any function, you just add an extra layer of Option<T> to the working type. The performance suffers but you always have fast counterpart functions tryDoSomething. Of course some poor soul has to write all those pairs of functions now — because of that I was seriously considering supporting two types at the same time, lightweight Nullable<T> and rich Option<T>.

Oh yes, there is another approach:

5. communicate failure (like in Icon),

I have no idea what the performance is, but the Icon code is simply beautiful. It was about time to read a book about it — I barely even opened The Implementation of the Icon Programming Language by Ralph E. Griswold and Madge T. Griswold when I found this inspiring passage:

(…) the design of programming languages should not be overly inhibited by perceived implementation problems, since new implementation techniques often can be devised to solve such problems effectively and efficiently.

I am sold — I want beauty, and I want performance! Nothing less. I want stackable (nested) Option<T> type, easy to use with speed of raw null values. Until now I was focusing on optimizing nested Option<T> type, but maybe I could somehow add a stack to null… wait a second.

Let’s consider what happens on the first nesting level (think of Option<T>) — we have either the actual value or a null. On the second level (Option<Option<T>>) — the actual value or a null again. At both levels when we have the actual value we can recognize that we didn’t end up with no result because our reference is not equal to null. It is the null which has to be wrapped (because null from the first level of nesting plays a role of true data on the second level), the real, actual value does not need any wrapping. It is some progress but we still have to do a little wrapping, right? No, we don’t — just turn the microscope on and take a look. In the first case the failure is indicated by null¹, in the second case by another, different null — null².

Click, click, click — do you hear this sound?

Option<T> type. Who said it is a regular type in the first place? It does not have to be — our Option<Option<T>> is a disjoint union of types. It is Null² type, or Null¹ type, or T type — Null²|Null¹|T.

Because we have to tell compiler what type we would like to get, it knows at what level it operates — i.e. how many layers it should pretend unwrapping to get the value. If it is any of the null value cases, it will also know how long it should pretend it has real data — the show with single null keyword is just for the user. Say the pointer is set to null¹ and our current type is Option<Option<String>> — do we have real value? Sure thing, it is not null² and that’s all we have to care about.

Thanks to all those lies there is no wrapping values in the runtime, the speed is the same as working with plain old null values. The only difference comparing to rich Option<T> school is with down casting — we will be able to tell what type we hold in hand (String for example), but we will not be able to deduce what union of types it comes from (Null¹|String or maybe Null²|Null¹|String).

Could it be this cookie is absolutely for free? Unfortunately — no. Union types does not work well with generic types (at least if you want to keep static control over types), but since we have here very specific case of union we can enrich type info with option counter. Whenever there is Option<T> type used we have to take option counter from type T and increment it.

This leaves me a syntax to think about and supporting three valued logic to consider — this could add an interesting twist to the language.

From goal-driven execution to macros

I keep reading how goal-driven execution model looks like, and while I really like natural feel of such condition:

if (a < b < c)

yesterday I found the piece of code which put me off. Consider printing out the just-read line in Icon:

write(read())

When read hits the EOF it fails, and thus entire chain of expressions fails — here it means write won’t be called. Let’s say you would like to catch the content of the line nevertheless, so you do a little rewrite:

line := read()
write(line)

OK, we have 2 lines, but the code basically does the same thing… Wait, it does not. It is not even close. Now when read hits the EOF the chain of expressions ends at the assignment, so it will be omitted and line will keep its previous value. Since the second line is completely unrelated it will be executed with the old line value.

Sure, I am surprised, and maybe part of the surprise is because I am not used to the way Icon works. But I think transparency of adding temporary objects to keep intermediate results is pretty fundamental concept, and Icon breaks it — something I cannot like.

From that point there are two approaches — give up or design improved goal-driven model (it might the same path though).

As for giving up I went back to my old idea of returning tuple — pair of function outcome and error:

if ((success:@,value) = dict["hello"]).success then
  // we have direct access to value
  // success is dropped

The syntax has a lot to be desired, so how about adding when construct which could handle exactly such cases:

when value = dict["hello"] then
  // we have direct access to value

This would be syntactic sugar only — pure internal rewrite to the if presented above.

And then it struck me — forget about Lisp homoiconicity, uniformity stupid! Lispers have to struggle with ugly syntax:

(if (= 4 (+ 2 2)) (...

instead of clean “syntax for the masses”:

if 4 = 2+2 then...

but Lisp syntax is uniform, and because of that Lispers don’t have to wait for anyone to bring new construct to the language. They add their when whenever they like — it blends in perfectly.

On the other hand in C-derivatives all you have is function-like expansion, you cannot add another for or while. Any attempt to bring Lisp-macro to C world would require allowing the user to alter the grammar of the language.

I didn’t solve anything with goal-driven model — instead I added yet another puzzle for myself. What to sacrifice in the design to bring power of macros to Skila?

If you find this topic interesting, more readings for you — The Nature of Lisp, Homoiconicity isn’t the point and Lisp: It’s Not About Macros, It’s About Read.

Function result — how to read it?

Lately I spent quite some time thinking about perfect “else if” syntax (don’t confuse it with “if-else”), and I just wonder how much time I will spend on this subject.

Let’s start from easy one — in all C-like languages function call and condition would like this:

if ((idx = str.indexOf("hello")) != -1)
  ...

This is ugly. Not only we have to add extra scope (not shown here) to avoid variable leak, but we are using magic number. Worse — what about such values as “-2”? They are valid for “idx” but they are invalid in context of “indexOf”.

Before you even warm up consider more difficult case — reading values from a map (associative array, dictionary):

value = dict["hello"];

This won’t fly, if there is no such key we will get an exception. So maybe like this:

if (dict.ContainsKey("hello"))
  value = dict["hello"];

OK, this is safe, but now we hit map twice. In case of C# the following is the best approach:

if (dict.TryGetValue("hello",out value))
  // we have valid value here

One can say — not so bad — but! First of all you have to mark your “trying” functions in some way to distinguish them from “regular” functions. For me it is not appealing to have bunch of “TryRead”, “TryConcat” and alike. Secondly, such syntax is inconsistent with already existing notion:

result ← expression

I have nothing against switching the flow (to — from left to right), however what bothers me here is inconsistency. I wouldn’t like to see this happening to Skila.

What are the other options? “Options” indeed:

if (!(value = dict["hello"]).isEmpty())
  Console.WriteLine(value.get());

Such code is efficient (kind of), with result being kept on left, but it is too elaborate. And all the time you use value, you have to use “get” method for it (or introduce temporary variable).

I was struggling with some mad ideas of dual variable, which is partially and implicitly converted to “bool”, or solving the issue with tuples (Skila style):

if ((success:@,value) = dict["hello"]).success then
  // we have direct access to value
  // success is dropped

But the concept of failure from Icon struck me as really elegant. Map indexer could “return” failure in case of missing key, and this would cause “if” to fail.

I didn’t write even “hello world” in Icon, so better read about Icon from somewhere else, Wikipedia may be a good starting point.

At first glance it looks, oh, so charming — I am afraid though that this beauty might come at cost of problematic maintenance, workflow which is hard to understand, not to mention complex implementation.

But this approach will be my first to investigate. As usual, if you have better, or simply other ideas worth considering, I am all ears.