As if to start, this evening I went to a Glasgow TechMeetup. There are Edinburgh equivalents that I never attended so I don’t know how they compare. The format was:

• beer and pizza at 6.30. Next time I must remember a bottle opener!
• a welcome about 7pm including an introduction from everyone in the room. There must have been about fifty people in the room so it took about half an hour!
• a few minutes to chat amongst ourselves, particularly with anyone who sounded interesting or employable.
• a talk about web design
• a talk about http caching

The first talk was pretty dull and I was ready to call it a day at that point. Thankfully I didn’t cos the second talk was well worth the entry fee, even if it was free. If you bump into serialseb he gives a great talk - funny and informative.

Section 3 of the paper uses the same technique from Section 2 and adapts it to tackle a larger and more interesting problem: array bound checking. To motivate the problem they follow in the footsteps of the dependent type literature by implementing binary search with statically-checked “safe” array lookups.

To explain the approach they use we’ll start with an unsafe implementation that uses runtime bounds checking on each index.

bsearch :: (Ord a) => (a -> a -> Ordering) -> a -> Array a -> Maybe (Int,a)
bsearch cmp key arr = go 0 (length arr - 1)
  where go lo hi = if hi >= lo
                      then let midpoint = lo + ((hi - lo)`div`2)
                               curvalue = arr ! midpoint
                           in case cmp key curvalue of
                                  LT -> look lo (m - 1)
                                  EQ -> Just (m, curvalue)
                                  GT -> look (m + 1) hi
                      else Nothing

The index operation (!) checks at runtime to see if the supplied index is within the bounds defined by the array (we still have to catch an exception, which I haven’t shown here, but at least it won’t blow everything up). This check is not necessary if we can prove that the index will always be in range.

The general approach outlined in Section 2 was to brand “safe” operations and types which go together. Once a list had been branded non-empty (ie had type FullList) it could be freely passed around and later safely destructured into head and tail without worry.

In the case of lists the values were branded “safe” and the head/tail operations could only work on “safe” values. The head operation is analogous to indexing the 0th element in an array. If we were to extend the list approach to arrays we would need a deref0 for the 0th element, deref1 for the 1st element and so on. This is both unwieldy and needlessly restricts the type of indexing possible.

The solution proposed by the paper is a set of “safe” indexing operations which only work when both the array and the index are branded as being compatible. An index is compatible with an array if it is within that array’s bounds.

get :: BrandedArray b a -> BrandedInt b -> a

The type variable b here is a phantom type which brands arrays and indices. The get operation can only be used if the same brand is applied to both array and the index.

Initially the only “safe” array indices are the offsets of the upper and lower element. The brands for an array are created alongside the upper and lower index in one operation, providing the assurance to the type system that whatever type b represents it is identical for these three values:

brand :: Array -> (BrandedArray b a, BrandedInt b, BrandedInt b)

(In fact this is not the proper type signature. The proper signature is reproduced below, with existential quantification over the branding type b and continuations for passing/failing conditions. This signature is just a gentle introduction to get the idea across)

Further operations can be created with transformations on the provided indices: increments, decrements and midpoints.

Whatever b is here it’s the same type, but next time brand is called b may well be a different type. The type system can’t unify them since it doesn’t know the type — it’s never specified. So only indices created alongside the branded array can be used with it, since only in that situation can the type checker be sure that the type of b matches.

This seems a powerful use of phantom types and it’s one I’d never seen before. The paper uses the phrase “type eigenvariable” at this point, which I’m not familiar with but it’s such a nice phrase that I’ll have to keep an eye out for it.

As before the type constructors to enforce this safety are hidden and only accessible under specific and well-guarded conditions.

-- Not exported!
newtype BrandedArray b a = BrandArray a
newtype BrandedInt   b i = BrandInt i
 
brand :: Array -> (forall b. (BrandedArray b, BrandedInt b, BrandedInt b) -> w)
               -> w -> w
brand arr cont stop = if lo <= hi
                         then cont (BrandArray, BrandInt lo, BrandInt hi)
                         else stop

The user provides functions to deal with the two cases that the bounds are valid and invalid. This same pattern follows through to other parts of the implementation: condition checking is done internally and the user provides functions to deal with various cases. This keeps some of the constraints (like the existential quantification over b, the brand type) wrapped up. If the user tries to subvert this the type checker objects:

Inferred type is less polymorphic than expected
  Quantified type variable `b' escapes
In the second argument of `brand', namely `Just'
In the expression: brand testarray Just Nothing
In the definition of `it': it = brand testarray Just Nothing

In order to implement the rest of binary search over arrays some arithmetic functions on indices are introduced. These functions exist inside the trusted kernel so some effort is made to ensure they are correct. They also require passing/failing functions for operations that may fail (such as finding the successor of a valid index: the successor may be out of bounds):

-- If i1 and i2 are within bounds then their midpoint
-- is within bounds.
bmiddle :: BrandedInt b -> BrandedInt b -> BrandedInt b
bmiddle i1 i2 = BrandInt (i1 + (i2 - i1) `div` 2)
 
bsucc :: BrandedInt b -> BrandedInt b
                      -> (BrandedInt b -> w)
                      -> w -> w
-- If i is less than the upper bound we use continue calculation
-- with the inside else we fail with outside.
bsucc (BrandInt upper) (BrandInt i) inside outside = 
    let i' = i + 1
    in if i' <= upper then inside (BrandInt i') else outside

There is also a predecessor function which works analogously. First we need to rework the binary search for this style of programming:

bsearch :: (Ord a) => (a -> a -> Ordering) -> a -> Array a -> Maybe (Int,a)
bsearch cmp key arr = brand arr find Nothing
  where find (barr,lower,upper) = go lower upper
    where go lo hi = let m = bmiddle lo hi
                         x = unsafeGet barr m
                     in case cmp key x of
                        LT -> bpred lo m (\i -> go lo i) Nothing
                        EQ -> Just (unbrand m, x)
                        GT -> bsucc hi m (\i -> go i hi) Nothing

On each iteration of the algorithm finding the value at the midpoint is always a safe operation provided the two arguments are safe. Incrementing or decrementing the upper/lower bound might fail if the index goes out of range, but this check can be considered part of the end condition, ie the search finishes unsuccessfully anyway if the index goes off the end because it’s an indication that the value we’re looking for is not stored.

The call to unsafeGet is, in fact, safe, since the index m is guaranteed to be in range for this array. No runtime checks are required over those needed to indicate the end condition. As before this problem is always dealt with formally using a similar mechanism for the empty-list problem, using Strict and Lax languages and mappings from one to the other. I’ll look at that next time.

The next section of the paper is more theoretical. The authors introduce two notional languages, Strict and Lax, and their typing mechanisms. Strict has a “fancy” type system which specifies two list types (List and List+) and only the latter type can be deconstructed with head and tail functions. The List+ type can only be created by a special nonempty conversion, which cannot be applied to empty lists.

The Lax language has no such advanced typing facility and will happily accept nonempty(nil) as a legal construct. This language is more like a traditional functional language with a possibly-empty list type. In terms of typable operations Strict defines a smaller set than Lax, and every legal Strict program is a legal Lax program too. The authors define a mapping from Strict to Lax programs to allow embedding a statically safe program in Haskell, ML, etc.

This embedding is ultimately what we always attempt to do when programming. The set of things which the compiler will accept is necessarily larger than the set of things which will get us the right answer. Type systems aim to constrain the “acceptable” set slightly though there is still plenty of scope for bugs. :-) Some programming languages make it easier to write programs which are dangerously but subtly wrong. Using the full capabilities of a language like C all the time is incredibly foolhardy: not every situation demands pointer arithmetic. If we can use the type system to wall off “dangerous” code then all the better. Another example is the use of IO actions or unsafe* operations in Haskell. These are often used only where necessary in the former case, or walled off in an opaque module in the latter case. This walling-off allows us to concentrate our verification efforts to areas which can cause the most problems.

The resulting “embedded” code is the trusted kernel mentioned in the previous post. The reduction rules defined for Strict and Lax are contained in this kernel. The reduction rules for the non-empty list case are:

head (nonempty (cons E1 E2)) -> E1
tail (nonempty (cons E1 E2)) -> E2
indeed nil E1 E2             -> E1
indeed (cons E E') E1 E2     -> E2 (nonempty (cons E E'))

The first two rules define the operation of head and tail on a non-empty list. These operations aren’t defined for empty lists. The second two rules show how the non-empty type is constructed if the list is truly non-empty. Notice the similarity between indeed and Haskell’s maybe function (with arguments rearranged):

indeed :: List a -> b -> (FullList a -> b) -> b
maybe :: Maybe a -> b -> (a          -> b) -> b

The maybe function forces the user to deal with all cases in the option type by requiring functions and default values for each case. The indeed function goes one step further by not allowing any other way to take apart the list type — in order to use head and tail you have to convert the List to a FullList and only indeed can do that.

In the spirit of the maybe function for Maybe types and the either function for Either types it would seem sensible to have a list function for List types, which would take a function for each case suggested by a list. After a bit of reflection I think this is the fold, or rather maybe and either are specific folds.

Which all suggests that the toy example provided (reversing a linked list) can be safely recreated using a fold. If we look again at the safe version defined in the previous post we see two cases we have to deal with:

reverse :: List a -> List a -> List a
reverse xs acc = case full xs of
            Nothing -> acc
            Just ls -> reverse (tail ls) (cons (head ls) acc)

Stripping away all the plumbing stuff that’s going on here, we have two vital bits:

1. The return of the accumulator when the list is no longer non-empty, ie is empty.
2. Consing the head of the list onto the accumulator at each stage that the list is non-empty.

Step 1 is done automatically by the fold: it lets you iterate over values without being concerned about them running out. Step 2 is just the cons operation we know, with the order of arguments reversed. To actually reverse a list we supply an empty accumulator.

reverse in = foldl combiner acc in
  where combiner xs x = cons x xs
        acc = []

We know we can define this simple function as a one-liner but not all functions are as simple. The next section looks at array indexing operations. The motivating example this time is binary search over arrays — checking that all indexing operations remain in-bounds and that these constraints are ensured by the type system. By contrast with list reversal a binary search deals with several separate elements (the array, the lower and upper index) which cannot be hidden completely if we are to allow general array operations.

The area the paper looks at is labeled “lightweight dependent typing” — dependent typing without dependent types, essentially. Dependent type systems allow the programmer to specify types based on the values, so the programmer can ensure that lists will necessarily be non-empty and denominators necessarily non-zero, among other useful examples. The paper looks at ways of creating type safe “capabilities” without dependent types.

I will take the paper a little bit at a time, and as I read on I might have to correct previous misreadings/misinterpretations. The first motivating example given on page 2 provides a program to reverse a list, building up the result in an accumulator.

reverse :: List a -> List a -> List a
reverse ls acc = if nil ls
                    then acc
                    else reverse (tail ls) (cons (head ls) acc)

The standard list destructors head and tail are partial functions. They are not defined for the empty list and will produce a runtime error if accidentally applied to the empty list. For this reverse function we can be reasonably certain that it won’t blow up at runtime, but it’s a very simple function and we can’t get the same assurances in all situations. Let’s call these standard head and tail functions by more explicit names:

unsafeHead :: List a -> a
unsafeTail :: List a -> List a

In order to provide static guarantees that all uses of head and tail are safe a new type called FullList is introduced which is a rebranded standard list type. The key element is that head and tail can only operate on FullList types, and a guarantee is provided that values in this type are non-empty lists. With this static assurance we can actually use our unsafe functions from earlier:

-- unfull converts a FullList to an ordinary List type
head :: FullList a -> a
head = unsafeHead . unfull
 
tail :: FullList a -> List a
tail = unsafeTail . unfull

The FullList type becomes a one-use token which the value has when it moves through the program. This token can only be created in one place (which we can lock inside a separate module). We can easily control under what conditions this token is assigned:

module (FullList, unFull, full) where
 
newtype FullList a = Full { unFull :: List a }
 
full :: List a -> Maybe (FullList a)
full ls = if null ls then Nothing else Just (Full ls)

To verify that head and tail are always “safe” we must only verify that a FullList is always non-empty. The paper goes deeper into the formal verification than I’m comfortable talking about (at least at the moment) but it should be reasonable to say that there is only one line of code, the definition of full, which needs examined.

This is the kernel of trust concept that the paper revolves around. Shrinking down this kernel means it’s easier to examine and verify; and from there the type system extends our trust to the rest of the program.

From the user perspective, the type system forces us to check that lists are not empty before invoking head or other unsafe list operations:

reverse :: List a -> List a -> List a
reverse xs acc = case full xs of
            Nothing -> acc
            Just ls -> reverse (tail ls) (cons (head ls) acc)

Ideally we’d be doing this anyway so there isn’t a greater burden — but it does provide us with compilation errors if we forget to do it. (From personal experience the use of an option type like Maybe makes it much harder to forget these things, even before getting the compiler error. And calling fromJust or other unsafe extraction method feels immoral.)

This takes us up to Section 2.1, with only one omission. The paper at this point switches from explicit wrapping and unwrapping of option types to continuation-passing style. The option types are straightforward but slow things down. Depending on how things go I might follow suit and use continuation-passing from here on, but it might prove a better learning experience to convert to option types. We’ll see when I come to write up the next section.

The upload fails if you supply a non-empty list of tags to the hsflickr library. In short, uploading images works like this:

uploadPhoto filename title description tags attributes

The upload will fail if the tags argument is non-null. That is, supplying tags at the point you upload the image will result in an error. Thankfully the workaround is short:

photoID <- uploadPhoto filename title description [] attributes
addTags photoID tags

Supply an empty list of tags for the initial upload, and when that succeeds use the resulting photo ID to set the real tags afterwards.

When I work out what the underlying issue is I’ll post a patch here (and punt one upstream).

Come on, form mail isn’t difficult! Get your act together, and check the damn things before you hit send.

Back to the topic of the study group. Reading SICP is deceptively easy at times. Each step is a simple progression from the last, such that each idea seems obvious and trivial. Then suddenly some trivial new concept makes no sense at all and you find yourself backtracking through pages of explanation to find some firm handhold from which to start moving forward again. Most of the time I feel that I’m not learning anything but I realised today that some things which were not intuitive in the past are now familiar and natural. I was reading The Arrow Calculus and realised that I could understand all of the notation and type rules for lambda calculus and arrows given. It was the environment stuff in particular that felt “obvious” in the way that it wouldn’t have in the past, and I’ve been doing a lot of interpreter writing and environment-jigging in recent weeks with SICP. It’s all coming together.

I’ve added some unit tests, which was a simple thing to do but not very easy to get right. Unit tests are a bit like Cluedo, in that you have to convince yourself of something very complicated with a few well-chosen questions. And it’s easy to forget your assumptions and miss something vital.

Just today I spotted something by playing with the game. I realised it was possible to hurl or shove a piece over another to reach the enemy on the far side. The logic only checked that the target was within reach but not that it was within proper line of sight. (This goes back to the original implementation too, so it’s not just a feature of my mucking about with the code that deals with the game rules and breaking something.) The rules are silent on whether this is even allowed (I suspect not) so I’ve disallowed it for now.

This is not the kind of bug I would have caught with unit tests because it would never have occurred to me to check for it. Of course the butler didn’t commit the murder, that’s preposterous! I only spotted it during play because I was purposely lining up foes next to each other in order to manually test their capture capabilities and realised that I could hurl a dwarf further than necessary… and the game allowed it.

So I’ve been repeatedly rewriting the file that deals with game rules so that scenarios like this become more obvious. I’m getting to the point where I’m happy with this small segment so I need to push on to other things.

As I mentioned on another post it’s nice having an issue tracker for the code so I can write down any ideas that occur to me, whether they’re short-term or long-term concerns. I think it’s time to look at some of the more medium-term goals now, such as network interaction, otherwise I’ll spend the rest of my days endlessly fiddling with the game rules.

Okay, so Scheme typically works on the everything-is-a-tuple philosophy. Other structures can be built out of nested or chained tuples. A tuple, or pair, looks like this:

(1,2)

Immutable linked lists can be made by inserting one pair inside the second part of another pair, with an empty marker at the end.

(1,(2,null))

The functions for accessing the left and right elements in a tuple are called, for historical reasons, car and cdr. Applying car to (1,2) gives you the 1, and applying cdr gives you the 2.

When applied to the linked list implementation, car and cdr are no longer really left and right but head and tail. And so for the past few months working on SICP I’ve been thinking about car as head and cdr as tail and mostly getting away with it.

But sooner or later the inaccuracy of this correspondence was going to become clear, and the other day it did. Single linked lists point from the head to the tail but not in the other direction. From each element you can delve deeper but you cannot move higher up the chain. An obvious reason for this is that there may not be a single “higher up” to link to. Immutable list tails can be shared.

Double linked lists can be traversed in either direction. If you’ve only got two places to store something (left and right elements in a pair) and three things to store (the left side of the list, the right side of the list and current element) you need to double up somewhere. At which point it is necessary to remember that cdr isn’t the same thing as tail.

Which makes me wonder why the idiomatic way of traversing lists is with car and cdr? Why is the meaning of tail intentionally blurred with the meaning of cdr? SICP puts a lot of emphasis on abstraction of implementation from intent yet it still litters its list-manipulation code with car and cdr, references to the underlying representation that doesn’t have anything to do with the meaning of the lists at all.

Of course we may as well ask why cdr, an acronym for “Contents of the Decrement part of Register number”, itself is a reference to an implementation detail on a 1950s computer. I’m sure left and right or something slightly more meaningful wouldn’t have hurt.