They had released a teaser challenge as a sample of the main competition which seemed like it was worth trying. It took us a couple of days to get to the answer, not least of all because we actually had work to do!

Since the teaser challenge is now well over and the “official” method has been published on the challenge page (move your mouse over the big white space to see the solution) I thought it would be interesting to go through my solution and see (a) what the puzzle-setters had done (b) how the answer was stumbled upon (c) how the answer could have been derived and (d) where my method differed from the official approach, despite reaching basically the same answer. It’s been a while since I thought about the practical techniques of cryptanalysis so my methods are neither the best nor the most sensible, but at least I can be honest. Helen’s just finished reading Cryptonomicon so will probably tell where I went wrong!

If you want to try out your skills the original problem is still available and will probably be there for some time. At least you can see what the starting point was.

I’ll write up my results and methods in the next few days, posting each section as it goes. I’ll also publish the code I used to extract the solution, which was written in Haskell. I’ve never done anything like this before but the Haskell list-munging functions and point-free style made it simple. The effortless compositional style of creating pipelines of functions meant that I could play with little sections, joining and splicing them as necessary, until I found something which seemed like a good result. But there will be more details in the coming days.

When the primary router is working normally the secondary doesn’t do much. Its only role is to monitor the liveliness of the primary machine. The redundant router can often be used for other things when the primary is operating — and many times the primary acts as a redundant router for the secondary’s clients. Each provides backup for the other.

So when I found out recently that I was being made redundant I thought “great! I’ll just sit and watch other people working and take over if they burst into flames”. But it turns out that when people are redundant it’s totally different from when routers are redundant. Instead of being relied on for backup in case of failure, it means “no longer working”. Strange but true! I can see why it wouldn’t catch on very well in networking.

My last day at Cisco is this Friday (30 July). It’s been an interesting few years and provided novel experiences, silly conversations about Star Trek and given me a bit more confidence. I’m sad to be going, and though there will always be loose ends to tie up and the promise of interesting projects on the horizon, the team I’m leaving behind seems to have a glut of these at the moment. I’m also disappointed that the study group at work will continue reading SICP without me. Obviously I can read it alone but the discussion and peer support/pressure was a useful part of it.

Meanwhile, the job hunt continues. Recruitment agencies make this process at least ten times harder by hiding the employer, the industry and the specifics of the job for their own ends. I have had a few friends pass on job details, and had some telephone discussions, but no success yet. Watch this space, or one very much like it.

The first and obvious question is, What’s an IP address? It’s an identifier that your computer uses to talk to other computers on the network. It bears a lot of resemblance in form and function to a telephone number. There are prefixes which are shared by every address in the area, and then there’s a bit specific to you.

An IP address has two important pieces of information embedded in it. The first is the host ID — the identifier of my specific computer. The second is the network address — which is the number of the network my computer is found in, and is analogous to a telephone area code.

Just to make things difficult though, the two numbers are joined together so you can’t tell where one part ends and the other begins. So when talking about addresses we need another piece of information to tell us which part is network and which part is host.

First, let’s write out 192.168.0.3 in binary. This makes a very long number but it will make everything much clearer from here on. Each number between the dots is converted separately. This number is not the same as 19,216,803.

192.168.0.3 = 11000000 10101000 00000000 00000011

I’ve left spaces where the dots were previously. Each section is 8 binary digits long, so each section can represent a maximum number of 11111111 — which is 255 in decimal.

The next bit we come to is choosing a point on that line so that all the digits on the left represent the network address and all the ones on the right show the host address. Looking back up at the address I gave at the top you’ll see a “/24” sitting at the end. This is called the network mask and it works just like a piece of card with a hole in it. You write out your address and then align 24 bits underneath: everything with a 0 underneath is masked out, leaving the network address. We are getting the result of 1 whenever both the address and the mask is 1.

192.168.0.3  = 11000000 10101000 00000000 00000011
mask 24 bits = 11111111 11111111 11111111 00000000
result       = 11000000 10101000 00000000 00000000

Back in decimal land, that network address is 192.168.0.0. Since the network mask is also a binary number it is often written like an IP address, as four decimal numbers separated by dots. The same address can be written as 192.168.0.3/24 or 192.168.0.3/255.255.255.0.

We can invert the network mask to give a host mask and use the same procedure to find out the host ID, which turns out to be 00000011, otherwise known as 3.

192.168.0.3 = 11000000 10101000 00000000 00000011
mask 8 bits = 00000000 00000000 00000000 11111111
result      = 00000000 00000000 00000000 00000011

You probably knew that from inspection but computers ain’t so clever!

Another useful number we can learn from the address and mask is the broadcast address. This can be used to send messages to everyone on the network. We calculate this by inverting the mask again, so we’ve got 8 bits on the right instead of 24 bits on the left, and taking the result to be 1 wherever the address or the mask is 1.

192.168.0.3 = 11000000 10101000 00000000 00000011
mask 8 bits = 00000000 00000000 00000000 11111111
result      = 11000000 10101000 00000000 11111111

The result can then be written as 192.168.0.255 in dotted-decimal format. Packets sent to this address will be examined by every host on the local network. This is useful if you don’t know the address of your recipient!

Photo is The World’s Network by saschaaa.

A few friends came over and we got fajitas from Los Cardos across the road — and despite the utilitarian appearance the food was pretty good. The staff were friendly too. I got a “fajita burrito”, which is kinda what I’d call a fajita if I was making it at home, except they loaded it with rice as well as the usual fried onions, peppers, cheese, salsa and spicy chicken. (There are a number of other fillings available besides chicken, including haggis…) The food was more or less what you’d make yourself, simple but plentiful, and they had big, good quality tortillas which seems to be the hardest part when it comes to make-your-own fajita meals.

The film was weird as all hell. I borrowed the special edition with audio commentary and a separate making-of disc — I wonder what explanation they’ll have for some of the stranger scenes. Some didn’t even seem to connect at all to the rest of the story. It certainly wasn’t what I expected. I knew there was action in light-striped arenas that probably represented some gaming system, but I didn’t realise that most of the protagonists would be computer programs. Needless to say, if you know anything about computers you have to plug your ears at some points or risk bursting into entirely inappropriate laughter. The plot is a bit like The Lord of the Rings (take the magical item into the evil overlord’s domain to free everyone from tyranny).

The sequel — coming out in December I think, so aiming for the family Christmas market I suppose — follows the action twenty five years later in the same inner-computer environment. Hopefully they won’t be relying on the crutch of dazzling graphics and spectacle instead of coherent plot. But that’s probably a foolish hope…

Helen’s out tonight because one of her colleagues is leaving the lab, so she didn’t get to see it. Maybe we’ll watch it tomorrow with extras before I give the DVD back.

At the weekend Helen and I went an epic (miniature epic?) walk up the Water of Leith, from Bonnington to Juniper Green. This map shows the extent of the Water of Leith Walkway but we didn’t cover all of it. We started near the firth outlet, about halfway between Craigleith and the docks I think, and went to Juniper Green which is one of the yellow-marked waypoints. It was probably about 8 or 9 miles in total, uphill but mostly on a very slight incline and broken up by lots of down and flat bits.

We passed by the Water of Leith Visitor Centre (why does a river have a visitor centre? who knows) but it was closed. We saw a couple of herons like in the photo on the visitor centre website. I took my new camera and spent the day playing with different settings in Manual mode, trying to get a feel for how things turn out. I’m not confident but I’m much closer than I was, and I’m enjoying it greatly.

When we got to Juniper Green we had tea at Ristorante Al Borgo, which is owned and run by the parents of one of Helen’s colleagues. After a half day walking in serious heat it was a relief to hop on the bus and get back home again.

But that’s not the extent of the new, exercising, Dougal. Oh no! Near to the flat is the Out of the Blue arts centre, an ex-drill hall that’s been gutted and refurbished to include studio spaces, exhibition areas, a cafe and so on. We go to regular art fairs there and I’ve often seen a poster on the wall advertising their capoeira classes. Last night I popped over to the beginners class organised by the Mão no Chão. They were very friendly and the guest tutor was genial and enthusiastic in equal measure. The class lasted an hour longer than advertised: it should have been 8–9.30 so by the time I got home at 10.40 I was completely wiped out.

It was great fun though. I last did any capoeira about 8 years ago so some of the ideas seemed familiar but it’s not like falling off a bike. It can be forgotten!

The issue that needed clarified was whether I could justify treating the null-value argument as a full-blown identity element. The text says:

a null-value that specifies what base value to use when the terms run out.

(It’s worth noting that the definition of combiner also isn’t specified very well: a “procedure (of two arguments) that specifies how the current term is to be combined with the accumulation of the preceding terms”.)

This isn’t abundantly clear but it suggests that null-value is the last element that can be included in the calculation, so that:

5 + 4 + 3 + 2 + 1 + 0

is allowed but

0 + 1 + 2 + 3 + 4 + 5

is not allowed, because I’m starting with my null-value (ie, 0) and adding the remaining values to it. In this case there is no difference in result. So I needed to find a situation where some element work as an identity on the right but not on the left. Wikipedia to the rescue!

The number 1 as an identity element for exponents if it’s the power but not if it’s the base. Using ^ as an infix exponent function, for any x:

x ^ 1 = x

but

1 ^ x = 1

We can create a chain of odd-numbered exponents to test our implementations of filter-accumulate:

(define (odd-exponent-chain a b)
  (filtered-accumulate odd? expt 1 identity a inc b))

An implementation of filtered-accumulate which cleaves as closely as possible to the definitions should implement (odd-exponent-chain 2 6) as (expt 3 (expt 5 1)).

This conclusion that null-value only works as a right-identity has further repercussions — most of the answers published for iterative version of accumulate are wrong. Why?

(define (accumulate combiner null-value term a next b) 
  (define (go acc n) 
    (if (> n b)
        acc
        (go (combiner acc (term a)) (next n)))) 
  (go null-value a))

The iterative versions always start with the null-value in the accumulator and slowly combine each new term until the end. But the definition above clearly states that null-value is to be combined with the results when there are no more terms left.

This is possible but not as elegant as the recursive solution by a long shot. There’s probably a good way of cleaning it up that I haven’t spotted yet.

(define (accumulate-iter combiner null-value term a next b)
  (define (go acc n)
    (if (n > b)
        (combiner acc null-value)
        (go (combiner acc (term n)) (next n))))
  (if (a > b)
      null-value
      (go (term a) (next a))))

Having got this far we realise we can no longer implement filtered-accumulate in terms of accumulate: the first term forms the seed for the accumulator whether it is appropriate or not. In this case we do have to create a new filtered-accumulate which is more powerful than accumulate on its own.

(define (filtered-accumulate-iter filter combiner null-value term a next b)
  (define (go acc n)
    (if (n > b)
        (combiner acc null-value)
        (go (combiner acc (term n)) (next n))))
  (define (start n)
    (cond ((n > b) null-value)
          ((filter n) (go (term n) (next n)))
          (else (start (next n)))))
  (start a))

After all this it seems prudent to ask “who’s being pedantic here?” and whether the mistakes identified are the right ones.

• The lack of formal specification of parameters like null-value encourages assumptions. Most people doing these exercises assumed that null-value works as an identity element per the examples of (+,0) and (*,1). These examples reinforced rather than highlighted the assumptions.
• With this assumption it is possible to implement filtered-accumulate in terms of accumulate, contra the question. I haven’t found an instance online where someone spotted this mismatch.
• The “obvious” ways of writing the iterative and recursive accumulate have different semantics if you don’t pay close attention. (Which is another way of saying one of them is wrong.) Hence the problem with making poor assumptions.
• The only way that requires a rewrite of filtered-accumulate is one that doesn’t assume a left-identity element.
• There is an official tutor’s book published but not available online (presumably to prohibit cheating for universities who use this textbook). So we don’t know what the writers had in mind when they wrote that section.

It’s been an interesting few days of thinking! Many thanks to Brent for making me re-examine the text and my assumptions a further time. I think we got to the bottom of it in the end.

You can obtain an even more general version of accumulate by introducing the notion of a filter on the terms to be combined. That is, combine only those terms derived from values in the range that satisfy a specified condition. The resulting filtered-accumulate abstraction takes the same arguments as accumulate, together with an additional predicate of one argument that specifies the filter. Write filtered-accumulate as a procedure.

The tricky bit I see is the “even more general version” instruction. As far as I can tell filtered-accumulate can be written in terms of accumulate. It’s not more general or more powerful than the plain accumulate.

The previously-defined accumulate takes the following arguments:

• a null-value and a combiner-function (which is to say the binary operator and identity for some monoid)
• a pair of numbers representing the start and end value in the range we want to accumulate over
• a function which maps values in this range to values in the monoid described above (and it could be the identity)
• a function to get to the next value in the range given the current one

My (recursive) accumulate looks like this:

(define (accumulate combiner null-value term a next b)
  (if (> a b)
    null-value
    (combiner (term a) (accumulate combiner null-value term (next a) next b))))

Exercise 1.33 wants you to pre-filter values so that they don’t all get added together. Only those a which pass some predicate get combined into the result. But since the combining function comes specified with a null value it’s not necessary to skip over undesired results. All we need to do is convert any values which fail the test into the null value.

Put it this way: given a list of numbers between one and ten, you could find the sum of primes by either removing all the non-primes, or converting them into zeroes, because adding zero doesn’t affect the result. Similarly if you want the product of odd numbers between 14 and 103 you can filter out all the even numbers or turn them into 1s, because multiplying by 1 doesn’t affect the end result.

And it just so happens that there is already a way in accumulate to convert numbers before they are combined together, the term argument. This is used to square input values or apply some other function. The key argument is that this function can check if the input meets some criterion and then either process it further or replace it with null-value.

(define (filtered-accumulate filter combiner null-value term a next b)
  (define (filterterm n) (if (filter n) (term n) null-value))
  (accumulate combiner null-value filterterm a next b))

All the solutions to this exercise that I have seen online basically re-implement the rest of accumulate with extra filtering. This doesn’t seem to be necessary. My implementation above works for the following two exercises (sum of squares of primes, and product of all relatively prime integers).

But the exercise plainly says that filter-accumulate is “an even more general version of accumulate”, which to my mind means that filter-accumulate should be able to produce answers which standard accumulate cannot. (And that filter-accumulate cannot be implemented in terms of accumulate.) What is the counter-example that I’m missing, or is the book wrong?

Note: I’ve starting putting my solutions for the SICP exercises online, at http://www.dougalstanton.net/code/sicp/.

After seeing this post on keeping a “lab book” I realised I could make extra use of this environment variable that I already used.

I save this file in ~/bin/note and I can write quick scribbles to myself in whichever window I’m using and it will go to the correct “book”:

#!/bin/bash
 
LOGDIR=$HOME/labbook
 
if [ "" == "$BUGID" ]
then LOGFILE=$LOGDIR/GENERAL.log
else LOGFILE=$LOGDIR/$BUGID.log
fi
 
if [ "" == "$*" ]
then
    echo "No log written: message empty"
else
    DATE=$( date +%Y%m%d-%H:%M )
    mkdir -p $LOGDIR
    echo $DATE $* >> $LOGFILE
fi

It’s not complex but I think it will do wonders for how much I remember to document about my work from day to day.

Update: I learned today the delight of this one-liner which prints everything I’ve done today in order. It’s quite pleasing.

$ grep -h `date +%Y%m%d` labbook/* | sort

Scheme is not so interesting so far. The parentheses still bug me and we haven’t got to the stage where dynamic typing has proven useful so I’m smarting from the lack of a decent type system. From what I remember when I skimmed the book in the past most of Scheme development involves creating ad hoc type systems in the runtime. :-) Many people insist that dynamic languages are good for something so hopefully this something will become apparent.

I have enjoyed several of the exercises which require rewriting recursive algorithms in an iterative style. Seeing the contrast and thinking up different routes to the end product is fun, and I’m glad we’ve got a unit test framework (SchemeUnit) to help in that regard. I tried to track down a Scheme version of QuickCheck but while one used to exist it’s disappeared off the net.

• Measuring the Universe by Kitty Ferguson

  A kind of history of geometry and cosmology, to answer the question, “how do we measure things that we can’t hold a yardstick against?”. It covers the history of the ancient Hellenic philosophers who calculated the diameter of the Earth, the distance to the Moon, and so on, right up to the discovery of the size and shape of the Milky Way and the distance to the furthest reaches of the Universe. It was interesting to see how much we currently know is very modern knowledge. Before the 1920s we didn’t know there were other galaxies! I also liked learning about the contributions made by people who I’m familiar with through their namesakes — the Cassinis, the Hubbles, the Oorts.

• The Steel Remains by Richard Morgan

  Richard Morgan is more known for his hard-edged post-cyberpunk science fiction. This is his first foray into “high fantasy”, the realm of dragons, gods and barbarian adventurers with big swords. This book had most of the above, including what I believe to be the first gay sword-wielding hero — a move so obvious in hindsight it is hard to believe it’s taken this long. I enjoyed the book a lot but felt the point of the villains eluded me, and the ending lacked satisfaction. More than enough sex and blood-letting to count as a Richard Morgan book though.

• Wicked by Gregory Maguire

  This one took me by surprise. I suppose it’s not totally odd that book with the tagline “inspired the hit musical and sold two million copies” would be worth trying but I was still very pleased by everything that it delivered. It’s the life story of Elphaba, the Wicked Witch of the West from L Frank Baum’s The Wonderful Wizard of Oz. She is an outcast from the start, being born green and allergic to water, in a little rural town in Munchkinland. I felt quite caught up in the events of her life and as the events of The Wonderful Wizard of Oz began to overlap with Wicked it was quite painful, knowing that she would soon die from a bucket of water, her dreams unfulfilled and her friends tormented by the despotic Wizard. I’ve added the two sequels to my wishlist.

• Dark Entries by Ian Rankin and Werther Dell’Edera

  The one book on this list which didn’t come from the book group, this is a graphic novel written by the same Ian Rankin who writes the Rebus novels. It’s a very short (I read it in an evening) John Constantine novel about a Big Brother-style reality show whose contestants are having horrifying visions. Constantine is brought in to exorcise the house or otherwise get to the bottom of the problem, though things don’t go according to plan. The drawing was strangely out-of-kilter with the story — the London graffiti with American spelling, the dialogue describing Highland Park as a whiskey-with-an-e alongside a picture of the bottle showing whisky-without-an-e. It all just seemed badly put together, and the failure to decide whether the story was darkly comic or properly horrific didn’t help. Maybe not bother.