Archive for the 'Programming' Category

Jan 02 2011

Password too short

Published by Dougal under Programming

This is the kind of post that I write on occasion, and people tell me for weeks later, “yeah, I read your blog but I don’t understand it”. This is another one of those posts.

I was upgrading a password system for a quick web application. The initial design stored plain text passwords in the database, with the only constraints being they had to be alphanumeric strings less than 100 characters long. I had created some text users which I wanted to keep using after moving over to a salted-hash password system.

I decided to store the salt and hashed password together in the original password field, which is easy to do if you know how long your salt is. The salt is also used inside the hash function, but we store it on the front of the result too.

salt = "12345678"
pass = "secret"
hash = salt + digest(salt + pass)

I thought I would be sneaky and change my password in the database to use this scheme before I enabled the code. That way I wouldn’t lose the ability to log in as this user — because there’s no way I could find a password that could be hashed to “secret”!

$ php -r 'echo ("saltsalt" . md5("saltsalt" . "secret")) . "\n";'
saltsaltb857da9b2f51247b63107cc8a3e38c02

So I manually change my password to saltsaltb857da9b2f51247b63107cc8a3e38c02, upload the new authentication code and get locked out.

Why? Well I changed the password constraints at the same time, and my password was too short! After all that I registered again anyway. It never hurts to exercise some of the code a bit more.

One response so far

Dec 10 2010

Flickr authentication process (war stories)

Published by Dougal under Programming

I’ve recently been trying to add publishing support to ComicBake, so that comics can easily be uploaded to remote servers. I started looking at Flickr support because that’s where I would put my comics if I had any. Also it means I would be able to upload work-in-progress shots much quicker.

I spent some time reading over Flickr’s extensive but not very clear API documents. I registered my app with Flickr and received in return a “key” and a “secret”, which are used for signing in. Now I’m not quite sure how the key/secret pair work with open source software. There’s no way I can get round the fact that these two items will be stored in the source. I can obfuscate them in some way, but inevitably the obfuscation mechanism will be documented in the code, because I have to be able to undo it!

On top of that, if someone wants to get a Flickr key they can get one for free in two minutes like I did. The only real reason to secure the key is to prevent other people from writing apps with that key, thus masquerading as your app, and abusing the service. In that case the key would be revoked which would be a pain requiring a new key and new binaries to be distributed. But the chances of someone going to that effort are practically zero, I think.

Anyway, once you’ve got the key and secret to identify your app you need to authenticate yourself in order to change things on Flickr. The authentication process can be boiled down to three discrete steps:

  1. Get a frob. Now I have no idea what a “frob” is, but it appears to be a nonce, which in security terms is a one-use random number. You log in with the key and secret and say, “gies a frob!” and you get a random number in return. This random number is stored by Flickr for an hour and then forgotten, so you have an hour to complete the next step.
  2. With your new frob and your authentication details you construct a web address which is unique to this instance, since the frob is (probably) unique and so is the key/secret. The user then has to load the page up and authorise your app.

    http://api.flickr.com/services/auth/?api_key=1234&frob=6789&perms=write&api_sig=162738

  3. Once that’s done you can request a token from the Flickr server. The token is only handed out once per login, so if it’s lost you need to start again at step one. The token can be saved to disk and re-used as many times as needed by this application, but the user may revoke access at any time from the Flickr website, so be ready to discard saved tokens.

The specification requires that you let users “log out” too. Since there is no way to revoke your own permissions remotely you can simply delete any local tokens you have, which is as good.

I managed all of the above fairly easily with the Haskell Flickr bindings put together by Sigbjorn Finne. The uploading stage is where I stalled and it seems to be a problem in the library since the included example programs give the same error message I’m getting:

toRequest: POST request contains 1 files; unable to represent as query string
Defaulting to multiform/form-data instead
Flickr error:
 
 Code: 96
 Type: Flickr API error
 Details: Invalid signature

I haven’t had a chance to capture what’s being sent yet and decipher where the problem lies. I’ve emailed the author to see if he knows any more but haven’t received a reply yet.

Comments Off

Nov 23 2010

Rounded rectangles

Published by Dougal under Programming

The latest addition to ComicBake is speech bubbles with rounded corners. The GD library bindings don’t currently support many drawing operations (though the library itself is quite complete) so I’m having to awkwardly construct shapes using the few primitives I have at my disposal.

The problem is simple — to round off the edges of an otherwise brutish rectangle:

roundRectExample0

The method is quite straightforward. The rounded rectangle is decomposed into six separate shapes. There is a full-height rectangle overlaid with a full-width rectangle, resulting in a fat cross shape. The little gaps in the corners are where the rounded corners need to go.

roundRectExample1

At this stage we’ve already decided something vital: how much of a curve should these rectangles have? I started off with the assumption that the radius of the curve shouldn’t be so large that two adjacent curves meet. There should always be a bit of straight line in between them.

I also wanted the radius of the horizontal and the vertical to be the same, so that the resulting shape is a proper circle not an ellipse. This cuts down on the number of things to reason about.

The radius here is calculated as 1/3 of the smallest dimension (width or height) of the rectangle. This works okay in the “middle ground” but gets silly if the rectangle is too big or too small. If the rectangle is too small then the curves look a bit rubbish because they get too triangular. So the minimum radius of the curve is set to the font size.

If the rectangle gets too big then the corner pieces become huge and the curved area starts to look more like a blob, which encroaches on the text inside. So the maximum is 3 times the font size.

With the given radius we draw the rectangles above and then create little circles of that size in the corners, centred on point which is 1 radius-length in from each edge:

roundRectExample2

When we use white instead of lots of bright colours for this we get a uniform shape which I’m happy to call a rounded rectangle.

roundRectExample3

Comments Off

Nov 21 2010

Hurling dwarves over other dwarves: probably not legal

Published by Dougal under Bugs, Games, Programming

It’s been a quiet week on the programming front but I’ve made steady changes to Thud since committing the initial code to BitBucket ten days ago.

I’ve added some unit tests, which was a simple thing to do but not very easy to get right. Unit tests are a bit like Cluedo, in that you have to convince yourself of something very complicated with a few well-chosen questions. And it’s easy to forget your assumptions and miss something vital.

Just today I spotted something by playing with the game. I realised it was possible to hurl or shove a piece over another to reach the enemy on the far side. The logic only checked that the target was within reach but not that it was within proper line of sight. (This goes back to the original implementation too, so it’s not just a feature of my mucking about with the code that deals with the game rules and breaking something.) The rules are silent on whether this is even allowed (I suspect not) so I’ve disallowed it for now.

This is not the kind of bug I would have caught with unit tests because it would never have occurred to me to check for it. Of course the butler didn’t commit the murder, that’s preposterous! I only spotted it during play because I was purposely lining up foes next to each other in order to manually test their capture capabilities and realised that I could hurl a dwarf further than necessary… and the game allowed it.

So I’ve been repeatedly rewriting the file that deals with game rules so that scenarios like this become more obvious. I’m getting to the point where I’m happy with this small segment so I need to push on to other things.

As I mentioned on another post it’s nice having an issue tracker for the code so I can write down any ideas that occur to me, whether they’re short-term or long-term concerns. I think it’s time to look at some of the more medium-term goals now, such as network interaction, otherwise I’ll spend the rest of my days endlessly fiddling with the game rules.

Comments Off

Nov 11 2010

Thud now visible on Bitbucket

Published by Dougal under Friends, Programming

I’ve set up an account on BitBucket and published the initial commits into a Thud repository. Mat’s also joined up so hopefully we can get this project ticking along nicely with some good changes. Also some testing, which it is currently lacking. I’ll have to look into whatever unit test frameworks are used in the C# world. (I’ve just started contributing to Banshee too, so I hope to take some of the practices from that larger project and try my hand at them on Thud.)

Most of what I’ve done to Thud so far is just importing the old code into Mercurial and uploading it to the BitBucket servers. Since the original code is all Mat’s my first problem was learning how to commit code in someone else’s name. Thankfully that wasn’t a difficult problem:

$ hg commit --user "A N Other <another@example.com"

Visual Studio and MonoDevelop both spew auto-generated files around your project directory and I’m not sure what most of them do. I tried not to include anything that wasn’t too much like source code, but it’s still possible I’ve included useless extras or omitted something that is needed for building the project. So that’ll be another learning curve where I must patch in necessary files.

The only new code I have added cleans up the logic used when moving pieces on the board. There are still plenty more changes to be made, especially to improve testability, and I’ve started storing these in the bug tracker. Having a bug tracker is quite cool. :-)

Comments Off

Nov 08 2010

Rewriting COND as a series of nested IF statements

Published by Dougal under Programming

It’s been a while since I mentioned SICP and this time I’ve got something worth mentioning. We’re on section 4 of the book now, which is writing a Scheme interpreter in Scheme. At the start the idea is just to get the basic syntax down but later on you change the evaluation model for fun and effect.

I’ve been working on exercise 4.5, which involves supported an “extended” syntax for conditional evaluation. It lets you pass the result from a test to a receiving function if the result evaluates to “true”.

(cond ((cons 1 2) => cdr)
      (else false))

That evalutes to 2. The standard is not clear if the predicate has to be pure. If I run the following in DrScheme:

(cond ((begin (print "once") (cons 1 2)) (cdr)))

I receive 2 back and see “once” printed once to the screen. But if the interpreter assumes that the test is (a) pure and (b) cheap to compute we can rewrite it like this:

(if (cons 1 2)
    (cdr (cons 1 2))
    (else false))

Rewriting the side-effecting code in this fashion would print “once” twice. Not good.

The following is more-or-less the way the book presents it to you, with my small changes to evaluate the extended syntax. It rewrites a series of cond expressions into nested if statements, which are evaluated until you find something that’s true or you reach the final else.

;; Assumes pure test predicates for (test => func) syntax because
;; the test is evaluated twice.
(define (expand-clauses clauses)
  (if (null? clauses)
      'false                          ; no else clause
      (let ((first (car clauses))
            (rest (cdr clauses)))
        (if (cond-else-clause? first)
            (if (null? rest)
                (sequence->exp (cond-actions first))
                (error "ELSE clause isn't last -- COND->IF" clauses))
            (make-if
             (cond-predicate first)
             (if (cond-extended? first)
                 (list (cond-extended-actions first) (cond-predicate first))
                 (sequence->exp (cond-actions first)))
             (expand-clauses rest))))))

The alternative is not to rewrite the cond syntax as a series of nested if statements. Instead we evaluate the predicate and pass that pre-evaluated value to the receiving function if it’s true. But damn if that isn’t a really difficult thing to do.

In the end I had to interleave evaluation and application of values, since the evaluator would always attempt to further evaluate any primitive value I gave it. (If I’m missing something really obvious here I’ll kick myself.)

;; Attempt to evaluate test predicate only once so that impure code
;; can be used without affecting program state twice.
(define (eval-cond-extended clauses env)
  (if (null? clauses)
      'false
      (let ((this (car clauses))
            (rest (cdr clauses)))
        (if (cond-else-clause? this)
            (if (null? rest)
                (eval (sequence->exp (cond-actions this)) env)
                (error "ELSE clause isn't last -- EVAL-COND" clauses))
            (let ((testval (eval (cond-predicate this) env)))
              (if (true? testval)
                  (if (cond-extended? this)
                      (apply (eval (cond-extended-actions this) env) (list testval))
                      (eval (sequence->exp (cond-actions this)) env))
                  (eval-cond-extended rest env)))))))

Notice the (apply (eval .. a few lines from the end there? That’s where it gets very ugly because the evaluator doesn’t accept pre-computed values.

The third alternative, and one that I haven’t looked at too closely because it seems fraught with other “interesting” issues, is inserting the computed value back into the environment so that the evaluator can see it.

Basically we would translate to this statement:

(let (result (test))
  (if result
      (function result)
      false))

This avoids the issue of computing something twice and running state-changing code twice but you have to drop variables into the interpreted environment which might already be used. If you go this route you either use some variable naming scheme that you can ensure doesn’t clash or deal with it. And for a toy evaluator, I think that’s too deep for me.

4 responses so far

Nov 04 2010

Do you know C-pound? How about C-thud?

Published by Dougal under Friends, Programming, Work

During this period of tedious and unfruitful job applications I’ve come to the conclusion that I’m either a horrible person on paper (I refuse to believe I’m a horrible person in real life) or I need more experience in the object-oriented managed-runtime languages. To wit, Java and C#.

The Java language has failed to get its act together in the last few years and, at least on paper, I have some experience with it since all my university work used Java. (I’m also watching Nick’s updates to remind myself. An RSS feed attached to his bitbucket account makes him dead easy to stalk learn from.) So I thought I’d get myself some experience in C#. I’ve nabbed a project that Mat made a couple of years ago (a game of Thud! from the Terry Pratchett novel of the name, itself based on a Norse board game) and, with his permission, I’m going to whip it into shape.

I know for a fact that it runs fine in Mono, since I knew that two years ago when he first knocked it out, and Mono has come on a long way since then. At the moment it doesn’t do very much and Mat admits that it isn’t great code on the inside either. That’s all fine by me though. I can familiarise myself with the code base by cleaning it up before I decide where the new features need to go.

I hope to keep this and ComicBake going alongside each other, since there are a number of differences besides the subject matter:

  1. Thud! doesn’t have to think much for itself. The program pits two users against each other across the network, so there’s no AI involved. The set of valid moves is very small so the game logic is not tricky. ComicBake, on the other hand, is me trying to encode a set of heuristics to simulate what an artist would do in the same situation.
  2. One is old-fashioned imperative code, in the object-oriented style. It’s fairly staid C# too, not using many of the recent innovations of the language which make it differ nowadays from Java. The other is obviously purely-functional Haskell: higher-order, expressively-typed and immutable.
  3. I can update and release ComicBake when I like. I haven’t checked with Mat whether he’s happy for me to publish the code/changes or whether he wants it just “between friends”. It can just be a learning experience, though obviously it’s a better one if I can point potential employers to it and say look, I can code.

The title is a reference to this Daily WTF classic.

Comments Off

Nov 02 2010

ComicBake hits version 0.1 and becomes almost usable!

Published by Dougal under Programming

It’s been such a long time since I looked at this stuff that I almost needed reminding what it was all about myself. So I’ll explain things more or less from the start.

The basic principle is that ComicBake can turn “scripts”, which resemble scripts as written by playwrights or screenwriters, and turn them into comic strips. This is the kind of output it produces (excuse the less-than-stellar dialogue) — speech bubbles tied to users, margins around the perimeter and multiple panels stitched together.

generated web comic

This is the script which defined the above image:

-- An Insult
-- A Series of Examples
-- 05 September 2009
-- Dougal Stanton
 
-- Photo CC-licenced from:
-- http://www.flickr.com/photos/clairity/154640125/
 
Scene 1. <scenes/conversation.png>
 
Man: This is going to be some one-sided
     chat as a demonstration of that
     kind of comic.
 
Scene 2.
 
Man: One character will have loads of
     stuff to say while the other will
     remain unnaturally silent for an
     extremely long time...
 
Scene 3.
 
Man: ...until finally, they leap out
     and attack with something both
     insightful and witty.
 
Woman: Yer maw.

The data which ties these two together is a “map” file for each background image. The script points to scenes/conversation.png so there’s also a conversation.map which has sets of co-ordinates for two entities called “man” and “woman”.

The magic incantation to pull all this together is:

$ comicbake --input scriptfile

Standard behaviour will produce comicstrip.png and comicstrip.N.png for panels 1 to N. (The individual panels are stitched together after they are written to disk.) You can choose different output names, and put the temporary panel files in a separate location:

$ comicbake --input scriptfile --tmpdir tmp/ --output mycomic

Full help available by running comicbake --help.

Things to note at the moment.

  • Obviously the speech placement isn’t perfect: the “punchline” should be further down the page so the reading order isn’t confused. Right now I’m just using the centre points of each rectangle, where really I should be ensuring that the top of the next bubble is below the bottom of the last bubble. All in good time!
  • I could probably serve to shrink the text or maybe the bubble margins a bit. In the second panel the man is nearly being decapitated by his own speech. Similarly, if there’s plenty of room it wouldn’t help to spread the bubbles out a bit. It’s hard to program this stuff without many examples at once: each change improves one comic but ruins another.
  • The preamble is parsed (title, author, etc.) but nothing is done with it yet. In future I want to place this information on the image (maybe in the margin at the bottom).
  • The text layout of each speech is drawn directly from how it’s written in the script. This makes it easier to adjust if you want to, but if you don’t really care then you still have to think about it. That being said, positioning of line breaks is pretty important with so small an amount of text. It would be foolish to remove that for greater simplicity.
  • I read somewhere that the library I’m using for text output respects inline HTML tags for styling. I haven’t played with that yet so it may not work. More importantly it may screw things up. Something to experiment with.
  • I’ve been experimenting with thought bubbles (you know, the clouds with trailing smaller clouds) but haven’t decided how to treat that in text. What would be the best way of choosing a thought bubble over a speech bubble? Would this be awkward?

    Garfield: (Thinking) I hate Mondays

    Would I want something less obtrusive?

  • For lack of a better system I’ve riddled the code with TODO and guesswork comments. These show where a shortcut was taken or where a design decision was made without empirical evidence. My hope is to go back and rectify some of those issues.

As always the code is publically available and I’m happy to take patches, suggestions, ideas and links to pictures of cats. The repository does have a README file which is sadly not as accurate or helpful as it could be. My next proper step should probably be documentation so others can use the program and understand the code.

5 responses so far

Oct 24 2010

More comic generation work, finally

Published by Dougal under Programming

I’ve spent the last few days heavily reworking my comic book generation code in order to finally get over some of my conceptual and logistical humps. I’m not yet at the stage where I have redone all my undoing, ie I’ve taken the entire contents of the cupboard and spread it out across the floor in order to tidy up. :-) So it doesn’t compile at the moment.

Placing speech bubbles has always been the nub of this program and one that I’ve never satisfactorily solved. Last time I tried simulated annealing but was never able to come up with good heuristics to guide the evolution of a solution. I’ve still got the code, so I may experiment with it in the future if I have the inclination.

The current angle of attack is more straightforward. Speech bubbles can be placed at any point along a set of “arms” which emanate from the character. (I was actually thinking about the clip clock we have on our kitchen wall when I imagined this.) For simplicity’s sake the arms are the four compass points and their intermediate diagonals.

I’m then just searching through each speaker’s arms for the first valid position and trying again with the next speaker until all the elements in the panel are done. If at any point we can’t place an element I’m just backtracking: depth-first search on element positions.

I haven’t got to the stage where I can see output from this approach but there will inevitably be problems. The most obvious at the moment is that the search stops when all elements can be placed, rather than being in their optimal positions. As always it comes down to the thorny problem of teaching the computer what “optimal positioning of speech bubbles” actually means. And sadly I don’t have an answer, though every new approach I take results in some interesting ideas. This is inevitably what happens when you try to automate artistic judgement.

For now I will follow the worse-is-better dictum and stop fiddling with speech placement until the rest of it is in place. I’m currently looking at the next stage in the pipeline, putting the graphics together. As with all things graphics-related there’s a lot of fiddling with co-ordinates and printing out (x,y) values and munging in order that various primitives (drawing ellipses, rectangles, text) all work off the same origin. Hopefully I’ll soon have some clean abstraction so that the elements I want will be drawn where I want them. Til next time!

Comments Off

Oct 07 2010

Learn Web Development in PHP in 24 hours

Published by Dougal under Programming, Work

NB: I wrote this some time ago. I didn’t get through to the next round of interview.

I just spent the last ~24 hours writing the biggest chunk of PHP I’ve ever had to work with. This isn’t much of a record, as up until now I’ve never done any PHP/web work outside of the occasional excursion into the WordPress plugins to debug problems. So I went from no PHP experience to writing a simple web app, which gave me quite a feeling of accomplishment.

This was all part of an interview for IDE Group. A fortnight ago I went for an interview, and then they sent me a problem to code the solution for. The problem involved setting up a database for users and related data, querying and updating it via web forms and so on.

Since this was the first time I’d ever done something like this (ie web programming) it was a massive learning experience. Every few steps I rediscovered things, like the need for session management, or data validation procedures, or authentication processes, and so on. There were many many things that I just had to skip over for being too cumbersome to implement quickly — a proper DB schema with validating constraints rather than a jumbled mess, salted-hashed passwords in the database, web pages that conform to any standard, never mind a recent one. The list goes on.

But I tried to get the basics done. I specifically wanted to concentrate on security of the application, since that’s something which is important for the integrity of the application. I didn’t want Little Bobby Tables to make a mess of my database. To that end I was really limiting in what counted as a valid username/password character. I didn’t know how to properly validate password data so I went the route of only accepting alphanumeric sequences. Little Bobby Tables wouldn’t be able to sign up, but at least the DB should still be standing if he tries. The web page itself looked very 1994. No CSS for you!

I handed in my solution about a day after receiving the problem. I was worried that taking this long would count against me, but there was nothing I could really do about that since I didn’t have any experience in what I was doing. I spent a lot of time researching each step and a lot of time debugging because of complete unfamiliarity with the language. The problem statement suggested a couple of frameworks for “bonus points”, though I didn’t use them. I considered that learning how to set up, integrate and use two additional systems over and above what I needed to get the job done would really have killed me.

In hindsight it seems this was the wrong choice. I have since learned from the interviewer that other candidates took much longer (about 3 days in one case). My thinking at the time was that anyone who knew what they were doing would get the whole thing finished in a working day so I wanted to give myself no more than 50% over that. I didn’t want to hand in something really slick that took me a whole week longer than everybody else, so I erred on the side of speed.

In future maybe I will be less cautious about investing extra time, though maybe there won’t be a future. These programming problems don’t seem very common in the UK. Anyone else done this kind of “interview homework” problem before? I can see why they are useful but I can also imagine they would be impossible for someone in a full-time job.

Comments Off

« Prev - Next »