Sam Frances' Blog

"Smart constructors" in Typescript

2021-08-19T13:55:00+01:00

I want to share a small but very useful pattern which I discovered on the pages of two highly recommended books:

In functional languages this pattern generally gets called "smart constructors", and it is the partial fulfilment of a wish that I have had for some time:

I wish the type system could enforce more interesting constraints than "this must be a string" or "this must be a number", like "this must be a number between 1 and 5" or "this string must be a valid email address".

A motivating example

At Cydar, we use a distributed content-addressable storage system we call "The Disthashbin", which lets you store and access files using their sha256 hash - or as we say around here, its "hashbin ref".

Here's a very simplified outline of the client interface.

class HashbinClient {

  get_blob(ref: string) {
    console.log(`Getting blob: ${ref}`);
  }

}

You use it as follows:

const hb = new HashbinClient();

const validRef = "beefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeef";

hb.get_blob(validRef);

In reality it would return some sort of Blob object with the content of the file, but you get the idea.

An obvious issue with this code is that it will just as happily accept a string which is not a valid sha256 hash. The following will compile just fine:

const badRef = "this is not a valid sha256 hash";

hb.get_blob(badRef);

A validation function

If you're anything like me, your first instinct is to write a function like this:

function validateRef(ref: string) {
  const sha256regex = /^[A-Fa-f0-9]{64}$/;
  if (!sha256regex.test(ref)) {
    throw new Error(`Invalid ref: ${ref}`);
  }
}

class HashbinClient {

  get_blob(ref: string) {
    validateRef(ref);
    console.log(`Getting blob: ${ref}`);
  }

}

However, now if we extend the API of the HashbinClient class, we have to remember to use the validateRef function in every public method that accepts a hashbin ref.

class HashbinClient {

  get_blob(ref: string) {
    validateRef(ref);
    console.log(`Getting blob: ${ref}`);
  }

  get_file(ref: string) {
    validateRef(ref);
    console.log(`Getting file: ${ref}`);
  }

  exists(ref: string) {
    validateRef(ref);
    console.log(`Checking if ref exists: ${ref}`);
  }

  del(ref: string) {
    validateRef(ref);
    console.log(`Deleting: ${ref}`);
  }

}

We are also faced with a dilemma when it comes to helper methods. Should a helper method assume that the ref has already been validated at in the API method that calls it, or should it revalidate?

class HashbinClient {

  get_blob(ref: string) {
    validateRef(ref);
    console.log(`Getting blob: ${ref}`);
    helper(ref);
  }

  private helper(ref: string) {
    // validateRef(ref);   // Yay or nay?
    console.log("Do I revalidate or don't I?");
  }

}

Ultimately the only way to be sure is to check every caller of the helper function, or if that caller itself receives a ref, each of the caller's callers, and so on.

Eventually, you (or another developer who is less familiar with the codebase) will forget to validate a ref in the right place, and you'll end up with an invalid hashbin ref somewhere deeper in your call stack.

A smart constructor

The first step towards a better solution is to stop thinking in terms of "strings which are valid hashbin refs", and instead think of hashbin refs as their own type. In other words, we need to get over our primitive obsession.

We move our validation code into the Ref constructor, making it impossible to create an invalid hashbin ref.

// ref.ts

const sha256regex= /^[A-Fa-f0-9]{64}$/;

export default class Ref {
  value: string;
  constructor(value: string) {
    if (!sha256regex.test(value)) {
      throw new Error(`Invalid ref: ${value}`);
    }
    this.value = value;
  }
}

We then update our HashbinClient accordingly.

class HashbinClient {

  get_blob(ref: Ref) {
    console.log(`Getting blob: ${ref.value}`);
  }

  get_file(ref: Ref) {
    console.log(`Getting file: ${ref.value}`);
  }

  exists(ref: Ref) {
    console.log(`Checking if ref exists: ${ref.value}`);
  }

  del(ref: Ref) {
    console.log(`Deleting: ${ref.value}`);
  }

This falls short of the dream of enforcing validity in the type-system itself, but it does use a combination of type-checking and runtime validation to ensure that you can never encounter an invalid hashbin ref.

In other words, if you have a function (method, class etc.) which requires a Ref instance, you can be sure that the body of that function will never run with an invalid hashbin ref, because to do that you would have to first create an invalid hashbin ref to pass to the function, and we've made that impossible.

const validRefString = "beefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeef";

const badRefString = "this is not a valid hashbin ref";

// This runs fine
hb.get_file(new Ref(validRefString));

// This throws an error *before* get_file() has a chance to run
hb.get_file(new Ref(badRefString));

// This won't compile
hb.get_blob(badRefString);

Making sure values stay valid

You've probably already spotted a flaw in the code above, for although we have prevented you from creating an invalid hashbin ref, you can take a valid ref and then make it invalid.

const validRef = new Ref(validRefString);
validRef.value = badRefString;

In functional languages, this wouldn't be an issue, since data structures are generally immutable by default. But this is easy to fix with Typescript's readonly modifier.

export default class Ref {
  readonly value: string;
  constructor(value: string) {
    if (!sha256regex.test(value)) {
      throw new Error(`Invalid ref: ${value}`);
    }
    this.value = value;
  }
}

Structural typing gotchas

Typescript's structural type system also presents another escape hatch. This is because the type checker isn't checking whether items passed to a Ref parameter are instances of the Ref class. It only checks if they have the same structure as an instance of the Ref class, which in this case just means having a field called value which stores a string.

// This compiles
const sneakyRef: Ref = { value: "not a valid ref" };

To get around this, we have to use a neat trick with unique symbol to simulate nominal typing.

// ref.ts

const sha256regex= /^[A-Fa-f0-9]{64}$/;

declare const RefType: unique symbol;

export default class Ref {
  [RefType]: void;
  readonly value: string;
  constructor(value: string) {
    if (!sha256regex.test(value)) {
      throw new Error(`Invalid ref: ${value}`);
    }
    this.value = value;
  }
}

This unique symbol type is private to the ref module. As long as we don't export it, nothing outside of the ref module can create something using that symbol.

Now, the following will not compile:

// This won't compile any more
const sneakyRef: Ref = { value: "not a valid ref" };

The great thing about this technique is that it only applies at the type level. If you look at the generated JavaScript code, the [RefType] field is removed.

const sha256regex = /^[A-Fa-f0-9]{64}$/;
export default class Ref {
    constructor(value) {
        if (!sha256regex.test(value)) {
            throw new Error(`Invalid ref: ${value}`);
        }
        this.value = value;
    }
}

This means that if you need to use instances of Ref in contexts where you only want to use serializable values (e.g. as part of redux state), this won't present an issue.

Private constructor

Another way to get around our smart constructor is to create a subclass of Ref that will nullify our validation.

const validRefString = "beefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeef";

export default class SneakyRefSubclass extends Ref {
  value: string;
  constructor(value: string) {
    super(validRefString);
    this.value = value;
  }
}

Any parameter which requires a Ref will also accept an instance of a subclass of Ref.

// Disaster!

// This runs without raising an error
const sneakyRef = new SneakyRefSubclass("bad ref");

// And this compiles!
hb.get_file(sneakyRef);

To stop this from happening, we have to forbid subclassing of Ref. At the time of writing, Typescript doesn't have a final modifier, so you can't do this:

final class Ref {
    // etc.
}

What you can do is make the constructor private, which prevents subclassing, and also stops code outside of the class from creating an instance using new.

export default class Ref {
  [RefType]: void;
  readonly value: string;
  private constructor(value: string) {
    if (!sha256regex.test(value)) {
      throw new Error(`Invalid ref: ${value}`);
    }
    this.value = value;
  }

  // This is how we create instances now
  static create(value: string) {
    return new Ref(value);
  }
}

Now our SneakyRefSubclass will no longer compile.

Complete example

Here's the complete example, with a bit of tidying up here and there.

// ref.ts

const sha256regex= /^[A-Fa-f0-9]{64}$/;

declare const RefType: unique symbol;

class Ref {
  [RefType]: void;
  readonly value: string;
  private constructor(value: string) {
    if (!sha256regex.test(value)) {
      throw new Error(`Invalid ref: ${value}`);
    }
    this.value = value;
  }

  static create(value: string) {
    return new Ref(value);
  }
}

// Only export the type, since clients of this module
// don't need access to the runtime class
export type { Ref };

// No need to reveal externally that we're using a class,
// So export a factory function
export default function create(value: string) {
  return Ref.create(value);
}

// hashbin-client.ts

import type { Ref } from "./ref";

export default class HashbinClient {

  get_blob(ref: Ref) {
    console.log(`Getting blob: ${ref.value}`);
  }

  get_file(ref: Ref) {
    console.log(`Getting file: ${ref.value}`);
  }

  exists(ref: Ref) {
    console.log(`Checking if ref exists: ${ref.value}`);
  }

  del(ref: Ref) {
    console.log(`Deleting: ${ref.value}`);
  }

}

// index.ts

import createRef from "./ref";
import HashbinClient from "./hashbin-client";

const hb = new HashbinClient();

const validRefString = "beefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeef";

const badRefString = "this is not a valid hashbin ref"

hb.get_file(createRef(validRefString));

// Throws an error
hb.get_file(createRef(badRefString));

Conclusion: Why care?

Ultimately, this pattern reduces the burden of validation by removing the dilemma of where and when to run the validation function. You run validation code when creating a value, and from that point onwards you can trust that you have a valid item. That alone is worth the price of having to wrap your primitive values.

But the truth is, you should probably be wrapping your primitive values anyway. Conceptually, a string is rarely a string, but a name, an email address, a uuid etc.; an integer is rarely an integer - rather it's a temperature, a timestamp, a width, an age etc.

NASA found this out to their peril in 1999.

The Mars Climate Orbiter crashed and disintegrated in the Mars atmosphere because a component developed by Lockheed provided momentum measured in pound-force seconds, while another component developed by NASA expected momentum as Newton seconds. -- Vlad Riscutia

Who knows? Maybe this disaster could have been averted by function parameters which accepted NewtonSeconds or PoundForceSeconds rather than int.

I also want to emphasize that smart constructors are not the only way to enforce complex constraints using the type system.

Take the related example of validating a hex colour. You could use a smart constructor very similar to Ref, or you could do something like the following (warning: Not the most efficient way to store hex colours!):

type HexDigit = "0"|"1"|"2"|"3"|"4"|"5"|"6"|"7"|"8"|"9"|"a"|"b"|"c"|"d"|"e"|"f";

type HexColor = [HexDigit, HexDigit, HexDigit, HexDigit, HexDigit, HexDigit];

const red: HexColor = ["f", "f", "0", "0", "0", "0"];

const wont_compile: HexColor = ["f", "f", "0", "@", "0", "0"];

const wont_compile_either: HexColor = ["f", "f"];

How far these approaches will take you, and under what circumstances, is a discussion for another day.

Retiring the "Grauniad News" twitterbot

2021-04-03T16:00:00+01:00

After more than four years, I'm retiring the "Grauniad News" twitterbot. I'll be honest, I sort of forgot it was running! If you want to see misspelled Guardian headlines, you'll have to go to @guardian.

What I'm doing now

2021-04-03T15:08:00+01:00

(This is my now page. It tells you what I'm doing at the moment. It also reminds me what I have decided I want to be doing at the moment.)

Reading:

~~The Pragmatic Programmer, 20th Anniversary Edition by David Thomas and Andrew Hunt (finished)~~
~~Test Driven Development: By Example by Kent Beck (finished)~~
Algebra DeMYSTiFieD, Second Edition by Rhonda Huettenmueller
- Trying to slowly get my maths back to the level it once was, and beyond, with the ultimate goal of getting a solid grasp of linear algebra. Next after this will be Foundation Mathematics and then possibly Coding the Matrix or Linear Algebra: Step by Step.

Looking forward to reading:

Amazon Web Services in Action by Michael Wittig & Andreas Wittig
JavaScript Allongé, the "Six" Edition
Developing a Redux Edge by Johannes Lumpe, Karl Purkhardt, Art Muller, Darío Cravero & Ezekiel Chentnik
Docker in Action by Jeff Nickoloff
RabbitMQ in Depth by Gavin M. Roy
How Linux Works by Brian Ward
Foundations of Python Network Programming by Brandon Rhodes
ZeroMQ by Pieter Hintjens
Ansible for DevOps by Jeff Geerling
Haskell programming from first principles by Christopher Allen and Julie Moronuki
Foundation Mathematics by K.A. Stroud
Learn You Some Erlang for Great Good! by Fred Hebert
How to Design Programs (2nd Edition) by Matthias Felleisen, Robert Bruce Findler, Matthew Flatt and Shriram Krishnamurthi
So many books!!!

Studying:

UBCx: Software Development MicroMasters on Edx

Working on:

Lots of interesting stuff for Cydar, involving things like Django Rest Framework, Celery, Ansible, and AWS

What I'm doing now

2019-08-05T22:50:00+01:00

(This is my now page. It tells you what I'm doing at the moment. It also reminds me what I have decided I want to be doing at the moment.)

Reading:

Kubernetes in Action by Marko Lukša
~~Functional Programming in JavaScript by Luis Atencio~~ (finished)
~~Elixir in Action, Second Edition by Saša Jurić~~ (finished)
Algebra DeMYSTiFieD, Second Edition by Rhonda Huettenmueller
- Trying to slowly get my maths back to the level it once was, and beyond, with the ultimate goal of getting a solid grasp of linear algebra. Next after this will be Foundation Mathematics and then possibly Coding the Matrix or Linear Algebra: Step by Step.

Looking forward to reading:

Amazon Web Services in Action by Michael Wittig & Andreas Wittig
JavaScript Allongé, the "Six" Edition
Developing a Redux Edge by Johannes Lumpe, Karl Purkhardt, Art Muller, Darío Cravero & Ezekiel Chentnik
Docker in Action by Jeff Nickoloff
RabbitMQ in Depth by Gavin M. Roy
How Linux Works by Brian Ward
Foundations of Python Network Programming by Brandon Rhodes
ZeroMQ by Pieter Hintjens
Ansible for DevOps by Jeff Geerling
Haskell programming from first principles by Christopher Allen and Julie Moronuki
Foundation Mathematics by K.A. Stroud
Learn You Some Erlang for Great Good! by Fred Hebert
How to Design Programs (2nd Edition) by Matthias Felleisen, Robert Bruce Findler, Matthew Flatt and Shriram Krishnamurthi
So many books!!!

Studying:

UBCx: Software Development MicroMasters on Edx

Working on:

Lots of interesting stuff for Cydar, involving things like Django Rest Framework, Celery, Ansible, and AWS

What I'm doing now

2019-01-16T22:50:00+00:00

(This is my now page. It tells you what I'm doing at the moment. It also reminds me what I have decided I want to be doing at the moment.)

Reading:

Learn You Some Erlang for Great Good! by Fred Hebert
~~Test-Driven Development with Python, 2nd Edition by Harry J.W. Percival~~ (finished)
~~Practical Object-Oriented Design in Ruby by Sandi Metz~~ (finished)
How to Design Programs (2nd Edition) by Matthias Felleisen, Robert Bruce Findler, Matthew Flatt and Shriram Krishnamurthi
Algebra DeMYSTiFieD, Second Edition by Rhonda Huettenmueller
- Trying to slowly get my maths back to the level it once was, and beyond, with the ultimate goal of getting a solid grasp of linear algebra. Next after this will be Foundation Mathematics and then possibly Coding the Matrix or Linear Algebra: Step by Step.

Looking forward to reading:

Amazon Web Services in Action by Michael Wittig & Andreas Wittig
JavaScript Allongé, the "Six" Edition
Developing a Redux Edge by Johannes Lumpe, Karl Purkhardt, Art Muller, Darío Cravero & Ezekiel Chentnik
Docker in Action by Jeff Nickoloff
RabbitMQ in Depth by Gavin M. Roy
How Linux Works by Brian Ward
Foundations of Python Network Programming by Brandon Rhodes
ZeroMQ by Pieter Hintjens
Ansible for DevOps by Jeff Geerling
Haskell programming from first principles by Christopher Allen and Julie Moronuki
Foundation Mathematics by K.A. Stroud
So many books!!!

Studying:

UBCx: Software Development MicroMasters on Edx

Working on:

Lots of interesting stuff for Cydar, involving things like Django Rest Framework, Celery, Ansible, and AWS

What I'm doing now

2018-07-07T17:11:00+01:00

(This is my now page. It tells you what I'm doing at the moment. It also reminds me what I have decided I want to be doing at the moment.)

Reading:

~~Introducing Elixir by Simon St. Laurent and J. Eisenberg~~ (finished)
Test-Driven Development with Python, 2nd Edition by Harry J.W. Percival
~~Javacript with Promises by Daniel Parker~~ (finished)
+ Practical Object-Oriented Design in Ruby by Sandi Metz
+ How to Design Programs (2nd Edition) by Matthias Felleisen, Robert Bruce Findler, Matthew Flatt and Shriram Krishnamurthi
+ Algebra DeMYSTiFieD, Second Edition by Rhonda Huettenmueller
- Trying to slowly get my maths back to the level it once was, and beyond, with the ultimate goal of getting a solid grasp of linear algebra. Next after this will be Foundation Mathematics and then possibly Coding the Matrix or Linear Algebra: Step by Step.

Looking forward to reading:

Amazon Web Services in Action by Michael Wittig & Andreas Wittig
JavaScript Allongé, the "Six" Edition
Developing a Redux Edge by Johannes Lumpe, Karl Purkhardt, Art Muller, Darío Cravero & Ezekiel Chentnik
Docker in Action by Jeff Nickoloff
RabbitMQ in Depth by Gavin M. Roy
How Linux Works by Brian Ward
Foundations of Python Network Programming by Brandon Rhodes
ZeroMQ by Pieter Hintjens
Ansible for DevOps by Jeff Geerling
Haskell programming from first principles by Christopher Allen and Julie Moronuki
Foundation Mathematics by K.A. Stroud
So many books!!!

Studying:

UBCx: Software Development MicroMasters on Edx

Working on:

Lots of interesting stuff for Cydar, involving things like Django Rest Framework, Celery, Ansible, and AWS

Using generators to build async agnostic API clients

2018-06-28T20:02:00+01:00

I'm a big fan of asyncio, couroutines and async/await syntax in Python 3.5+. However, they come with some well documented downsides. Not least among these is the red/blue function problem. Coroutines can call plain functions, but only a coroutine can await another coroutine (unless you count running it by directly invoking the event loop). Meanwhile, if a coroutine calls a plain function which triggers some sort of blocking IO, the whole event loop is blocked.

I have found the red/blue divide particularly annoying in the context of web API clients. Take boto3, the popular client for AWS's various APIs. This only works in a synchronous context (unless you don't mind blocking the event loop for the duration of each request). Another project, aiobotocore, provides some of these capabilities in an async context, but doesn't support all services and operations.

Yet, many of the things API clients do have nothing directly to do with IO, async or otherwise. Preparing a HTTP request - determining what headers to use, constructing a request body, calculating signatures and other authentication details - is a purely functional, IO-free business. The same goes for interpreting a HTTP response.

A function or method in an API client is like a sandwich where constructing the request and interpreting the response are the two pieces of bread, and sending the request is the filling. We ought to be able to use the same bread for any type of sandwich, without caring if the filling is ham, cheese, or something else.

Ok, enough with the shaky metaphors. Time for some code.

The strategy I have stumbled upon is to write the pure request-construction / response-interpretation logic as a generator function, which yields in the middle. The generator yields an object providing details of the HTTP request it wishes to be sent, ceding control to "something else, I care not what" which executes the actual web request, and sends an object representing the response back into the generator. The generator then processes the response.

Here's a toy API client for The Internet Chuck Norris Database.

# agnostic_client.py

from typing import Iterable, NamedTuple, Dict, Any, Optional

import requests
from aiohttp import ClientSession

URL_TEMPLATE = "https://api.icndb.com/jokes/{id}/"


class Request(NamedTuple):
    method: str
    url: str
    json: Optional[Dict[str, Any]] = None


class Response(NamedTuple):
    status: int
    json: Optional[Dict[str, Any]] = None


def get_joke(id: int) -> Iterable[Request]:

    response = yield Request("GET", URL_TEMPLATE.format(id=id))

    try:

        if response.status != 200:
            raise JokeApiError("API request failed")

        data = response.json
        if data.get("type") == 'NoSuchQuoteException':
            raise NoSuchJoke(data.get("value", ""))
        if data.get("type") != "success":
            raise JokeApiError("API request failed")

        raise Return(data["value"]["joke"])

    except (JokeApiError, Return, StopIteration):
        raise
    except Exception as e:
        raise JokeApiError() from e


def call_api_sync(it):
    try:
        for req in it:
            response = requests.request(req.method, req.url, json=req.json)
            it.send(
                Response(status=response.status_code, json=response.json())
            )
    except Return as e:
        return e.value


async def call_api_async(it):
    async with ClientSession() as session:
        try:
            for req in it:
                async with session.request(
                        method=req.method,
                        url=req.url,
                        json=req.json) as res:
                    json = await res.json()
                    response = Response(status=res.status, json=json)
                    it.send(response)
        except Return as e:
            return e.value

class JokeApiError(Exception):
    pass


class NoSuchJoke(JokeApiError):
    pass


class Return(Exception):

    def __init__(self, value):
        self.value = value

I still haven't decided if this is a good strategy overall, although it certainly seems to achieve its immediate goal of allowing a large chunk of code to be shared between sync and async clients. Comments welcome.

¡Hasta pronto!

What I'm doing now

2018-05-28T00:23:00+01:00

(This is my now page. It tells you what I'm doing at the moment. It also reminds me what I have decided I want to be doing at the moment.)

Reading:

~~Learn Functional Programming with Elixir by Ulisses Almeida~~ (finished)
+ Introducing Elixir by Simon St. Laurent and J. Eisenberg
+ Test-Driven Development with Python, 2nd Edition by Harry J.W. Percival
~~Lightweight Django by Julia Elman and Mark Lavin~~ (finished)

Looking forward to reading:

Amazon Web Services in Action by Michael Wittig & Andreas Wittig
JavaScript Allongé, the "Six" Edition
Developing a Redux Edge by Johannes Lumpe, Karl Purkhardt, Art Muller, Darío Cravero & Ezekiel Chentnik
Docker in Action by Jeff Nickoloff
RabbitMQ in Depth by Gavin M. Roy
How Linux Works by Brian Ward
Foundations of Python Network Programming by Brandon Rhodes
ZeroMQ by Pieter Hintjens
Ansible for DevOps by Jeff Geerling
Haskell programming from first principles by Christopher Allen and Julie Moronuki
Foundation Mathematics by K.A. Stroud
Head First Algebra by Tracy Pilone & Dan Pilone
So many books!!!

Studying:

UBCx: Software Development MicroMasters on Edx

Working on:

Lots of interesting stuff for Cydar, involving things like Django Rest Framework, Celery, Ansible, and AWS

What I'm doing now

2018-03-28T00:23:00+01:00

(This is my now page. It tells you what I'm doing at the moment. It also reminds me what I have decided I want to be doing at the moment.)

Reading:

+ Learn Functional Programming with Elixir by Ulisses Almeida
+ Introducing Elixir by Simon St. Laurent and J. Eisenberg
+ Test-Driven Development with Python, 2nd Edition by Harry J.W. Percival
+ Lightweight Django by Julia Elman and Mark Lavin
~~Learn you a Haskell for Great Good by Miran Lipovaca~~ (finished)
~~Concurrency in Go by Katherine Cox-Buday~~ (finished)
~~Head First Ruby by Jay McGavren~~ (finished)
- I read this to provide a foundation for "Practical Object-Oriented Design in Ruby" by Sandi Metz.

Looking forward to reading:

Amazon Web Services in Action by Michael Wittig & Andreas Wittig
JavaScript Allongé, the "Six" Edition
Developing a Redux Edge by Johannes Lumpe, Karl Purkhardt, Art Muller, Darío Cravero & Ezekiel Chentnik
Docker in Action by Jeff Nickoloff
RabbitMQ in Depth by Gavin M. Roy
How Linux Works by Brian Ward
Foundations of Python Network Programming by Brandon Rhodes
ZeroMQ by Pieter Hintjens
Ansible for DevOps by Jeff Geerling
Haskell programming from first principles by Christopher Allen and Julie Moronuki
Foundation Mathematics by K.A. Stroud
Head First Algebra by Tracy Pilone & Dan Pilone
So many books!!!

Studying:

UBCx: Software Development MicroMasters on Edx

Working on:

Lots of interesting stuff for Cydar, involving things like Django Rest Framework, Celery, Ansible, and AWS

What I'm doing now

2017-12-11T00:32:00+00:00

(This is my now page. It tells you what I'm doing at the moment. It also reminds me what I have decided I want to be doing at the moment.)

Reading:

+ Concurrency in Go by Katherine Cox-Buday
~~Go in Action by William Kennedy~~ (finished)
Ansible for DevOps by Jeff Geerling
Haskell programming from first principles by Christopher Allen and Julie Moronuki
Foundation Mathematics by K.A. Stroud
Head First Algebra by Tracy Pilone & Dan Pilone

Looking forward to reading:

Test-Driven Development with Python, 2nd Edition by Harry J.W. Percival
Amazon Web Services in Action by Michael Wittig & Andreas Wittig
JavaScript Allongé, the "Six" Edition
Developing a Redux Edge by Johannes Lumpe, Karl Purkhardt, Art Muller, Darío Cravero & Ezekiel Chentnik
Docker in Action by Jeff Nickoloff
RabbitMQ in Depth by Gavin M. Roy
How Linux Works by Brian Ward
Foundations of Python Network Programming by Brandon Rhodes
ZeroMQ by Pieter Hintjens
So many books!!!

Studying:

UBCx: Software Development MicroMasters on Edx

Working on:

Lots of interesting stuff for Cydar, involving things like Django Rest Framework, Celery, Ansible, and AWS

What I'm doing now

2017-11-19T00:09:00+00:00

(This is my now page. It tells you what I'm doing at the moment. It also reminds me what I have decided I want to be doing at the moment.)

Reading:

~~Learning React by Alex Banks & Eve Porcello~~ (finished)
~~Understanding ECMAScript 6 by Nicholas C. Zakas~~ (finished)
~~Introducing Go by Caleb Doxsley~~ (finished)
Go in Action by William Kennedy
Ansible for DevOps by Jeff Geerling
Haskell programming from first principles by Christopher Allen and Julie Moronuki
Foundation Mathematics by K.A. Stroud
Head First Algebra by Tracy Pilone & Dan Pilone

Looking forward to reading:

Test-Driven Development with Python, 2nd Edition by Harry J.W. Percival
Amazon Web Services in Action by Michael Wittig & Andreas Wittig
JavaScript Allongé, the "Six" Edition
Developing a Redux Edge by Johannes Lumpe, Karl Purkhardt, Art Muller, Darío Cravero & Ezekiel Chentnik
Docker in Action by Jeff Nickoloff
RabbitMQ in Depth by Gavin M. Roy
How Linux Works by Brian Ward
Foundations of Python Network Programming by Brandon Rhodes
ZeroMQ by Pieter Hintjens
So many books!!!

Studying:

Working on:

Lots of interesting stuff for Cydar, involving things like Django Rest Framework, Celery, Ansible, and AWS

What I'm doing now

2017-04-24T08:32:00+01:00

(This is my now page. It tells you what I'm doing at the moment. It also reminds me what I have decided I want to be doing at the moment.)

Reading:

Ansible for DevOps by Jeff Geerling
~~The Goldfinch by Donna Tartt~~ (finished)
+ Understanding ECMAScript 6 by Nicholas C. Zakas
+ Haskell programming from first principles by Christopher Allen and Julie Moronuki
Foundation Mathematics by K.A. Stroud
Head First Algebra by Tracy Pilone & Dan Pilone

Looking forward to reading:

Test-Driven Development with Python, 2nd Edition by Harry J.W. Percival
Amazon Web Services in Action by Michael Wittig & Andreas Wittig
JavaScript Allongé, the "Six" Edition
~~Understanding ECMAScript 6 by Nicholas C. Zakas~~
Developing a Redux Edge by Johannes Lumpe, Karl Purkhardt, Art Muller, Darío Cravero & Ezekiel Chentnik
Docker in Action by Jeff Nickoloff
RabbitMQ in Depth by Gavin M. Roy
How Linux Works by Brian Ward
Foundations of Python Network Programming by Brandon Rhodes
ZeroMQ by Pieter Hintjens
So many books!!!

Studying:

Working on:

Lots of interesting stuff for Cydar, involving things like Django Rest Framework, Celery, Ansible, and AWS

Other things:

I'm getting back into studying Spanish after a few months off.

What I'm doing now

2017-02-14T22:25:00+00:00

(This is my now page. It tells you what I'm doing at the moment. It also reminds me what I have decided I want to be doing at the moment.)

Reading:

~~JavaScript: The Good Parts by Douglas Crockford~~ (finished)
Ansible for DevOps by Jeff Geerling
The Goldfinch by Donna Tartt
+ Foundation Mathematics by K.A. Stroud
+ Head First Algebra by Tracy Pilone & Dan Pilone

Looking forward to reading:

Test-Driven Development with Python, 2nd Edition by Harry J.W. Percival
Amazon Web Services in Action by Michael Wittig & Andreas Wittig
JavaScript Allongé, the "Six" Edition
Understanding ECMAScript 6 by Nicholas C. Zakas
Developing a Redux Edge by Johannes Lumpe, Karl Purkhardt, Art Muller, Darío Cravero & Ezekiel Chentnik
Docker in Action by Jeff Nickoloff
RabbitMQ in Depth by Gavin M. Roy
How Linux Works by Brian Ward
Foundations of Python Network Programming by Brandon Rhodes
ZeroMQ by Pieter Hintjens
So many books!!!

Studying:

Working on:

Lots of interesting stuff for Cydar, involving things like Django Rest Framework, Celery, Ansible, and AWS

Other things:

I'm getting back into studying Spanish after a few months off.

What I'm doing now

2017-02-11T16:53:00+00:00

(This is my now page. It tells you what I'm doing at the moment. It also reminds me what I have decided I want to be doing at the moment.)

Reading:

JavaScript: The Good Parts by Douglas Crockford
Ansible for DevOps by Jeff Geerling
The Goldfinch by Donna Tartt
+ Foundation Mathematics by K.A. Stroud
+ Head First Algebra by Tracy Pilone & Dan Pilone

Looking forward to reading:

Test-Driven Development with Python, 2nd Edition by Harry J.W. Percival
Amazon Web Services in Action by Michael Wittig & Andreas Wittig
JavaScript Allongé, the "Six" Edition
Understanding ECMAScript 6 by Nicholas C. Zakas
Developing a Redux Edge by Johannes Lumpe, Karl Purkhardt, Art Muller, Darío Cravero & Ezekiel Chentnik
Docker in Action by Jeff Nickoloff
RabbitMQ in Depth by Gavin M. Roy
How Linux Works by Brian Ward
Foundations of Python Network Programming by Brandon Rhodes
ZeroMQ by Pieter Hintjens
So many books!!!

Studying:

Working on:

Lots of interesting stuff for Cydar, involving things like Django Rest Framework, Celery, Ansible, and AWS

Other things:

I'm getting back into studying Spanish after a few months off.

What I'm doing now

2017-02-02T00:26:00+00:00

(This is my now page. It tells you what I'm doing at the moment. It also reminds me what I have decided I want to be doing at the moment.)

Reading:

~~TypeScript for Javascript Programmers by Steve Fenton~~
JavaScript: The Good Parts by Douglas Crockford
+ Ansible for DevOps by Jeff Geerling
The Goldfinch by Donna Tartt

Looking forward to reading:

Test-Driven Development with Python, 2nd Edition by Harry J.W. Percival
~~Ansible for DevOps by Jeff Geerling~~
Amazon Web Services in Action by Michael Wittig & Andreas Wittig
JavaScript Allongé, the "Six" Edition
Understanding ECMAScript 6 by Nicholas C. Zakas
Developing a Redux Edge by Johannes Lumpe, Karl Purkhardt, Art Muller, Darío Cravero & Ezekiel Chentnik
Docker in Action by Jeff Nickoloff
RabbitMQ in Depth by Gavin M. Roy
How Linux Works by Brian Ward
+ Foundations of Python Network Programming by Brandon Rhodes
+ ZeroMQ by Pieter Hintjens
So many books!!!

Studying:

Working on:

Lots of interesting stuff for Cydar, involving things like Django Rest Framework, Celery, Ansible, and AWS

Other things:

I'm getting back into studying Spanish after a few months off.
I have ordered a running machine! This way I can exercise in my pyjamas while watching Netflix. My plan is to make exercise so convenient that I can't find an excuse not to do it.

What I'm doing now

2017-01-30T22:12:00+00:00

(This is my now page. It tells you what I'm doing at the moment. It also reminds me what I have decided I want to be doing at the moment.)

Reading:

TypeScript for Javascript Programmers by Steve Fenton
+ JavaScript: The Good Parts by Douglas Crockford
~~Apprenticeship Patterns: Guidance for the Aspiring Software Craftsman by David H. Hoover & Adewale Oshineye~~ (finished)
The Goldfinch by Donna Tartt

Looking forward to reading:

Test-Driven Development with Python, 2nd Edition by Harry J.W. Percival
Ansible for DevOps by Jeff Geerling
Amazon Web Services in Action by Michael Wittig & Andreas Wittig
JavaScript Allongé, the "Six" Edition
Understanding ECMAScript 6 by Nicholas C. Zakas
Developing a Redux Edge by Johannes Lumpe, Karl Purkhardt, Art Muller, Darío Cravero & Ezekiel Chentnik
Docker in Action by Jeff Nickoloff
RabbitMQ in Depth by Gavin M. Roy
+ How Linux Works by Brian Ward
So many books!!!

Studying:

Working on:

Lots of interesting stuff for Cydar, involving things like Django Rest Framework, Celery, Ansible, and AWS

Other things:

I'm getting back into studying Spanish after a few months off.
I have ordered a running machine! This way I can exercise in my pyjamas while watching Netflix. My plan is to make exercise so convenient that I can't find an excuse not to do it.

What I'm doing now

2017-01-28T10:55:00+00:00

(This is my now page. It tells you what I'm doing at the moment. It also reminds me what I have decided I want to be doing at the moment.)

Reading:

TypeScript for Javascript Programmers by Steve Fenton
Apprenticeship Patterns: Guidance for the Aspiring Software Craftsman by David H. Hoover & Adewale Oshineye
The Goldfinch by Donna Tartt

Looking forward to reading:

Test-Driven Development with Python, 2nd Edition by Harry J.W. Percival
Ansible for DevOps by Jeff Geerling
Amazon Web Services in Action by Michael Wittig & Andreas Wittig
JavaScript Allongé, the "Six" Edition
Understanding ECMAScript 6 by Nicholas C. Zakas
Developing a Redux Edge by Johannes Lumpe, Karl Purkhardt, Art Muller, Darío Cravero & Ezekiel Chentnik
Docker in Action by Jeff Nickoloff
RabbitMQ in Depth by Gavin M. Roy
So many books!!!

Studying:

Working on:

Lots of interesting stuff for Cydar, involving things like Django Rest Framework, Celery, Ansible, and AWS

Other things:

I'm getting back into studying Spanish after a few months off.
I have ordered a running machine! This way I can exercise in my pyjamas while watching Netflix. My plan is to make exercise so convenient that I can't find an excuse not to do it.

More twitterbot silliness

2017-01-23T23:41:00+00:00

Not content to rest on my laurels after the roaring success that was @fight_guybrush, I have written another twitter bot, @grauniad_news, an exact replica of the twitter feed for The Guardian newspaper, but with added spelling mistakes.

(For any non-British readers, The Guardian is a UK newspaper with a reputation for spelling mistakes, among other things, giving rise to the nickname "The Grauniad".)

This bot is a slightly more sophisticated project than the last one, which was just a python script triggered hourly by cron. The new bot is written in node.js, uses Twitter's streaming API to monitor and respond to tweets in real time, and is hosted on an Amazon Web Services t2.nano instance, launched using Ansible. I use the Twit library to communicate with Twitter's API, and forever to ensure that the bot keeps running continuously.

The method of generating badly spelled tweets is quite simple, but even so, it taught me some important lessons. The most important lesson: watch out for infinite loops in node. Early on, I found that the twitterbot would freeze after a seemingly random interval during which it worked normally. No errors were logged, and forever list showed the program as still running.

The problem turned out to be in my simple function for introducing misspellings.

function misspellRandomWords(sentence) {
    let words = sentence.split(" ");
    const limit = words.length;

    // Choose a first word, filtering out urls
    var iFirstWord = Math.floor(Math.random() * limit);
    while (isLink(words[iFirstWord]) || words[iFirstWord][0] === "@" ) {
        iFirstWord = Math.floor(Math.random() * limit);
    }

    // Choose second misspelled word, and make sure it isn't the first or an URL
    var iSecondWord = Math.floor(Math.random() * limit);
    while (isLink(words[iSecondWord]) ||
            iSecondWord === iFirstWord ||
            words[iSecondWord][0] === "@") {
        iSecondWord = Math.floor(Math.random() * limit);
    }

    words[iFirstWord] = swapRandomLetters(words[iFirstWord]);
    words[iSecondWord] = swapRandomLetters(words[iSecondWord]);

    return words.join(" ");
}

This worked most of the time, but would get into an infinite loop in certain circumstances - for example, if every word in the tweet starts with "@".

I don't know if this is how a node.js expert would do it, but after resolving this bug, I started guarding my loops to catch any unforeseen conditions that would cause them to loop infinitely, and make sure that these were logged as errors.

function misspellRandomWords(sentence) {
// ...
// ...
    let nLoops = 0; // Infinite loop error detection
    while (!isSwappable(words[iFirstWord])) {
        iFirstWord = Math.floor(Math.random() * limit);

        // Infinite loop detection
        if (++nLoops > 1000) {
            console.error("Infinite loop detected on sentence: " + sentence);
        }

    }
// ...
// ...
}

If you're interested, you can take a look at the code on Github, or have a look at the bot on Twitter. Be warned, the bot is not sophisticated enough to know when a randomly misspelled headline might cause offence.

A Monkey Island inspired twitterbot

2015-08-08T01:07:00+01:00

Just for fun, I've made a rather silly twitterbot inspired by Monkey Island's 'insult sword-fighting'. For example, tweet "@fight_guybrush you fight like a dairy farmer" and it will reply "How appropriate! You fight like a cow!" within the hour. Other Monkey Island insults should work too. It uses the python tweepy library, which was a bit of a challenge as the documentation seems to be quite incomplete. I've used cron to schedule the bot script to run every hour. The code is Here: https://github.com/samfrances/threepwood

Relaunching with Pelican

2015-08-06T21:51:00+01:00

After realising that my django-powered site was running on a dangerously old version of django, I've decided to relaunch the site using pelican, a python static site generation framework. I reason that if it's good enough for django legend Daniel Roy Greenfeld it's good enough for me.

Most of the old posts aren't up yet, and I'm currently using a prepackaged theme, but I'm going to be working on adding the old content and writing my own pelican theme so that the site ends up looking like the old one. I'll also be adding disqus comments in the fullness of time, or working out a way of hosting my own comments.

¡Hasta pronto!

MVC todo list demo

2014-01-29T00:00:00+00:00

Here's a simple javascript todo list application, which I created as an exercise in javascript MVC without a framework.

It is influenced by this article. However, it differs by making each todo list item its own MVC triad.

The source code is on github.

I may explain the code in a later post.

A canvas drag and drop example

2012-10-08T00:00:00+01:00

I've recently unearthed an old piece of code which uses multiple layered HTML5 canvas elements to create a multi-layered drag and drop effect. The code is probably due some refinements, but I thought I'd post a link just for fun.

Drag and drop the characters and shapes (if you like):

Creating a blog with django, part 1: models

2012-07-03T23:43:00+01:00

In this tutorial, I'll walk you through one possible way of creating a basic blog application using django 1.3's class-based generic views, built-in admin interface and syndication framework.

Note: this tutorial was written for django 1.3, and is therefore out of date.

I'll assume that you are familiar with some django fundamentals, specifically:

Creating a new project
Creating a new app
Setting up a database and configuring django's database settings
The basics of django models, templates, views, urlconfs etc.

If you aren't familiar with these things, then I suggest you work through the 'first steps' tutorial on the django website.

Start with an existing django project or create a new one, and make sure the database settings are up and running. We'll call the project directory mysite for convenience. Create a new app called blog and open mysite/blog/models.py. Our models will include a Post class to represent blog posts and a Tag class to represent the different categories a post can be placed in:

# models.py

from django.db import models
from datetime import datetime

class Tag(models.Model):
    name = models.CharField(max_length=20, unique=True)

class Post(models.Model):
    title = models.CharField(max_length=120)
    slug = models.SlugField(max_length=120, unique_for_date='publication_date')
    publication_date = models.DateTimeField(default=datetime.now)
    body = models.TextField()
    tags = models.ManyToManyField(Tag)

Each Tag instance has only one attribute, a name, which is implemented as a CharField with a unique value (unique=True).

The Post class is somewhat more interesting. It has a title attribute, a main body of text a publication_date that defaults to the time at which the individual post was created.

The tags attribute implements a many-to-many relationship with Tag; Each blog post can fall under many categories, and each category can include many blog posts.

The slug attribute is a SlugField, which contains a 'slug' - a short, human-readable label for a blog post, used as part of its URL, and consisting only of letters, numbers, underscores and/or hyphens. Setting the unique_for_date option to 'publication_date' ensures that no two blog posts can have both the same publication_date and the same slug.

Finally, append 'mysite.blog' to INSTALLED_APPS in settings.py. Run python manage.py validate to check that your models are valid, and create the database tables corresponding to your models by typing `python manage.py syncdb.

Now you are ready for the next installment of this tutorial, which will intergrate our model with django's admin framework.

Creating a blog with django, part 2: admin interface

2012-07-03T23:43:00+01:00

In part2 of this series, we defined the models for our blog application. However, aside from python manage.py shell, we don't yet have a way to create and edit blog posts or categories. For this, we need to enable django's admin interface.

Note: this tutorial was written for django 1.3, and is therefore out of date.

If you haven't already done so, enable the admin interface by adding django.contrib.admin to the INSTALLED_APPS setting, and uncommenting the relevant lines in the main project urlconf, mysite/urls.py.

from django.conf.urls.defaults import patterns, include, url

# Uncomment the next two lines to enable the admin:
from django.contrib import admin
admin.autodiscover()

urlpatterns = patterns('',
    # Examples:
    # url(r'^$', 'mysite.views.home', name='home'),
    # url(r'^mysite/', include('mysite.foo.urls')),

    # Uncomment the admin/doc line below to enable admin documentation:
    # url(r'^admin/doc/', include('django.contrib.admindocs.urls')),

    # Uncomment the next line to enable the admin:
    url(r'^admin/', include(admin.site.urls)),
)

Create a file called admin.py in the directory containing the blog app.

First, we need to import the admin module and the models which we wish to add to the admin interface.

# admin.py

from django.contrib import admin
from mysite.blog.models import Post, Tag

Next we register our models with the admin site.

6 7	admin.site.register(Post) admin.site.register(Tag)

At this point if you start the test server and go to http://127.0.0.1:8000/admin/ you will see that the Post and Tag models have been added.

However, if you create a couple of blog post and then try to view the list of posts, this is what you will see:

In order to rectify this, we need to add a __unicode__ method to each of our models.

# models.py

class Post(models.Model):
....
    def __unicode__(self):
        return self.title

class Tag(models.Model):
....
    def __unicode__(self):
        return self.name

Finally, we want the admin interface to generate the slug field automatically from the title field. To do this, we add the following lines to the admin.py file.

6 7	class PostAdmin(admin.ModelAdmin): prepopulated_fields = {"slug": ("title",)}

We then add 'PostAdmin' as the second argument to the call which registers Post with the admin interface.

9	admin.site.register(Post, PostAdmin)

Now you will see that the slug field is automatically filled in when you enter a title for your blog post. Here's the complete code for reference:

# admin.py

from django.contrib import admin
from mysite.blog.models import Post, Tag

class PostAdmin(admin.ModelAdmin):
    prepopulated_fields = {"slug": ("title",)}

admin.site.register(Post, PostAdmin)
admin.site.register(Tag)

# models.py

from django.db import models
from datetime import datetime

class Tag(models.Model):
    name = models.CharField(max_length=20, unique=True)

    def __unicode__(self):
        return self.name

class Post(models.Model):
    title = models.CharField(max_length=120)
    slug = models.SlugField(max_length=120, unique_for_date='publication_date')
    publication_date = models.DateTimeField(default=datetime.now)
    body = models.TextField()
    tags = models.ManyToManyField(Tag)

    def __unicode__(self):
        return self.title

That concludes this installment. Next time we'll get to the real meat of the project - using Django's generic class-based views.

Sam Frances' Blog

"Smart constructors" in Typescript

A motivating example

A validation function

A smart constructor

Making sure values stay valid

Structural typing gotchas

Private constructor

Complete example

Conclusion: Why care?

Further reading

Retiring the "Grauniad News" twitterbot

What I'm doing now

What I'm doing now

What I'm doing now

What I'm doing now

Using generators to build async agnostic API clients

What I'm doing now

What I'm doing now

What I'm doing now

What I'm doing now

What I'm doing now

What I'm doing now

What I'm doing now

What I'm doing now

What I'm doing now

What I'm doing now

More twitterbot silliness

A Monkey Island inspired twitterbot

Relaunching with Pelican

MVC todo list demo

A canvas drag and drop example

Creating a blog with django, part 1: models

Creating a blog with django, part 2: admin interface