Fun With Software

17 January 2020

The impetus for personal projects is, for me, often like coming up with a joke. You want to tell it as soon as possible, but you also want to tell it as well as possible. And you don’t want anybody else to do it before you, because nobody cares about you coming up with the same joke a bit later.

That means my thought process for making things is often a bit like this:

  • Somebody should do X
  • That sounds quite easy.
  • If I don’t do it, somebody else will.
  • If it’s easy, that will be quite soon.
  • I should do X as soon as possible.

As time has gone on, my difficulty threshold has gone down: if it’s going to take more than an evening’s work, I’m not really sure I can be bothered. Especially if it requires hosting and maintenance. But sometimes, the perfect set of conditions arrive, and I need to make some nonsense.

This is how I ended up writing a somewhat silly SparkAR filter in an afternoon.

SparkAr is Facebook’s platform for making “augmented reality” effects for Instagram and Facebook camera. That translates into “realtime 2D/3D image manipulation”, rather than anything remarkable involving magic glasses. For the time being, anyhow.

Effects might work in 2D, through pixel processing or compositing, or by using 3D assets and technologies like head- and gesture-tracking to map that 3D into the scene. Some “effects” are like filters. They stylize and alter an image, much like the photo filters we know and love, and you might want to use them again and again.

Others are like jokes. They land strongly, once, and from that point on with diminishing returns. But the first point of landing is delightful.

Mgs point

In the popular Playstation game Metal Gear Solid from 1998, the hero, Solid Snake, sneaks around an Alaskan missile base, outgunned and outnumbered. In general, he will always do best by evading guards.

When a guard spots Snake, the guard’s attention is denoted by an instantly recognizable “alert” sound, and an exclamation mark hovering over his head. At this point, Snake must run away, or be hunted down.

The exclamation mark – and sound – have become a feature throughout the entire franchise, and gone on to be a bit of a meme.

That is all you need to know about Metal Gear Solid.

I wanted to to make a filter that recreated the effect of the alert, placing the exclamation mark above a user’s head, tracking the position of their head accurately, and – most importantly – playing that stupid sound.

In Spark this is, by most programming standards, trivial. Facebook supply many, many tutorials with the product (and, compared to the dark days of their API documentation nine or ten years ago, their new documentation standards are excellent).

I started with a template for making 3D hats, which matches a 3D object to a user’s head, and occludes any part of the the object falling ‘behind’ their head. Then I just had to lightly adjust it, replacing the hat with a flat plane that displayed a texture of the exclamation mark I’d done my best to recreate. That was half the work: putting the image in the right place.

What was more interesting was determining how the mark should pop-in, playing the iconic sound effect at the same time. The face-tracker offers lots of ways to extract the positions of facial geometry, and I spent a while tracking how far a mouth was open in order to trigger the effect. Eventually, though, I settled on the “raised eyebrows” outlet as being a much better vector for communicating surprise.

Spark screengrab2

Some brief faffing in Spark’s node-based programming environment later, and now raising eyebrows triggered the brief animation of the exclamation mark popping in, and the corresponding sound effect.

Aside: I am not the biggest fan of graph/node-based coding, partly because I’m not a very spatial thinker, and partly because I’m already comfortable with the abstractions of text-based code. But this model really does make sense for code that’s functioning more like a long-running pipeline than a single imperative script. You find this idiom in game-engine scripting (notably in Unreal 4), visual effects tools, realtime graphics tools, and similar, and it is a good fit for these tools. Also: the kind of people coming to work in SparkAR will likely have experience of 3D or video tools. Increasingly, as I pointed out to a friend, “everything is VFX“, and so graph-based coding is in increasingly more places than it used to be.

I also made sure I supported tapping the screen to toggle the effect’s visibility. After all, not everyone has reliably detectable eyebrows, and sometimes, you’d like to use this imagery without having to look surprised all the time.

Finally, I added some muddy scanlines to capture the feel of 90s-era CRT rendering: a slight tint for that authentic Shadow Moses feel, coupled with rolling animation of the texture.

With this all working, I promptly submitted it for review, so I could share it with friends.

The output looks like this:


As with every single workflow where there is a definite, final submission – be it compilation, code review, or manufacture – this is exactly the point in time when I realise I wanted to make a bunch of changes.

Firstly, I felt the effect should include some textual instructions; if you haven’t been shown it by me, you might not know how it works. So I worked up adding them.

Secondly, I felt the scanlines should be optional. Whilst they’re fun, some visual jokes might be better off without them. So I wrote some Javascript to make a native option picker at the bottom of the effect, with the default being “scanlines on”.

Once version one had been approved, the point release was quickly accepted. And then I shared the filter in my story, and wondered if friends would borrow it.

(“Borrow”: the only way to share filters/effects it to use them yourself. Once they’re published, you have to use them in a story post of your own. Then, if a friend wants to “borrow” it, they can grab the filter/effect from your story featuring that effect. In short: the only way to distribute them is through enforced virality. I can even share them outside my protected account. I am fine with this; I think it’s rather smart).

When I first shared it with friends, one wrote back: “yay for fun with software“.

It’s a while since I’ve worked on a platform that’s wanted to be fun. I’ve made my own fun with software, making tools to make art or sounds, for instance. But in 2020, so much of the software I use wants me to not have fun.

I made my first bot on Twitter a long time ago, and from there, Twitter became my platform of choice for making software-based jokes.

Twitter is now very hard to make jokes on. The word ‘bot’ has come to stand for not ‘fun software toy’ but ‘bad actor from a foreign state’. The API is increasingly more restricted as a result. I’m required to regularly log in to prove an account is real. My accounts aren’t real: they’re toys, automatons, playing on the internet.

I get why these restrictions are in place. I don’t like bad actors spreading misinformation, lies, and propaganda. But I’m still allowed to be sad the the cost of that is making toys and art on the platform. (An art project I built for the artist Erica Scourti was finally turned off once API restrictions made it unviable).

Most software-managed platforms are not places you can play any more. I understand why, but I still wish that wasn’t the case.

Yes, I know Facebook, who are hardly a great example of a good actor, are getting me to popularise their platform and work on it for free. It’s a highly closed platform, and I’m sure they’ll monetise it when they work out how. I’m giving them free advertising just by writing this.

But. Largely, what I made is a stupid visual effect that neither they nor I can effectively monetise, and it’s a joke is better told than not told. In that case, let’s tell the joke.

I showed it to my friend Eliot:

Every single time I hear that sound and see it working, I laugh. Actually, sometimes I don’t: I’m busy holding a straight face with my eyebrows up, and somebody else is laughing. Either way, somebody laughs. And that’s good enough for me.

I know, intuitively, this is a not unproblematic development platform. I know it’s not really ‘free’ in the slightest. But I write this because, right now, it was a quiet delight to be allowed to make toys and play on somebody’s platform, and one of the more pleasant platforms they run (if you keep it small, private, and friendly). I’m sure they’ll mall-ify this like everything else, but for now, I’m going to enjoy the play it enables.

It’s a while since I’ve made a toy that was so immediate to build, so immediately fun for others.

Yay for fun with software.

(If you want to try the effect yourself, open this link on your phone. It’ll open Instagram and let you try the effect.)


10 February 2014

Time to write up something that’s been sitting around on various disks for a while.

Many months ago, I saw Plotagon. It’s best explained as Xtranormal by way of The Sims: reasonable resolution, 3D-animated videos based on scripts; a desktop tool to generate them, and a site to host them.

Most interestingly, it’s scripted with what actually looks like movie scripts, and that got me thinking: what would it look like to feed it with procedurally generated scripts? Could you make the machine make videos? All I knew was two things:

  1. I know, for good or ill, how Markov chains work.
  2. All the scripts for Friends are transcribed on the internet.

After all, given Plotagon’s focus on semi-realistic forms, I decided that it was best suited to the great American artform of the 20th century: the sitcom.

The Infinite Friends Machine was born.

The machines does a few simple things. First, it scrapes Friends transcripts. For now, it works for most of Series 1. It then parses those scripts and chops them up into episodes, scenes, and lines attributed to individual characters. It also strips out some directions. Then, using all that, it offers ways to generate new scripts.

Markov Chains, as Leonard has frequently pointed out, are not always the best way of generating text alone, especially when the corpus you’re working from isn’t particularly consistent. He is, of course, right. Still, I enjoy the mental leap readers make in order to make generative prose actually make sense, and for this project, I mainly wanted to get to scripts as fast as I could.

Still, I didn’t want to hamper their relative crudeness, so I tried to skew things in their favour. To that end, the Infinite Friends Machine generates scripts by copying the structure of existing scripts. When it makes a new “episode”:

  1. it finds the scenes that are in the original episode it’s being copied from
  2. for each scene, it finds each line – who says a line at what point in the episode
  3. then, it generates a new line for the speaking character from their own corpus. That is: Joey only ever things derived from Everything Joey Has Ever Said. What this means is that the main cast have quite diverse things they might say, and the bit players pretty much only say the same thing. Gunther is quite boring.

That’s it. A few seconds later, it spits out a nonsensical episodes of friends. Here’s a scene:

Friends script
and this is a full episode.

The machine isn’t online because it’s quite crude and processor-intensive, but you can get at the sourcecode from Github.

Anyhow: machine to generate scripts. Next stage: get them into Plotagon.

This was where my troubles began. For starters, despite having a nice format for scripts, Plotagon really demands you enter them via its UI – you can’t paste a big block of text in, you have to enter it by hand. Painful.

Next: Plotagon only lets scenes have two characters in. I decided to make a single scene – the tag on the end of the episode. But this turned into many scenes in Plotagon, as four people in an apartment was a bit much for it. I had to keep track of who was where, who was talking to whom at any point.

And then I had to deal with the unfortunate truth: Plotagon is horrible. I mean, Xtranormal used its non-realistic avatars and computer-voices to comic extent. By contrast, here we had disappointing voice acting with clunky visuals. Also, I had to add some ‘acting’. This largely consisted of making Chandler say everything whilst doing the (crazy) emote, to really capture that Series 1 Matthew Perry vibe.

A quick sting later, and Infinifriends S1E1 existed:

It is not exactly high art.

Just one scene took long enough, and I think, proved my point to an extent, but probably can’t be improved on for now. I’m not sure if I’ll ever return to the Infinite Friends Machine, but it was an entertaining enough exercise, and the video rendition is probably worth it for the cringe factor alone.

Theme tune. Credits. Tune in next time.


I’ve been playing a little with Toca Boca‘s latest iOS toy: Toca Builders.

As you’d expect from Toca Boca, it’s charming: a straightforward, paired down implementation of an idea, with unambiguous UI and lovely character design.

Builders is Toca’s take on a block construction toy for small children. Initially, it might seem a bit clunky, a Minecraft pastiche that’s not nearly so sophisticated as Mojang’s original. After all, it’s a tiny play area compared to Minecraft – six blocks of height and relatively small X-Y dimensions.

It’s the plural in the title that makes it so interesting, though: builders. This is not (just) a game in which the user is a builder; it is a game about six individual builders (pictured above). Each has their own different ability: most can both construct and desturct; almost all can control the colour of blocks; some are better at changing blocks after the fact, others at sketching with. They each control slightly differently – and they each manifest in the landscape. You can swap between builders with a simple or menu, or by tapping on any that you can see.

It’s the manifestation and personification of the four builders that suddenly clicked for me. As I played this, I realised what it really was: Minecraft through the eyes of Seymour Papert. Lego as LOGO.

Papert explained the LOGO turtle as an “object-for-thinking-with“. Not just a device to command, attached to a programming language; a device that you see the world through. Or as Papert says in Mindstorms, his wonderful book about the development and intent behind LOGO:

“objects in which there is an intersection of cultural presence, embedded knowledge, and the possibility for personal identification.”

He makes what I think is a clearer point later, though, and which I think Toca Builders captures perfectly:

Even the simplest Turtle work can open new opportunities for sharpening one’s thinking about thinking: Programming the Turtle starts by making one reflect on how one does oneself what one would like the Turtle to do. Thus teaching the Turtle to act or to ‘think’ can lead one to reflect on one’s own actions and thinking. And as children move on, they program the computer to make more complex decisions and find themselves engaged in reflecting on more complex aspects of their own thinking.

Papert is sometimes quite wordy. When I explain this to people, I tend to say: the value of the Turtle is that when you are stuck, you solve the problem by pretending to be the Turtle. This is especially valuable for young learners; as Papert points out later:

Children can identify with the Turtle, and are thus able to bring their knowledge of their bodies and how they move into their work of learning formal geomtetry

There are some lovely photos in Mindstorms of kids on a playing field, practicing Logo; one of them is being the Turtle, and the others are telling her what to do. The second you act out a program for yourself (or watch another child follow your instructions to the letter), you see how literal you need to be, or which line of code is ambiguous. You begin to see how the computer processes information (“thinks” being, unfortunately, an entirely inaccurate word).

This kind of embodied representation of computational logic is very rare. Often, the hard things in computer science are very abstract. I do not know how to “pretend to be the compiler”; I just have to trust input and output.

Toca Builders takes the abstract building of Minecraft – tools attached to a disembodied perspective (albeit one hindered by some degree of personhood – factors such as gravity, and so forth) – and embodies them to help younger children answer the question which tool would you use to place a block where you need to? Or sometimes backwards: which block shall we place next? It is not quite as freeform as Minecraft, but it actually forces the user to think a little harder about planning ahead, lining up his builders, and which builders go together well. Measure twice, cut once.

To that end, it’s much more like real-world building.

Papert was very clear about one particular point: the value of this is not to think in mechanical ways; it’s actually the opposite. By asking children to think in a mechanical way temporarily, they end up thinking about thinking more: they learn that there are many ways to approach a problem, and they can choose which way to think about things; which might be most appropriate.

And so Toca Builders is, in many ways, like all good construction toys: it’s about more than just building. It’s about planning, marshalling, making use of a limited set of tools to achieve creative goals. And all the while, helping the user understand those tools by making them appear in the world, taking up space in it, colliding with one another, and needing moving. All so that you can answer the question when you’re stuck: well, if you were Blox the Hammer, what would you do?

Some of what looks like clunkiness, then, is actually a subtle piece of design.

If you’re interested in the value of using computers to teach – not using computers to teach about computers, but using computers to teach about the world, then Mindstorms is a must-read. It’s easy to dismiss LOGO for its simplicity, and to forget the various paradigms it bends and breaks (more so than many programming languages) – and it’s remarkable to see just how long ago Papert and his collaborators were touching on ideas that are still fresh and vital today.

  • "At the moment, however, the prevailing wisdom seems to be that audiences have to be tricked into buying digital toys. Toys have to be disguised as something else. They don't yet have the framework of expectations around them that allows people to decide whether the proposition is worth it on its own or not, whatever that phrase really means. They're yet to feel entirely legitimate." Lots of lovely stuff in Christian's article here, but this stood out particularly: having to disguise toys to sell them to current expectations and the current marketplace.
  • "Over the past few months I have been collaborating with her to curate her first ever career-spanning exhibition. Retrospective Posy Simmonds: Essentially English opens on June 12th at the beautiful Art Nouveau, Victor Horta-designed Belgian Comic Strip Centre in Brussels and continues until November 25th 2012. I’ll be adding photos from the exhibition shortly, but below are the texts I have written for the explanatory graphic panels." Paul Gravett on Posy Simmonds – some great sketches in here and details of early work.
  • "[Bradbury] told them about a child he had watched, teased by his friends for wanting to enter a toy shop because they said it was too young for him, and how much Ray had wanted to persuade the child to ignore his friends and play with the toys." That, forever.
  • "It should be pointed out, however, that physics is not the only systemic toy upon which fun games can be built. Probability fields, such as those forged by the colours, numbers and suits in a deck of cards, and the stochastic patterns that emerge from mixing those cards up, are another well-known toy upon which many great games are built. In fact, there is a literal infinity of foundational systemic toys upon which meaningful games can be built, yet for the most part, the game industry focuses on building baseline game engines that simulate one single toy that is proven to only be marginally fun: physical reality."