Fun With Software

17 January 2020

The impetus for personal projects is, for me, often like coming up with a joke. You want to tell it as soon as possible, but you also want to tell it as well as possible. And you don’t want anybody else to do it before you, because nobody cares about you coming up with the same joke a bit later.

That means my thought process for making things is often a bit like this:

  • Somebody should do X
  • That sounds quite easy.
  • If I don’t do it, somebody else will.
  • If it’s easy, that will be quite soon.
  • I should do X as soon as possible.

As time has gone on, my difficulty threshold has gone down: if it’s going to take more than an evening’s work, I’m not really sure I can be bothered. Especially if it requires hosting and maintenance. But sometimes, the perfect set of conditions arrive, and I need to make some nonsense.

This is how I ended up writing a somewhat silly SparkAR filter in an afternoon.

SparkAr is Facebook’s platform for making “augmented reality” effects for Instagram and Facebook camera. That translates into “realtime 2D/3D image manipulation”, rather than anything remarkable involving magic glasses. For the time being, anyhow.

Effects might work in 2D, through pixel processing or compositing, or by using 3D assets and technologies like head- and gesture-tracking to map that 3D into the scene. Some “effects” are like filters. They stylize and alter an image, much like the photo filters we know and love, and you might want to use them again and again.

Others are like jokes. They land strongly, once, and from that point on with diminishing returns. But the first point of landing is delightful.

Mgs point

In the popular Playstation game Metal Gear Solid from 1998, the hero, Solid Snake, sneaks around an Alaskan missile base, outgunned and outnumbered. In general, he will always do best by evading guards.

When a guard spots Snake, the guard’s attention is denoted by an instantly recognizable “alert” sound, and an exclamation mark hovering over his head. At this point, Snake must run away, or be hunted down.

The exclamation mark – and sound – have become a feature throughout the entire franchise, and gone on to be a bit of a meme.

That is all you need to know about Metal Gear Solid.

I wanted to to make a filter that recreated the effect of the alert, placing the exclamation mark above a user’s head, tracking the position of their head accurately, and – most importantly – playing that stupid sound.

In Spark this is, by most programming standards, trivial. Facebook supply many, many tutorials with the product (and, compared to the dark days of their API documentation nine or ten years ago, their new documentation standards are excellent).

I started with a template for making 3D hats, which matches a 3D object to a user’s head, and occludes any part of the the object falling ‘behind’ their head. Then I just had to lightly adjust it, replacing the hat with a flat plane that displayed a texture of the exclamation mark I’d done my best to recreate. That was half the work: putting the image in the right place.

What was more interesting was determining how the mark should pop-in, playing the iconic sound effect at the same time. The face-tracker offers lots of ways to extract the positions of facial geometry, and I spent a while tracking how far a mouth was open in order to trigger the effect. Eventually, though, I settled on the “raised eyebrows” outlet as being a much better vector for communicating surprise.

Spark screengrab2

Some brief faffing in Spark’s node-based programming environment later, and now raising eyebrows triggered the brief animation of the exclamation mark popping in, and the corresponding sound effect.

Aside: I am not the biggest fan of graph/node-based coding, partly because I’m not a very spatial thinker, and partly because I’m already comfortable with the abstractions of text-based code. But this model really does make sense for code that’s functioning more like a long-running pipeline than a single imperative script. You find this idiom in game-engine scripting (notably in Unreal 4), visual effects tools, realtime graphics tools, and similar, and it is a good fit for these tools. Also: the kind of people coming to work in SparkAR will likely have experience of 3D or video tools. Increasingly, as I pointed out to a friend, “everything is VFX“, and so graph-based coding is in increasingly more places than it used to be.

I also made sure I supported tapping the screen to toggle the effect’s visibility. After all, not everyone has reliably detectable eyebrows, and sometimes, you’d like to use this imagery without having to look surprised all the time.

Finally, I added some muddy scanlines to capture the feel of 90s-era CRT rendering: a slight tint for that authentic Shadow Moses feel, coupled with rolling animation of the texture.

With this all working, I promptly submitted it for review, so I could share it with friends.

The output looks like this:

Mgsthumb

As with every single workflow where there is a definite, final submission – be it compilation, code review, or manufacture – this is exactly the point in time when I realise I wanted to make a bunch of changes.

Firstly, I felt the effect should include some textual instructions; if you haven’t been shown it by me, you might not know how it works. So I worked up adding them.

Secondly, I felt the scanlines should be optional. Whilst they’re fun, some visual jokes might be better off without them. So I wrote some Javascript to make a native option picker at the bottom of the effect, with the default being “scanlines on”.

Once version one had been approved, the point release was quickly accepted. And then I shared the filter in my story, and wondered if friends would borrow it.

(“Borrow”: the only way to share filters/effects it to use them yourself. Once they’re published, you have to use them in a story post of your own. Then, if a friend wants to “borrow” it, they can grab the filter/effect from your story featuring that effect. In short: the only way to distribute them is through enforced virality. I can even share them outside my protected account. I am fine with this; I think it’s rather smart).

When I first shared it with friends, one wrote back: “yay for fun with software“.

It’s a while since I’ve worked on a platform that’s wanted to be fun. I’ve made my own fun with software, making tools to make art or sounds, for instance. But in 2020, so much of the software I use wants me to not have fun.

I made my first bot on Twitter a long time ago, and from there, Twitter became my platform of choice for making software-based jokes.

Twitter is now very hard to make jokes on. The word ‘bot’ has come to stand for not ‘fun software toy’ but ‘bad actor from a foreign state’. The API is increasingly more restricted as a result. I’m required to regularly log in to prove an account is real. My accounts aren’t real: they’re toys, automatons, playing on the internet.

I get why these restrictions are in place. I don’t like bad actors spreading misinformation, lies, and propaganda. But I’m still allowed to be sad the the cost of that is making toys and art on the platform. (An art project I built for the artist Erica Scourti was finally turned off once API restrictions made it unviable).

Most software-managed platforms are not places you can play any more. I understand why, but I still wish that wasn’t the case.

Yes, I know Facebook, who are hardly a great example of a good actor, are getting me to popularise their platform and work on it for free. It’s a highly closed platform, and I’m sure they’ll monetise it when they work out how. I’m giving them free advertising just by writing this.

But. Largely, what I made is a stupid visual effect that neither they nor I can effectively monetise, and it’s a joke is better told than not told. In that case, let’s tell the joke.

I showed it to my friend Eliot:

Every single time I hear that sound and see it working, I laugh. Actually, sometimes I don’t: I’m busy holding a straight face with my eyebrows up, and somebody else is laughing. Either way, somebody laughs. And that’s good enough for me.

I know, intuitively, this is a not unproblematic development platform. I know it’s not really ‘free’ in the slightest. But I write this because, right now, it was a quiet delight to be allowed to make toys and play on somebody’s platform, and one of the more pleasant platforms they run (if you keep it small, private, and friendly). I’m sure they’ll mall-ify this like everything else, but for now, I’m going to enjoy the play it enables.

It’s a while since I’ve made a toy that was so immediate to build, so immediately fun for others.

Yay for fun with software.

(If you want to try the effect yourself, open this link on your phone. It’ll open Instagram and let you try the effect.)

code , , , , , , , ,
  • "In this context, Google+ is not the company’s most strategic project. That distinction goes to Glass, to the self-driving cars, and to Google Maps, Street View, and Earth—Google’s model of the real, physical world. Maybe in twenty years we’ll think of Google primarily as a vision company—augmenting our vision, helping us share it—and, oh wow, did you realize they once, long ago, sold ads?" This is good. I like the distinction between pictures and vision a lot.
    (tags: robinsloan facebook google seeing vision pictures photography computervision robotreadableworld )
  • ""And thus ends all that I doubt I shall ever be able to do with my own eyes in the keeping of my journal, I being not able to do it any longer, having done now so long as to undo my eyes almost every time that I take a pen in my hand; and, therefore, whatever comes of it, I must forbear: and, therefore, resolve, from this time forward, to have it kept by my people in long-hand, and must therefore be contented to set down no more than is fit for them and all the world to know; or, if there be any thing, which cannot be much, now my amours to Deb. are past, and my eyes hindering me in almost all other pleasures, I must endeavour to keep a margin in my book open, to add, here and there, a note in short-hand with my own hand." Well put. Well done, Sam. Well done, Phil.
    (tags: samuelpepys pepysdiary philgyford publishing finishing )
  • "I cannot think of a worse fate: hearing something worse than John Mayer when you have to click on a link that says John Mayer. (Consider clicking on a Google search result for your dentist’s office phone number and getting your ex-girlfriend instead.)" This line is very funny, but the whole post is a shrewd explanation of the importance of resolution, and the fist Facebook makes of it. I hope consumers will discover they care about this more than they thought, too.
    (tags: api music echonest facebook resolution )
  • "His base is too good, and I don’t have the choke. He proceeds to take a more dominant position, scores points, and my body is burning from the effort. The choke he applies toward the end of the match is almost a formality, since I’m far too tired to do much more than hang on. Second place. Second place because I’m learning the triangle choke, not learning Jiu Jitsu. Chipp never wins tournaments." A fantastic piece of writing, about beat-em-ups and combat sports, and the mindset you get into as you play both. I'm not a combat sports man, but it nails some of the inside of your brain when you've played a lot of beat-em-ups, for sure.
    (tags: mma brazilianjiuitsu insertcredit fighters games writing )
  • "But to my eye, GIF is the most popular animation and short film format that's ever existed. It works on smartphones in millions of people's pockets, on giant displays in museums, in web browsers on a newspaper website. It finds liberation in constraints, in the same way that fewer characters in our tweets and texts freed us to communicate more liberally with one another. And it invites participation, in a medium that's both fun and accessible, as the pop music of moving images, giving us animations that are totally disposable and completely timeless."
    (tags: culture gifs animation internet )
  • "My wife and I talk about this. We talk about the protocol of the fertility clinic. We talk about her support group, and failure to produce. We talk about adoption, which is expensive and ambiguous. We talk about giving up on the process and living our lives without the ghosts of unconceived children (the most adorable ghosts there are). We talk, and talk, and wait." Powerful, sad, brave writing from Paul Ford. Sometimes, you wish things were nice for the good people in the world.
    (tags: paulford writing fertility themorningnews medicine )
  • "I needed to get up to speed with doing recursive node structures so I coded up a project that would put a dot on the screen. When you tapped this dot, it would create a bunch of orbiting child-dots. These children could also be tapped, creating even more child nodes. This prototype took less than a day to create and I naively thought we would be done with the whole thing in a week, max. Silly me."

    Marvellous, dense post from Robert on designing Planetary: lots of show-everything, material exploration, and plussing. What detail looks like.

    (tags: design software bloom planetary detail plussing )
  • "Years later, when recounting his conversations with Beckett (which he did often), André the Giant revealed that they rarely talked about anything besides cricket."
    (tags: history wrestling beckett cricket )
  • "In this scenario one sunny day you're working on low-level NoSQL projects at the Gootch or wherever, and you get an email from Facebook and you go for the interview and Zuckerberg is talking about scaling PHP and suddenly pauses, gets this look in his eye, pulls his hoodie over his head and says “You have sixty seconds. You should be running.” Because engineers, as we are often reminded, are the ultimate prey."
    (tags: facebook google plus social interviewtechnique paulford )
  • "A problem with the human mind – your human mind – is that it's a horrific kludge that will fail when you most need it not to. The Ugh Field failure mode is one of those really annoying failures. The idea is simple: if a person receives constant negative conditioning via unhappy thoughts whenever their mind goes into a certain zone of thought, they will begin to develop a psychological flinch mechanism around the thought. The "Unhappy Thing" – the source of negative thoughts – is typically some part of your model of the world that relates to bad things being likely to happen to you."
    (tags: psychology mind stupidlizardbrain )
  • "Designers get handed a tool kit that has as many tools as a good swiss army knife, and the maps reflect these tools. Millions of people use them to make appointments across town, find restaurants, and drive home for the holidays.

    But what if, instead of a swiss army knife, we used a box of crayons? Or charcoal and newsprint? Or play-doh? What would those maps look like? What could they tell us about the world?"

    (tags: data design maps mapping stamen whynotmakeitpretty )
  • "One thing that I learned during the launch of the original Macintosh in 1984 was that the press usually oversimplifies everything, and it can't deal with the reality that there are many people playing critical roles on significant projects. A few people always get too much credit, while most people get too little, that's just the way it has always worked. But luckily, it's 2011 and I can use the service that I helped to create to clarify things." This is Good And Proper. (Also it's good management).
    (tags: management design credit google googleplus )
  • How ads used to be made. Some beautiful photographs here.
    (tags: design advertising sugarwater history )
  • Useful notes on the modern way of deploying Rails applications with Bundler and Capistrano.
    (tags: rails capistrano bundler ruby deployment )
  • '"The expectation is slightly weird here, that you can do this stuff without killing yourself," added McNamara. "Well, you can't, whether it's in London or New York or wherever; you're competing against the best people in the world at what they do, and you just have to be prepared to do what you have to do to compete against those people."'

    This is what McNamara considers responding to controversy. I'm furious that men like this are allowed to manage other human beings.

    (tags: management industry games workingpractice quality brendanmcnamara asshat )
  • "This is an atlas, then, made by that other nature, seen through other eyes. But those eyes have been following me, unseen and without permission, and thus I consider provoking breach a necessary act." This is good.
    (tags: book location privacy mapping breach robots )
  • "csvkit is a library of utilities for working with CSV, the king of tabular file formats." Ooh.
    (tags: data csv shell commandline )
  • "Synapse is an app for Mac and Windows that allows you to easily use your Kinect to control Ableton Live, Quartz Composer, Max/MSP, and any other application that can receive OSC events. It sends joint positions and hit events via OSC, and also sends the depth image into Quartz Composer. In a way, this allows you to use your whole body as an instrument." Oooh. OSC into anything; really nice, dead simple, and exactly the sort of thing I've been considering poking.
    (tags: kinect osc computervision )
earlier posts tagged as 'facebook'