"pv – Pipe Viewer – is a terminal-based tool for monitoring the progress of data through a pipeline. It can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion." Looks very handy.
"xsv is a command line program for indexing, slicing, analyzing, splitting and joining CSV files. Commands should be simple, fast and composable." iiinteresting.
Nice, clear explanation (as clear as it seems anything involving M4 can be) of this process, which I've used a lot but not thought about that much.
Very good post on Just Using make as your task runner.
"There was also one late night when a stranger opened the door and walked into the house when August should have auto-locked the door. (The stranger was trying to enter our next-door neighbor’s house and didn’t realize he was at the wrong door.)" YOU HAD ONE JOB etc.
Rather looking forward to seeing this play out: thirty days of processing and spelunking CSV, from Paul Downey. Lots of new tools and tricks emerging already.
Really nice exploration of a small stack for poking data at the commandline. I'm a fan of jq and its ilk already, so this extends some of those techniques.
"I'm less nostalgic for old kinds of HTML than for the part of myself that was young and fearless and desperate to connect to the wider world. I get a kick out of the under construction images but, I mean, they actually are hosted and served on a perfectly modern boxes into browsers that are essentially virtualized supercomputers." Paul Ford: still the best.
"As serious intellectuals often do, we spent hours discussing these questions, what data we would want to collect to answer them, and even how we might go about collecting it. It sounded like a fun project, so I wrote a program that takes video captures of our Mario Kart 64 sessions and picks out when each race starts, which character is in each box on the screen, the rank of each player as the race progresses, and finally when the race finishes. Then I built a web client that lets us upload videos, record who played which character in each race, and browse the aggregated stats. The result is called Kartlytics, and now contains videos of over 230 races from over the last year and change." Yes, it's a plug for manta, but it's also a nifty piece of engineering.
09 August 2013
I’m working on a hardware project at the moment that’s more complex than a basic microcontroller-build that I could have implemented with an Arduino. So I’ve been using a Raspberry Pi as the centre of the project, and writing my code in the high-level languages I’m most proficient in. (In this case, Ruby).
As the project nears completion, it’s really important that the device doesn’t manifest as a computer in any way: it’s just a physical object. To that end, I’d like all the software inside the miniature computer to run at startup without any manual intervention.
My Pi is running Raspbian – the Pi-focused version of Debian – so, fortunately, there’s a tool easily available to do that for us: upstart.
We’ll write an upstart configuration for our script, which will turn our script into an upstart service, which we can then start at login.
First, let’s install upstart:
sudo apt-get install upstart
This will issue some warnings, because – on my install – it was replacing the traditional
init.d setup. Don’t worry; everything will continue to work.
Once you’ve installed upstart, reboot your Pi, either from the command line or with a power-cycle.
Let’s now make an upstart config file. Here’s a very basic one:
Put this code into
/etc/init/myscript.conf. You should then be able to run the script by typing
sudo start myscript, and kill it with
sudo stop myscript. And, of course, you’ll discover it’ll start automatically on startup.
That’s a very simple example, with no dependencies. But that won’t work for the script I’d like to use. In this example, I’m running a Ruby script (using the Pi Piper library) that blinks an LED attached to GPIO Pin 17. That script needs to be run as root to get access to the GPIO pins, and it needs to reference the directory’s Gemfile. Upstart scripts are run as root, so that’s not an issue – but we need to set up the environment correctly. That’s not so hard:
As you can see, we just have to export the
BUNDLE_GEMFILE variable so that Bundler will know where the Gemfile is located.
Also, you’re going to have to make sure that all references to files in your code are done with absolute paths. That wasn’t a problem with the simple shell script example, but becomes an issue with more real-world type code – especially the program I’m ultimately running, which has various includes, dependencies, and data files to load. An obvious place you’ll run into this with Ruby is when setting up the
Rather than starting my Ruby script
$LOAD_PATH << 'lib'
I had to do this:
$LOAD_PATH << File.expand_path(File.dirname(__FILE__)) + '/lib'
And similarly, any other references to file loading will need to be absolute – and, ideally, derived using tricks like
File.dirname(__FILE__) rather than through hardcoded paths.
Anyhow: it took me a while to piece together, but now that I have, it felt worth writing down – because I’ve now got reasonable complex computation-backed hardware working in an entirely headless enclosure – and one that’s resilient to power-cycling.
Simple dynamic site generator with standardised templating tools: certainly looks nice for building those early-stage prototypes before you need a full backend.
"The history of switching power supplies turns out to be pretty interesting." It really does: long, fascinating post about a history of AC-DC conversion and how new technologies affected it.
"Remote terminal application that allows roaming, supports intermittent connectivity, and provides intelligent local echo and line editing of user keystrokes." As recommended by Matthew Somerville. Looks useful!
"Modern digital cameras capture gobs of parsable metadata about photos such as the camera's settings, the location of the photo, the date, and time, but they don't output any information about the content of the photo. The Descriptive Camera only outputs the metadata about the content." Lovely: a camera powered by Mechanical Turk.
"All operating systems know when they were born. Their internal clocks start counting then, so they can calculate the date and time in the future. It is unclear whether it was Mr Ritchie or Mr Thompson who set the so-called start Unix time at January 1st, 1970. That moment came to be known as the epoch. Mr Ritchie helped bring it about. And with it, he ushered in a new era." Which is as poetic a way as any of expressing how deeply rooted K&R are in the modern world.