I have a few audio courses, with each lecture as a separate mp3. I wanted to be able to listen to them using ​AntennaPod, but that means having an RSS feed for them. So I wrote a simple utility​ to take a directory of mp3s and create an RSS feed file for them.

It uses the ​PyRSS2Gen module, available in Fedora with dnf install python-PyRSS2Gen.

$ ./adhoc-rss-feed --help
usage: adhoc-rss-feed [-h] [--feed-title FEED_TITLE] [--url URL]
                      [--base-url BASE_URL] [--filename-regex FILENAME_REGEX]
                      [--title-pattern TITLE_PATTERN] [--output OUTPUT]
                      files [files ...]

Let's work through a concrete example.

An audio version of the King James version of the Bible is available from ​Firefighters for Christ; they provide a ​990MB zip of mp3s, one per chapter of each book of the Bible.

wget http://server.firefighters.org/kjv/kjv.zip
unzip kjv.zip
mv -- "- FireFighters" FireFighters # use a less cumbersome directory name

There are a lot of chapters in the Bible:

$ ls */*/*/*.mp3 | wc -l
1189

We can create an RSS2 feed with as little as

./adhoc-rss-feed \
    --output rss2.xml \
    --url=http://example.com/rss-feeds/kjv \
    --base-url=http://example.com/rss-feeds/kjv/ \
    */*/*/*.mp3

However, that's going to make for an ugly feed. We can make it a little less awful with

./adhoc-rss-feed \
    --feed-title="KJV audio Bible" \
    --filename-regex="FireFighters/KJV/(?P<book_num>[0-9]+)_(?P<book>.*)/[0-9]+[A-Za-z]+(?P<chapter>[0-9]+)\\.mp3" \
    --title-pattern="KJV %(book_num)s %(book)s chapter %(chapter)s" \
    --output rss2.xml \
    --url=http://example.com/rss-feeds/kjv \
    --base-url=http://example.com/rss-feeds/kjv/ \
    */*/*/*.mp3

That's simple, and good enough to be useful. Fixing up the names of the bible is beyond what that simple regex substitution can do, but we can also do some pre-processing cleanup of the files to improve that. A bit of tedius sed expands the names of the books:

for f in */*/*; do
    mv -iv $f $(echo "$f" | sed '
        s/Gen/Genesis/
        s/Exo/Exodus/
        s/Lev/Leviticus/
        s/Num/Numbers/
        s/Deu/Deuteronomy/
        s/Jos/Joshua/
        s/Jdg/Judges/
        s/Rth/Ruth/
        s/1Sa/1Samuel/
        s/2Sa/2Samuel/
        s/1Ki/1Kings/
        s/2Ki/2Kings/
        s/1Ch/1Chronicles/
        s/2Ch/2Chronicles/
        s/Ezr/Ezra/
        s/Neh/Nehemiah/
        s/Est/Esther/
        s/Job/Job/
        s/Psa/Psalms/
        s/Pro/Proverbs/
        s/Ecc/Ecclesiastes/
        s/Son/SongOfSolomon/
        s/Isa/Isaiah/
        s/Jer/Jeremiah/
        s/Lam/Lamentations/
        s/Eze/Ezekiel/
        s/Dan/Daniel/
        s/Hos/Hosea/
        s/Joe/Joel/
        s/Amo/Amos/
        s/Oba/Obadiah/
        s/Jon/Jonah/
        s/Mic/Micah/
        s/Nah/Nahum/
        s/Hab/Habakkuk/
        s/Zep/Zephaniah/
        s/Hag/Haggai/
        s/Zec/Zechariah/
        s/Mal/Malachi/
        s/Mat/Matthew/
        s/Mar/Mark/
        s/Luk/Luke/
        s/Joh/John/
        s/Act/Acts/
        s/Rom/Romans/
        s/1Co/1Corinthians/
        s/2Co/2Corinthians/
        s/Gal/Galatians/
        s/Eph/Ephesians/
        s/Php/Philipians/
        s/Col/Colosians/
        s/1Th/1Thesalonians/
        s/2Th/2Thesalonians/
        s/1Ti/1Timothy/
        s/2Ti/2Timothy/
        s/Tts/Titus/
        s/Phm/Philemon/
        s/Heb/Hebrews/
        s/Jam/James/
        s/1Pe/1Peter/
        s/2Pe/2Peter/
        s/1Jo/1John/
        s/2Jo/2John/
        s/3Jo/3John/
        s/Jde/Jude/
        s/Rev/Revelation/
    ')
done

There are a couple of errors generated due to the m3u files the wildcard includes as well as 'Job' already having its full name, but it will get the job done.

Run the same adhoc-rss-feed command again, then host it on a server under the given base url, and point your podcast client at the rss2.xml file.

AntennaPod lists episodes based on time, and in this case that makes for an odd ordering of the episodes, but by using the selection page in AntennaPod, you can sort by "Title A->Z", and books and chapters will be ordered as expected. And then when adding to the queue, you may want to sort them again. While there is some awkwardness in the UI with this extreme case, being able to take a series of audio files and turn them into a consumable podcast has proven quite helpful.

"I'm not building a Mark III."

Famous last words.

I made the mistake of asking my parents if they had any 3.5" floppy disks at their place.

They did.

And a couple hundred of them were even mine.

Faced with the prospect of processing another 500-odd disks, I realized the Mark III was worth doing. So I made a few enhancements for the Floppy Machine Mark III:

• Changed the gearing of the track motor assembly to increase torque and added plates to keep its structure from spreading apart. The latter had been causing the push rod mechanism to bind up and block the motor, even at 100% power.
• Removed the 1x4 technic bricks from the end of the tractor tread, and lengthened the tread by several links and added to the top of the structure under those links. This reduced the frequency that something got caught on the structure and caused a problem.
• Extended the drive's shell's lower half by replacing the 1x6 technic bricks with 1x10 technic bricks; and a 1x4 plate on the underside flush with the end. This made the machine more resilient to the drive getting dropped too quickly.
• Added 1x2 bricks to lock the axles into place for the drive shell's pivot point, since they seemed to be working their way out very slowly.
• Added 1x16 technic bricks to the bottom of all the legs, and panels to accommodate that, increasing the machine's height by 5" and making it easier to pull disks out of the GOOD and BAD bins.
• Added doors at the bottom of the trays in the front to keep disks from bouncing out
• Added back wall at bottom of the trays in the back to keep disks from bouncing out.
• Moved the ultrasonic sensor lower in an attempt to reduce the false empty magazine scenario. This particular issue was sporadic enough that the effectiveness of the change is hard to determine. I only had one false-empty magazine event after this change.
• Added a touch sensor to detect when the push rod has been fully retracted in order to protect the motor. Before this, the machine identified the position of the push rod by driving the push rod to the extreme right until the motor blocked. This seems to have had a negative effect upon the motor in question. Turning the rotor of that poor, abused motor in one direction has a very rough feel. This also used the last sensor port on the NXT. (One ultrasonic sensor and three touch sensors.)
• Replaced the cable to the push rod motor with a longer one from ​HiTechnic.
• Significantly modified the controlling software to calibrate locations of the motors in ways that did not require driving a motor to a blocked state.
• Enhanced the controlling software to allow choosing what events warranted marking a disk as bad and which didn't.
• Enhanced the data recovery software to allow bailing on the first error detected. This helps when you want to do an initial pass through the disks to get all the good disks archived first. Then you can run the disks through a second time, spending more time recovering the data off the disks.
• Enhanced the controlling software to detect common physical complications and take action to correct it, such as making additional attempts to eject a disk.

With those changes, the Mark III wound up much more rainbow-warrior than the Mark II:

And naturally, I updated the model with the changes:

The general theme for the Mark II was to rebuild the machine with a cleaner construction, reasonable colors, and reduced part count. The general theme for the Mark III was to improve the reliability of the machine so it could process more disks with less baby-sitting.

All told, I had 1196 floppy disks. If you stack them carefully, they'll fit in a pair of bankers boxes.

And with that, I'm done. No Mark IV. For real, this time. I hope.

Previously: the Mark II

Four and a half years ago, I built a machine to archive 3.5" floppy disks. By the time I finished doing the archiving of the 443 floppies, I realized that it fell short of what I wanted. There were a couple of problems:

• many 3.5" floppy disk labels wrap around to the back of the disk
• disks were dumped into a single bin
• the machine was sensitive to any shifts to the platform, which consisted of two cardboard boxes
• the structure of the frame was cobbled together and did not use parts efficiently
• lighting was ad-hoc and significantly affected by the room's ambient light
• the index of the disks was cumbersome

I recently had an opportunity to dust off the old machine (quite literally), and do a complete rebuild of it. That allowed me to address the above issues. Thus, I present:

The Floppy-Disk Archiving Machine, Mark II

The Mark II addresses the shortcomings of the first machine.

Under the photography stage, an angled mirror provides the camera (an Android Dev Phone 1) a view of the label on the back of the disk. That image needs perspective correction, and has to be mirrored and cropped to extract a useful image of the rear label. OpenCV serves this purpose well enough, and is straight forward to use with the Python bindings.

The addition of lights and tracing-paper diffusers improved the quality of the photos and reduced the glare. It also made the machine usable whether the room lights were on or off.

The baffle under disk drive allows the machine to divert the ejected disks into either of two bins. I labeled those bins "BAD" and "GOOD". I wrote the control software (also Python) to accept a number of options to allow sorting the disks by different criteria. For instance, sometimes OpenCV's object matching selects a portion of a disk or its label instead of the photography stage's arrows. When that happens, the extraction of the label will fail. That can happen for either the front or back disk labels. The machine can treat such a disk as 'BAD'. When a disk is processed, and bad bytes are found, the machine can treat the disk as bad. The data extraction tool supports different levels of effort for extracting data from around bad bytes on a disk.

This allows for a multiple-pass approach to processing a large number of disks.

In the first pass, if there is a problem with either picture, or if there are bad bytes detected, sort the disk as bad. That first pass can configure the data extraction to not try very hard to get the data, and thus not spend much time per disk. At the end of the first pass, all the 'GOOD' disks have been successfully read with no bad bytes, and labels successfully extracted. The 'BAD' disks however, may have failed for a mix of different reasons.

The second pass can then expend more effort extracting data from disks with read errors. Disks which encounter problems with the label pictures would still be sorted as 'BAD', but disks with bad bytes would be sorted as 'GOOD' since we've extracted all the data we can from them, and we have good pictures of them.

That leaves us with disks that have failed label extraction at least once, and probably twice. At this point, it makes sense to run the disks through the machine and treat them as 'GOOD' unconditionally. Then the label extraction tool can be manually tweaked to extract the labels from this small stack of disks.

Once the disks have been successfully photographed and all available data extracted, an html-based index can be created. That process creates one page containing thumbnails of the front of the disks.

Each thumbnail links to a page for a disk giving ready access to:

• a full-resolution picture of the extracted front label
• a full-resolution picture of the extracted back label
• a zip file containing the files from the disk
• a browsable file tree of the files from the disk
• an image of the data on the disk
• a log of the data extracted from the disk
• the un-processed picture of the front of the disk
• the un-processed picture of the back of the disk

The data image of the disk can be mounted for access to the original filesystem, or forensic analysis tools can be used on it to extract deleted files or do deeper analysis of data affected by read errors. The log of the data extracted includes information describing which bytes were read successfully, which had errors, and which were not directly attempted. The latter may occur due to time limits placed on the data extraction process. Since a single bad byte may take ~4 seconds to return from the read operation, and there may be 1474560 bytes on a disk, if every byte were bad you could spend 10 weeks on a single disk, and recover nothing. The data recovery software (also written in Python) therefore prioritizes the sections of the disk that are most likely to contain the most good data. This means that in practice everything that can be read off the disk will be read off in less than 20 minutes. For a thorough run, I will generally configure the data extraction software to give up if it has not successfully read any data in the past 30 minutes (it's only machine time, after all). At that point, the odds of any more bytes being readable are quite low.

So what does the machine look like in action?

(Also ​posted to YouTube.)

Part of the reason I didn't disassemble the machine while it collected dust for 4.5 years was that I knew I would not be able to reproduce it should I have need of it again in the future. Doing a full rebuild of the machine allowed me to simplify the build dramatically. That made it feasible to create an Ldraw model of it using LeoCAD.

Rebuilding the frame with an eye to modeling it in the computer yielded a significantly simpler support mechanism, and one that proved to be more rigid as well. To address the variations of different platforms and tables, I screwed a pair of 1x2 boards together with some 5" sections of 1x4 using a pocket hole jig. The nice thing about the 5" gap between the 1x2 boards is that the Lego bricks are 5/16" wide, so 16 studs fit neatly within that gap. The vertical legs actually extend slightly below the top of the 1x2's, and the bottom horizontal frame rests on top of the boards. This keeps the machine from sliding around on the wooden frame, and makes for a consistent, sturdy platform which improves the machine's reliability.

The increase in stability and decrease in parts required also allowed me to increase the height of the machine itself to accommodate the inclusion of the disk baffle and egress bins.

What about a Mark III?

Uhm, no.

I have processed all 590 disks in my possession (where did the additional 150 come from?), and will be having these disks shredded. That said, the Mark II is not a flawlessly perfect machine. Were I to build a third machine, increasing the height a bit further to make the disk bins more easily accessible would be a worthwhile improvement. Likewise, the disk magazine feeding the machine is a little awkward to load with the cables crossing over it, and could use some improvement so that the weight of a tall stack of disks does not impede the proper function of the pushrod.

So, no, I'm not building a Mark III. Unless you or someone you know happen to have a thousand 3.5" floppy disks you need archived, and are willing to pay me well to do it. But who still has important 3.5" floppy disks lying around these days? I sure don't. (Well, not anymore, anyway.)

Previously: the Mark I

Update: the Mark III

While working to transfer data between two Android devices, I ran into an error like this:

$ adb install pkg.apk
8043 KB/s (38782490 bytes in 4.709s)
        pkg: /data/local/tmp/pkg.apk
Failure [INSTALL_FAILED_UID_CHANGED]

The answers I found on how to fix the problem generally involved deleting the user's data or resetting the device, and did not address what the underlying issue was.

The underlying issue here is that you are installing an application, but there is already a /data/data/<application-name> directory on the device which is owned by a different UID than the application is now being installed under.

This can be fixed without deleting the data, but does require root.

And, like anything you read on the net, this is provided in the hope it will be useful, but with no warranties. If this breaks your device, you get to keep the pieces. But you're here reading about low-level Android error messages you're getting from a developer tool, so you knew that already, right?

Out of an abundance of caution, I chose to run a number of these steps from ​TWRP recovery mode. TWRP supports adb shell and drops you into a root shell directly. You may be able to take these steps while running the system Android, but since I took the additional steps, I will show them here.

First, we'll rename the directory. For this step, boot the device into TWRP, go to the mount menu and mount the Data partition. Then rename the directory like this:

[user@workstation]$ adb shell
~ # cd /data/data
/data/data # mv <application-name> <application-name>-backup

Boot the device back to Android, and attempt to install the application again:

[user@workstation]$ adb install pkg.apk
7176 KB/s (38782490 bytes in 5.468s)
        pkg: /data/local/tmp/pkg.apk
Success

Then fix the permissions and the lib symlink in the backup directory:

[user@workstation]$ adb shell
shell@device:/ $ su
root@device:/ # d=<application-name>
root@device:/ # UID=$(ls -ld $d | awk '{print $2 ":" $3}')
root@device:/ # rm $d-backup/lib
root@device:/ # find $d-backup | while read f; do chown -h $UID "$f" done
root@device:/ # cp -P -p $d/lib $d-backup/lib

Now swap the old data directory back into place. For this step, I booted the device into TWRP:

[user@workstation]$ adb shell
~ # cd /data/data
/data/data # mv <application-name> <application-name>-fresh
/data/data # mv <application-name>-backup <application-name>

Reboot back to Android. Your application is now installed, and has its old data.

Once everything checks out, you can cleanup the leftover directory:

[user@workstation]$ adb shell
shell@device:/ $ su
root@device:/ # rm -rf /data/data/<application-name>-fresh

I'm finally getting around to upgrading my trusty old ​Android Dev Phone 1 from the original Android 1.5 firmware to ​Cyanogenmod 6.1. In doing so, I wanted to take my contacts with me. The contacts application changed its database schema from Android 1.x to Android 2.x, so I need to export/import. Android 2.x's contact application supports importing from VCard (.vcf) files. But Android 1.5's contact application doesn't have an export function.

So I wrote a minimal export tool​.

The Android 1.x contacts database is saved in /data/com.android.providers.contacts/databases/contacts.db which is a standard sqlite3 database. I wanted contact names and phone numbers and notes, but didn't care about any of the other fields. My export tool generates a minimalistic version of .vcf that the new contacts application understands.

Example usage:

./contacts.py contacts.db > contacts.vcf
adb push contacts.vcf /sdcard/contacts.vcf

Then in the contacts application import from that file.

If you happen to have a need to export your contacts from an Android 1.x phone, this tool should give you a starting point. Note that the clean_data function fixes up some issues I had in my particular contact list, and might not be very applicable to a different data set. I'm not sure the labels ("Home", "Mobile", "Work", etc.) for the phone numbers are quite right, but then, they were already a mess in my original data. Since this was a one-off task, the code wasn't written for maintainability, and it'll probably do something awful to your data--use it at your own risk.

August 31st of last year, at the age of 89, my Grandfather passed away. I'm a computer geek, as was he, though his machines filled rooms, and mine, merely pockets. His software flew fighter aircraft. He worked on the Apollo missions. He wrote the first software by which to operate a nuclear reactor. That is a hard act to follow.

But as a computer geek, he had accumulated a large stack of 3.5" floppy disks: 443, of them in fact. And when he passed away, it became my responsibility to deal with those. I was not looking forward to the days of mindless repetition inherent in that task. So, I did what any self-respecting software engineer would do: I automated it.

Start with Lego Mindstorms, add a laptop running Fedora Linux, an Android Dev Phone 1, a good bit of Python code, and about the same number of hours of work, and you get this:

​Watch it in action on YouTube

There are a number of interesting details in this build which I plan to write about in the coming weeks, so stay tuned.

Follow up articles: NXT control software, The Floppy-Disk Archiving Machine, Mark II