Taming Twitter with the Command Line

I thought I was done with the command line for the week, but then I did something cool that I thought I'd share with you. Linux users only ... others will think this is silly ... join me below the fold.

OK, are we alone? Good. It's nice to be away from all those Windows Symps for a while. Oh, I see there's a couple of Mac OSX users lurking in the back of the room. Come right over and join us, this will probably work for you too!.

Here is my problem. I use twitter to promote my blog, and I therefore follow almost 3,000 people on twitter. That means that every few seconds there is an update on my "people I follow" list, and it is almost always something I am not even slightly interested in.

This makes Twitter kind of useless for me. Just the other day, I did something that a lot of people apparently found terribly objectionable (I have no idea what it was) and later got an email from a friend that said "Hey, did you see what so and so and so and so and so and so said about you on Twitter?"

"Of course not" was my reply. "I can't use Twitter. Too much data, not enough good data. If something happens on Twitter there is no way for me to know it."

Then, yesterday, I heard: "Hey, did you hear that so and so and so and so are getting married? It was the talk on Twittter!"

"Of course not" was my reply... and so on and so forth, you get the point.

I know there are many possible solutions to this problem (feel free to mention them in the comments if you like) but having a second Twitter account (one of the obvious solutions) with my actual 'friends' on it was not one I wanted to do. Instead I wanted to filter the data.

Yes, there are some software solutions out there to filter data, and maybe some day I'll use one of them, but I really felt that this was a case of too much mucking around for a simple solution. What I really wanted to do was no more than this:

1. Produce a file of recent tweets by those I "follow."
2. Manually maintain a file of the names of people whose tweets I want to actually see. An a-list, if you will.
3. Write a script that would use these two data sources to come up with a list of tweets by the smaller list of people, culled from the larger fire hose list.
4. Format that list minimally so it shows up in a web browser as a local page.
5. Put a button on my Gnome toolbar that makes all this happen.

I call it a-tweet. Example results are depicted in the graphic above, and the script looks like this:

#! /bin/bash

twidge lsrecent -lsu > ~/twidge_data/data

awk 'NR==FNR {u[$1];next} ($2 in u)' ~/twidge_data/a-list ~/twidge_data/data > ~/twidge_data/a-tweets

cut -f2,4 ~/twidge_data/a-tweets |

sed 's/^\(.*\)\t\(.*\)/<DT>\1<\/DT><DD>\2<<\/DD><\/br><\/br>/g

> ~/twidge_data/a-tweet.html

firefox ~/twidge_data/a-tweet.html

This script is not all squishy and one-liney like it could be. I use intermediate files and drawn out constructs so that I can later mess with it more easily. This also makes it easier to explain.

The first line uses Twidge. Twidge is a great find. sudo apt-get install twidge will add it to your system.

After you install twidge you will run

twidge setup

which will ask you for your user name and password for the account you want to twidge around with.

Twidge lets you do cool things with Twitter on the command line. Learn about it here. Among other things, you can post tweets from the command line, or get a list of your followers or followees, and so on. I found it by just searching for "twitter" in my package manager.

So let's break it down.

twidge lsrecent -lsu > ~/twidge_data/data

Twidge always works by using 'twidge' followed by a command. In this case, lsrecent, which lists (ls) recent tweets. There are arguments one can use to make this list come out differently, but the -l argument results in a one-tweet-per-line tab delimited list of tweets, and is essential for thi script to work. The -s and -u paramaters make Twidge keep track of where it last looked and gives you stuff only since then. Not shown but available is the -all option.

The -all option (not shown) is ... optional. If you have a couple of hundred followees and want to see the most recent among, say, a couple of dozen, you probably don't need it. However, I'm scanning for about twenty twits among nearly three thousand, and I may check only every few days. Twidge tends to be conservative, only fetching the most recent several dozen tweets. I believe the -all option may work better for the scenario I just described. In any event, you can play with it because it is your command line. You can even create different versions of the script to run under different circumstances.

Here, I dump the output into a file in a special subdirectory where I keep the twidge-related data. The data file that holds these data is called, enigmatically, "data." After the guy on star Trek.

Next line:

awk 'NR==FNR {u[$1];next} ($2 in u)' ~/twidge_data/a-list ~/twidge_data/data > ~/twidge_data/a-tweets

This line takes two files, a-list and data, and using the programing language awk filters the data list based on matches between the second field (which happens to be user name) and the list of user names in the file a-list. The output, the filtered subset of tweets that are only by "a-list" twits (if that is what they are called) is then dumped into the file "a-tweets" with any old content in that file blotto'd out of existence.

The next line is:

cut -f2,4 ~/twidge_data/a-tweets |

sed 's/^\(.*\)\t\(.*\)/<DT>\1<\/DT><DD>\2<<\/DD><\/br><\/br>/g'

> ~/twidge_data/a-tweet.html

This is two (well, really, more, but I'm simplifying slightly) commands connected with a pipe. The first command uses cut to isolate the second and fourth fields from the data, which happen to be the user name and the tweet itself. Left off, then, is a blank field, an ID number, and the date/time stamp. One could argue for leaving on the date/time stamp, but I chose not to.

This stream of data, consisting of one username/tweet pair per line, is then sent to sed, which inserts code to make an HTML definition list. This formats the data as I like it. It also inserts two HTML line breaks after each tweet.

This sed command could be used in a computer science textbook to illustrate all the strangeness of sed. I love sed.

This stream of HTML formatted data is then dumped, unceremoniously, into a file where, if opened with a web browser, it will be formatted as I want it. Since the filename is created with an 'html' extension, opening the file directly will likely open it in an appropriate app, depending on how your desktop is configured.

The last line simply opens up a Firefox instance or tab with the file. If there were no tweets form the a-list twits, then there is nothing in the file and you get blankness. Otherwise, you get something that looks like the picture posted above.

There is a lot that can be done to enhance this. URL's in the tweets could be identified (using sed) and packaged as links, for instance. A bit of code allowing responses or retweets can be added. Eventually, with enough mucking around, one can have a full featured application (like those that may or may not be available) but made entirely from scratch.

Suggestions for mods welcome!