Lately, I’ve been on this crazy kick in looking for some sort of lifestreaming software or application. Basically, I (and most likely you — if you’re reading this and one of my internet friends) create a ridiculous amount of data each day. From my tweets, to my foursquare checkins, to my Instagram photos, to uploading things to Flickr, to blogging, to liking videos on YouTube, and sharing articles on Google Reader.
I’ve been thinking about this a lot lately for one reason: this would make an incredible diary of my life. I’m not the first to think this (just read the Wikipedia article I linked to — people were thinking about this in the 1990’s), but it’s something I’ve found myself becoming obsessed with.
When FriendFeed was announced in 2007, I thought, “this is perfect!” It aggregates data from nearly every web service you can imagine. I happily started plugging things in and letting it archive all my data. It ended up being awesome for a number of reasons.
“Oh, man! What was that thing I tweeted about 2 years ago, about some guy bunting a home run?” Well, Twitter search goes back about 7 days, so that was useless. FriendFeed to the rescue! I could easily search for things I tweeted about (and [website-verb]ed about) from the moment I started importing things.
In August of 2009, Facebook acquired FriendFeed and proceeded to let the site rot. Since then, there’s been no easy way to export your data, and their search function eventually broke, making the site useless for searching archived data. To this day, FriendFeed is happily pulling in everything I do on the internet, but sadly, I have no way to search for it.
Earlier this week, I found a brilliant PHP script by Claudio Cicali. It scrapes your FriendFeed profile and saves all your data to a JSON file.
After accumulating over 3 years of data, I ran the script (which took an entire evening) and it scraped something like 300K different things I’ve done on the internet in the past few years. The resulting JSON file is over 300MB (now I need to work on a way to parse the data and feed it back into a MySQL database). Incredible!
Sadly though, I don’t think this is a tenable solution. It’s great for fetching all my past data, but who knows how long FriendFeed will remain around. I’d like something more permanent, open-source, and that I can potentially run on my own server.
Locker sounds like it may be what I’m looking for, but it still has a ways to go. Momento on the iPhone sounds exactly like what I need, but you need to manually kick it off (and it won’t pull in data too far in the past).
Anyone have any ideas or thoughts on this?