flickr set parser for GPSVisualizer

Geekier than normal post today. If you aren't interested in python programming, flickr's API and GPS display of images in Google maps or Google Earth then I'd stop reading now. Here's another picture from Death Valley. See you again tomorrow.

Right. Still here ? I've been using GPSVisualizer to combine a GPS tracklog, with geotagged images on flickr. Images are entered into GPSVisualizer as a series of CSV values, in a fairly flexible format. The first line provides the layout:

latitude, longitude, name, url, thumbnail, desc

Each line after that is one entry for an image, with the location in decimal degrees, followed by various text fields containing the description, title and links to the actual image. For example:

36.366953, -117.391867, "shot up car", "http://flickr.com/photos/mcgregorphoto/2355449153/", "http://farm3.static.flickr.com/2323/2355449153_c733f5ee1e.jpg", "a long way from nowhere, burnt out and shot up, just off the access road"
36.442947, -117.435447, "half way point", http://flickr.com/photos/mcgregorphoto/2356283640/, http://farm3.static.flickr.com/2388/2356283640_be78036888.jpg, "lunch about half way into the hike"

Initially I generated this file by hand, extracting the EXIF location from the image files using exiftool. I then went through each image, entered a title, found the URL for the image on flickr, extracted the URL for a thumbnail image and added a longer description. This was painful to say the least. It was only 16 images but it was a pain. Particularly as all the information was already there, in a flickr set. So I looked up the information on the flickr API, found a python library to access it and wrote the script below in half an hour. Given the URL or set id for a flickr set, it iterates over all of the photos and produces a CSV formatted list that's suitable to load straight into GPSVisualizer. This can then be linked along with the original GPS track log to generate a map with images and also the path traveled. It can be used for both Google Earth and Google maps path generation and runs pretty quickly. I'm posting it here in case it proves useful to anyone. To run, you'll need an up to date python install and additionally will need to download and install the flickrAPI for python. The final step is to obtain an API key for flickr. This key has to be added to the script, in the marked location. Once installed the program is run with:

python flickr2gpsv.py -s set_id > results.csv

or, alternatively,

python flickr2gpsv.py -s http://www.flickr.com/set_url/ > results.csv

If you find this useful, please let me know. You can download the script here. (probably better than cutting and pasting from below, because the download will keep the correct indentation). The archived version also includes a patch from Brad Crittenden to add unicode support.

# Author : Gordon McGregor
# Contact: http://gordonmcgregor.blogspot.com
#
# License : public domain
#
# Purpose: parses a flickr set to extract information to generate a map overlay, via http://gpsvisualizer.com
#
# Usage: python flickr2gpsv.py -s 72157604221838137
# or
#        python flickr2gpsv.py -s http://flickr.com/photos/mcgregorphoto/sets/72157604221838137/
#
#   If the URL is given, the set_id is automatically extracted

# Typically, you'll want to  redirect the output to a file, as errors & comments will appear on stderr (not in the file)
#
# e.g., python flickr2gpsv.py -s 72157604221838137 > output_list.csv
#
#
# only generates entries for photos with geographic information attached
#
# Required libraries:
# the flickrapi python libraries, from http://flickrapi.sourceforge.net/
# installation info here http://flickrapi.sourceforge.net/installation.html
#

import flickrapi
import time

import sys
from optparse import OptionParser
from urlparse import urlparse

# enter your api_key here to connect to flickr
# obtain one from http://www.flickr.com/services/api/keys/apply/
#

api_key = 'your_key_goes_here'

def getURL(sizes, size):

for element in sizes.sizes[0].size:
if element['label'] == size:
    return element['source']

raise flickrapi.exceptions.FlickrError, "No " + size+ " URL found."

# the main routine
# pass in a flickr set id
# outputs the appropriate data for GPSVisualizer to stdout
def parseSet(set_id):

cnt = 0

flickr = flickrapi.FlickrAPI(api_key)

photoset = flickr.photosets_getPhotos(photoset_id=set_id)

# iterate over list of photos in list

print 'latitude, longitude, name, thumbnail, url, desc'
for photo in photoset.photoset[0].photo:

try:
# get the various bits of data

    sizes = flickr.photos_getSizes(photo_id = photo['id'])
    info  = flickr.photos_getInfo(photo_id =  photo['id'])

# extract the required fields
    try:
        lat = info.photo[0].location[0]['latitude']
        lon = info.photo[0].location[0]['longitude']
    except AttributeError:
        raise flickrapi.exceptions.FlickrError, "No Geographical data found."

    name = photo['title']
    thumbnail =  getURL(sizes, 'Small')
    url = info.photo[0].urls[0].url[0].text
    desc = info.photo[0].description[0].text.strip()  # strip to remove extra newlines

    if(len(desc)):
        print '%s, %s, "%s", "%s", "%s", "%s"' % (lat, lon, name, thumbnail, url, desc)
    else:
        print '%s, %s, "%s", "%s", "%s",' % (lat, lon, name, thumbnail, url)
    sys.stderr.write('.')
    cnt = cnt + 1

except flickrapi.exceptions.FlickrError, e:
    sys.stderr.write( '\n'+photo['title'] +' : ' + e.__str__() + ' Skipping.\n')

return cnt



def main(argv=None):
if argv is None:
argv = sys.argv

usage = "usage: %prog [options]\nPass in a flickr set to produce output suitable for GPSVisualizer's google maps overlay"
opt_parser = OptionParser(usage=usage)

opt_parser.add_option('-s', '--set', dest='set', help="flickr set to process (full url or set number)")

(options, args) = opt_parser.parse_args()

if(options.set == None):
opt_parser.print_help()
return 1

# treat the command line value as a url
url = urlparse(options.set)

# even if it is just a set id, the value ends up in field 3 after urlparse (url[2])
# this removes any trailing /, explodes around any remaining /'s and takes the last value [-1]
# works for a full URL or simple set id
set_id = url[2].strip('/').split('/')[-1]

if (not set_id.isdigit()):
sys.stderr.write("set_id" + set_id + "is not a number. This is not expected.")
return 1

sys.stderr.write("Processing set id "+set_id+"\n")
start_time = time.time()
num = parseSet(set_id)
end_time = time.time()

total_time = end_time - start_time

average = 0
if(num):
average = total_time/num

results = "\nProcessed %d valid photos in %0.1f seconds (%0.2f seconds/photo). Finished.\n" % (num, total_time, average)
sys.stderr.write(results)

if __name__ == "__main__":
sys.exit(main())

GPS tagging

50mm, 1/40sec @f1.4, ISO 1600

I have looked in to some options of combining GPS data with the photos I take. For a few hours I thought about writing my own software, I also found the open source Happy Camel project and considered using that. Eventually I realised that the download software that I already use, Downloader Pro, has a GPS option in the latest version. I tried it out briefly prior to traveling to Death Valley but didn't really spend much time working out how to use it. Turned out that it was exceptionally easy to use. I'm really impressed. There's a 30 day evaluation available if you were interested in trying it out. The GPS isn't connected to the camera at all. There's no wires or wireless connection. I don't have anything new mounted on the camera hot shoe, unlike some other GPS tagging solutions. All I have to do is switch on the GPS unit and put it in my bag. It is creating a tracklog of where I go and when. The time on the GPS and the time on the camera need to be synchronised, or you at least need to be aware how different the time is (you can enter the difference in the downloader software). Once you've got that worked out, the GPS knows where you are at a given time and the camera knows when you take a picture (it is recorded in the photo's EXIF data). That's all you need to know to pretty accurately tag where the picture was taken. All that is left is to combine the two and fill out the GPS data field in the photo's EXIF data. That's where Downloader Pro comes in. I have a Garmin GPSmap 60CSx GPS unit. It has a 10,000 point automatic track log, so when I switch it on, it starts tracking. I'm sure this sort of track log can be generated by any GPS unit that can connect to a computer. (GPSBabel might be useful to convert to a format that Downloader Pro can understand). When I get back to my computer, I plug the camera's compact flash card into a card reader and hook up the GPS unit via USB. I switch the Garmin into the USB mass storage mode so that it appears as a hard drive on the computer. In that mode the latest track logs are available in the root of the drive. Downloader Pro is configured to scan removable drives for track logs. (Under GPS Settings, enable Geo-Tagging, set the camera and GPS clock times and for Track Log Settings select 'Search removable drives'). Once Downloader Pro is configured, I simply plug in the GPS, plug in the CF card, set a job reference name and hit download. Everything else happens automatically. I don't have to convert track formats, it already understands the Garmin GPX log format. Simple. You can also produce Google Earth and Google Maps viewable versions of the track log. Next thing I want to try is integrating the images into Google earth to have the shots hanging in space. For now though, when I load up the images in Lightroom there is a new field filled out in the EXIF - the GPS co-ordinates where the image was taken. The 'location' set of metadata browsing in Lightroom shows all this in a concise form. You can then click on the arrow to the right of the GPS data and a web page is opened to Google maps, showing the satellite view of where the image was taken. I also found out that in Flickr, you can change your account permissions, so that it will automatically extract the location information when you upload images. This then places them on the map and indicates where the image was taken automatically. The option is off by default and is under You->Your Account->Privacy & Permissions in the Import EXIF location data option. You can set different permissions on each image for who can see the location data and who can see the image. So you could let anyone see the pictures but only friends and family know exactly where it was taken, for example.

If you click on the shot of Hamilton pool, above, you can use the show on map option in the lower right to see where it was taken. I am interested in any good examples of using geo-tagging in photos. It seems like a neat technology and fun to be able to see exactly where you were when you took a shot. I did use it quite a bit while scouting locations too, to mark a spot that I wanted to return to at a better time of day. Really then all I could have done was hit the mark button on the GPS and put a suitable name in. So far I've found this useful for sharing the location with others, more than anything else. When talking through the images, I could click on the GPS link and pull up the satellite view - handy to show just how far away from anything I was. In some other shots, it has been useful to be able to describe the route and thought process taken to finding a particular location. I have considered using it when out shooting in an urban area, to tag locations with great backgrounds for portraits - there are plenty around Austin and I occasionally forget where they are. Again there the information would be more useful for sharing those spots with others that might want to visit. I can already see my shots appearing automagically in Google Earth, when I switch on the flickr layers, which is quite neat. I'd actually like to work out if it is possible to include my images as floating billboards in Google Earth, when you fly past the location where they were shot. That might require directional information though, to really line up the scene. Useful Links

• Geotagging on Wikipedia
• Geotagging on Flickr
• Geocoding photos on Wikipedia
• Imageingester pro (another tagging program that I hear works well on the Mac)