2015-02-05: What Did It Look Like?

Having often wondered why many popular videos on the web are time lapse videos (that is videos which capture the change of a subject over time), I came to the conclusion that impermanence gives value to the process of preserving ourselves or other subjects in photography. As though a means to defy the compulsory fundamental law of change. Just like our lives, one of the greatest products of human endeavor, the World Wide Web, was once small, but has continued to grow. So it is only fitting for us to capture the transitions.

What Did It Look Like? is a Tumblr blog which uses the Memento framework to poll various public web archives, take the earliest archived version from each calendar year, and then create an animated image that shows the progression of the site through the years.

To seed the service we randomly chose some web sites and processed them (see also the archives). In addition, everyone is free to nominate web sites to What Did It Look Like? by tweeting: "#whatdiditlooklike URL". 

In order to see how this system is achieved, consider the architecture diagram below. 

Source code

The system is implemented in Python and utilizes Tweepy and PyTumblr to access the Twitter and Python APIs respectively, and consists of the following programs:

1. timelapseTwitter.py: This application fetches tweets (with "#whatdiditlooklike URL" signature) by using the tweet ID of the last tweet visited as reference to know where to begin retrieving tweets. For example, if the application initially visited tweet IDs 0, 1, 2. It keeps track of the ID 2, so as to begin retrieving tweets with IDs greater than 2 in a subsequent tweet retrieval operation. Also, since Twitter rate limits the number of search operations (180 requests per 15 minute window), the application sleeps in between search operations. The snippet below outlines the basic operations of fetching tweets after the last tweet visited: for tweet in tweepy.Cursor(api.search, q="%23whatdiditlooklike", since_id=sinceIDValue).items(30): #update since_id if( tweet.id > sinceIDValue ): sinceIDValue = tweet.id print localTimeTweet, ",tweet_id:", tweet.id, ",", tweet.user.screen_name, " - ", tweet.text print '...sleeping for 15 seconds' time.sleep(15)
2. usingTimelapseToTakeScreenShots.py: This is a simple application with invokes timelapse.py for each nomination tweet (that a tweet with the "#whatdiditlooklike URL" signature).
3. timelapse.py: Given an input URL, this application utilizes PhantomJS, (a headless browser) to take screenshots and utilizes ImageMagick to create an animated GIF. It should also be noted that the GIFs created are optimized due to the snippet below in order to reduce their respective sizes to under 1MB. This ensures the animation is not deactivated by Tumblr. #optimize params = ['../gifsicle-static','--lossy=80', '--optimize', '--colors', '160', '--resize-width', '800', '-o', newfile, file] subprocess.check_call(params)
4. timelapseSubEngine.py: this application executes two primary operations:
1. Publication of the animated GIFs of nominated URLs to Tumblr: This is done through the PyTumblr API create_photo() method as outlined by the snippet below:client = pytumblr.TumblrRestClient(CONSUMER_KEY, CONSUMER_SECRET, OAUTH_TOKEN, OAUTH_TOKEN_SECRET) postID = client.create_photo(globalBlogName, tags=[tagsString], state='publish', caption=[captionString], data=gifAnimationFilename)
2. Notifying the referrer and making status updates on Twitter: This is achieved through Tweepy's api.update_status() method as outlined by the snippet below which tweets the status update message:auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_KEY) auth.set_access_token(OAUTH_TOKEN, OAUTH_TOKEN_SECRET)data=gifAnimationFilename) # Creation of the interface, using authentication api = tweepy.API(auth) api.update_status(statusUpdateString)   However, a simple periodic Twitter status update message could result in the message to be flagged eventually as spam by Twitter. This comes in form of a 226 error code. In order to avoid this, timelapseSubEngine.py does not post the same status update tweet message or notification tweet. Instead the application randomly selects from a suite of messages and injects a variety of attributes which ensure status update tweets are different. The randomness in execution is due to a custom cron application which randomly executes the entire stack beginning from timelapseTwitter.py down to timelapseSubEngine.py.

How to nominate sites onto What Did It Look Like?

If you are interested in seeing what a web site looked like through the years:

1. Search to see if the web site already exists by using the search service in the archives page; this can be done by typing the URL of the web site and hitting submit. If the URL does not exist on the site, go ahead to step 2.
   
2. Tweet "#whatdiditlooklike URL" to nominate a web site or tweet "#whatdiditlooklike URL1, URL2, ..., URLn" to nominate multiple URLs.

Tweet "#whatdiditlooklike URL" to nominate a web site or tweet "#whatdiditlooklike URL1, URL2, ..., URLn" to nominate multiple URLs.

How to explore historical posts

To explore historical posts, visit the archives page: http://whatdiditlooklike.tumblr.com/archives

Examples 

What Did cnn.com Look Like?

What Did cs.odu.edu Look Like?

What Did apple.com Look Like?

"What Did It Look Like?" is inspired by two sources: 1) the "One Terabyte of Kilobyte Age Photo Op" Tumblr that Dragan Espenschied presented at DP 2014 (which basically demonstrates digital preservation as performance art; see also the commentary blog by Olia Lialina & Dragan), and 2) the Digital Public Library of America (DPLA) "#dplafinds" hashtag that surfaces interesting holdings that one would otherwise likely not discover.  Both sources have the idea of "randomly" highlighting resources that you would otherwise not find given the intimidatingly large collection in which they reside.

We hope you'll enjoy this service as a fun way to see how web sites -- and web site design! -- have changed through the years.

--Nwala