Web Science and Digital Libraries Research Group

I have written posts detailing how an archives modifications made to the JavaScript of a web page being replayed collided with the JavaScript libraries used by the page and how JavaScript + CORS is a deadly combination during replay . Today I am here to announce the release of a suite of high fidelity web archiving tools that help to mitigate the problems surrounding web archiving and a dynamic JavaScript powered web.To demonstrate this, consider the image above: the left-hand screen shot shows today's cnn.com archived and replayed in WAIL, whereas the right-hand screen shot shows cnn.com in the Internet Archive on 2017-07-24T16:00:02 . In this post, I will be covering: Updates to WAIL Release of node-warc Release of node-cdxj Release of Squidwarc WAIL Let me begin by announcing that WAIL has transitioned away from using Heritrix as the primary preservation method . Instead, WAIL now directly uses a full Chrome browser (Electron provided) as the pres...

CNN.com has been unarchivable since 2016-11-01T15:01:31 , at least by the common web archiving systems employed by the Internet Archive , archive.is , and webcitation.org . The last known correctly archived page in the Internet Archive's Wayback Machine is 2016-11-01T13:15:40 , with all versions since then producing some kind of error (including today's;   2017-01-20T09:16:50 ). This means that the most popular web archives have no record of the time immediately before the presidential election through at least today's presidential inauguration. Given the political controversy surrounding the election, one might conclude this is a part of some grand conspiracy equivalent to those found in the TV series The X-Files . But rest assured, this is not the case; the page was archived as is, and the reasons behind the archival failure are not as fantastical as those found in the show.  As we will explain below, other archival systems have successfully archived CNN.com ...

I'm from Louisiana and used Archive-It to build a collection of webpages about the September flood there ( https://www.archive-it.org/collections/7760/ ). One of the pages I came across, Hundreds of Louisiana Flood Victims Owe Their Lives to the 'Cajun Navy' , highlighted the work of the volunteer "Cajun Navy" in rescuing people from their flooded homes. The page is fairly complex, with a Flash video, YouTube video, 14 embedded tweets (one of which contained a video), and 2 embedded Instagram posts. Here's a screenshot of the original page (click for full page): Live page, screenshot generated on Sep 9, 2016 To me, the most important resources here were the tweets and their pictures, so I'll focus here on how well they were archived. First, let's look at how embedded Tweets work on the live web. According to Twitter : "An Embedded Tweet comes in two parts: a <blockquote> containing Tweet information and the JavaScript file on T...

Trip Report for the IIPC Building Better Crawlers Hackathon in London, UK.                            On September 22-23, 2016, I attended the IIPC Building Better Crawlers Hackathon ( #iipchack ) at the British Library in London, UK. Having been to London almost exactly 2 years ago for the Digital Libraries 2014 conference , I was excited to go back, but was more so anticipating collaborating with some folks I had long been in contact with during my tenure as a PhD student researcher at ODU. The event was a well-organized yet loosely scheduled meeting that resembled more of an "Unconference" than a Hackathon in that the discussion topics were defined as the event progressed rather than a larger portion being devoted to implementation (see the recent Archives Unleashed 1.0 and 2.0 trip reports ). The represented organizations were: British Libr...

The development, state, and future of 🐳 Web Archiving Integration Layer. 💄∨🐷?                                                                  Some time ago I created and deployed Web Archiving Integration Layer (frequently abbreviated as WAIL ), an application that provides users pre-configured local instances of Heritrix and OpenWayback. This tool was originally created for the Personal Digital Archiving 2013 conference and has gone through a metamorphosis. The original impetus for creating the application was that the browser-based WARCreate extension required some sort of server-like software to save files locally because of the limitations of the Google Chrome API and JavaScript at the time (2012). WARCreate would perform an HTTP POST to thi...