Web Science and Digital Libraries Research Group

(with apologies to Charles Dickens, Robert Frost, and Dr. Seuss) "It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, ..." (A Tale of Two Cities, by Charles Dickens). At the end of this part of my journey; it is time to reflect on how I got here, and what the future may hold. Looking back, I am here because of answering two simple questions.  One from a man who is no longer here, one from a man who still poses new and interesting questions.  Along the way, I've formed a few questions of my own. The first question was posed by my paternal uncle, Bertram Winston.  Uncle Bert was a classic type A personality.  Everything in his life was organi

Mat Kelly, Justin F. Brunelle and Dr. Michael L. Nelson travel to London, UK to report on the Digital Libraries 2014 Conference.                            On September 9th through 11th, 2014, Dr. Nelson ( @phonedude_mln ), Justin ( @justinfbrunelle ), and I ( @machawk1 ) attending the Digital Libraries 2014 conference (a composite of the JCDL (see trip reports for 2013 , 2012 , 2011 ) and TPDL (see trip reports for 2013 and 2012 ) conferences this year) in London, England. Prior to the conference, Justin and I attended the DL2014 Doctoral Consortium , which occurred on September 8th. The main conference on September 9th opened with George Buchanan ( @GeorgeRBuchanan ) indicating that this year's conference was a combination of both TPDL and JCDL from previous years. With the first digital libraries conference being in 1994, this year marked the 20th year anniversary of the conference. George celebrated this by singing Happy Birthday to the conference and introduced t

On Monday (Sep 15), Michael and I attended the NEH Office of Digital Humanities Project Directors' Meeting at their new location in the Constitution Center in Washington, DC. We were invited based on our "Archive What I See Now" project being funded as a Digital Humanities Implementation Grant . There were two main goals of the meeting: 1) provide administrative information and advice to project directors and 2) allow project directors to give a 3 minute overview of their project to the general public. The morning was devoted the first goal.  One highlight for me was ODH Director Brett Bobley 's welcome in which he talked a bit about the history of the NEH (NEH's 50th anniversary is coming up in 2015).  The agency is currently in the process of digitizing their historical documents, including records of all of the grants that have been awarded (originally stored on McBee Key Sort cards ). He also mentioned the recent article "The Rise of the Machines&

(or how I learned to embrace the new) I am reaching the end of this part of my professional, academic, and personal life.  It is time to reflect and consider how I got here. The trail ahead. When I started, I thought that I knew the path, the direction, and the work that it would take.  I was wrong.  The path was rugged, steep, and covered with roots and stones that lay in wait to trip the unwary.  The direction was not straight forward.  At times I wasn't sure how to set my compass, and which way to steer.  In the end, there was more work than I thought in the beginning.  But the end is nigh.  The path has been long.  At times the was direction confusing.  The work seemed never ending.  This is a story of how I got to the end, using a little help from "a friend" at the end of this post. Bringing the initially disparate disciplines of graph theory, digital preservation, and emergent behavior together to solve a particular class of problem, is/was non-trivial. 

After exploring London on Sunday, I attended the first DL2014 session: the Doctoral Consortium. Held in the College Building at the City University London , the Doctoral Consortium offered early-career Ph.D. students the opportunity to present their research and academic plans and receive feedback from digital libraries professors and researchers. Edie Rasmussen chaired the Doctoral Consortium. I was a presenter at the Doctoral Consortium in 2012 with Hany SalahEldeen , but I attended this year as a Ph.D. student observer. Session I: User Interaction was chaired by Jos é Borbinha . Hugo Huurdeman was first to present his work entitled "Adaptive Search Systems for Web archive research". His work focuses on information retrieval and discovery in the archives. He explained the challenge with searching not only across documents but also across time. Georgina Hibberd presented her work entitled "Metaphors for discovery: how interfaces shape our relationship with

WARCMerge is the name given to a new tool for organizing WARC files. The name describes it -- merging multiple WARC files into a single one. In web archiving, WARC files can be generated by well-known web crawlers such as Hertrix and Wget command, or by state-of-the-art tools like WARCreate/WAIL and Webrecorder.io which were developed to support the personal web archiving. WARC files contain records not only for HTTP responses and metadata elements but also all original HTTP requests. By having those WARC files, any replay tools (e.g., Wayback Machine) can be used to reconstruct and display the original web pages. I would emphasize here that a single WARC file may consist of records related to different web sites. In other words, multiple web sites can be archived in the same WARC file. This Python program runs in three different modes. In the first mode , the program sequentially reads records one by one from different WARC files and combines them into a new file in which an extr