Web Science and Digital Libraries Research Group

    I've previously written about the Memento Damage project, originally started by Dr. Justin Brunelle , a Web service designed to estimate the amount of damage to a web archive by assessing it's missing resources. Previously, I had been specializating some of the project  while working on the Memento Tracer project, funded by the Alfred P. Sloan Foundation , to take special considerations regarding the damage weighting for Web hosted repository pages.  I have been making further updates to the Memento Damage project over the course of this year that helps improve this analysis and damage estimation. The most prominent is the implementation of a secondary crawler component for analyzing an archived repository and its source tree. Web-hosted Git repositories are hosted on centralized Web platforms, the largest being GitHub along with other major platforms such as GitLab, Bitbucket, and Sourceforge. The source files for a Git project are hosted "behind the scen...

Hello Internet and Archivists!      I'm back for another blog post and I'm excited to share some overhauls, updates, and research that I have been working on for the Memento Damage service, previously developed by Dr. Justin Brunelle , Erika Siregar , and Grant Atkins . The project page is still currently running the original project build but I wanted to share some behind-the-scenes updates before we roll out the new build soon! Under the Hood Fig. 1: Homepage for the Memento Damage web service      When I took on this project, there were many components needing update due to age; the code base had been on Python version 2 still as it had sat over the years as Dr. Brunelle and previous students had graduated and moved on to other endeavors. Updating the code base to Python version 3 was one of the top to-do items! Over a lot of time learning the code base and refactoring I was able to clean up and modernize the code a bit thanks to the syntax and language...

Paywall page from The Advertister Paywalls  have become increasingly notable in the Internet Archive over the past few years. In our recent investigation into news similarity for U.S. news outlets, we chose from a list of websites and then pulled the top stories. We did not initially include subscriber based sites, such as The Financial Times  or Wall Street Journal , because these sites only provided snippets of an article, and then users would be confronted with a "Subscribe Now" sign to view the remaining content. The New York Times , as well as other news sites, also have subscriber based content but access is only limited once a user has exceeded a set number of stories seen. In our study of 30 days of news sites, we found 24 URIs that were deemed to be paywalls, and these are listed below: Memento Responses All of these URIs point to the Internet Archive but result in an HTTP status code of 404. We took all of these URI-Ms from the homepage of thei...