2011-06-23: How Much of the Web is Archived?

There are many questions to ask about web archiving and digital preservation - why is archiving important? what should be archived? what is currently being archived? how often should pages be archived?

The short paper "How Much of the Web is Archived?" (Scott G. Ainsworth, Ahmed AlSum, Hany SalahEldeen, Michele C. Weigle, and Michael L. Nelson), published at JCDL 2011, is our first step at determining to what extent the web is being archived and by which archives.

To address this question, we sampled URIs from four sources to estimate the percentage of archived URIs and the number and frequency of archived versions. We chose 1000 URIs from each of the following sources:

1. Open Directory Project (DMOZ) - sampled from all URIs (July 2000 - Oct 2010)
2. Delicious - random URIs from the Recent Bookmarks list
3. Bitly - random hash values generated and dereferenced
4. search engine caches (Google, Bing, Yahoo!) - random sample of URIs from queries of 5-grams (using Google's N-gram data)

For each of the sample URIs (4000 in all), we used Memento to discover archived versions, or mementos, of the URI.

We categorize the archives as Internet Archive (using the classic Wayback Machine), search engine caches (Google, Bing, and Yahoo!), and other (e.g., Diigo, Archive-It, UK National Archives, WebCite).

Our first set of graphs (click on each graph for a larger version) shows the mementos discovered for each URI, ordered by the first observation date. These are separated by sample set, so the y-axis on each graph runs from 0-1000. Brown dots indicate mementos discovered at the Internet Archive, blue dots indicate those found in search engine caches, and red dots indicate mementos found at other archives.

DMOZ	Delicious
Bitly	Search engines

There are a few interesting observations:

• DMOZ URIs are well-represented, especially in the Internet Archive. There are two likely reasons for this: DMOZ is the primary source for seed URIs for the Internet Archive and the DMOZ sample contains more old URIs than the other sources.


• Bitly URIs are very poorly represented. The majority of Bitly URIs are not found in any archive. This is currently under further investigation.


• There is a large gap in mementos found in the Internet Archive, starting in 2008. We suspect this is because of the use of the classic version of the Wayback Machine.

The second set of graphs shows the relationship between the density of mementos for a URI and the URI's age. The x-axis is the estimated creation date of the URI, and the y-axis is the number of mementos found for this URI. Large dots indicate that several URIs had similar creation dates and number of mementos. We show density guidelines for 0.5, 1, and 2 mementos created per month.

DMOZ	Delicious
Bitly	Search engines

From these graphs, we make the following observations:

• Many of the DMOZ URIs are archived at least once every 2 months.
• Older Delicious URIs have many mementos.
• A few Bitly URIs have many mementos, but most URIs have 0-10 mementos.

So, how much of the web is archived? Depends on which "web" you mean.




Ahmed presented this work at JCDL 2011. His presentation slides are below:

How Much of the Web is Archived? JCDL 2011 from Ahmed AlSum

This work supported in part by the Library of Congress and NSF IIS 1009392.

-Michele

2012-12-27 Update: The long version of this paper has been posted to arXiv, along with the URI datasets described in the paper.