Web Science and Digital Libraries Research Group

2012-01-23: Release of Warrick 2.0 Beta

2012-01-24T11:36:00.000-05:00

After a long hiatus, the Warrick tool has been resurrected with some modifications. Warrick is a free utility for reconstructing (or recovering) a website. The original version of Warrick discovered archived versions of resources by searching the Web Infrastructure (which includes search engine caches and the Internet Archive) for archived versions of web resources. It would automatically download and organize the best versions of the archived resources and package them into a copy of the deleted site.

As discussed by Warrick's creator, Frank McCown, the original version of Warrick was prone to breaking due to frequent changes to search engine APIs and archive URLs. Warrick 2.0, adapted from Dr. McCown's original code by Justin F. Brunelle, interfaces with the Memento framework via the mcurl program (developed by Ahmed AlSum). By incorporating Memento timemaps, Warrick no longer has the responsibility of directly searching and communicating with the caches and archives, or learning about new repositories. Instead, Memento handles the interface and communication with the archives, allowing Warrick to remain unaffected by API or URL changes. This makes Warrick more resistant to failures when repositories change, appear, or disappear. Memento allows Warrick to provide additional functionality, such as the ability to recover sites from a specific point in time by utilizing timemaps and the mcurl program.

Warrick 2.0 has already been helping individuals recover lost web sites. Dag Forssell reached out with the following message:

"I just Googled the idea of restoring a website from the Wayback Machine and discovered your work on Warrick. .... Perhaps you can use my project as one of your guinea pigs.

I am restoring a book by Professor of Law Hugh Gibbons and found when listing his references that he created a website on the principles of law in 2002, then abandoned it in 2006 or so, when he retired. I have downloaded 143 htm files from the Wayback Machine. The site looks complete. But of course, each file comes with its own folder (css, jpgs and such) and the links all point back to the wayback machine. Cleaning it all up will be a lot of work.

If it is in the cards for you ... to take this under your wing, I will be overjoyed."

Dag's site was successfully recovered, and helped us to work out some last remaining bugs before our beta release. Further, Dag mentioned that utilizing Warrick eliminated much of the effort on his part by allowing Warrick to deduplicate resources downloaded from the Internet Archive and arrange the resources in the correct site structure. Further, it allow him a deeper understanding of how the resources interact within the page. His recovered content will reportedly be available live at http://www.biologyoflaw.org/ (the website is not available at the time of this blog posting).

We are happy to announce the release of the Beta source of the project which can be downloaded from its Google Code site. Installation and usage instructions are available from the Google Code site.

Warrick is run with a series of command line flags, or options. These are largely unchanged from the original Warrick, but some flags are new. For example, the user now has the -dr and -R options. The -dr option allows the user to specify the date at which the site should be recovered. For example:

warrick.pl -dr 2004-02-01 http://www.cs.odu.edu/

will recover the ODU Computer Science homepage as close as possible to February 1st, 2004.

-R is the resume flag, which allows a user to resume a suspended reconstruction job from a saved file.

Let's say we run the following recovery job:

warrick.pl -D MyRecoveryDirectory -k -n 100 http://www.justinfbrunelle.com/

This will recovery Justin Brunelle's homepage into the directory MyRecoveryDirectory with the -D flag, convert all links from absolute to be relative to the local disk with the -k flag, and stop the recovery after 100 resources are recovered with the -n flag.

When Warrick completes the recovery session of 100 files, it saves the recovery state in a save file. A user can resume the state by using the -R flag as follows:

warrick.pl -R MYSAVEFILE.save

This will resume the suspended job stored in MYSAVEFILE.save. This will recover an additional 100 files.

For a visual example, let's look at one of the aforementioned commands and demonstrate how it can recover a page.

warrick.pl -dr 2004-02-01 http://www.cs.odu.edu/

We can visit the current (as of 2012-01-23) ODU CS website (http://www.cs.odu.edu/) to see the following representation:

To get an idea of what Warrick will recover, we can observe the ODU CS homepage archived at the Internet Archive on 2004-02-06.

After running Warrick, we can view the reconstructed page at my local directory.

This resource is nearly identical to the copy at the Internet Archive. The branding at the top of the page has been removed to keep the representation as faithful as possible to the original resource at the time of archiving. However, we can see from the reporting that all 10 of the recovered resources that make up the recovered page came from the Internet Archive.

#############################################
Memento Timegate Accesses: 11
Internet Archive Contributions: 10
Bing Contributions: 0
Google Contributions: 0
WebCitation Contributions: 0
Diigo Contributions: 0
UK Archives Contributions: 0
URIs obtained from lister Queries: 0
####
Total recoveries completed: 10
Number of cache resources used: 0
Number of resources overwritten: 0
Number of avoided overwrites: 0
Total failed recoveries: 1
Images recovered: 8
HTML pages recovered: 1
Other resources recovered: 1
URIs left in the Frontier: 0
#############################################

More examples can be viewed in the README file available in the source archive or at Warrick's Google Code Wiki.

Please note that this is only a beta version of the software. Also, it is only runnable via Perl on a Linux command-line. A new version of warrick.cs.odu.edu is in development and will be release soon. This web interface will allow users to run Warrick from a browser which will provide tech-savvy and non-tech-savvy users, alike, to benefit from Warrick.

If you utilize Warrick to recover a web site, we are very interested in learning about your experience; this will help us improve Warrick for future users. Please reach out to us via email by joining the WarrickRecovery Google Group (warrickrecovery@googlegroups.com) to learn how you may help.

--Justin F. Brunelle

2012-01-221: 2011 NFL Season Conference Championship

2012-01-22T14:18:00.000-05:00

The NFL Conference championship games are today. Our models have a tendency to reward teams that can pass the ball well as Passing efficiency correlates with wins rather well. Therefore it is no surprise that two out of the three models predict New England will win over Baltimore. However the Neural Network is predicting that it will be a close game and that New England will not cover the spread of 7 points.

The San Francisco / New York game is going to be a good game to watch. Both teams are very close but the Giants have the edge on passing efficiency.

Favorite	Spread	Underdog	Discrete	Pagerank
At NE	4	BAL	NE	BAL
At SF	1	NYG	NYG	SF

-- Greg Szalkowski

2012-01-01: 2011 NFL Season Week 17

2012-01-01T13:27:00.000-05:00

The last week of the regular season games is here. Week 17 traditionally exhibits greater statistical dispersion than the other weeks. Teams that have locked in playoff spots will be resting the starting players and teams that do not have a chance at the playoffs may be looking for a better draft pick for next year.

Our algorithms once again have picked Green Bay to win but most likely they will rest Aaron Rodgers and most of the starters and Detroit will win the game. Green Bay is an enigma this year, they are 14-1 so far and they have given up more yards than they have gained over the year which invites some interesting analysis.

Favorite	Spread	Underdog	Discrete	Pagerank
At PHI	10	WAS	PHI	PHI
At ATL	14	TB	ATL	ATL
SF	5	At STL	SF	SF
At MIN	6	CHI	CHI	CHI
At GB	8	DET	GB	GB
At NYG	4	DAL	NYG	NYG
At NO	4	CAR	NO	NO
At HOU	5	TEN	TEN	TEN
BAL	2	At CIN	BAL	BAL
PIT	7	CLE	PIT	PIT
At JAX	3	IND	JAX	JAX
At MIA	3	NYJ	MIA	NYJ
At NE	11	BUF	NE	NE
At OAK	3	SD	OAK	SD
At DEN	3	KC	DEN	KC
At ARI	3	SEA	ARI	SEA

-- Greg Szalkowski

2011-12-15: 2011 NFL Season Week 15

2011-12-17T08:47:00.000-05:00

So far this year all three of the prediction algorithms are 68% correct straight up. This is better than the predictions of most of the NFL "experts" such as the guys at ESPN. Last year we ended up right below 70% correct as well. Breaking the 70% barrier over the season seems to be rather hard to do as seen on the Prediction Tracker. Looking into the statistics of those games reveals some interesting information. In the majority of those games, the losing team had better box scores but still lost the game. We had thought that incorporating the betting line data this year would have had impact but the accuracy of the straight up predictions is not significantly better than last year.

The season isn't over yet and anything can happen so here are the predictions for week 15.

Favorite	Spread	Underdog	Discrete	Pagerank
DAL	7	at TB	DAL	DAL
at NYG	10	WAS	NYG	NYG
GB	9	at KC	GB	GB
NO	9	at MIN	NO	NO
at CHI	3	SEA	CHI	SEA
at BUF	2	MIA	BUF	BUF
at HOU	4	CAR	HOU	HOU
TEN	4	at IND	TEN	TEN
CIN	7	at STL	CIN	CIN
at OAK	4	DET	OAK	OAK
NE	9	at DEN	NE	NE
at PHI	3	NYJ	PHI	NYJ
at ARI	3	CLE	ARI	ARI
at SD	4	BAL	BAL	BAL
PIT	2	at SF	PIT	PIT

-- Greg Szalkowski

2011-12-14 Python & Memento Presentation for the ODU ACM

2011-12-14T12:36:00.007-05:00

Earlier this semester, I was invited to present Python at an ODU ACM meeting. I presented a brief overview of the Python language and followed up with a code walk through of the code I use to parse Memento timemaps in my current research.

Python, of course, has advantages and disadvantages compared to other languages. Since most ODU undergrads have experience with C++, the presentation presents Python with respect to C++. Pythons advantages include a fast development cycle and an extensive collection of community libraries. Its primary disadvantage compared to C++ is execution speed. My experience is that Python is sometimes over 100 times slower.

Python's basic syntax and semantics are straight forward, so the presentation focused on the Python equivalents of commonly-used C++ constructs and the differences between static (C++) and dynamic (Python) typing. Python's implementation of high-level data types (lists, dictionaries, tuples, and sets) and functional code were compared to the complexity of the C++ equivalents.

ODU ACM Python & Memento Presentation

To bring all the pieces together, I did a code walk through of the python.py module I use to parse Memento timemaps (see the Memento Introduction and Internet Draft for more information). The module has two classes. The TimeMap class is a parser and dictionary for timemap data. The TimeMapTokenizer class is a tokenizer for link-style timemaps.

To load a timemap, a new instance of TimeMap is created using the timemap's URI, which is the constructor's only argument. A TimeMapTokenizer instance returns individual tokens, simplifying the parsing code in the get_next_link function. TimeMap implements the __getitem__ function, allowing it to act as a Python dictionary. TimeMapTokenizer implements the __iter__ and next functions, which the use of Python iteratation constructs over the list of tokens.

— Scott G. Ainsworth

2011-12-14: CS 495/595 Web Server Development for Spring 2012

2011-12-14T11:28:00.004-05:00

The only WS-DL related class that will be offered in spring 2012 is CS 495/595 "Web Server Development". I had planned to offer CS 751/851 "Introduction to Digital Libraries, but I've taught that the last two springs and it has been a while since I've taught the web server development class (the last offering was actually from Martin Klein in spring 2010).

The premise of this course is that the best way to really get to know HTTP is to build a fully-functional web server from scratch in the language of your choice. That sounds simple enough, but it becomes quite challenging, in part because if you do a poor job at design at the beginning you have to live with the consequences the entire semester. On the other hand, do a good job up front and each assignment will just drop into place (hello, software design). Along the way, you'll also become quite familiar with reading RFCs and the REST architectural model.

Take a look at past offerings of the class for an idea of what the structure will be. The CRNs are 35757 (CS 495) and 35758 (CS 595). The class will be on Tuesdays, 4:20 -- 7:00 pm in r. 2120.

--Michael

2012-01-09 edit: The class homepage is now available.

2011-12-08: Summer Microsoft Internship

2011-12-08T04:07:00.004-05:00

It all started in San Francisco airport while waiting to get my luggage on my way to the PDA2011 conference. The recruiter from Microsoft called me to inform me that I have been accepted to intern at Microsoft Silicon Valley this summer. I was ecstatic and after a couple of months of bureaucracy and a ton of documents I was ready to leave Norfolk by the end of May. Since I haven’t been on an adventure or a trip for a long time, and since I will definitely need a car in California for the three months of the summer, I decided to drive my car all across the continent. I have always wanted to make a road trip like that where I can stop in every city or town along the way, check out their attractions and eat from their authentic cuisines.

At the same time, our colleague and best friend Moustafa Aly managed to secure a job at Amazon’s engineering office in San Francisco. So when he knew I was going to drive all the way there he told me: “forget the plane, I will join you!”

We left Norfolk on the 24th, set the odometer of the car to 0 and having in mind since we are information retrieval and social networking people we will make our status updates and check-ins on Facebook our trip’s record keeper. We picked the route, filled up the car and drove. From Norfolk, stopping at Richmond and Nashville we drove through a tornado passing Tennessee, almost ran out of gas in Texas in the middle of no where, changed the clock twice in one day, eating the best steak I have ever had in Texas and the best burritos on earth in Las Cruses, playing with rockets in White sands missile range, passing over the Hoover dam and the burning the car’s AC compressor in the desert of Nevada we finally made it to Las Vegas where we wanted to spend an entire day relaxing. Next day we started driving and after 9 more hours we made it to San Francisco finishing 3559.6 miles in 5.5 days.

Working at Microsoft Silicon Valley definitely has its perks. The location was amazing and the engineers there are really incredible. I joined the office 365 server-side team for PowerPoint where I shared my office with another intern from UC Berkeley. Working with this team I had the most liberty I had in years working for companies. We sat together and set the goals I need to reach for this internship and they gave me the entire freedom to pick the way I was going to build it, which is more my style in working. I was supposed to start the implementation of a certain fraction of the distribution and investigate two other things but to my surprise they liked what I did with the first task so they decided to modify my internship goals to finish this project completely, reach ship quality and release it in the next version. With this I passed all the phases of software development from meeting with managers, architects and program managers to setting the design to development to finally quality and integration testing. Finally I had to demo my work to the three department managers to see if this could be incorporated in the next shipping release, and to my delight they were fascinated by it and it will be shipped!

The first day I attended the orientation and they gave us an overview to what we will be doing this summer and how are we going to be evaluated. Our mentors then came and took us and I was introduced to my team, the PowerPoint team. Immediately after that I was introduced to the available projects and I choose the one that was more appealing to me. Immediately after that I was granted permissions to access the codebase. Imagine having the source code of both PowerPoint and the server cloud back-end, it felt awesome! for the next two weeks I tried to break in the thousands of lines of code and produced a prototype proof of concept that I was on the right track. By the end of the first week I set my internship goals with my mentor but after the fast prototype I produced I was called to a meeting with both the test and the product management team, I was representing the development team. They decided to change my goals completely to actually build the entire feature and its backend support from scratch and have the opportunity to ship it. Knowing the task in hand of rebuilding the PowerPoint backend on the cloud with the appropriate interface to match the latest award-wining rich-client application I had to go back to the basics. I had several one-on-ones with the development team of PowerPoint client-side to understand piece by piece the functionality of each module of the application. The problem with a project like PowerPoint that it is fairly old and fairly stable with more than 20+ years of development and thousands of legacy code. I was completely lost in the beginning but my mentor didn't let me stumble much, I was practically staying in his office the first couple of weeks. We used C++ and C in the backend with javascript and C# for the matching interface. This was the trickiest part, the ability to match functionalities between two very different frameworks. At a certain point I found a severe gap in the design document related to the functionality. I talked with my manager and he told me a change like the one you want in the design document needs to be escalated. A couple of hours later I was sitting in a room full of Microsoft's elite developers, testers, PMs and managers, the least of which has 7 years work experience under his belt,...and me! That what I loved about Microsoft, even though I was just an intern I owned the project and they appreciated that. I explained my case and it was approved and the design document was changed! I was so proud of myself that day.

The atmosphere within the office was relaxing, cool, upbeat and always challenging. I can fairly say I was spoiled this summer. I was residing in the corporate housing complexes where I got a spacious studio apartment fully furnished with maid service that come clean weekly! Courts, swimming pool and a huge hot tub all provided for free within the apartment complex. Every other week the recruiters and the PR managers created an event, party or outing for all the interns on campus. We went hiking, bowling, watching movies and they even flew us to Seattle to visit the headquarters for the summer intern event. They paid flight tickets, the luxury hotel and even a car rental. Steven Sinofsky gave us a wonderful presentation where they show us classified sneak peeks to the all-new amazing Windows 8 and I was genuinely impressed. At the company store we got lots of t-shirts, games and gadgets with our employee discount. After that they rented the Zoo for us since we were about 1000 interns from all over the country and they got us the “Dave Matthews” band and gave each one of us a brand new xbox360 with Kinect!

It was definitely unique and rewarding to work with all those interns from the top universities all over the country: MIT, UC Berkeley, Stanford, …etc. I asked around and I found that I was the only representative from ODU so I was definitely proud and tried to behave. Me and the other interns became friends and since most of us are residing on the same apartment complex we gathered almost every night and on the weekends we went and discovered the city and the surrounding area. Unfortunately I didn’t join them in the Yosemite hiking/camping trip, as I was sick that day. One day we all decided to wear suits and sunglasses all day at work and call it "Brogramming" day. Someone took a photo of us and it gone viral on twitter and facebook!

In conclusion I feel honored and blessed for being able to work at this wonderful fascinating place with all those extremely intelligent colleagues. My manager/team lead told me on my first day one thing that I believe it changed everything. He said you were only an intern during the 2-hour orientation session, now consider yourself a full time software engineer and own your work. This definitely helped me to shine, participate, own my work, suggest enhancements, which actually were considered, and we changed the design document. Now, I can proudly say that my product is being used currently by millions of users; probably you are using it right now!

-- Hany SalahEldeen

2011-12-07: 2011 NFL Season Week 14

2011-12-07T22:26:00.001-05:00

Week 14 of the 2011 NFL season is upon us. Talk of play-off teams and Superbowl probabilities fill the airwaves even more than Christmas music. Sitting in traffic on the drive home from work tonight I was listening to a few on-air personalities discussing Green Bay and New England for the Superbowl. Green Bay has already clinched a playoff berth and many people would say they are headed to the Superbowl this year. The comment that caught my attention was that the defense for both teams was terrible this year and the only reason they were doing well this year is that their offenses were so good that they could "outscore their mistakes".

This led me to think about the Colts without Peyton Manning this year. For the past 3 or 4 years the Colts with Manning as their quarterback have dominated the sport. It would seem that they built the entire team around Manning. The Colts would run up the score on offense and then the opposing team would be forced to attempt to pass often just to catch up. Then the Colts defense would focus on the opposing teams quarterback to keep him from making plays. Now this year without Manning the Colts have no game. Are Green Bay and New England in a similar situation?

Contemplating statistics during rush hour traffic is a good way to become a statistic so I did not get much more in depth listening to the show, but after arriving home I ran some SQL queries to check the veracity of the claims made by the radio show.

Indeed it is true that the defense for both Green Bay and New England have given up more than the average number of yards this year. They are both almost dead last in defensive performance. Here is a list of the teams with the average number of yards given up per play on both passing and rushing plays.

Team	Yards given up per play
Atlanta	4.3638
Pittsburgh	4.7886
Baltimore	4.8224
Houston	4.9090
Cincinnati	5.1182
San Francisco	5.1645
New York	5.1786
Cleveland	5.2142
Jacksonville	5.2292
Seattle	5.3491
Tennessee	5.3568
Washington	5.4359
Detroit	5.5313
Arizona	5.5506
Miami	5.5726
Denver	5.6300
Kansas City	5.6887
Chicago	5.6887
Oakland	5.7423
Dallas	5.7464
San Diego	5.7867
Minnesota	5.8281
St. Louis	5.8436
Indianapolis	5.8923
Philadelphia	6.0145
Buffalo	6.0374
New York	6.0508
New Orleans	6.0600
Carolina	6.3130
New England	6.3642
Tampa Bay	6.4829
Green Bay	6.5041

I have a feeling that a team with a balanced offense and a good pass defense like Pittsburgh or Baltimore could give New England and/or Green Bay a tough time in the post season but maybe we will cover that next week.

The predictions for week 14 are:

Favorite	Spread	Underdog	Discrete	Pagerank
at PIT	9	CLE	PIT	PIT
at BAL	7	IND	BAL	BAL
HOU	5	at CIN	CIN	HOU
at GB	15	OAK	GB	GB
at NYJ	7	KC	NYJ	NYJ
at DET	7	MIN	DET	DET
NO	2	at TEN	NO	TEN
PHI	9	at MIA	PHI	PHI
NE	15	at WAS	NE	NE
ATL	4	at CAR	ATL	ATL
at JAX	5	TB	TB	JAX
SF	3	at ARI	SF	SF
DEN	2	CHI	DEN	CHI
at SD	12	BUF	SD	BUF
at DAL	3	NYG	DAL	DAL
at SEA	4	STL	SEA	SEA

-- Greg Szalkowski

2011-12-01: 2011 NFL Season Week 13

2011-12-01T13:08:00.000-05:00

Week 13 of the 2011 NFL season is upon us. This week New England is a 20 point favorite over Indianapolis. 20 points is rather rather significant for a line value. In fact since 2002 there have only been six games with a line value of 20 or greater. Of those six games, New England was the favorite in five of them. In none of the five games did New England cover the spread but they came close to covering the spread in the 2007 game against Miami winning by 21 points with a 22 point line value.

Favorite	Spread	Underdog	Discrete	Pagerank
PHI	5	at SEA	PHI	SEA
TEN	3	at BUF	BUF	TEN
at CHI	4	KC	CHI	CHI
MIA	7	at OAK	MIA	OAK
at PIT	6	CIN	PIT	PIT
BAL	1	at CLE	BAL	BAL
NYJ	1	WAS	NYJ	NYJ
at HOU	7	ATL	ATL	HOU
CAR	6	at TB	TB	CAR
at NO	7	DET	NO	NO
At MIN	6	DEN	DEN	DEN
at SF	10	STL	SF	SF
DAL	8	at ARI	DAL	DAL
GB	2	NYG	GB	GB
NE	10	IND	NE	NE
SD	4	JAX	SD	SD

-- Greg Szalkowski

2011-11-24: 2011 NFL Season Week 12

2011-11-24T15:31:00.001-05:00

Happy Thanksgiving!

I apologize for posting these a little late but I have been cooking food for the past three days. When I was not cooking I was reading papers about modifications to Support Vector Models to get some ideas to improve the accuracy of our predictions.
Adapting Ranking SVM to Document Retrieval concentrated on a modification of the hinge loss function when training the model to increase accuracy. Training a Support Vector Machine in the Primal points out that much literature jumps right to the dual optimization aspect of SVMs and does not pay enough attention to the primal problem. A portion of the paper mentions replacing the hinge loss function with one that is differentiable such as the Huber loss function.

While experimenting with SVM training I observed an interesting data point. Using NFL statistics from 2002 to 2010, one of the training methods assigned the following weights to the teams.

	1.1276	Indianapolis Colts
	1.1055	New England Patriots
	0.5704	Philadelphia Eagles
	0.4269	Pittsburgh Steelers
	0.3317	Tennessee Titans
	0.3184	San Diego Chargers
	0.2922	Baltimore Ravens
	0.2248	New Orleans Saints
	0.1976	Green Bay Packers
	0.1718	Jacksonville Jaguars
	0.1621	Tampa Bay Buccaneers
	0.0957	Carolina Panthers
	0.0946	Atlanta Falcons
	0.0877	New York Jets
	0.0704	Chicago Bears
	0.0504	New York Giants
	-0.0569	Dallas Cowboys
	-0.0689	Buffalo Bills
	-0.0740	Miami Dolphins
	-0.0812	Houston Texans
	-0.1085	Denver Broncos
	-0.1752	Cincinnati Bengals
	-0.2420	Kansas City Chiefs
	-0.3457	Cleveland Browns
	-0.3631	Minnesota Vikings
	-0.3634	Seattle Seahawks
	-0.4150	Arizona Cardinals
	-0.5037	St. Louis Rams
	-0.5121	Oakland Raiders
	-0.5633	Washington Redskins
	-0.6925	San Francisco 49ers
	-0.7623	Detroit Lions

Well it is time for me to break out the whipping cream and make some homemade whip cream for the pies I baked the other day. Here are the picks for this week.

Favorite	Spread	Underdog	Discrete	Pagerank
GB	6	at DET	GB	GB
at DAL	8	MIA	DAL	DAL
at BAL	3	SF	BAL	BAL
at STL	5	ARI	STL	STL
at NYJ	1	BUF	NYJ	BUF
at CIN	8	CLE	CIN	CIN
HOU	4	at JAX	HOU	HOU
CAR	7	at IND	CAR	IND
at TEN	20	TB	TEN	TEN
at ATL	11	MIN	ATL	ATL
at OAK	6	CHI	OAK	CHI
at SEA	5	WAS	SEA	SEA
NE	6	at PHI	NE	PHI
at SD	6	DEN	SD	DEN
PIT	11	at KC	PIT	PIT
at NO	3	NYG	NO	NO

-- Greg Szalkowski

2011-11-17: 2011 NFL Season Week 11

2011-11-17T16:02:00.000-05:00

Thursday Night Football, this week the NY Jets play at Denver. The Jets have a number of players on the injured list this week. Even with those injuries all three of our algorithms picked the Jets to win on Thursday.

The Jets injury list is not as bad as some of the other teams. Philadelphia's quarterback, Vick has two broken ribs and has not been at practice all week. Kansas City's quarterback Matt Cassel underwent hand surgery and will probably be out for the rest of the season.

A weakness of our algorithms is that they are heavily based on this years performance to date. A major injury to an important player that may or may not have an impact of game performance is not really taken into account. That is one of the reasons we have incorporated the Line data this year. Hoping that the "Collective Intelligence" of the crowd would help to point out teams that may not perform differently.

Favorite	Spread	Underdog	Discrete	Pagerank
NYJ	4.5	at DEN	NYJ	NYJ
at ATL	4	TEN	ATL	TEN
BUF	3.6	at MIA	MIA	BUF
at BAL	5	CIN	BAL	BAL
JAX	7	at CLE	JAX	JAX
at MIN	5	OAK	MIN	OAK
at DET	11	CAR	DET	DET
at GB	18	TB	GB	GB
DAL	8	WAS	DAL	DAL
at SF	8	ARI	SF	SF
at STL	2	SEA	SEA	SEA
at CHI	5	SD	CHI	CHI
at NYG	7	PHI	NYG	PHI
at NE	8	KC	NE	NE

-- Greg Szalkowski

2011-11-10: 2011 NFL Season Week 10

2011-11-10T20:20:00.002-05:00

Thursday Night Football is back! The match-up for tonight features the San Diego Chargers at home vs the Oakland Raiders.

This is a very close matchup according to the stats. San Diego has a offensive pass efficiency of 7.2 and Oakland 6.7. Oakland has a better run game but not by much. The defensive ratings are almost exactly the same with San Diego leading by a little bit.

The SVM and Neural Network both chose San Diego to win but the PageRank algorithm decided Oakland was a better choice.

Favorite	Spread	Underdog	Discrete	Pagerank
At SAN	8.5	OAK	SAN	OAK
PIT	1.5	At CIN	PIT	PIT
At KC	5	DEN	KC	KC
At IND	4	JAX	JAX	JAX
At DAL	3	BUF	DAL	DAL
HOU	7	At TB	HOU	HOU
At CAR	6	TEN	Car	TEN
At MIA	5	WAS	MIA	WAS
At ATL	6	NO	ATL	NO
At CHI	6	DET	CHI	CHI
At CLE	4	STL	CLE	STL
At PHI	11	ARI	PHI	PHI
BAL	4	At SEA	BAL	BAL
NYG	3	At SF	SF	SF
NE	10	At NYJ	NE	NYJ
At GB	14	MIN	GB	GB

-- Greg Szalkowski

2011-11-10: Day in the Life of a Computer Scientist

2011-11-10T11:27:00.000-05:00

Old Dominion University has a freshmen computer science course that focuses on what it means to be a computer scientist. This course discusses career opportunities, current research being performed, and serves to debunk myths and misconceptions about the field of computer science. Such myths include: we never talk to humans, we code our entire lives away, and we are nocturnal. I was invited to be a guest lecturer for the class last night. Even though the last myth is sometimes true, I did my best to touch on each of these talking points during the presentation embedded below.

Day in the Life of a Computer Scientist

View more presentations from Justin Brunelle.

The first topics I spoke about in the presentation generated the majority of questions and discussion. I spoke first about the digital preservation work being performed in the WS-DL group at ODU. This, of course, included discussing the ArchiveFacebok, Warrick, and Memento projects at a cursory level. During our discussion, we hit on the issues of copyright and web crawling, and why, as computer scientists, we find these problems interesting. We briefly talked about revisitation policies and change-rate studies of web pages that are important for search engines and archival methods. (Interested readers should direct their attention to Cho and Garcia-Molina's work (1999) for a canonical study on recrawl and page change rates.) We also discussed why computing theory (not just development) is important in the current research being performed in the academic and industrial communities.

The remainder of my talk was a description of what I do on a daily basis as a professional computer scientist. I mentioned that I worked at The MITRE Corporation as a developer and researcher, and discussed what my job entails. For example, I practice Agile engineering, work with people on a daily basis, and probably only spend less than a quarter of my time in actual development. The remainder of my time is spent in testing cycles, working with customers to find direction for products, writing documentation, and other "non-coding" aspects of software development. Further, I discussed that MITRE is unique company in that it is a Federally Funded Research and Development Center (FFRDC), and supports the US government in an advisory roll. This point illustrated that there are variety of opportunities available to computer scientists, and not all of them are at traditional corporations.

My lecture was meant to illustrate that a professional developer doesn't sit in a dark cubicle all night hammering out code, and goes weeks without human interaction. More importantly, this presentation provided examples of work being done in industry and academia, and how the degree they are earning will benefit them in their career.

--Justin F. Brunelle

2011-11-4: 2011 NFL Season Week 9

2011-11-05T23:21:00.001-04:00

Week 9 is the last of the bye weeks for this year. Two games that have a higher level of chatter this week and should be good games to watch are the New England vs Giants game and the Pittsburgh vs Baltimore game.

The Patriots with Brady and the Giants with Manning, both have potent veteran quarterbacks and these game will probably play on the passing efficiency of both teams. The offensive passing efficiency of both teams is comparable at about 7.8 yards, however the Giants pass defense is better with only 5.9 yards given up compared to 7.5 for New England.

The rhetoric for the Baltimore vs Pittsburgh game has been rather lively. The Baltimore defense is possibly one of the best this year with a pass defense of only 4.8 yards given up although the offense is has not been stellar with a pass efficiency of 5.8 yards. Pittsburgh's pass offense is better, rated at 7.02 yards while their pass defense, while still decent, is not as good as Baltimore's with a rating of 5.1 yards.

Favorite	Spread	Underdog	Discrete	Pagerank
ATL	0.5	At IND	ATL	ATL
At NO	12	TB	NO	NO
At HOU	9	CLE	HOU	HOU
At BUF	4	NYJ	BUF	BUF
At KC	3	MIA	KC	KC
SF	4.5	At WAS	SF	SF
At DAL	6	SEA	DAL	DAL
At OAK	5	DEN	OAK	OAK
At TEN	3	CIN	TEN	TEN
At ARI	7	STL	ARI	STL
At NE	6	NYG	NE	NYG
At SD	2	GB	GB	GB
At PIT	3	BAL	PIT	BAL
At PHI	8	CHI	PHI	PHI

-- Greg Szalkowski

2011-11-05: Agile Engineering - ODU's ACM Meeting

2011-11-05T06:59:00.000-04:00

I was invited to present an overview on Agile Development to Old Dominion University's ACM chapter. More specifically, I gave an overview of the Scrum method. My work in MITRE's Agile Engineering department has allowed me to practice Agile methodologies in the work force. Through this presentation, I shared my experiences with the members of the ACM.

Agile Engineering - ODU ACM

View more presentations from Justin Brunelle

Agile engineering's main focus is a shift from a linear development model. The Waterfall model is the classic example of a linear process model. Agile focuses on a cyclic and adaptive model. One of the main focuses of Agile is to receive and incorporate user feedback into the development process in order to produce a better product for the user. Also, it allows the product owner to garner greater control over a project.

Each cycle in Agile includes all of the traditional development steps: Requirements, Design, Implementation, Verification, and Assessment/Maintenance. These cycles are sometimes called sprints. At the conclusion of each sprint, a fully releasable product should be available. That is, the end of the sprint produces a product that has been through all of the necessary development steps and can be sold as a subset of the end-goal product. This provides the benefit of having a complete and deliverable product even if funding is cut or production must be halted.

An Agile development model provides the benefit of Failing Early. This means the development team can encounter and solve errors earlier in the development process and solve them when the costs are lower. An overly simplistic example would be the selection of a database. If MySQL is chosen at the beginning of a project using Agile, the development team would know earlier in the process if it was suitable for the solution. However, in the Waterfall model, it is possible to not understand the requirements until too late in the process to make a cheap switch.

An ODU WS-DL alumnus (Carlton Northern) has been instrumental in releasing a handbook for implementing Agile methods. This handbook provides guidelines for implementing Agile methods in the government (specifically the DoD) environment.

These resources should serve as an introduction to Agile methods and the benefits of using this development model.

-- Justin F. Brunelle

2011-10-28: 2011 NFL Season Week 8

2011-10-29T22:54:00.000-04:00

I am back from San Diego and while I ran into some computer problems while I was there, thankfully the results of my trip were much better than the results of last weeks predictions.

Our discrete winner predictor is based on a Sequential Minimization Optimization (SMO) method for training the Support Vector Model (SVM). In our experiments, the SVM has proven to be one of the best binary classifiers for predicting the winner/loser of NFL games.

As I mentioned a few weeks ago, this year we have incorporated the betting line data into the classification model as a form of collective intelligence. The betting line data quickly began to dominate the output of the prediction model followed by passing efficiency and turnovers in importance to the outcome. The result of favoring the betting line is that the classifier usually follows the favorite and when there are a number of upsets like last week, then our results are below expectations.

Indeed many of the experts did not fare that well either last week. This led me to think about how " the experts" and the hypothetical average NFL fan make their choices. Are the fans influencing the betting line with with their bets or is the line influencing the bets of the fans. Some form of endogeneity rearing its head and interfering with the model.

Washington and Buffalo will be playing in Toronto this week.

Favorite	Spread	Underdog	Discrete	Pagerank
At TEN	3	IND	TEN	TEN
At HOU	7.2	JAX	At HOU	At HOU
At CAR	3.4	MIN	CAR	CAR
NO	6.8	At STL	NO	NO
At BAL	9.3	ARI	BAL	BAL
At NYG	8.4	MIA	NYG	NYG
At BUF	4.4	WAS	BUF	BUF
At DEN	1.7	DET	DEN	DET
At PIT	0.8	NE	NE	NE
At SF	5.6	CLE	SF	SF
CIN	2.7	At SEA	CIN	CIN
At PHI	4.2	DAL	PHI	DAL
SD	0.8	At KC	SD	KC

-- Greg Szalkowski

2011-10-22: 2011 NFL Season Week 7

2011-10-23T12:21:00.000-04:00

I have been on travel in San Diego all this week and I have had computer issues the entire time. Therefore I am posting these picks at the last minute and do not have much in the way of commentary especially since I was on an airplane during most of the games this past Sunday. I would like to thank my wife for typing in the commands I told her over the phone so that we could run the algorithms in order to get the picks done. I would not have been able to do it without her.

Favorite	Spread	Underdog	Discrete	Pagerank
TB	2.2	CHI	TB	CHI
at CAR	4.9	WAS	CAR	WAS
SD	0.3	At NYJ	SD	NYJ
At CLE	2.7	SEA	CLE	SEA
At TEN	5	HOU	TEN	TEN
At MIA	1.3	DEN	MIA	DEN
At DET	3.6	ATL	ATL	DET
At OAK	5.5	KC	OAK	OAK
PIT	4.9	At ARI	PIT	PIT
At DAL	10.3	STL	DAL	DAL
GB	1.5	At MIN	GB	GB
At NO	4.5	IND	NO	NO
BAL	4	At JAX	BAL	JAX

-- Greg Szalkowski

2011-10-14: 2011 NFL Season Week 6

2011-10-16T00:15:00.000-04:00

Our neural network predictor was 68% correct straight up this past week but overall our results were not awe inspiring. Two of the games that almost everyone got wrong were the Eagles-Bills and Seahawks-Giants games. In both games the favorite lost and one of the crucial stats was interceptions. Michael Vick of the Eagles threw four interceptions and Eli Manning threw three for the Giants. This is completely out of character for either of the quarterbacks.

So far this year our Support Vector Machine (SVM) predictor has tracked the favorites very closely. With the addition of the line data this year, the line value has driven the output of the SVM. Ignoring the Line values, passing efficiency and turnovers forced by the defense have been two of the most dominant statistics.

Predictions for week 6:

Favorite	Spread	Underdog	Discrete	Pagerank
At GB	14.5	STL	GB	GB
At PIT	9.5	JAX	PIT	PIT
PHI	-0.7	At WAS	PHI	WAS
At DET	3.4	SF	DET	SF
At ATL	-0.3	CAR	ATL	ATL
At CIN	3.2	IND	CIN	CIN
At NYG	4.7	BUF	NYG	BUF
At BAL	5.6	HOU	BAL	BAL
At OAK	6.4	CLE	OAK	OAK
At NE	8.6	DAL	NE	DAL
NO	-4.5	At TB	NO	NO
At CHI	-1.7	MIN	CHI	MIN
At NYJ	3.8	MIA	MIA	NYJ

-- Greg Szalkowski

2011-10-06: Week 5 2011 NFL Season

2011-10-07T10:45:00.001-04:00

Week 4 performance was rather pleasing. Straight up and against the spread were 80% and 75% correct. Buffalo lost with that last minute field goal and as to why Philadelphia fell apart in the second half and lost a 20 point lead has been the subject of numerous commentator's discussions. Hopefully the predictions continue to perform at this level but pessimism indicates that they will regress to the mean.

Week 5 of the NFL season means the commencement of bye weeks. This week's teams on bye are the Baltimore Ravens, Cleveland Browns, Dallas Cowboys, Miami Dolphins, St. Louis Rams and Washington Redskins.

For comparison purposes we have included one of the better performing algorithms from the past two years. The PageRank algorithm that we modified to indicate strong teams averaged 68% for straight up predictions over the past two years. A more detailed explanation is provided in one of our previous posts.

The predictions for Week 5:

Favorite	Line	Underdog	Discrete	PageRank
At IND	6.3	KC	IND	KC
At MIN	4.6	ARI	MIN	ARI
At BUF	1	PHI	PHI	BUF
At HOU	3.4	OAK	HOU	OAK
At CAR	0.2	NO	NO	NO
CIN	3.2	At JAC	JAC	JAC
At PIT	1	TEN	PIT	TEN
At NYG	9.7	SEA	NYG	NYG
At SF	2	TB	TB	SF
At NE	11.3	NYJ	NE	NYJ
SD	1.3	At DEN	SD	DEN
GB	2.9	At ATL	GB	GB
At DET	5.3	CHI	CHI	DET

-- Greg Szalkowski

2011-10-02: 2011 NFL Season Under Way

2011-10-02T12:04:00.001-04:00

The 2011 NFL season is underway and we are ready to put some of our improved algorithms to the test. Last year we primarily used box score data for our predictions. This resulted in adequate performance but nothing spectacular.

This year we are increasing the collective intelligence quotient in our algorithm by incorporating betting line data and line movement. The purpose of the betting line is to make the sportsbooks money by splitting the betting population in half. The line will move as a result of betting pressure presented by the betting population. e.g. The favorite team is favored by 5 points. Many bettors may feel that the favorite team is not that good and place bets on the underdog. With an unbalanced wager profile the sportsbook has the potential to lose money so they will move the bet line until the incoming bets are equal on each side. This movement is a form of collective intelligence of the betting population.

Another change this year is that in addition to choosing the winner as a discrete value (winner or loser) we will also predict the line value as a continuous variable. This line value is what we think the line should be. If the favorite team is favored by 5pts and we predict 3pts it may be wise to vote on the underdog. However if the favorite team is favored by 2pts and we predict 7pts, the favorite is the better option.

Without further ado, here is what we are looking at for week 4:

Time	Favorite	Line	Underdog	Discrete
10/2 1:00 ET	At Dallas	3.1	Detroit	Dallas
10/2 1:00 ET	New Orleans	2.5	At Jacksonville	New Orleans
10/2 1:00 ET	At Philadelphia	8.6	San Francisco	Philadelphia
10/2 1:00 ET	Washington	2.2	At St. Louis	Washington
10/2 1:00 ET	Tennessee	4.3	At Cleveland	Tennessee
10/2 1:00 ET	At Cincinnati	2.3	Buffalo	Buffalo
10/2 1:00 ET	Minnesota	3.7	At Kansas City	Kansas City
10/2 1:00 ET	Carolina	0.7	At Chicago	Chicago
10/2 1:00 ET	At Houston	0.9	Pittsburgh	Houston
10/2 4:05 ET	Atlanta	2.0	At Seattle	Atlanta
10/2 4:05 ET	NY Giants	0.3	At Arizona	NY Giants
10/2 4:15 ET	At San Diego	5.2	Miami	San Diego
10/2 4:15 ET	At Green Bay	9.5	Denver	Green Bay
10/2 4:15 ET	New England	4.5	At Oakland	New England
10/2 8:25 ET	At Baltimore	6.8	NY Jets	Baltimore
10/3 8:35 ET	At Tampa Bay	1.3	Indianapolis	Tampa Bay

--Greg Szalkowski

2011-09-14: Dissertation Completed

2011-09-14T14:11:00.010-04:00

I am very happy to write about the successful completion of my dissertation work in the Computer Science Department at Old Dominion University.
My dissertation is titled "Using the Web Infrastructure for Real Time Recovery of Missing Web Pages" and, as the title suggests, it makes several contributions in the areas of digital data preservation and information retrieval. In brief, the dissertation evaluates multiple techniques for a "just-in-time" approach to web page preservation. We, for example, investigate the suitability of lexical signatures and web page titles to rediscover missing content. These two methods are based on old copies of the pages provided by the Memento framework. We also analyze the performance of tags that users have created to annotate pages as well as the most salient terms derived from a page's link neighborhood as methods to find missing pages.

On the practical side, the dissertation introduces Synchronicity, a Firefox add-on that implements all evaluated methods for web page recovery. It catches 404 "Page not Found" errors when they occur and offers alternatives in real-time, while the user is browsing. I concluded writing my thesis in June, defended on July 18th and got the degree officially awarded in August 2011.

Dissertation Defense

View more presentations from Martin Klein

It goes without saying this work would not have been possible without the outstanding support from my dissertation committee. It consisted of the internal members Dr. Michele C. Weigle, Dr. Yaohang Li and Dr. Mohammad Zubair and the external members Dr. Herbert Van de Sompel and Dr. Robert Sanderson.

I am deeply grateful to my advisor Dr. Michael L. Nelson for his eternal patience and superior guidance and mentoring. He truly is a role model for all aspiring academics (and he took me to a Hokies football game).

I am now looking back at six years of taking classes (MS and Ph.D. level), passing diagnostic and candidacy exams, conducting countless experiments, publishing over 20 research papers (and writing even more), teaching two classes, giving numerous guest lectures and I can finally give an answer to the ever annoying question: "When are you going to be done?".

As my next step, I am very excited to join the Research Library at the Los Alamos National Laboratory as a Postdoctoral Researcher. I will work with Herbert and Rob on Memento and on making time-based access of web resources more convenient and, of course, will enjoy the green and red chili!

--
martin

2011-08-28: KDD 2011 Trip Report

2011-08-28T19:39:00.008-04:00

Author: Carlton Northern

The SIGKDD 2011 conference took place August 21 - 24 at the Hyatt Manchester in San Diego, CA. Researchers from all over the world interested in knowledge discovery and data mining were in attendance. This conference in particular has a heavy statistical analysis flavor and many presentations were math intensive.

I was invited to present my masters project research at the Mining Data Semantics (MDS2011) Workshop of KDD. In this paper, we present an approach to find social media profiles of people from an organization. This is possible due to the links created between members an organization. For instance, co-workers or students will likely friend each other creating hyperlinks between their respective accounts. These links, if public, can be mined and used to disambiguate other profiles that may share the same names as those individuals we are searching for. The following figure shows the amount of profiles found from the ODU Computer Science student body for each respective social media site and the links found between them.

This picture represents the actual students themselves and the links between them. Black nodes are undergrads, green nodes are grads, and red nodes are members of the WS-DL research group.

These are the slides:

An Unsupervised Approach to Discovering and Disambiguating Social Media Profiles

View more presentations from carlton.northern.

Here is the paper:

MDS 2011 Paper: An Unsupervised Approach to Discovering and Disambiguating Social Media Profiles

View more documents from carlton.northern.

I've synopsized some of the interesting presentations from the conference:

Stephen Boyd - Stanford University "From Embedded Real-Time to Large-Scale Distributed". Stephen Boyd's talk focused on his current research area of convex optimization. He explained that convex optimization is a mathematical technique in which many complex problems of model fitting, resource allocation, engineering design, etc. can be transformed to a simple convex optimization problem to be solved and then transformed back into the original problem to get the solution. He went on to explain how this can be implemented in real-time embedded systems sych as a hard disk drive head seek problem, to large distributed system such as California's power grid.

Amol Ghoting - IBM "NIMBLE: A Toolkit for the Implementation of Parallel Data Mining and Machine Learning Algorithms on MapReduce". Use Hadoop to write a map function and a reduce function where you can map anything to a (key, value) pair. The problem with Hadoop is that it has a two-stage data flow which can be cumbersome for programming. Also, job scheduling and data mangement is handled by the user. Lastly, code-reuse and portability is diminished. This toolkit tries to make the key features of Hadoop available to developers but without a Hadoop specific implementation. NIMBLE actually decouples algorithm computation from data management, parallel communications and control. It does this through using a series of basic datasets and basic tasks that create a DAG. Tasks can spawn other tasks. With this structure in place, simultaneous data and tasks parallelism is achievable.

David Haussler – UC Santa Cruz “Cancer Genomics”. DNA sequencing cost has reduced dramatically. DNA sequencing was following Moore’s law but is now reducing cost 10 fold every two years. Can now cheaply sequence entire genomes. Created the Cancer Genome Atlas. 10,000 tumors will be sequenced in the next two years using this Atlas. Cancer genome sequencing will soon be a standard clinical practice. Because each persons DNA is different, and each tumor resulting from a persons DNA is different, a huge computational processing problem looms in the near distant future.

Ahmed Metwally - Google. "Estimating the number of people behind an IP Address". Most research assumes that there is 1 person using 1 IP address, but this is not the case. IP's also change size of users, for instance, a hotel with a conference will have many more users possibly using the same IP address than usual. So, how would one estimate the amount of these users in a non-intrusive way? One method is to look at trusted cookie counts. Another method is to look at diverse traffic. Google caps traffic volume per IP to stop people from gaming the system using the same IP address. Google knows how many users share an IP address because they are logged in with a username and password to Googles sites. However, some of Googles traffic is from users that don't have a Google account. This research is for those who want to filter users without asking them for any identification, thus preserving their privacy. This method is currently being used at Google for determining click fraud.

D. Scully - Google "Detecting Adverserial Advertisements in the Wild". An adversarial advertiser would be an advertiser that uses Google AdWords or AdSense to advertise misleading products like counterfeit goods or scams. Most ads are good, only a small amount are bad. Using in-house trained people to hand build rule based models. Allowing these people to hand-build the rules gave a great incentive and improved morale rather than just having them do repetitive tasks over and over again. Automated methods are being used as well, but this part of the presentation went right over my head.

Chunyu Luo - University of Tennessee "Enhanced Investment Decisions in P2P Lending: An Investor Composition Perspective". In this paper, they are trying to decide which loans are worthwhile to invest, in other words, what makes a good loan? Use a bipartite investment network with one side investors and the other investees and the edges between them loans. Each loan can be considered a composition of many investors. The idea is that by looking at the past performance of the other investors of a given loan, you can improve your prediction of the return rate for that loan. Performed experiment from dataset of prosper.com. The composition method far outperformed the average return of investment.

Susan Imberman - College of Staten Island "From Market Baskets to Mole Rats: Using Data Mining Techniques to Analyze RFID Data Describing Laboratory Animal Behavior". This paper presents the data mining techniques used in analyzing RFID data from a colony of Mole Rats. Much like we use RFID in cars for tolls like EZ Pass, they are using RFID on Mole Rats and when they pass specific points of the colony (a series of pipes and rooms) they collect that sample. They used k-means clustering which showed animal place preference. Used an adjacency matrix to get an idea of which Mole Rats liked to be near one another. This created 3 distinct sub graphs which corresponded well to the different colony structure of Mole Rats, queen workers, large workers and small workers. Next they correlated common transactions made in the grocery store with items in a basket to repeat behavior of Mole Rats.

After the conference ended on Wednesday, Hurricane Irene was on track for a direct hit to Hampton Roads. My flight was scheduled to arrive in Norfolk Friday night which was cutting it very close to the storm hitting on Saturday. So I decided to extend the trip till Monday and ride out the storm here in sunny San Diego. In total, I managed to miss a hurricane, a tornado, an earthquake, and a swamp fire. I think I made a good decision...

2011-08-28: Fall 2011 WS-DL Classes

2011-08-28T15:36:00.006-04:00

The Web Science and Digital Libraries Research Group is offering two classes for the fall 2011 semester. CS 895 Web-Based Information Retrieval will be offered on Tuesdays, 4:20-7:00 in room 2120 of the ECS building. This class will use the recent Croft, Metzler & Strohman book as the required text, and the Manning, Ragahavan, & Schutze book as the recommended text. By choosing the former book as the primary guide for the course, we are intentionally provided a strong engineering component to the class (i.e., a level of coding and development is expected) as opposed to just a theoretical exploration of information retrieval. CS 751/851 Introduction to Digital Libraries is not a prerequisite, but it would help to be familiar with the material covered in that class.

Dr. Weigle will be teaching CS 795/895 Information Visualization on Thursdays, 9:30-12:15 in room 2120 of the ECS building. This class is a follow-on to the CS 796/896 Visual Analytics Seminar from last semester, but that course is not a prerequisite for this semester's course. Unlike the seminar from last semester, this course will be a "regular" course that will count toward one of the four courses required in the "advanced-level course requirement".

Students wishing to become active in the WS-DL research group should take one or both of these courses.

--Michael

2011-07-26: Universal Access to All Knowledge

2011-08-02T14:57:00.004-04:00

On July 26, 2011, the Web Science and Digital Library group at Old Dominion University hosted Kris Carpenter Negulescu, Director of the Web Group at the Internet Archive who gave a talk entitled “Universal Access to All Knowledge”. The presentation started with an introduction about what the Internet Archive is, then, she gave us some information about what are the archived materials in Internet Archive for now: Text (+2.9M books), Moving Images (+542,500 items), Audio (+950,000 items), Television broadcast (+1M hours), Web Pages (+150 billion pages). Moreover, she gave an overview about some of the special collections such as K-12 students and NASA images.

After that, Kris explained the common collection strategies that are used by the Internet Archive to crawl the web. Frequently, they are doing a broad survey for the wide range domains such as .com, .net, .org, etc. They also considered the frequency of change for these websites and gave more support to the sites without archiving capabilities. Internet Archive has a special focus on the exhaustive websites when the web master decided to shutdown the website and would like to take a snapshot for the last time (for example, geocities). Another strategy is crawling by topics or specific collections based on a feedback from researchers or experts in this topic. In general, the key inputs for the URIs seeds are nominations from various partners (e.g., domain experts, trusted directories, Wikipedia).

Kris explained the methods of access to the web archive. The default method is "Host based" by browsing the website as it was using the WayBack Machine. In addition to other novel techniques such as full-text search and metadata/catalog look-ups; building an API's mirroring UI based access is also undergoing.

Finally, she gave an overview about some special projects such as:

Data mining and extraction
Link domain of 20thCF or of an entire domain (e.g., .uk) from 1996-2010
Dynamic, on-demand archiving of video and Wikipedia annotation
Semantic data extraction, metadata services

The colloquium was recorded and is available below.

-- Ahmed AlSum

2011-07-28: Web Video Discussing Preservation Disappears After 24 Hours

2011-07-28T12:09:00.016-04:00

One week ago (July 21, 2011) I was fortunate enough to be invited to speak about Web Archiving on Canada AM, sort of like the Today Show or Good Morning America in the US. I was asked to appear on the program in part because of the July 17, 2011 article in the Washington Post, which followed a July 6, 2011 blog post for the Chronicle of Higher Education, which was based on a June 23, 2011 blog post about our JCDL 2011 paper "How Much of the Web is Archived?". In other words, the process went like this: step 1 - get lucky & step 2 - let preferential attachment do its thing.

I was able to do the appearance in Washington DC, while attending the NDSA/NDIIPP 2011 Partner Meetup. The morning of July 21, I took a taxi to an ABC studio in DC, did the interview (about 4 minutes) and took a taxi back to the conference in time to make the morning session. I had not been on TV before and was both nervous and excited. The local and Canadian crew made the entire experience painless and the whole interview was over right as I started to get comfortable.

Given the short time, I tried to stress two topics: the first is that the ODU/LANL Memento project is not a new archive, but rather a way to leverage all existing web archives at once (this is a common misunderstanding we've experienced in the past). The other point I tried to make was that much of our cultural discourse occurs on the web and we should try to preserve as much of that as possible (including things like lolcats) because we (collectively) do a bad job at predicting what will be important in the future. Shortly after airing, the video segments was available on-line at:

http://www.ctv.ca/canadaam/?video=504307

As the URI suggests, this is the homepage for Canada AM (http://www.ctv.ca/canadaam/), but with an argument ("?video=504307") specifying which video segment (i.e., each individual story -- not the entire morning's show) to display. I shared the video URI with colleagues, friends, and family and was enjoying my 4 minutes of fame (I should still have 11 left in the bank). I had not made a local copy of the video because their web site obfuscated the actual URI of the streaming video, I had to finish the rest of the conference and drive back to Norfolk, and I thought I would have the time to figure it out after I returned.

So imagine my surprise on Friday at about lunch time when I reload the URI and do not see the video, but instead a newly redesigned Canada AM web page! The video of me making the point that we should save web resources lasted approximately 24 hours. I don't mean to seem ungrateful for the opportunity Canada AM afforded me, but as a professor I try to see everything as a teaching opportunity, so here it goes...

Sometime on Friday morning (July 22), the entire web site was redesigned and the old URIs no longer worked (cf. "Cool URIs Don't Change"). The video id was an argument and is now silently ignored, so even worse than a 404 you now get a "soft 404":

% curl -I http://www.ctv.ca/canadaam/\?video=504307
HTTP/1.1 200 OK
Server: Apache/2.2.14 (Ubuntu)
Content-Type: text/html
X-Varnish: 2550613724
Date: Thu, 28 Jul 2011 16:55:48 GMT
Connection: keep-alive

The soft 404 means people clicking on the original video link in Facebook, Twitter, email, etc. won't even see an error page -- they see the new site, but without the video or indication that the video is missing. The new site has a link titled "watch full shows", with the URI:

http://www.ctv.ca/canadaAMPlayer/index.html

Which is textually described as the "Canada AM Video Archive", but the archive begins on July 22, 2011 -- one day after my appearance! The new segments are available at URIs of the form:

http://www.ctv.ca/canadaAMPlayer/index.html?video=504933

The older videos are not available, not even as an argument to the new URI, which also returns a soft 404 (i.e., the video is not available despite the 200 response):

% curl -I http://www.ctv.ca/canadaAMPlayer/index.html\?video=504307
HTTP/1.1 200 OK
Server: Apache/2.2.14 (Ubuntu)
Content-Type: text/html
X-Varnish: 2550976182
Date: Thu, 28 Jul 2011 17:35:35 GMT
Connection: keep-alive

The video ids seem to be continuous (i.e., they did not appear to start over with "1"), so URL rewriting could easily make all the old video URIs continue to work, unless whatever CMS that hosted those videos has been retired with no migration path forward.

Here are some screen shots of the newly redesigned home page (left) and the video archive page (right) from July 22:

Of course, I did not think to make a screen shot of the original home page, or the page of my video because I thought it would live longer than 24 hours! I was able to find a recent (December 8, 2010) copy in the Internet Archive's Wayback Machine:

http://web.archive.org/web/20101208084455/http://www.ctv.ca/canadaam/

And I also pushed the two pages above to WebCite, which nicely contrasts two styles of giving URIs for archived pages (URI-M in Memento parlance):

http://www.webcitation.org/60NizRC0o
http://www.webcitation.org/60Nj60H8D

The IA's URIs violate the W3C "good practice" of URI opacity, but they sure are handy for humans. WebCite actually offers both styles of URIs, for example the latter of the two URIs above is equivalent to:

http://www.webcitation.org/query?url=http%3A%2F%2Fwww.ctv.ca%2FcanadaAMPlayer%2Findex.html&date=2011-07-22

But the resulting URI encoding, while technically correct, is not conducive to easy memorizing and exploration by humans. Different styles of using a URI as an argument to another URI will be explored in a future blog post.

Fortunately I was given a DVD of the session, from which I was able to rip a copy and upload it to YouTube, provided below with the dual interests of vanity and pedagogy. I'm not sure about its status with respect to copyright, so it might disappear in the future as well. It should be covered under fair use, but I would not count on it. However, that is also a topic for another blog post...

--Michael

2011-07-25: NDSA/NDIIPP Partner Meetup 2011 Trip Report

2011-07-21T19:05:00.190-04:00

The NDSA/NDIIPP (@ndiipp) Partner Meetup took place July 19-21 at the Hyatt Regency Washington on Capitol Hill in Washington, DC. Technical and non-technical joined together to form an aggregated consortium of archivists, librarians, digital media specialists and concerned parties. Three representatives from the ODU Web Sciences and Digital Libraries group attended to make archivists aware of tools they had developed to accomplish the common goal of web archiving.

WS-DL’s Comtributions to the NDSA/NDIPP Meetup

Mat Kelly presented the Mozilla Firefox add-on Archive Facebook to a breakout group of presentations specifically targeting web archiving. The redesigned and re-architected add-on allows a user to archive the content of his/her Facebook account with the result being truly WYSIWYG versus Facebook’s native offerings of a content dump.

NDIIPP/NDSA 2011 - Archive Facebook

Vivens Ndatinya showed the workings of a tool he is currently building with his presentation, “Creating Persistent Links to YouTube Music Videos”. The software serves as a medium between a user and YouTube where, if a music video has been deleted or removed, the proxy will search for a comparable or official substitute and seamlessly forward the user to the resource for which he/she was looking.

NDIIPP/NDSA 2011 - YouTube Link Restoration

Michael Nelson presented "How Much of the Web is Archived?", which was also presented at JCDL 2011. By examining links on DMOZ, delicious, bit.ly and search engines and cross-referencing the links with various archives, they were able to establish the criteria for likelihood of archival rate and conclude the amount of the web that is archived with, "It depends on the source of the URIs".

How Much of the Web is Archived? JCDL 2011

The Speakers

Martha Anderson (@MarthaBunton), the director of program management for the N ational Digital Information Infrastructure and Preservation Program (NDIIPP), exclaimed that “We are growing!” in her introductory presentation, citing the increasing numbers of members in the group and the larger breadth of the scope of the members’ specializations. “ She introduced the theme of the conference "Make It Work" and stated that the conference’s 3 days were broken up by the respective keywords of “Open”, in that all presenters were committed to openness, “Solve”, where all speakers presented studies on creative approaches toward solving their problems and “Connect”, which had a focus on community building and relationships.

Tim O’Reilly (@timoreilly), founder and CEO of O’Reilly Media Inc., kicked off the list of speakers providing insightful one-liners such as “Forgetting makes room for new things”, “Design more systems that have their own memory” and “We’re engaged in the wholesale destruction of our history”. He listed two of his pasts failures where his process of archiving could have been improved:

In 1993 he created one of the first websites but neglected to archive it. “Things that turn out to be historic”, he stated, “aren’t deemed to be historical at the time” – a theme that reverberated through many other presentations.

His second past failure of preservation was in 1998 when he attended the inaugural Open Source Summit (link?), where the term “Open Source” was officially born. Learning from his 1993 failure, he diligently built an archive and linked to all of the relevant content but neglected to deep link the archiving, which meant all of the information that was coupled with his coverage was no longer available at time of access.

O’Reilly rhetorically queried the audience, “What kind of tools do we need in the everyday practice of the digital world to encourage presentation?” He stated that we have to consider the widely divergent scenarios if we are to archive effectively. He reiterated that the tools we have should be adapted to assure that it is more likely that archived would survive when things went awry. “What matters?”, Tim stated, again referencing his two failures and answering his own question. He emphasized that our current perspective of what matters is temporally subjective and that we are likely neglecting to archive collections we now consider trivial.

To close up, Tim emphasized that there should be an exception in copyright for the sake of archiving so that our past will be preserved.

Yancey Strickler (@ystrickler) came on next to speak about his project Kickstarter, a funding platform for creative projects. Kickstarter works on an all-or-nothing approach of fund-raising where users can offer monetary support for projects they believe worthwhile with no commitment if the project fails to get funded. Yancey spoke of a tipping point in the funding process, where a large majority get funding to and sometimes beyond the threshold after attain 30% of their goal. Those that donate to the project are forbidden from being rewarded with equity but the fundee usually provides something priceless in return, like a photo for a donator from a project where a girl wished to sail around the world or the ability to be first to purchase a potentially popular iPod accessory that neglected to get traditional backing.

Kickstarter takes only a very small (5% of the raised funds) to remain sustainable but only receives these if the project gets funded. With this, Kickstarter and the projects both grow. “One day”, Yancey said, “we’ll hopefully be a cultural institution”.

Michael Edson (@mpedson) of the Smithsonian Institution came on next after a short break with his presentation “Let us go boldly into the present”. Michael emphasized that the time to archive is now and that “today is the future that all of the visionaries wrote about.” To do so, he gave five “design patterns” that we should exhibit to assure that the present is archived:

Extra-terrestrial Space Auditor is a concept best depicted by an extraterrestrial that examines an organization, blind to its current workings, and provokes the organization to do a self-analysis as to whether it is performing as it should in terms of business practices, HR, etc having been potentially skewed in operation by the baggage of the last epoch.
On Ramp and Loading Docks encourages the mindset that successful preservation is not about building infrastructure but rather creating movement.
Edge to Core suggests that the best work is done on the fringes of an institution where subject matter experts exist. “An organization”, Michael said, “should develop a process that brings in and bootstraps these experts so their ideas can scale.”
Self Awareness about organization change patterns states that there are predictable miscommunications and general crankiness in an organization between innovators and managements.
Focus on the mission was Michael’s observation that of the 80 to 90 organization that he had spoken to in the last few years, the ones that were not suffering their pursuit of worth know the outcomes they want in society.

After Michael, Aaron Presnall (blog) of the Jefferson Institute came on to speak about “Tools for informing public decision-making”. A continuing project of his was to assist those at the National Archives of Serbia in archiving their documents. Many of these documents are of great interest, as they document the recent struggles and secession of the country and have immediate application (such as implication for war crimes) if preserved. Using the tools available, some unconventional, Aaron assisted those interested in moving from the dissolving stacks of papers to a digital form. He then built a management tool and genericized the tool to allow it to be reused in instances beyond Serbia. He has since been queried by Bosnia, who wishes to do the same as Serbia and because of the generic setup Aaron has created, the information Bosnia has to offer will not be lost.

With Aaron being the last presenter for the day, Abby Rumsey moderated a panel discussion/Q&A with all of the Day One speakers., first hoping to address Martha's question, "How do we make it work?" She first asked Aaron how to connect demand of archiving with the supply of skill and if there is something that needs to be in-place to make these connections easier. He replied with the need to communicate the success of individual cases to much broader audience, convey the lessons learned and establish best practices for performing such an archiving session. He admitted that it's difficult "to make archiving sexy" but popularized projects such as History Pin get people thinking and both energize and popularize the task of archiving.

Tim O'Reilly expounded on Aaron's reply, referencing a collection of railway edition books from the 1880s that were bounded by people that found the works both valuable and beautiful. "When some individual finds something that would otherwise be disposable and finds it beautiful and a keepsake", Tim said, "that's a wonderful impulse for preservation". He continued, "When we allow things to be reused by individuals, it really appeals to value of fair use." He went on to speak about how intellectual property fights against preservation and what we can do to preserve things of value is to give them more freedom.

Abby then questioned Michael Edson about how his approach of Edge-to-Core has had an impact on The Smithsonian. Michael gave the example of how the Smithsonian handled the inception of the world-wide web with no business process in-place. "Because the institution took a decentralized approach to managing content and ideas", he said, "there was no existing infrastructure to make order out of the web. It's been a series of opportunistic efforts to pick the pieces of the low hanging fruit and bring them to the center of the organization to achieve scalability and a greater impact."

Yancey was then asked, "How do you get something where the connections are so profoundly personal into something that really scale to the level we think about with digital preservation?", citing Wikipedia's scaling issues. Yancey alluded to Wikipedia's moderation challenges in terms of curation with, "What happened if I'm a guy that knows a lot about a topic you're concerned with archiving and I decide to reach out and tell you everything I know and all of the ways to be wrong? What do I get to contribute? Do I have any voice whatsoever?" Aaron replied with, "Exactly, that's a tremendous challenge and whether 80% of time you're right, 20% of time you could be fundamentally, deeply, troublingly wrong."

The Q&A was followed up with a reception accompanied with 30-or-so poster displays. Of particular interest to the WS-DL members was the Ace Audit Manager and Integrity Management System, an integrity auditing system for archives, which would prove useful in both the Memento and Archive Facebook projects. This closed out day one.

Day two started with a presentation from Helen Hockx-Yu from the British Library. "In the UK", she said, "there are tow archives - The UK web archive and the UK Government Web Archive." She spoke further that there was pending legislation that would limit the viewing of archives to on-site within the library. "Web archiving in the UK", she said "is only 10 years old at the British library - much younger than Internet Archive." One notable part of the collection, to which she said the British Library found accidentally, is the oldest archived website - that of the British Library's website from 1995, which was found stored away on a library's server.

Tricia Cruse of the California Digital Library spoke next about "Curation approaches in a public university system", stating "We're seeing an ever-increasing amount and degree of diversity of content. While our budgets were going down, we have had to do more with less."She also spoke of EZID, a system for users to create unique identifiers for their archived content; UC3 Merritt, a place where collaboration for researchers can happen and data can be stored and shared and Digital Curation for Excel (DCXL), an open source Microsoft Excel add-in that allows working in Excel to be easier for versioning, archiving and applying unique identifiers.

Jack Brighton of WILL, a radio/television station in Illinois, spoke of "Archiving at Web Speed". Jack spoke of his efforts in preserving the stations broadcasts using PBCore and emphasized the need for the adaptation of the archiving process to make it as painless as possible for those that did not necessarily see the value of the content at the current time.

Ben Vershbow (@subsublibrary) from the New York Public Library finished up the first session of the day with his presentation, "Bringing in the Crowd". He cited a project his group created, "What's on the Menu?", which was a crowd-sourced effort to transcribe old menus. He believes that there is an untapped reservoir of time and through crowd coordination and building datasets, people will be willing to devote their time for free.

Subsequent to Ben's presentation, the crowd broke up into three groups for workshops. The three topics of the workshops were "And the winner is..: How does a community recognize achievement?", "Tales from the crypt: What are the emerging practices of large scale storage" and "Special Interest Session: Web Archiving: Pecha Kucha and discussion of emerging topics in Web archiving". Because Vivens and Mat presented at the latter of the three, the WS-DL members attended and participated in the third session.

Presentations resumed after the breakout session with the theme of Open Source Tools and Community. The first presentation was by MacKenzie Smith (website) of MIT with "Exhibit3@MIT: Lessons learned from 10 years of the Simile Projet for building library open source software". MacKenzie stated that "Everybody's a curator" and "If we're creating these tools for the public, how can we assure that these tools will flow into the organizations, as many die? When you're doing a project that's open source", he continued, "you need to design for that community from the beginning." MacKenzie went on to say that metrics should be used to assure that you can tell the chance of success of the open source project, you're more likely to have a sustainable project if you have an audience "outside of this room" (i.e. outside of the archiving community) and that maintenance of the code has to be done by those that are committed, not just casual developers.

Sharon Leon (website) of George Mason University then presented "Omeka: from digital exhibits to web publishing platform". Omeka is a plug-in based Content Management System (CMS) modeled off of Wordpress that emphasizes extensibility. Sharon repeatedly emphasized the openness of the platform and that her group "specifically fights against Flash for re-use", as wrapping content in a Flash-based application limits access to the content within. She also mentioned that in developing a grant-funded open source project, one should not spend all of the funds on the development of the project but rather should put funds toward workshops, outreach and marketing of the product.

Michele Kimpton (website) spoke of ways to go beyond grant funding once it's exhausted with "Building and sustaining open source communities through the life cycle: Dspace, Fedora and DuraCloud case studies". Her group has create a write-up on the Meetup.

Following Michele was another breakout session of concurrent workshops with each having the topics of "Tools at risk", "I can haz standardz" and "Developing cutting-edge internship programs in digital preservation: What are the essential elements?". The WS-DL group attended "I can haz standarz", which disappointingly was more about the inability of the non-technical in building a tool for data management rather than about the standards themselves. As the group were all of technical mind, this was clearly the wrong workshop of the three to attend.

After another short break was a third set of concurrent workshops: "Digital preservation in a box: What are the key resources for digital preservation and education and outreach?", "Slaying the dragons: What is at risk and how do we rescue it?" and "The Challenge challenge: What are ways we can spark digital preservation innovation". The WS-DL group attended the third of the three. There, the attendees were broken into groups with each group being tasked to discuss a single topic in-depth with varying concerns in each group. Unlike the previous workshop, one topic was specifically technical - that of investigating how one assures archive integrity from a host and how to go about performing an audit on the collections stored. The WS-DL group along with Michelle Gallinger (@mgallinger), Professor Micah Beck (website), Mike Smorul (@msmorul) and a couple others devised the Storage Ping concept, which would require those that host collections to enable a client induced check on the server's collection integrity.

Day 3 started out with an introduction by Martha Anderson and the followed with the first presenter, David Rosenthal (website) of Stanford University on "Cloud Storage for LOCKSS Boxes". LOCKSS (Lots Of Copies Keeps Stuff Safe) boxes are dedicated computers with local storage that communicate with each other and repair any damages of data. David discussed challenges of speed he encountered when developing his system and conveyed a method of assuring integrity of data and assurance of data's existence on a remote server by prepending a nonce. He has recently been working with students at Carnegie Melon University to develop a crawling process that he described as being "a pretty robust approach to form filling." He also expressed some difficulty he has had in the past with archiving AJAX-based contents but emphasized that his archiving process was different than others', as he does not use Heritrix, the crawler used by The Internet Archive.

After David, Cal Lee (website) of UNC Chapel Hill analyzed the four NDIIPP State projects:

Persistent Digital Archives and Library System (PeDALS) by Arizona
A Model Technological and Social Architecture for the Preservation of State Government Digital Information by Minnesota Historical Society
Geomap (GIS Data headed by North Carolina for Center Geographic Information and Analysis)
Multi-state Preservation Consortium by Washington State Archives

The questions he asked about each projects included:

What are the main factors that drove the project in the first place?
What brought these about?
Who was involved and why?
What were the activities they engaged in before this?

Following Cal was Robert Horton from the Minnesota Historical Society who presented his slide-less report of his NDIIPP-sponsored project. Cal spoke of a soon-to-be enacted uniform law for the preservation and authentication and access to electronic legislative records. The legislation will define the required usage of digital Signatures to sign all legislative content online.

Peter Krogh (@peterkrogh) of the American Society of Media Photographers spoke next with, "Extending the reach of www.dpBestflow.org". Peter had been investigating means of collaboration and methods to get people to archive by conveying the task of archiving in a way that will appeal to the would-be archivist.

After a break, summaries of the 2010 DPA finalists sponsored by the Library of Congress were presented. WS-DL's own Michael L. Nelson (website) reported on the Memento project (joint work with Herbert Van de Sompel (who gave the original presentation in London in December 2010) and Robert Sanderson of LANL) which was referenced multiple times by other presenters throughout the meetup. Dr. Nelson stated that there is currently a disconnect in viewing web archives, as there is no seamless way to go from the past and the present. Memento overcomes "being stuck in the perpetual now" by leveraging content that currently exists in the web archives and provides a bi-directional means to view different versions of a web site on-the-fly. Michael stated that Memento does not create web archives but rather puts the notion of time onto the web.

Following Michael's presentation was Fran Berman (website) of Rensselaer Polytechnic Institute with "Economics and Digital Preservation", a final report of the Blue Ribbon Task Force (BRTF), whose mission is to promote sustainable digital preservation and access. Fran spoke of BRTF's investigation of the technical, economical and social problem. "Infrastructure is not free", she said, "and the preservation and access to our data is not free. Because it is not free and because there are so many interesting solutions, you see it as a multivariate problem. " She stated that the Task Force wanted to do a deep dive into the economics of the problem of cost for digital preservation.

"Our charge was to do roughly three things", Fran enumerated:

Assemble a representative group of experts with broad perspective and influence.
Look at the problem space: how can we structure it and understand us in a way that helps us take action.
Come up with actionable recommendations.

The BRTF created a report with it recommendations.

The final presentation of the conference was by Kari Kraus (@karikraus) of the University of Maryland with "Preserving Virtual Worlds" (Jerry McDonough gave the original presentation in London). Kari spoke of her attempts at preserving virtual worlds with repeatedly referencing example from Second Life. The project was a multi-institution, multi-disciplinary project by University of Illinois at Urbana-Champaign, Stanford University, Rochester Institute of Technology and The University of Maryland that investigating preserving virtual worlds for their aesthetic merit as well as their economic significance. "We believe there is tremendous cultural importance to these artifacts.", she said, "We believe video games represent the limit case of what we can do with digital preservation. If we can figure out how to save a classic first-person shooter game like Doom, we'll have a better chance of preserving computational simulations of genetic evolution or climate change or the galactic behavior of star systems."

She said that their mission was very practical: they needed to ingest game bits into institutional repositories and provide packaging standards for doing that. Other examples of virtual worlds she mentioned were investigated were Spacewar, Adventure (interactive fiction) and Mystery House (interactive fiction) among others.

In Closing

Neither of the WS-DL student presenters had presented at a meetup/conference of this caliber before, which made the experience more than worthwhile. Much was learned about the various efforts of the archiving community and WS-DL's projects gained exposure. Further, we were made aware of others' efforts and found some resources that we hope to integrate into our research in the near future.

— Mat Kelly

2011-07-21: Towards a Machine-Actionable Scholarly Communication System

2011-07-21T17:48:00.004-04:00

I've told all the members of my research group they should watch this, so I thought I might as well make the same recommendation to the rest of the world... Herbert Van de Sompel presented "Towards a Machine-Actionable Scholarly Communication System" at LIBER 2011 in Barcelona, Spain on June 30, 2011.

You really have to simultaneously watch the video and review the slides to get the full impact of the presentation. The first part is a succinct review of various projects, but starting at slide 16 ("nanopublications") things really get interesting. Well worth the 40 minute investment.

Towards a Machine-Actionable Scholarly Communication System

View more presentations from Herbert Van de Sompel

--Michael

2011-07-05: JCDL 2011 Trip Report

2011-07-05T17:23:00.007-04:00

JCDL 2011 (#jcdl2011) was held June 13–16 in Ottawa, Ontario, Canada. The weather was beautiful and the conference sessions wonderful. The ODU Web Sciences and Digital Libraries team was fortunate enough to to have six of its members attend, present three short papers, and demonstrate the Synchronicity Firefox extension.

Our Contributions to JCDL 2011

Ahmed Alsum presented How Much of the Web is Archived? This paper approximates the amount of the Web that is archived using four URI sources. From this data, we observe significant variation in archival rate in URIs from different sources. So, how much of the web is archived? It depends on which web you mean. (pdf, slides).

How Much of the Web is Archived? JCDL 2011

Martin Klein presented Rediscovering Missing Web Pages Using Link Neighborhood Lexical Signatures, which details a method for discovering missing web pages (the dreaded 404). Martin also demonstrated Synchronicity, a Firefox extension that uses lexical signatures (and other methods) for automatically rediscovering missing web pages in real time (pdf, slides).

Rediscovering Missing Web Pages
Using Link Neighborhood Lexical Signatures

Abdulla Alasaadi presented Persistent Annotations Deserve New URIs, which describes a method for creating new persistent URIs for annotations and creating persistent, independent
archived versions of all resources involved in the annotation (pdf, slides).

Persistent Annotations Deserve New URIs

The Web Archive Globalization Workshop

After the conference, the members of the Web Science and Digital Libraries team attended the Web Archive Globalization Workshop. This workshop focused current initiatives and future possibilities. Eric Hetzner provided insight into the California Digital Library's web archiving activities. The Library of Congress's Nicholas Taylor told us about the Library's digital preservation initiatives. Brad Tofel of the Internet Archive gave us the low down the future of web archive formats (ARC, WARC, WAT) and the Wayback Machine. Robert Sanderson with the LANL Research Library provided an overview of current Memento infrastructure. There was much discussion about current archiving challenges including management of huge volumes of information, copyright considerations, and the challenges of making the archives accessable to researchers and the public. (slides)

The workshop was organized by:

Frank McCown, Harding University
Hector Garcia-Molina, Stanford University
Michael L. Nelson, Old Dominion University
Andreas Paepcke, Stanford University

Keynotes

The opening keynote, "Leaving the Cathedral and Entering the Bazaar: Library and Archives Canada Engages Canada’s Digital Society," was given by Daniel J. Caron, the current Librarian and Archivist of Canada. Mr. Caron discussed the issues and opportunities faced by national libraries as they transition from an analog to digital environment. He compared the situation to the cultural and process differences put forward by Eric. S. Raymond in The Cathedral and the Bazaar. It was an excellent talk and I really got the impression that Mr. Caron understood the transition required and the chaos inherent with a technological change of this magnitude.

Wednesday's open talk was given by IBM's Joan Morris DiMicco. "Data Narratives: Telling Stories with Data" (slides) focused on current reasearch at IBM into data visualization as storytelling medium. She defined at story as concrete, temporal, purposeful, and emotional. Brief presentations of visualizing legislative text with Many Bills, SaNDVis social relationship search, and the impact of visualizations on group behavior Second Messenger.

Christopher R. Barnes, the director of NEPTUNE Canada, described the NEPTUNE Canada cabled ocean observatory using many wonderful illustrations and photographs. He then went on to describe the digital library problem he and his team face: the 4+ (and growing) gigabytes of data collected daily by the project. This data is used by over 8,000 user. Storage, cataloging, and access are ever growing challenges the digital library and preservation communities can help with.

Session Highlights

Two or three session ran simultaneously durng the conference and I was not able to attend all presentations.

Session 1 presented automated methods to assist human understanding of texts. There were full papers on improving understanding of historical word sense variation (Measuring Historical Word Sense Variation) and improving information extraction from PDF books (Structure Extractions from PDF-based Book Documents); and a short paper on using syntactic dependency parse tree to learn expected patters between lexical arguments (Word Order Matters: Measuring Topic Coherence with Lexical Structure).

Session 5 explored rediscovery of missing web content, a topic near and dear to us. This session included two of our short papers and full papers on using patterns to efficiently implement web archiving (Archiving the Web Using Page Changes Patterns: A Case Study) and identifying academic home pages (On Identifying Academic Homepages for Digital Libraries).

The impact of copyright on access and use was covered in session 7. The attitudes of the social-media savvy were explored (The Ownership and Reuse of Visual Media) and the implications of data quality problems in national bibliographies were explored in (Using National Bibliographies for Rights Clearance).

Session 8 looked at methods to annotate the Web. Rob Sanderson presented SharedCanvas (preprint, slides). There was also a paper on combining superimposed information with digital libraries (Use of Subimages in Fish Species Identification: A Qualitative Study). Our Persistent Annotations Deserve New URIs short paper was also presented in this session.

Session 11 and 12 looked at the needs and abilities of user and improving the digital library experience. Understanding Digital Library Adoption: A Use Diffusion Approach and In the Bookshop: Examining Popular Search Strategies studied how users interact with digital libraries. Improving recommendations was looked at from several perspectives (A Social Network-Aware Top-N Recommender System using GPU, Serendipitous Recommendation for Scholarly Papers Considering Relations Among Researchers, and Product Review Summarization from a Deeper Perspective).

Other Perspectives on JCDL 2011

Heather's Darkroom has good descriptions of the first two keynotes and the The Ownership and Reuse of Visual Media paper.
Kayleigh Ayn Bohémier has a 4-part post (part 1) on the conference and the conference experience in Ottawa.
Some of the presentation slides are on the slideshare JCDL 2011 event page.
The Digital Repositories Workshop slides are also available.

— Scott G. Ainsworth

2011-06-23: How Much of the Web is Archived?

2011-06-23T10:30:00.001-04:00

There are many questions to ask about web archiving and digital preservation - why is archiving important? what should be archived? what is currently being archived? how often should pages be archived?

The short paper "How Much of the Web is Archived?" (Scott G. Ainsworth, Ahmed AlSum, Hany SalahEldeen, Michele C. Weigle, and Michael L. Nelson), published at JCDL 2011, is our first step at determining to what extent the web is being archived and by which archives.

To address this question, we sampled URIs from four sources to estimate the percentage of archived URIs and the number and frequency of archived versions. We chose 1000 URIs from each of the following sources:

Open Directory Project (DMOZ) - sampled from all URIs (July 2000 - Oct 2010)
Delicious - random URIs from the Recent Bookmarks list
Bitly - random hash values generated and dereferenced
search engine caches (Google, Bing, Yahoo!) - random sample of URIs from queries of 5-grams (using Google's N-gram data)

For each of the sample URIs (4000 in all), we used Memento to discover archived versions, or mementos, of the URI.

We categorize the archives as Internet Archive (using the classic Wayback Machine), search engine caches (Google, Bing, and Yahoo!), and other (e.g., Diigo, Archive-It, UK National Archives, WebCite).

Our first set of graphs (click on each graph for a larger version) shows the mementos discovered for each URI, ordered by the first observation date. These are separated by sample set, so the y-axis on each graph runs from 0-1000. Brown dots indicate mementos discovered at the Internet Archive, blue dots indicate those found in search engine caches, and red dots indicate mementos found at other archives.


DMOZ	Delicious

Bitly	Search engines

There are a few interesting observations:

DMOZ URIs are well-represented, especially in the Internet Archive. There are two likely reasons for this: DMOZ is the primary source for seed URIs for the Internet Archive and the DMOZ sample contains more old URIs than the other sources.

Bitly URIs are very poorly represented. The majority of Bitly URIs are not found in any archive. This is currently under further investigation.

There is a large gap in mementos found in the Internet Archive, starting in 2008. We suspect this is because of the use of the classic version of the Wayback Machine.

The second set of graphs shows the relationship between the density of mementos for a URI and the URI's age. The x-axis is the estimated creation date of the URI, and the y-axis is the number of mementos found for this URI. Large dots indicate that several URIs had similar creation dates and number of mementos. We show density guidelines for 0.5, 1, and 2 mementos created per month.


DMOZ	Delicious

Bitly	Search engines

From these graphs, we make the following observations:

Many of the DMOZ URIs are archived at least once every 2 months.
Older Delicious URIs have many mementos.
A few Bitly URIs have many mementos, but most URIs have 0-10 mementos.

So, how much of the web is archived? Depends on which "web" you mean.

Ahmed presented this work at JCDL 2011. His presentation slides are below:

How Much of the Web is Archived? JCDL 2011

This work supported in part by the Library of Congress and NSF IIS 1009392.

-Michele

2011-06-29: OAC Demo of SVG and Constrained Targets

2011-06-23T03:54:00.007-04:00

Online annotating service is a tool that helps to annotate different resources with different authors and give this annotation a separate URI that can be shared using a Facebook post, blog post, tweet, etc.

Web annotations can be described as a relation between different resources with different media types like text, image, audio, or video. The web annotation service will be able to provide:

A unique URI for every annotation.
Persistent annotations.
Annotate specific part of media.
Keep track of the resources.
Present annotation in browser.
Meet the OAC model requirements (alpha3 release) .

Open Annotation Model:

This service will generate annotations that meet the OAC model specification. In an annotation that contains different resources, the OAC will introduce a new resource that describes the relationships between the resources that make the annotation.

Example:

A user who is interested in wildlife is browsing a page about elephants in Africa, and he was interested in the map the shows where the elephants live exactly.

http://bioweb.uwlax.edu/bio203/s2007/shah_rach/map.JPG

The user relates this image to another image that shows how people kill the elephants in order to sell their expensive tusks in another website. Now, the tusks picture annotates the map, and shows the reason behind the decreasing number in elephants in central Africa.

How does it work?

The process starts at the client side, where the user creates his annotation using the SVG-Edit. SVG-Edit is an open source plugin that has been designed to create SVG graphs in local desks. SVG-Edit has been modified to meet our requirement where we can edit the graph online and send the results to our annotation online service.

SVG will enable the annotator to annotate specific parts of the image using any shape. This will solve the problem of the W3C media fragmentation specification that supports rectangular shapes only.

After creating the annotation using the SVG-Edit, the annotation data will be sent to our online service that does the following:

Pushes all related resources to the WebCite archive, in this case, each resource will have at least two copies with different URIs, one of them is the archived copy.
The service will generate an RDF file mentioning the relationships between the resources.
With all the different URIs generated of the resources and their archived copies, a resource map will be created for every annotation created. The associated resource map will aggregate all the resources that are related to this annotation. The resource map will be referred to by the link-header when the page gets dereferenced.
Since the generated URI of the annotation will be long, another short URI will be generated using the bit.ly URI shortening service. The new short URI will make it easy for the annotation to be shared on tweets or Facebook posts.
At the end of the annotating process, the user will get a simple and short URI that can be easily posted in user’s mail, twitter or Facebook.
When users dereference the URI they get the annotation back.

Pushing the annotation data to the web-service.

Retrieving the annotation using its URI.

You can check the video (http://bit.ly/Annotate) to watch a demonstration on how this service works.

For more details, you can refer to the paper "Persistent Annotations Deserve New URIs" which has been published in JCDL 2011, and the slides are below:

Persistent Annotations Deserve New URIs

This service will help you in:

Minting new URIs for the annotations.
Annotating the media fragments was made possible using the SVG and its media tags.
Using the web archives solves the issue of keeping the annotation persistent over time.
Keep track of all the related resources using ORE Resource Maps.

Links:

This work supported in part by the NSF IIS 1009392.

-- Abdulla

2011-06-18: Report on the 2011 Digging into Data Challenge Conference

2011-06-18T13:16:00.005-04:00

On June 9-10 I attended the 2011 Digging into Data Challenge Conference in Washington DC, which was a status report of the eight projects selected during the initial 2009 Digging into Data Challenge.

Unfortunately, due to traffic challenges to and from the conference, I was able to catch only one half of the sessions. Jennifer Howard of the Chronicle of Higher Education gives a good summary of the sessions (day 1 and day 2).

The highlights of the sessions I attended included the "Data Mining with Criminal Intent" project (whose poster is shown above), which includes the use of the Voyeur Tools for text collection summarization on the "Old Bailey", a corpus of criminal court proceedings in London 1674-1913. Also interesting was the "Mapping the Republic of Letters" project, which is basically social network analysis based on the letter exchanges of prominent scientists and intellectuals during the 18th century. Also of note was Tony Hey's keynote. Although his video/slides are not yet available from the conference website, you can get an idea of his presentation by looking at his OSCON 2009 presentation (slides, video), although Digging Into Data presentation was more recent and expanded. Interesting projects that I learned of included: Digital Narratives, NodeXL, and Zentity.

The conference had a new-to-me format about which I'm not entirely sure how I feel. The project PIs would present the status and highlights of their projects for 45 minutes, and then a respondent not involved with the project would present a rebuttal / evaluation / response / contextualization. The respondents that I saw were gracious and complimentary, but I heard during the breaks that was not necessarily the case for at least one of the respondents that I missed.

There is a 2011 Digging Into Data Challenge, although with less than a week between the conference and the due date of June 16 it is not clear to me how much the experiences of the previous participants could be incorporated into the 2011 submissions.

--Michael

201-06-17: The "Book of the Dead" Corpus

2011-06-17T15:43:00.006-04:00

We are delighted to introduce the "Book of the Dead", a corpus of missing web pages. The corpus contains 233 URIs all of which are dead meaning they result in a 404 "Page not Found" response. The pages were collected during a crawl conducted by the Library of Congress for web pages related to the topics of federal elections and terror between 2004 and 2006.

We created the corpus to test the performance of our methods to rediscover missing web pages introduced in the paper "Evaluating Methods to Rediscover Missing Web Pages from the Web Infrastructure" published at JCDL 2010. In addition we now thankfully have Synchronicity, a tool that can help overcome the 404 detriment to everyone's browsing experience in real time.

To the best of our knowledge the Book of the Dead is the first corpus of this kind. It is publicly available and we are hopeful that fellow researchers can benefit from it by conducting related work. The corpus can be downloaded at: http://bit.ly/Book-of-the-Dead

And one more thing... not only does the corpus include the missing URIs, it also contains a best guess of what each of the URIs used to be about. We used Amazon's Mechanical Turk and asked workers to guess what the content of the missing pages used to. We only provided the URIs and the general topics elections and terror. The workers were supposed to just analyze the URI and draw their conclusions. Sometime this can be an easy task, for example the URI:

http://www.de.lp.org/election2004/morris.html

is clearly about an election event in 2004. Maybe one could know that "lp" stands for Libertarian Party and "de" for Delaware. Now this URI makes real sense and most likely "Morris" was a candidate running for office during the elections.

All together the Book of the Dead now offers missing URIs and their estimated "aboutness" which makes it a valuable dataset for retrieval and archival research.
--
martin

2011-06-10: Launching Synchronicity - A Firefox Add-on for Rediscovering Missing Web Pages in Real Time

2011-06-10T15:14:00.013-04:00

Today we introduce Synchronicity, a Firefox extension that supports the user in rediscovering missing web pages. It triggers on the occurrence of 404 "Page not Found" errors, provides archived copies of the missing page as well as five methods to query search engines for the new location of the page (in case it has moved) or to obtain a good enough replacement page (in case the page is really gone).

Synchronicity works in real time and helps to overcome the detriment of link rot in the web.

Installation:

Download the add-on from https://addons.mozilla.org/en-US/firefox/addon/synchronicity and follow the installation instructions. After restarting Firefox you will notice Synchronicity's shrimp icon in the right corner of the status bar.

Usage:

Whenever a 404 "Page not Found" error occurs the little icon will change colors and turn to notify the user that it has caught the error. Just click once on the red icon and the Synchronicity panel will load up.

Synchronicity utilizes the Memento framework to obtain archived copies of a page. On startup you are in the Archived Version tab where two visualizations of all available archived copies are offered.

The TimeGraph is a static image giving an overview of the number of copies available per year. Three drop down boxes enable you to pick a particular copy by date and have it display in the main browser window.

The TimeLine offers a "zoomable" way to explore the copies in dependence of the time they were archived. Each copy is represented by the icon of its hosting archive. You can click on the icon to receive metadata about the copy and see a link that will display the copy. You can also filter the copies by their archive.

Based on these copies Synchronicity provides two content based methods:

the title of the page
the keywords (lexical signature) of the page

that both can be used as queries against Google, Yahoo! and Bing. The idea is that these queries represent the "aboutness" of the missing page and hence make a good query to discover the page at its new location (URI) or a discover a good enough replacement page that satisfies the user's information need.

Synchronicity can further obtain tags from Delicious created by users to annotate the page. Even thought tags are sparse, if available they can make a well performing search engine query. Additionally Synchronicity will extract the most salient keywords from pages that link to the missing page (link neighborhood lexical signature) that again can be used as a query.

Lastly Synchronicity offers a convenient way to modify the URL that caused the 404 error and try. The idea is that maybe shortening the path will get where you want to go.

These last three methods can be applied if no archived copy of the missing page can be found.

Synchronicity provides a straight forward interface but also enables more experienced users to modify all parameters underlying the extraction of titles, keywords, tags and extended keywords. The Expert Interface lets you for example show the titles of the last n copies where you specify the value of n. It also enables you to pick a particular copy to extract the keywords from and change many more parameters.

Notes:

Synchronicity is a beta release so do not let it perform open-heart surgery on your mother-in-law!

It was developed within the the WS-DL research group in the Computer Science Department at Old Dominion University by Moustafa Aly and Martin Klein under supervision of Dr. Michael L. Nelson.

Please send your feedback, comments and suggestions for improvement to

synchronicity-info@googlegroups.com

martin

2011-05-20: Report on the 2011 IIPC General Assembly

2011-05-20T10:46:00.005-04:00

I spent the week of May 9--13 at the KB in The Hague, the Netherlands for the 2011 IIPC General Assembly. Joining me there was Rob Sanderson of LANL. Rob had attended the 2010 GA in Singapore, but this was my first IIPC and I learned a great deal.

The first day was open to the public in a special session entitled "Out of the Box: Building and Using Web Archive Collections", of which I missed most because I was taking a nap after arriving the morning of May 9. Fortunately, Inge Angevaare prepared a comprehensive summary of the first day. I believe presentations and a video of highlights from the first day will be available from the IIPC site shortly.

The next three days were spent in the IIPC plenary and working groups. Rob gave a high-level Memento status report on Tuesday, and Rob and I gave a more detailed tutorial later in the day:

Memento: Updated technical details (May 2011)

View more presentations from Herbert Van de Sompel

Wednesday and Thursday were largely spent meeting with the Access Working Group discussing a pilot project that would, using Memento, allow harvesting and re-exposing of web page metadata from various IIPC member national libraries to the public. The goal is to have a large-scale, working demo of using Memento to aggregate the metadata about IIPC members' archives for the 2012 GA (to be held in Washington DC).

One of the things I learned at the IIPC is that many national libraries are archiving their national top-level domains (e.g., BNF archiving *.fr web sites), but rather restrictive intellectual property laws prevent the libraries from opening their archives off-site (in other words, you have to travel to the BNF to view their *.fr archives). I suppose I had been spoiled by the relatively unencumbered approach afforded to the Internet Archive. Of course, we'd like to see these archives completely opened in the future, but the ability to advertise their contents in a machine-readable manner is a good first step.

Friday was an excellent hands-on tutorial led by Brad, Aaron, and Vinay from the Internet Archive (bios) about processing WARC, CDX, and WAT files using Hadoop and Pig. Vinay provided a page that gathers all the appropriate links into one place (the data files were distributed via thumb drive).

Rob left on Saturday morning, passing Herbert on the train as he arrived Saturday for his 2 month visit to DANS. Herbert and I spent the day catching up while touring The Hague and Delft.

Inge also blogged about the closing of the conference ("Memento Sparks Optimism at Closing of IIPC 2011"), and the Twitter hashtag was "#iipc". I'll update this entry when additional information from IIPC is posted.

--Michael

2011-04-13: Implementing Time Travel for the Web

2011-04-12T18:04:00.020-04:00

Recent trends in digital libraries are towards integration with the architecture of the World Wide Web. The award-winning Memento Project proposes extending HTTP to provide protocol-level access to mementos (archived previous states) of web resources. Using content negotiation and other protocol operations, rather than archive-specific methods, Memento provides the digital library and preservation community with a standardized method to navigate between the original resource and its mementos.

Memento Client State Chart

The ODU Web Sciences and Digital Libraries Research Group has partnered with the LANL Research Library to create Memento and develop prototype Memento-compliant client and server implementations. A variety of Memento clients have been created, tested, and co-evolved along with the Memento protocol. There is now a FireFox extension, Internet Explorer browser helper object, and WebKit-based Android browser. The design and technical solutions identified during the development of these clients will be of interest to those considering implementation of a Memento-based platform, especially on the client side, and the interactions are also important for building conformant server-side systems.

MementoFox Screenshot

The full article can be found at:

Robert Sanderson, Harihar Shankar, Scott Ainsworth, Frank McCown, and Sam Adams. Implementing Time Travel for the Web. code{4}lib Journal, Issue 13, 2011-04-11. http://journal.code4lib.org/articles/4979.

-- Scott G. Ainsworth

2011-04-08: Radiation Map of Japan

2011-04-08T16:06:00.002-04:00

The devastation wrought by the 11 March earthquake in Japan,
and the depths of the human misery left in the wake of the massive Tsunami have left many people awestruck. The size of the quake itself was enormous and many people have had a hard time comprehending just how big this earthquake was. Some sites like Japan Quake Map help us to comprehend the magnitude of this event. As a result of the earthquake and tsunami the nuclear reactor at Dai-ichi was severely damaged and has been leaking radiation. The radiation readings have been made available by WIDE and Japan's Nuclear Safety Division.

The idea was to use R to create an informative map of Japan showing the radiation levels of the different prefectures. Python was used to import the data from both of the web sites and insert it into a MySQL database. The format of both of the pages was understandably quite dynamic and resulted in the python script needing to be tweaked quite often. Sometimes it was easier to just copy and paste the data in a spreadsheet and then export as a csv to import into the database.

For the map, the shapefiles included in the R distribution were not working out so shapefiles for Japan from Harvard Asia Studies were used. These shapefiles combined with the plotPolys() command produced a higher quality map than the standard shapefiles.

The readings for most prefectures were rather reliable however in Fukushima and in Miyagi the readings were sporadic. Miyagi was hardest hit by the tsunami and most of the area was destroyed. It appears that most of the readings were from mobile units and there are gaps in the coverage. If there were no readings available for a given day they were estimated using the surrounding readings both spatially and temporally. In Fukushima which is the location of the reactor, there were many monitoring sites set up but they seemed to come and go over the course of time. For the purposes of this map, the sites located between 20km to 30km from the reactor were averaged together to give a reading for the Fukushima prefecture.

Using R, the average daily radiation for each of the 47 prefectures was calculated. The maximum and minimum values were used to create a color gradient for the map. Most of the readings were low with only one or two high readings. This did not lend itself well to a smooth color gradient so the log of the values was used to create the color gradient.

R code for Map

The radiation level values were each assigned a hex color value and then merged into a vector that matched the prefecture names in the shapefile. Nested for loops are usually a bad idea, especially in R, suggestions for a more elegant solution are welcome. plotPolys() takes care of making the map and the only thing left was to reverse the logarithm function to get the real values back and add a legend.

All of the maps from 16 March to 4 April were combined into an avi using:

mencoder mf://*.png -mf fps=1:type=png -ovc lavc -lavcopts vcodec=mpeg4 -oac copy -o output.avi

Shapefiles from China Historical GIS Project, "Tokugawa Japan GIS, Demo Version." Feb 2004

-- Greg Szalkowski

2011-04-07: MITRE Records Expo Trip Report

2011-04-08T06:02:00.000-04:00

I have just returned from MITRE's Records Expo on MITRE's Campus in McLean, VA. The Records Expo is designed to raise awareness of the archival responsibilities of employees within MITRE, and also inform our sponsors about the archives and records management work we're doing. I was invited to present some of the research being done in digital preservation at ODU and MITRE. (George Despres and I have recently received funding to perform digital preservation research on the digital objects living within the corporate intranet. Our research was explained at the Expo.)

Records expo

View more presentations from Justin Brunelle

We set up booths in the MITRE 2 building, equipped with big-screen TVs with slide shows about other archival and records management systems being pioneered at MITRE (the slides are For Office Use Only, and cannot be shared in this blog). Several MITRE employees attended and listened to presentations given by the archives team and the records management teams at MITRE, as well as George and I. A former Lockheed Martin employee that had worked on the NARA records management system was also in attendance.

I spoke to all of the attendees about Memento and how it will be used to allow users to browse archives within MITRE's intranet as part of George and my research. Most of the attendees had some experience with the WayBack Machine, and had a cursory knowledge of web archiving, but most weren't familiar with Memento. All were extremely interested in hearing about the research further, and some have already been in touch with me requesting additional information.

It was helpful and informative to meet with other professionals working in "The Real World" and attempting to solve the same problems being researched in academia. There were also some additional approaches to archival and records management problems, such as using the Cloud as a repository, and archiving corporate social networking content.

--Justin F. Brunelle

2011-03-25: OAC Phase II Workshop Trip Report

2011-03-25T14:32:00.004-04:00

I've just finished attending the Open Annotation Collaboration (OAC) Phase II Workshop in Chicago, IL (March 24-25, 2011). The quality of the presentations was very high and I was surprised at how much the OAC community has grown in a relatively short time. Although I've served on OAC technical review panels before and my student, Abdulla Alasaadi, has worked on a small prototype (to be presented at JCDL 2011) for using SVG instead of the W3C Media Fragments for specifying an annotation target, I haven't been keeping up with the OAC community as closely as I should.

The Workshop has all the presentations online, as well as a wiki that contains various commentary, use cases, etc. (also, the hash tag is "#oacwkshp"). Although all of the presentations generated a lot of discussion from the attendees, the presentations that I learned the most from were:

Annotation Supporting Collaborative Development of Scholarly Editions (Jane Hunter and Anna Gerber) -- a detailed description of AustLit
Annotating the Biomedical Literature through Text Mining (Karin Verspoor) -- automatically annotating / extracting triples from the biomedical literature
Annotation Ontology and SWAN Annotation Tool (Paolo Ciccarese) -- similar to the above (perhaps with a broader spectrum of manual <--> automatic annotation)
Shared Canvas: Interoperability for Digitized Medieval MSS Repositories (Ben Albritton and Rob Sanderson) -- I never realized how complicated medieval manuscripts could be...
Historic Map Annotations with YUMA (Bernhard Haslhofer) -- an impressive geospatial demo
An OAC-Compliant Toolbox (Shannon Bradshaw) (slides not available yet) -- a toolkit approach for implementing the "scholarly primitives": discover, examine, compare, annotate, organize, synthesize, cite.

Also of special note was the OAC model overview from Herbert and Rob, but I was already pretty familiar with that. Again, I encourage you to look through all the slides since each presentation was well received.

Thanks to Tim Cole for organizing such a successful workshop.

-- Michael

2011-03-21: Grasshopper, prepare yourself. It is time to speak of graphs and digital libraries and other things.

2011-03-21T20:58:00.012-04:00

Announcing the publication of an Old Dominion Computer Science Department technical report and an homage to Davide Carradine, Keye Luke and the television series Kung Fu.

"Grasshopper."

"Yes, Master Po?"

"Grasshopper, you have passed many tests of strength, agility and stamina. But that is not enough. There are other trials you must pass before you are permitted to attempt to lift the fiery brazier. I will ask you a series of questions.

“Let us begin. What is a graph?"

"Master; a graph is a mathematical construct made of objects that may, or may not be connected to each other."

"Grasshopper, how does a graph relate to digital libraries and the world where we live?"

"Master; a graph is composed of nodes (or vertices) that can be connected in a pairwise manner with edges (or arcs). In the world of Facebook, people take the place of nodes and the connection that is made when one person “friends” another creates an edge. In the World Wide Web, pages can be nodes and navigational links can be edges. In digital libraries, a complex digital object, with all its contents and metadata, can be a node and the URIs of other digital objects are its edges. In our Shaolin temple, you and I can be nodes and your teachings are our edge."

"Grasshopper, explain what you mean by objects that in a graph may or may not be connected to each other."

"Master; one can think of the Internet as a graph made up of routers as nodes and cables as edges. If a cable between two routers is severed then the Internet can still function. Not as fully as before, but it will still function."

"Grasshopper, are you saying that a graph that is not connected can still function, al beit at a lower level?"

"Yes, Master."

"Grasshopper, I have in one hand a graph and in the other a hira shuriken. I will answer three questions and then you must cause as much damage as you can to the graph."

"Master; may I see the graph?"

"No. That is one question. I will tell you the name of one node. It is called 5."

"Master; to whom is 5 connected?"

"5 is connected to 4, 6, 8 and 9. That is two questions."

"Master who is connected to 4, 6, 8 and 9?"

"4 is connected to 2, 3 and 5. 6 is connected to 5, 7 and 11. 8 is connected to 2, 5 and 7. 9 is connected to 5 and 10. That is three questions. Now Grasshopper, you must select one node to remove with the shuriken."

"Master, I choose to attack node 5 because it has a the highest a 1 as its vertex centrality betweenness, while all others are far less than 0.5."

"Grasshopper, you have chosen wisely. Now, when node 5 is removed, how much damage has been done to the graph you have discovered?"

"Master, the damage is 0.29 after the first deletion."

"Grasshopper, here is the total graph. What will be the damage to the
discovered graph and to the total graph after the 5th deletion if I were to tell you the friends of the friends of the friends of 5?"

"Master, the damage to the discovered graph would be 0.89 and 0.68 for the total graph."

"Grasshopper, you have answered well. How and where did you learn these things?"

"Master; I read the technical report: Connectivity Damage to a Graph by the Removal of an Edge or a Vertex.”

“Grasshopper, tell me more about this report.”

"Master; it has an abstract that reads:

“The approach of quantifying the damage inflicted on a graph in Albert, Jeong and Barab´asi’s (AJB) report “Error and Attack Tolerance of Complex Networks” using the size of the largest connected component and the average size of the remaining components does not capture our intuitive idea of the damage to a graph caused by disconnections. We evaluate an alternative metric based on average inverse path lengths (AIPLs) that better fits our intuition that a graph can still be reasonably functional even when it is disconnected. We compare our metric with AJB’s using a test set of graphs and report the differences. AJB’s report should not be confused with a report by Crucitti et al. with the same name.

“Based on our analysis of graphs of different sizes and types, and using various numerical and statistical tools; the ratio of the average inverse path lengths of a connected graph of the same size as the sum of the size of the fragments of the disconnected graph can be used as a metric about the damage of a graph by the removal of an edge or a node. This damage is reported in the range (0,1) where 0 means that the removal had no effect on the graph’s capability to perform its functions. A 1 means that the graph is totally dysfunctional. We exercise our metric on a Collection of sample graphs that have been subjected to various attack profiles that focus on edge, node or degree betweenness values.

“We believe that this metric can be used to quantify the damage done to the graph by an attacker, and that it can be used in evaluating the positive effect of adding additional edges to an existing graph.”

“Grasshopper, where did you find this report?”

"Master; I found it at: http://arxiv.org/abs/1103.3075"

"Grasshopper, go and prepare for your next trial."

"Yes, Master."

Dr. Michael Nelson played the part of the Master Po. The part of Grasshopper is poorly played.

Chuck Cartledge

2011-03-09: Adventures with the Delicious API

2011-03-09T14:28:00.005-05:00

I recently conducted an experiment on tags provided from the bookmarking site delicious.com. The goal was to obtain a decent sized sample set of URIs and tags that users have used to annotate the URIs. The website provides a recent tool that automatically redirects to a somewhat random URI that was recently annotated by some Delicious user. By parsing the HTTP headers I was able to grab the redirect URI and therefore build a corpus of 5000 unique URIs. The URI for the tool is http://www.delicious.com/recent/?random=1.
As the second step I needed to obtain the corresponding tags for each URI. I tried to be a good programmer and used the Delicious API to query for the tags instead of parsing the web interface. In order to use the API (v1) you need an account with Delicious/Yahoo. The request for

https://username:pwd@api.del.icio.us/v1/posts/suggest?url=http://www.google.com/

for example returns an XML formated response with the top five popular tags:

search google search engine engine web

The API returns at most the top five tags per URI even though there may be more than five visible through the web interface.
However, I split my URI set into 5 batches and ran five times a thousand queries with the same account and from the same IP address, all within 30 minutes. To my surprise I noticed that roughly 50% of the URIs did not return any tags even though they are indexed by Delicious. My intentions were good but a 50% loss was too much so I turned my attention to screen scraping the HTML page. You need to generate the md5 hash value for each URI (including http://) and append to the proper URI. For example for http://www.google.com you need to request

http://www.delicious.com/url/ff90821feeb2b02a33a6f9fc8e5f3fcd

By parsing the source with simple regular expressions you can extract at most the top 30 tags and the frequency how often users have used this tag for this URI. This path turned out to be fast, reliable and provides better results since you get more than just five tags.

The discrepancy between the API and the web interface however raised some questions and so I will share some statistics about my data and provide theories trying to explain the observed behavior:
I only collected 4969 unique URIs. Apparently the recent tool distinguishes between e.g. google.com and www.google.com and possibly www.google.com/

The API did not return any tags for 78 URIs but the web interface provided tags for all 4969 URIs. Maybe the API accesses a smaller index than the web interface? The recent tool however may pull data from the "live" index. Similar behavior was observed by Frank McCown for search engine caches (JCDL 2007).

I got down to 78 URIs from originally 50% by distributing the queries over five different IP addresses and re-querying the API dozens of times stretched over an entire day. The API seems to be sensitive to high frequency requests or is simply not very powerful.

For the 78 URIs I obtained a mean of 23.2 tags with a standard deviation of 7.8. The minimum number of tags was two (for one URI) and the maximum was 30 (for 38 URIs). 51 of the 78 URIs had 20 or more and 73 URIs had 10 or more tags through the web interface. This just underlines the point: the API is not reliable.

I further found that in 465 cases the API returned less than five tags where the web interface returned more tags. This "under reporting" (meaning the API should have reported the top five) is another strong indicator for the API pulling from a smaller and possibly dated index.

One can argue whether or not the order of tags matters. I found that out of the 4891 URIs with tags from the API 1759 had a different order compared to the web interface data. 191 times I observed a change at rank 1. These changes account for 718 times where terms were added or removed from the union of both tag sets (API vs web interface). On average 1.11 moved in or out of the intersection of both sets.

The moral of all this? As much as you may appreciate an API, in the case of Delicious you can obtain more (better?) data by screen scraping the HTML page.

--
martin

2011-03-04: Personal Digital Archiving Conference 2011

2011-03-04T02:25:00.082-05:00

Last week, along with Dr. Nelson, I attended the 2nd annual conference of Personal Digital Archiving held at the Internet Archive in the heart of the foggy city, San Francisco. The weather was not on our side as the sunny state was facing the worst weather in quite a while. This didn't turn my spirit down as I was excited to be in room with experts and passionate geniuses whose collective IQ could cause an integer over-flow!

The general atmosphere was really nice; participants were very friendly and eager to introduce themselves and get to know you. I got exposed to a ton of ideas, projects and insights over coffee sometimes while other times just going up and down the stairs. My only regret is that I don't have a contact card as I got a bunch of them; I got to get me some of these!

So that the readers can relive this experience with me I have divided the conference into two days each in turn is divided into sessions. I will try to highlight a thing or two from each session and I will try to find the videos for the entire conference.

Day 1:

At 9am the conference started with Brewster Kahle and Jeff Ubois introducing Cathy Marshall from Microsoft Research who gave an amazing keynote entitled: "People are People and things change". It was a really insightful speech summarizing problems we face in dealing with data and backups. She gave examples from her own experience with her computer and the process of "not" backing it up, her tweets which needed to be backed up as well. She came across the note that even when we backup stuff we tend to replicate the entire folders and make copies not maintain an organized list of the resources, people always think that archiving data should be done by someone else.

Gary Wrightfrom Family search gave an insight from his paper entitled "Preserving your family history records digitally" (Legacy Dox) about best archiving practices. He also introduced the Millenniata Disc for data preservation. Jeremy Leighton John from the British Library came up next and discussed ways of processing digital manuscripts. Evan Caroll , the author and founder of TheDigitalBeyond, followed. He discussed a very interesting question: What happens to your digital assets when you die? Who gains access to them? Do you want them to be destroyed? He also discusses why certain assets grow to have more importance according to the sentimental value behind them.

After the break Ellysa Cahoy and Scott McDonald from Penn State University proposed their ideas and project by which faculty members help in the archiving process on their level further more. Judith Zissman used her software design expertise to discuss a very interesting idea about Agile Archiving. She discussed how to implement the Agile Manifesto but in the personal archiving process. I wish she could have given further examples though. After that Stan James , who later became my friend, introduced “The Smallest Day”. A project Stan and his father set up together to collect, archive, arrange, tag and connect all their family photos, documents, letters, postcards and more. It is an awesome project that utilized lots interesting technologies like Dragon software for voice recognition, Mechanical Turk, Ancestry.com…etc. Lori Kendall followed discussing and defining the concept of “personal” in regards to archiving by giving an example from her ancestors photos. Jason Scott followed with a wonderful speech first presenting himself as a collector. Then discussing that it is not enough to keep things safe but we need to make those collections accessible and available online. He discussed in agony the catastrophe of dropping down Geocities and its consequences. On a side note Jason Scott’s cat has 1.4 million followers on twitter and ranked on the top 200 to be followed!

Lunch was next, it was quite refreshing to discuss with other people ideas and thoughts. We had a tour in the internet archive and I took some photos, we saw microfilm readers, Book scanners (not the first time to see one, I saw a whole bunch of them in Alexandria Library). After lunch Birkin Diana from Brown University talked about Metadata on archived items and ways to enhance this metadata. Kate Legg from NCAR (National Center for Atmospheric Research) presented the center’s project in enabling ease of access to archived content and stated Warren Washington’s collection they have as a model that could be adopted in subsequent collections. From Bookism, Jay Datema discussed the issue of compatibility and creating standards of archiving (suggested why not make archive.txt file same as robots.txt and humans.txt files). He also talked about the paper of Jeremy John “The future of saving our past”. The next session was by Ben Gross and Evan Prodromou where they started by discussing the Social Data aspect added to archiving. They discussed FOAF, SIOC, ATOM, OpenDD, POCO and Activity Streams. Marc A. Smith discussed in a very interesting way where in the social graph an individual is located. He utilized MapXL, NodeXL to visualize the US Senates and to my surprise (and my lack of political knowledge) clusters started to appear which showed the Right and Left wings. From Berkeley Ray Larson came up next and introduced the SNAC project discussing also the Authority control and show the two possibilities: Having several names for one person, and having several persons with same name.

Financials and Economics were the theme of the next panelwhere Jeff Ubois, Brewster Kahle, Steve Griffin and David Rosenthal discussed the costs of archiving and showed that the bottleneck was in the scanning process as 80% of the cost goes in the human part of the process. 10 cents per page was the claimed number and discussed that a box of paper can cost from $200 to $680 to be archived. Those are the pay-once models like Presto Prime which costs $2000 per Terabyte to be preserved forever. You can notice that the hardware is the least cost as it is merely $50/TB. The LOCKSS system was introduced as well by Rosenthal. The closing keynote was made by Brian Fetzpatrick from Google and DataLiberation.org. He showed statistics of the Internet cut-off on Egypt the last month as an example of control over the data. He argued that there is a necessity to make data free from the framework beneath and introduce the Import/Export button to all products.

After dinner and reception the demos session started by Joanne Lang and her project “About One” which is an amazing tool to gather information, data, documents and content for the family to help organize and manage life. The slogan was small pieces of information can build a connected life. Michael Ashenfelder from the Library of Congress talked next. Debbie Weissman discussed the possibility of claiming ownership of preserved content. Laura Welcher from the Long Now Foundation came up next and introduced the Rosetta Project aiming to archive 7000 language for fear that some languages can go extinct. She also introduced a hierarchy of languages, language commons and sources in a wiki-like theme. Susam Kostal from San Francisco magazine discussed the concept of digital Hoarding and its relation to physical hoarding. Then Jonathan Good (whom I had a very nice chat earlier about the Egyptian revolution) gave a demo on his project 1000memories.com how friends and loved ones can remember a passed person by collectively gather his/her photos, testimonials, or even start a grant in his/her name. He also showed a dedicated page for the 384 martyrs who died in the Egyptian revolution each photo linked to a dedicated profile so that people don’t forget who were those people and get to know what their lives were. The day was concluded by Denim Smith’spresentation on his project “My Internet Cooperation”.

Day 2:

The second day was also really interesting but definitely shorter. It started with a keynote speech by Clifford Lynch where he gave a very insightful talk about the different forms of exposure of the personal documents to the public. He discussed how the personal archiving concept evolved from just individual private shards accessed individually to shared content with the spread of social media till it finally reached the public domain. He also argued that we need an archive “button” in lots of the digital media.

Danial Reetz with the DIY Book Scanner gave a very interesting talk with a different focus, it was on cameras and technologies. Initially it started when he made “an instructable” on how to build a cheap book scanner. He discussed how cameras vary in power and how the production is affected by users requirements in enhancements. He argued that sometimes users wanted the best modified photo not the best “real” one, slimming cameras, face enhancement lenses…etc. He wondered if it would have been better to invest in adding “document capturing” capabilities to cameras, perhaps OCR too. Dwight Swanson followed up next discussing Home Movies, their evolution and archival. Rich Gibson from Gigapan Project showed how Extreme close-up images can give more stories (like in the Italian fashion runway zooming to the designer on the tag).

After the break 2 poets and writers, Devin Becker and Collier Nogues , made a survey on a broad group of writers and their methods of saving their documents and writings, their archiving and organization. Hong Zhang a PhD student from the University of Illinois followed then Jason Zalinger from Rensselaer Polytechnic Institute in New york. Jason presented his study on possible enhancements to Google’s Gmail by adding these concepts: Forget label for unwanted emails, Digital Regret for undoing the send, Sleep on It for postponing confirming the send, Word Cloud,…etc. Aiden Doherty and Cathal Gurrin from Dublin University presented a very interesting and intriguing concept which is LifeLog. A small wearable device that logs, takes snapshots, GPS coordinates, temperature sensors…etc and store them in a searchable memory platform. They have been wearing these devices for the last 4.5 years!

Ted Nelson gave an awesome speech about how if things were designed differently from the beginning it would have been better. He argued that the documents on the computer are the biggest example. The slides he had didn’t work initially but later that day he showed an amazing demo for Xanadu , a project he is working on for a long time and it introduced a very new data structure which is the multidimensional cells which I found fascinating! Ed Feigenbaum from Stanford introduced SALT (Self Archiving Legacy Toolkit) and talked about the initiative they started at Stanford. Then Christina Englebart , daughter of Douglas Englebart (the inventor of the mouse), gave a presentation about her institute's work in collecting the digital artifacts regarding her father’s legacy.

Lunch was as the day before a good opportunity to mix, mingle and exchange ideas. It helped a lot that it was sunny so most of the people had lunch outside in the sun. When we came back Cal Lee from the University of North Carolina introduced the Forensics aspect in Digital preservation. Richard Cox from University of Pittsburgh followed up next then Mark Matienzo from Yale University Library and Amelia Abreu from University of Washington.

Gordon Bell from Microsoft Research came up next with a talk about his life experience in the health aspect. He illustrated how he gathered all his records from the very first one from decades ago in order to have a better picture on his health situation after he had a heart attack. The project MyLifeBits shows this initiative. Khaled Hassounah came up next and introduced a very successful PHR (Personal Health Record) service named MedHelp. Then Linda Branagan from Medweb argued the difference between EMR (Electronic Medical Record) and PHR (Health Panel Video).

After this final break, Elizabeth Churchill from Yahoo! Research lead a panel discussing forensics in the digital world. Kam Woods from the University of North Carolina presented the Forensic Toolkit Imager and Sam Miester from the University of Maryland discussed data from failed businesses like the Sherwood case.

As a grand finale, the Author Rudy Rucker gave a very interesting talk filled with insightful thoughts, humor and sarcasm discussing Digital Immortality by creating a digital replica of thought and memories which he named it LifeBox. LifeBox acts as a bot that can imitate your responses and be able to answer and give opinions based on your thoughts of memories it can stay forever even after your death for your great grand children.

As a summary it was an amazing conference, not just because I attended those 48 sessions but it gave me a priceless opportunity to meet those bright individuals and broaden my scope of thoughts. As a matter of fact I was inspired to come up with several ideas for my thesis proposal!

Also on other note, I found out that the size of the internet is 20x8x8 ft and it is located in a parking lot in Santa Clara California.

For more about the conference from another prospective please check out Collin Thorman's blog posts, the Library Of Congress's news page, Dick Eastman's blog, Christina Engelbart collective IQ post, Ellysa Cahoy's blog, Don Hawkins's article, collection of posts on The Waki Librarian's day1, day2 and #PDA2011 on twitter.

(2011-03-20 Update:) I have associated links to video recording of each of the sessions, press theand it will display it, photos from the conference can be seen here, courtesy of The Internet Archive and Jeff Ubois.

-- Hany SalahEldeen

2011-02-08: An Evaluation of Link Neighborhood Lexical Signatures to Rediscover Missing Web Pages

2011-02-08T20:29:00.000-05:00

The final project for my master's degree focused on the problem of “missing” web pages, those URIs that return an error result when retrieved. When a web page is no longer available at a given URI, it may be available at a new URI, and this research proposes and demonstrates a new method for finding the new URI.

Prior research has proposed using the lexical signature of a page as a search query to find the same or similar content at a new URI. A lexical signature (LS) is a few words that are used in that page much more often than they are used in other pages on the Web, and so are thought to describe what the page is about. That LS is then used as a search query which will hopefully find the target page in its results.

Previously-proposed methods for using an LS to find a new URI required either that the page be analyzed before being lost (ref: P&W) or that cached or archived versions of the page be available for analysis. If the page had not previously been analyzed and no cached copies existed, then these methods could not hope to recover the missing page.

We propose a new method for calculating an LS of a page using only its backlinks, that is the pages that link to the missing page. The target page together with its backlinks make up the link neighborhood, a trivial example of which is shown above.

Backlinks are retrieved from a search engine (we used Yahoo!). We process each backlink page to find the text it uses to link to the target page. This text is known as ‘anchor text’. We take all of the terms from the anchor texts of all of the backlinks, and calculate the Term Frequency - Inverse Document Frequency (TF-IDF) value of each. That is, we find the terms that are used commonly to link to the target page and are less common on the rest of the Internet. We take the terms with highest TFIDF to be the lexical signature. We use the LS as a query back to the search engine, and if the method is successful, we find our target URI at the top of the results. An example of this process is shown below.

We consider several variations of this method in order to draw conclusions about the most effective parameters. First, we find that including second-level backlinks is not helpful; they add too much noise to the LS and decrease its effectiveness. This confirms the intuition that second-level backlinks, those pages that don’t link directly to a target page but instead link to the target page’s backlinks, are not as closely related to the target page, and will therefore provide less relevant terms.

Second, we show that only the anchor text provides useful terms to the LS. We also experimented using the anchor text plus five words on either side of the link, or anchor +/-10 words, as well as using all words on the page. We found that using only the anchor text provides the best-performing LS, and every step further away from the anchor text led to worse performance.

Third, we show that a passable LS can be created using only the first ten backlinks retrieved from the search engine. Using the first hundred or thousand backlinks yields a slightly-better performing LS in some cases, but we argue that due to the increased cost associated with retrieving and processing 90 or 990 extra pages, the increased performance isn’t worth it.

Lastly, we recommend using a 4-term LS. This is noteworthy in that most other LS research has concluded that a 5- or 7-term LS is ideal, depending on the desired performance characteristics. We posit that fewer terms are preferable because we are drawing the terms from pages other than the page for which we are searching, therefore we run the risk of including terms which do not appear in the target page, which would likely exclude the target page from a search for the LS. By using fewer terms, there is less risk that we include a word from a backlink page that doesn’t exist in the target page.

Using our recommended method, the target URI appears as the first result for the LS in 56% of our test cases.

The full report can be found at:

Jeb Ware, Martin Klein, Michael L. Nelson, An Evaluation of Link Neighborhood Lexical Signatures to Rediscover Missing Web Pages, Technical Report arXiv:1102.0930, February 2011.

-- Jeb Ware

R Tutorial

2011-01-12T18:29:00.014-05:00

As part of Dr. Weigle's CS 796/896 Visual Analytics Seminar I will offer a tutorial on the statistical computing software R. I will give an hands-on introduction into data input/output, simple data manipulation, and (of course) plotting. If you for example always wondered how to fill data vectors, import data from MySQL databases, compute the mean and standard deviation, execute logic, ranking and sorting operations, compute correlations and linear regressions, use loops and write functions as well as create scatter-, bar-, box- and other plots and all that in R, you will enjoy this tutorial. The tutorial is targeted towards course participants which have a natural interest in data visualization but it also has merit for other MS and Ph.D. students doing research and consequently dealing with and plotting data.

While this introduction naturally can not cover all aspects of R and does not claim to be exhaustive, it will help students getting started with the software and the material will serve as an "almanac" in the future to generate pretty graphs such as the one shown here.

CS796/896 is not open to the public but the material covered in the tutorial can be obtained here.

If you have any questions, comments or would like to contribute to augment the materials do not hesitate to contact me.

martin

2011-01-11: WS-DL Spring 2011 Classes

2011-01-11T00:39:00.002-05:00

We are fortunate to have two classes of interest to WS-DL members this semester: Dr. Nelson's CS 751/851 Introduction to Digital Libraries and Dr. Weigle's CS 796/896 Visual Analytics Seminar. They'll run back-to-back on Tuesdays in ECSB r. 2120, with 796/896 beginning at 1:30pm and 751/851 beginning at 4:20pm.

--Michael

2010-12-27: Google Summer Internship, Zürich Switzerland

2010-12-27T00:04:00.008-05:00

"Hello Hany!...We are glad to inform you that you have been accepted in the summer internship program this year in Google Zürich GmBH!". Call me a geek but these were the best words I have ever heard! I now work for Google, well in one way or another!

After struggling with the visa issues I finally got my Swiss Schengen visa and the work permit. The Swiss people are very strict and precise, they thought I was 2 persons, one named Hany Khalil, and the other Hany SalahEldeen! Well I don't blame
them (fyi, in Egypt we don't have the concept of family name, your name is a concatenation of your ancestors names, my name then my father's, then his father's...etc). All my life I have been called Hany SalahEldeen but for some reason the American embassy in Cairo decided that my grandfather's name Khalil suits me better.

"Ich spreche kein Deutsch!" or "I don't speak German" Was the sentence I was repeating to my self on the plane to Zürich, you will never know when it could become handy sometimes! I was brushing up my old French as well, which seemed useless after I arrived to Zürich to realize that French is the main language in Geneva not Zürich. But I didn't care...I was in Google!....I am a Googler!...I even got an email address with my first name @google.com!

On the 6th of July I landed in Geneva, then I took the train to Zürich Hauptbahnhof (which means main station, try to keep up with the German words, or should I say Swiss-German words?). The Swiss really fascinate me, they know the real concept of time (well, they have the best clocks in the world). If you want to call something really punctual or accurate you say it's Swiss, or clock-work which also implies... Swiss. I was dragging my bag from the station, still can barely walk from my leg surgery I reached the tram station. When they say it will arrive 6:43 they actually mean it. I arrived to the student residence of ETH University where I sublet a room for the next 3 months, settled my stuff and fell asleep.

At 9 am next morning I was in the Google Zürich GmBH lobby. I met other interns and after an introductory session we were taken on a tour through all the huge 3 buildings (I used to lose my way for the first 3 days, well but maps were every where). I met there some fellow interns who became my great friends later on. The first two weeks were scheduled to be the training phase, including sessions and tutorials. I got to say when you get access to all these foods, candies, games and entertainment facilities (fussball tables, ping-pong tables, xbox, ps3, rockband, pool, musical instruments, they even got a massage and meditation room!) You get really distracted at the beginning, but that was trivial the following weeks and I loved the idea, if you spoil your employees and make them happy they will feel ownership to the company and commitment thus they will produce amazing work, that was the motto.

My host and manager was very excited and eager to start, so was I. I was the first intern to work under his supervision. He was a mentor, always there to help and give good advice, give me room to work, create and think outside the box and above all he was a good friend. Mostly that's the theme within all employees there, lieght weight, informal but respectful of course. later that week I had a standup coffee meeting with a guy who I later knew that he invented the automated language detection in Google translate! I was working in the MENA (Middle East and North Africa) team on a project allied with the Google translate team. I wish I was able to describe my project but the NDA (Non-Disclosure Agreement) I signed with Google prevents me as it is a new cool project and by the end of the three months I built successfully a huge portion of it. When it is released I will let you know!

Transparency and trust, that's what I was thinking of when I was working. You have access to all the resources and individuals, all available to help you proceed in your project. You can mail anyone and say hey I wanna ask you something! He/She will answer immediately. If you are stuck with a certain program or library you can ask, there experts in it on the mailing list. Maybe you can find the guy who actually invented it and wrote the whole thing! (Like the case in Vim, also you can find Sergey, Page and Cerf on the mailing list too!). Development process is totally different in Google, yes it is Agile and standup meetings are more common than coffee in Italy but there are other considerations. You want to meet deadlines and race to be innovative but also you have to produce code that is extremely scalable, dependable, throughly tested, following style convention and very readable. Handover time to another engineer shouldn't take a long time. I had to throw all most of what I know in C++ and adapt to the new framework of libraries, bigtables, mapreduces ...etc. If you required a functionality someone probably wrote it before so go directly to Code search and acess the code base.

TGIF (Thank Google It's Friday!) are the best weekly gatherings ever! You meet people from different teams in a social manner, relax, laugh, have fun and even karaoke which was a bad idea for me to participate! Every Friday night me and the other interns used to go discover the city and dine in a new place serving a new cuisine, ranging from Swiss cheese fondue to flaming duck Phad Thai. It was delicious and enlightening!

I have been to several parts of Switzerland, learned a little German,one of my friends at Google actually taught me the Blues Harp (AKA. Harmonica) and we used to practice three times a week. I travelled back to Spain to see friends, did water skiing on the lake in Zürich and was scheduled to do a sky-dive on top of the Alps but it was cancelled for bad weather, I was pissed!

Walking through the city was a pleasure itself. Enjoying a cup of coffee down one of the curling streets was amazing. Reading a book by the lake was a quality time. The only bad thing about Zürich is its prices!...I saw a suit in a shop and I kept looking for its price tag because I thought the numbers on the tag in front of it were the serial number not price!

The student residence I used to live in was amazing. Imagine living in a place where 100 different students live from more than 35 countries. We laughed together, we watched World-cup together and cheered for all teams! we cooked, watched movies and partied together too. It was friendly, brotherly and definitely educating. I met there people who definitely left a mark on my life.

In conclusion it was an amazing summer, educating, life changing experience. Working for the best company, living in an amazing city and meeting great people, what more can one ask for?!

2010-12-06: Memento Wins the 2010 Digital Preservation Award

2010-12-06T18:33:00.008-05:00

The Memento Project won the 2010 Digital Preservation Award in London on December 1, 2010. The DPA is sponsored by the Digital Preservation Coalition, and the Memento Project is sponsored by the Library of Congress (see also: LC's project page).

Details about the DPA are provided in several press releases, including ones from the DPC, ODU, LANL and LC. DPC has also posted a short video of an interview with Herbert. And for posterity, the original tweet from William Kilbride announcing the winner (more information from the award ceremony will be announced on #dpa2010).

Thanks to the DPC, the DPA judges, the Library of Congress, and everyone on the Memento team!

--Michael

2010-12-02: NASA IPCC Data System Workshop

2010-12-02T13:31:00.003-05:00

I attended a NASA Intergovernmental Panel Climate Change(IPCC) Data System Workshop in Greenbelt Maryland, November 9 - 10. The IPCC is an international committee overseeing the assessment of global climate change.

The purpose of this workshop is to discuss technical plan to prepare, incorporate and share IPCC-relevant NASA satellite observational datasets to support the Coupled Model Intercomparison Project Phase 5 (CMIP5). CMIP is a standard protocol and framework for evaluating climate model simulation (hindcast) and predictions/simulation of future climate change. CMIP5 is the 5th evaluation and being organized and lead by the Program for Climate Model Diagnosis and Intercomparison (PCMDI) mission at Lawrence Livermore National Laboratory. All of this activity will help contribute to the IPCC 5th Assessment Report (APCC AR5) and beyond. In prior assessments, NASA observational datasets were not used (or very little). NASA HQ has recognized the richness and important of NASA datasets and encouraged the satellite project teams to get involve and collaborate with the PCMDI on CMIP5.

An interesting overview talk on Earth System Grid (ESG) was presented. ESG is a distributed computational environment of grid services to support next generation climate modeling research. More technical details of ESG can be found in this paper by Bernoldt et al (2005). Technical talks from JPL,GSFC, NCAR, NOAA, and ORNL discussed progress from each group to support CMIP5. While most group are 1 or more year into the effort, we (at LaRC) are newbie. Our group presented an overview of relevant CERES datasets and new tool for ordering and retrieving CERES data. The biggest hurdle and question is how do we make satellite observations look like model output. This is critical for intercomparison. Lots of talk on CF NetCDF compliant formats, technical notes and metadata for each dataset, and selection of relevant observation dataset to include into CMIP5. A couple groups have gateways into the ESG while most have data nodes. With tight deadline in April 1, 2011, we agree to let ORNL host our CERES dataset on their ESG data node. We agree to set up a data node at Langley in the near future.

-Louis

2010-11-15: Memento Presentation at UNC; Memento ID

2010-11-15T21:48:00.005-05:00

I recently had a chance to return to the School of Information and Library Science, UNC Chapel Hill, where I had a most enjoyable post-doc during the academic year 2000-2001. Jane Greenberg was nice enough to invite me to speak about Memento in her INLS 520 "Organization of Information" class on Tuesday, November 9th as well as give an invited lecture about Memento to the UNC Scholarly Communications Working Group on Wednesday, November 10th.

When I first went to UNC I had the office next to Jane and she was just an assistant professor, now she's a full professor and director of the Metadata Research Center. I enjoyed catching up with her and my many other friends and colleagues at SILS.

My slides are available on slideshare.net; they are mostly a combination of slides I've posted before, but with some updates in the HTTP headers. Although the changes are very slight, the recently submitted (11/12/10) Memento Internet Draft takes precedence over all of our prior published papers and slides. For those who don't know, IETF Internet Drafts are the first step in the process of issuing an RFC (cf. "I'm Just a Bill...").

Memento: Time Travel for the Web

View more presentations from Michael Nelson.

--Michael

2010-11-05: Memento-Datetime is not Last-Modified

2010-11-05T17:16:00.020-04:00

One of the key contributions of the Memento Framework is the HTTP response header "Memento-Datetime" (previously called "Content-Datetime" in our earlier publications & slides). Memento-Datetime is the sticky, intended datetime* for the representation returned when a URI is dereferenced. The presence of the Memento-Datetime HTTP response header is how the client realizes it has reached a Memento.

Rather than formally explain what we mean by "sticky, intended datetime", it is easier to explain how it is neither the value in the HTTP response header Last-Modified, nor is it the creation date of the resource (which has no corresponding HTTP header, for reasons that will become clear). For the examples below, we'll define the following abbreviations:

CD (Creation-Datetime) = the datetime the resource was created
MD (Memento-Datetime) = the datetime the representation was observed on the web
LM (Last-Modified) = the datetime the resource last changed state

Case 1: CD == MD == LM

We'll begin with a case in which all three datetime values could be the same. Consider the case of this index page at Archive-It.org:

http://wayback.archive-it.org/927/*/http://www.nyu.edu/fas/projects/vcb/case_911_FLASHcontent.html

The index page has a link to a single Memento. For simplicity, we'll assume Archive-It.org created this index page and the Memento it references at the moment of the crawl, thus the various datetimes of the Memento would all be equal:

Creation-Datetime: Wed, 05 Mar 2008 20:16:49 GMT
Memento-Datetime: Wed, 05 Mar 2008 20:16:49 GMT
Last-Modified: Wed, 05 Mar 2008 20:16:49 GMT

Case 2: CD == MD < LM

If we click on the Memento (http://wayback.archive-it.or/927/20080305201649/http://www.nyu.edu/fas/projects/vcb/case_911_FLASHcontent.html), we see that it has a disclaimer banner ("You are viewing an archived web page...") that many archives employ to inform the reader that they are looking at a Memento and not the original resource. Although there are many techniques for inserting such a banner, the Archive-It example directly modifies the original HTML to insert this banner (as well as handle URI rewriting, etc.).

Now pretend the wording of the banner needs to be changed (for example, to address a new legal requirement). The CD and MD of the Memento are unchanged, but the LM must reflect when the wording of the banner changed:

Creation-Datetime: Wed, 05 Mar 2008 20:16:49 GMT
Memento-Datetime: Wed, 05 Mar 2008 20:16:49 GMT
Last-Modified: Fri, 05 Nov 2010 23:25:19 GMT

Both your lawyer and your HTTP cache consider this an important change, so you have to update LM. But it also clear that the essence of March 2008 observation of the Memento by Archive-It.org is unchanged by the wording change of the archive banner, so MD is not updated. And certainly the CD is unchanged by this modification.

Case 3: MD < CD <= LM

Now pretend you are making a new web archive, and you are populating it by crawling other web archives such as Archive-It.org (simulated with the king of browsers in the image to the left). You are effectively copying:

http://wayback.archive-it.org/927/20080305201649/http://www.nyu.edu/fas/projects/vcb/case_911_FLASHcontent.html

to:

http://archive.example.org/20101105232519/http://www.nyu.edu/fas/projects/vcb/case_911_FLASHcontent.html

The presence of the Memento-Datetime header from Archive-It.org indicates that the resource is an encapsulation of the state of another resource, at the MD datetime value. The link between the Memento and the original resource is indicated with an HTTP Link response header:

Link: rel="original"; <http://www.nyu.edu/fas/projects/vcb/case_911_flashcontent.html>

Thus, MD is sticky in that the new Memento at example.org retains the MD value it observed from Archive-It.org. However, the CD and LM values reflect the datetime relative to example.org:

Creation-Datetime: Fri, 05 Nov 2010 23:25:19 GMT
Memento-Datetime: Wed, 05 Mar 2008 20:16:49 GMT
Last-Modified: Fri, 05 Nov 2010 23:25:19 GMT

The MD and LM datetimes can also vary for the example.org Memento as described in Case 2. (In the unlikely case that the intent of example.org was to create an archive of how resources were archived, the MD could be reset to 05 Nov 2010 and the Link header would point to the Archive-It.org resource as the original resource instead of the nyu.edu resource; however, this is not the point of this discussion.)

Case 4: CD < MD <= LM

This scenario is probably less common, but you could imagine situations in which CD is the earliest datetime value. This might happen in situations in which the resource was created with something akin to fork() & exec() semantics: the resource was technically created at a certain datetime , but it did not acquire its own state until a later datetime, reflected in the MD & LM values.

For example, a transactional archive might record as CD the first datetime in which a resource returns a 200 response, but might choose to delay archiving Mementos until the resource's state is something other than "Welcome to Apache". In this scenario, you could have:

Creation-Datetime: Wed, 05 Mar 2008 20:16:49 GMT
Memento-Datetime: Fri, 05 Nov 2010 23:25:19 GMT
Last-Modified: Fri, 05 Nov 2010 23:25:19 GMT

The MD and LM datetimes could also vary as described in Case 2.

Creation Datetime Is Often Unavailable

To illustrate the differences between the various datetime concepts, the above examples have discussed Creation Datetime as if it is a commonly available value. However, this is most often not the case -- in fact, there is no defined HTTP response header that corresponds to Creation Datetime. This is due to the historical limitation of Unix inodes (i.e., metadata for files), which track three notions of time: atime (access time of the file), mtime (modification time of the file), and ctime (modification time of the inode). Modern content management systems might keep track of Creation Datetime, but it is not formally defined at the HTTP level.

Summary

The above examples should provide illustrations of how the three notions of datetime, although obviously related, have slightly different semantics. It should be clear that a Memento's Memento-Datetime is also not just Creation-Datetime or Last-Modified inherited from the original resource for which it is a Memento. Rather than overload an existing HTTP response header (such as Last-Modified), we have introduced the Memento-Datetime (nee Content-Datetime) response header. Additional information about Memento headers, Link rel types, and HTTP interactions can be found at mementoweb.org.

-- Michael

* Datetime = neologism of "date" & "time": the former is often understood to have a granularity of days, and the latter a granularity of seconds.

2010-10-21: RRAC Presentation

2010-10-21T11:40:00.005-04:00

Tuesday, I gave a presentation introducing some of the research we are doing in our WSDL group to the Records and Archivists (RRAC) national meeting. This group is made of archivists at Federally Funded Research and Development Centers (like MITRE and Aerospace) and University Archivists.

Digital Preservation - ODU

View more presentations from Justin Brunelle.

I used slides from several of Dr. Nelson's and Martin Klein's presentations (credits recently given in the last slide).

I also gave the same presentation to the Agile Development department (of which Carlton is a member) on Tuesday. Both groups widely received the research and had very interesting ideas and comments. The RRAC folks (who were of non-technical backgrounds) questioned the projected lifespan and availability of archives like the Internet Archive (IA). We also discussed the possibility of the Twitter virus being stored in the IA (and I have yet to investigate this possibility). The other interesting topic of discussion was how to use robots.txt files.

I thought the presentation went well, and I can provide more information on the other, less interesting questions offline.

--Justin F. Brunelle

2010-10-11: A Blast from the past: My road to Ws-Dl!

2010-10-11T23:39:00.011-04:00

Hello everyone, I am Hany SalahEldeen, a PhD student in my first year and I am honored to be a new member of the Ws-Dl group at Old Dominion University and supervised by Dr. Michael Nelson.

I have been in the group for a couple of months now so I thought I should introduce myself and give a background summary on my career before Ws-Dl because I believe if you didn't know where you were, you will never know where you are going.

I received my BSc. in Computer Systems Engineering at Alexandria University, Egypt in 2008. My graduation project entitled "VOID: The web-based integrated development environment" was selected to win the first prize in the graduation projects competition in the University for year 2008. For the last 2 years of my degree I was working in a software company back home called eSpace technologies, I worked in developing systems using Ruby on Rails, and was one of the members who developed Neverblock (an open source project to enable easy development of non-blocking concurrent code.) along with fellow student and friend Mostafa Aly who is also in the Ws-Dl group.

I started my masters program in Universitat Autonoma de Barcelona, Spain. I worked in CVC (the Computer vision center) in the colour group under the supervision of Robert Benavente, Maria Vanrell and Joost Van de Weijer, and on July 2009 I defended my thesis entitled "Colour naming Using Context-Based learning through a Perceptual Model", published a paper and the second is still under development. In a nutshell we were able to create a parametric model for the Lab color space based on psychophysical experiments using real life images in the machine learning process to reach a better model near to human perception of color in context. In August 2008 I participated in the CVC team competing in PASCAL VOC2009 image classification world challenge in Kyoto, Japan and won 2 gold medals.

In September 2009 I started my internship at Cairo Microsoft Innovation center "CMIC" working on creating recommendation systems based on social networks with Nayer Wanas and wrote a paper which is under review. Also performed a study that the research center presented by CMIC's director Tarek Alabady to the minister of communication and information technology Tarek Kamel in December 2009.

In January 2010 I arrived Norfolk and started my first semester at ODU. Later in the same month I was invited by Google to attend the 2010 Google Grad CS Forum. An all-paid trip from Norfolk to San Fransisco including two day stay at Hilton downtown, who can say no?! Hanah Kim the University Programs Specialist contacted me giving me the details and the agenda.

On the 21 I was with 82 other fellow PhD students from all over the states attending the opening reception hosted by Alfred Spector, Google's VP of Research and Special Initiatives. He discussed with us several topics and answered all our questions. Surrounded by all these brilliant minds. I was so proud to represent old Dominion University in this prestigious event. Early next day a shuttle came to take us to the GooglePlex. There, Marissa Mayer, VP Search Products & User Experience welcomed us and gave along with Kevin McCurley, Research Scientist an amazing keynote and answered all our questions in regards to research in Google, publishing, research in the industry in general. After that we were taken on a tour around the humongous GooglePlex campus. The tour took more than an hour and yet we just skimmed some of the public areas (some areas are restricted to outsiders and guests).

After lunch, some of the students in their final years of their PhD were selected to give presentations about their work, which was definitely enlightening. After that we had two Tech-Talks, the first was by Hector Gonzalez, Research Scientist, in which he described to us new briefly techniques in extremely Large Scale data collaboration and integration. The second was by T.V. Raman, Research Scientist, which I think was the most amazing talk I attended in a long time. T.V. is blind, but he leads one of the biggest accessibility teams at Google and he specializes in auditory user interfaces and structured electronic documents.

Finally there were some round tables with scientists from different fields who gladly answered our questions. Andrea frome which leads one of the teams in Google Maps specialized in Street View described to us her work and answered all our questions. Later that day we had dinner in an amazing Italian restaurant in the heart of San Fransisco named Palio d'Asti. The next day I flew back to Norfolk.

That was a quick snapshot of the highlights in my career before Ws-Dl, I joined in february of 2010. I hope this post wasn't too long!

For more details check out my Blog and Website.

--Hany

2010-10-11: ArchiveFacebook Version 1.2 is released

2010-10-11T14:13:00.004-04:00

Celebrating a year from the very first release of ArchiveFacebook the development team is releasing the new version 1.2. Throughout the last couple of months we have received feedback from the users asking for enhancements and resolving issues. We also received lots of compliments and thumbs up! This feedback was channeled and analyzed to give us an idea on how to enhance the user experience.

We released version 1.2 3 days ago with lots of bug fixes and new features, among which the expansion of stories and posts on comments. Several users suggested that it would be useful to be able to archive all the posts and comments on a certain activity (status update, event attendance, photo...etc). Now V 1.2 can support this and any activity stream within your Facebook profile.

The new version seems to be highly anticipated to an extent that the number of downloads within the first 3 days even before announcing the release reached 2000 according to Mozilla:

https://addons.mozilla.org/en-US/firefox/addon/13993/

Try out the new version and let us know what do you think. Development is triangle and feedback is one of its edges!

Please join the ArchiveFacebook group to post issues and stay tuned with the latest updates and future releases:

http://groups.google.com/group/archivefacebook

--Hany

2010-10-04: WAC Kickoff Meeting; LC Storage Architectures Meeting, DPC Award Shortlist

2010-10-04T18:35:00.003-04:00

On September 24, I attended the kickoff meeting at Stanford for the Web Archiving Cooperative (WAC) Project, a joint NSF project (~$2.8M) between Stanford, Old Dominion and Harding. A summary of the meeting will be published at a later date, but it was attended by several members of our Advisory Board (from memory: Chris Borgman (UCLA), Trisha Cruse (CDL), Rick Furuta (TAMU), Alon Halevy (Google), Carl Lagoze (Cornell), Raghu Ramakrishnan (Yahoo), Herbert Van de Sompel (LANL)) and several members and friends of the Stanford Infolab.

I gave two presentations, the first was a quick review of the state of web preservation (with the obligatory heavy emphasis on Memento), and the second was some of my ruminations about future things that we should (or should not) explore in the context of WAC.

Review of Web Archiving

View more presentations from Michael Nelson.

My Point of View: Michael L. Nelson Web Archiving Cooperative

View more presentations from Michael Nelson.

That night I caught a redeye back to Norfolk so I could be in DC the following Monday for the Library of Congress Designing Storage Architectures for Preservation Collections Meeting. While I believe this is their fourth such meeting, it is the first one I attended and while (because?) I did not present or speak, I learned a great deal. The meeting featured a good mix of academicians and storage industry leaders discussing very large scale storage architectures -- scales that we don't typically approach in our research at ODU. The majority of the presentations were limited to 5 minutes each, so a good breadth of topics was covered and perusing the slides will be worth your time.

Finally, Memento has been named one of five finalists for the Digital Preservation Coalition 2010 Digital Preservation Award. It is an honor to be a finalist amongst the other projects (see the DPC Press Release for a descriptions of all the projects). The Library of Congress has also issued a press release as well as ODU. The final announcement will come in December -- here's hoping Memento can bring in the prize.

--Michael

2010-08-28: A Lookup for Nicknames and Diminutive Names

2010-08-28T17:08:00.007-04:00

I created a simple lookup file that contains United States given names (first names) and their associated nicknames or diminutive names. For example "gregory" -> "greg", or "geoffrey" -> "geoff". The file can be downloaded and contributed to from here http://code.google.com/p/nickname-and-diminutive-names-lookup/.

This lookup was started from http://www.tngenweb.org/franklin/frannick.htm which is used for genealogy purposes. It was a good source to start from but because it is used for genealogy purposes there are some pretty of old names in there. There was also a significant effort to make it machine readable, i.e. separate names with commas, remove human readable conventions, like "rickie(y)", so that it would be made into two different names "rickie", and "ricky".

This is a large list with about 700 entries. Any help from people to clean this list up and add to it is greatly appreciated. Think of it as a wiki where you can contribute or change it as needed. CSV was the easiest format to use. Maybe I'll release this in XML or something later, or maybe a kind soul who uses this library wants to contribute another format they converted it into?

I was rather surprised that I couldn't find anything like this on the web. The best I could find was the pdNickname database and it costs $500. So, I created my own and released it as open source so that others could benefit from my work.

--Carlton