Web Science and Digital Libraries Research Group

I am excited to announce that Old Dominion University (ODU) is part of the multi-university grant awarded by the Open Philanthropy Foundation  to support the development of a systematic benchmark assessing how effectively large language models (LLMs) can evaluate, replicate, and conduct scientific research. The leading institution is the Center for Open Science (CoS, Dr. Brian Nosek and Dr. Tim Errington ) and the participation institutions are Pennsylvania State University ( Dr. Sarah Rajtmajer , Dr. Qingyun Wu ), Notre Dame University ( Dr. Meng Jiang ), and ODU ( Dr. Jian Wu , myself).  The team will conduct a test of whether LLMs are capable of determining whether claims are true or false. Here a claim means a statement that conveys a research finding in a scientific paper. Our operationalization of this question is whether LLMs can assess a scientific paper and predict whether primary findings would replicate or reproduce successfully in an independent test. In the fund...

ETD 2024 took place in Livingstone, Zambia I had the privilege of participating in the 27th International Symposium on Electronic Theses and Dissertations (ETD 2024) , which took place as a hybrid event in Livingstone, Zambia, from November 4th to 6th, hosted by the University of Zambia . The conference provided a unique opportunity for professionals in the fields of digital libraries, open science, and graduate education to gather, collaborate, and explore advancements in Electronic Theses and Dissertations (ETDs). The hybrid nature of the event made it possible for global audiences to participate, with sessions spanning a wide range of topics, including ETD implementation use cases, open access to ETDs, the intersection of open science and ETDs, long-term preservation, the global visibility of ETDs, and the transformative role of large language models in ETD research. Day 1 at ETD 2024 Workshops ETD 2024 kicked off with workshops designed for all experience levels. " ETDs 101: N...

Figure 1:  An example of a Coyote ad on social media in Spanish. (Figure 1 in  Kodandaram et al. )   Introduction Social media has revolutionized how information flows, connecting people across the globe at an unprecedented scale. However, this same connectivity has made these platforms fertile ground for illicit activities, including human smuggling. This issue, though not immediately visible to most people, poses a severe humanitarian crisis. We explore in our paper published at the 35th ACM Conference on Hypertext and Social Media 2024 titled " Unveiling Coyote Ads: Detecting Human Smuggling Advertisements on Social Media, " how traffickers exploit these platforms to lure vulnerable individuals, particularly those seeking better opportunities or fleeing dangerous situations. The study focuses on identifying and analyzing “ coyote ads, ” which are advertisements posted by human smugglers. Our goal was clear: to develop an effective detection mechanism that exposes thes...

         Figure 1: Overall process of applying conformal prediction for uncertainty quantification in LLMs (Figure 2 in paper ) Introduction As large language models (LLMs) gain prominence in academia and industry, evaluating their performance becomes increasingly critical. Popular platforms such as the HuggingFace Open LLM Leaderboard focus solely on accuracy, neglecting an essential dimension: uncertainty . Uncertainty quantification (UQ) is vital for comprehensive model evaluation, as two models may achieve similar accuracy but show different levels of uncertainty as shown in Figure 2 below. In this blog, we summarize the paper " Benchmarking LLMs via Uncertainty Quantification " [1] by Fanghua Ye et al., presented at the  NeurIPS 2024 . This paper introduces a novel benchmarking framework that incorporates uncertainty quantification using conformal prediction [2]. The paper quantifies the uncertainties of nine LLMs across five core natural language ...

During my senior year as a Ph.D. candidate at Old Dominion University (ODU) , I accepted an internship opportunity with the Center for Drug Evaluation and Research (CDER) / U.S. Food and Drug Administration (FDA) in the Division of Quality Intelligence III under the Office of Pharmaceutical Quality (OPQ) in Silver Spring, Maryland. I joined the FDA as a summer fellow through the Oak Ridge Institute for Science and Education (ORISE) . This is my third remote internship opportunity after two consecutive internships with Los Alamos National Laboratory and Bhirle Applied Research Inc . Although the internship was remote, I was called in for two days to provide a fingerprint, collect my FDA badge, and meet with my supervisor on-site. I was assigned to one of the regulatory projects to enhance a site selection model algorithm that leverages a machine learning technique for comprehensive quality surveillance. I was supervised by John Wan (Supervisory Operations Research Analyst) and L...

  According to a report published by National Academies of Sciences, Engineering, and Medicine , " reproducibility refers to instances in which the original researcher's data and computer codes are used to regenerate the results, while replicability refers to instances in which a researcher collects new data to arrive at the same scientific findings as a previous study." In this blog, we will focus on several milestone papers for replicability assessment of SBS  (social and behavioral sciences)  papers. SBS  includes psychology, economics, politics, sociology, etc. SBS papers typically use statistical testing to test hypotheses. A p-value below a significance level (typically 0.05) can be used as evidence to exclude the null hypothesis. However, as pointed out by a previous study , a p-value provides only  partial information about the probability of a tested hypothesis being true.  Statistically significant results from small  studies are ...