Web Science and Digital Libraries Research Group

         The “reproducibility crisis” in scientific research refers to growing concerns over the reliability and credibility of published findings, in many fields including biomedical, behavioral, and the life sciences ( Laraway et al. 2019 , Fidler et al. 2021 ). Over the past decade, large-scale reproducibility projects revealed failures to replicate findings. For example, in 2015 the Open Science Collaboration reported that a larger portion of replicated studies produced weaker evidence for the original findings despite using the same materials. Similarly, in fields like machine learning researchers may publish impressive new methods, but if others can’t reproduce the results, it hinders progress. Therefore, reproducibility matters. It’s science’s version of fact-checking. In this blog, we’ll break down our recent effort to reproduce the results of the paper Network Deconvolution by Ye et al. (hereafter, "original study"), published in 2020, which claime...

I am excited to announce that Old Dominion University (ODU) is part of the multi-university grant awarded by the Open Philanthropy Foundation  to support the development of a systematic benchmark assessing how effectively large language models (LLMs) can evaluate, replicate, and conduct scientific research. The leading institution is the Center for Open Science (CoS, Dr. Brian Nosek and Dr. Tim Errington ) and the participation institutions are Pennsylvania State University ( Dr. Sarah Rajtmajer , Dr. Qingyun Wu ), Notre Dame University ( Dr. Meng Jiang ), and ODU ( Dr. Jian Wu , myself).  The team will conduct a test of whether LLMs are capable of determining whether claims are true or false. Here a claim means a statement that conveys a research finding in a scientific paper. Our operationalization of this question is whether LLMs can assess a scientific paper and predict whether primary findings would replicate or reproduce successfully in an independent test. In the fund...

  According to a report published by National Academies of Sciences, Engineering, and Medicine , " reproducibility refers to instances in which the original researcher's data and computer codes are used to regenerate the results, while replicability refers to instances in which a researcher collects new data to arrive at the same scientific findings as a previous study." In this blog, we will focus on several milestone papers for replicability assessment of SBS  (social and behavioral sciences)  papers. SBS  includes psychology, economics, politics, sociology, etc. SBS papers typically use statistical testing to test hypotheses. A p-value below a significance level (typically 0.05) can be used as evidence to exclude the null hypothesis. However, as pointed out by a previous study , a p-value provides only  partial information about the probability of a tested hypothesis being true.  Statistically significant results from small  studies are ...

    Figure 1: Homepage of ACM REP ‘24 The ACM Conference for Reproducibility and Replicability (ACM REP) is a recently launched venue for computer and information scientists whose research covers reproducibility and replicability. It covers practical issues in computational science and data exploration, emphasizing the importance of community collaboration to adopt new methods and tackle challenges in this relatively young field. The conference is linked with the ACM Emerging Interest Group for Reproducibility and Replicability . The 2024 ACM Conference on Reproducibility and Replicability (ACM REP ‘24) combined in-person and virtual participation to broaden participation. Held from June 18th to June 20th at the Inria Conference Center in Rennes, France, the conference featured tutorials, keynote speeches, paper presentations, and discussions on reproducibility in computational research. The conference had four tutorial sessions; each divided into three to four tracks, s...