PDF] Reproducibility via Crowdsourced Reverse Engineering: A

The significant success of reverse-engineering the important accomplishments of DeepMind's Alpha Zero exemplifies the leverage that can be achieved by a concerted effort to reproduce results. The reproducibility of scientific findings are an important hallmark of quality and integrity in research. The scientific method requires hypotheses to be subjected to the most crucial tests, and for the results to be consistent across independent trials. Therefore, a publication is expected to provide sufficient information for an objective evaluation of its methods and claims. This is particularly true for research supported by public funds, where transparency of findings are a form of return on public investment. Unfortunately, many publications fall short of this mark for various reasons, including unavoidable ones such as intellectual property protection and national security of the entity creating those findings. This is a particularly important and documented problem in medical research, and in machine learning. Fortunately for those seeking to overcome these difficulties, the internet makes it easier to share experiments, and allows for crowd-sourced reverse engineering. A case study of this capability in neural networks research is presented in this paper. The significant success of reverse-engineering the important accomplishments of DeepMind's Alpha Zero exemplifies the leverage that can be achieved by a concerted effort to reproduce results.

PDF) Experiences in integrated data and research object publishing using GigaDB

PDF] Reprowd: Crowdsourced Data Processing Made Reproducible

Scientific Utopia III: Crowdsourcing Science - Eric Luis Uhlmann, Charles R. Ebersole, Christopher R. Chartier, Timothy M. Errington, Mallory C. Kidwell, Calvin K. Lai, Randy J. McCarthy, Amy Riegelman, Raphael Silberzahn, Brian

Lost in translation: the valley of death across preclinical and clinical divide – identification of problems and overcoming obstacles, Translational Medicine Communications

An artificial intelligence framework on software bug triaging, technological evolution, and future challenges: A review - ScienceDirect

Accelerating antiviral drug discovery: lessons from COVID-19

Crowdsourcing in Cognitive and Systems Neuroscience - Brian P. Johnson, Eran Dayan, Nitzan Censor, Leonardo G. Cohen, 2022

PDF) Reproducible biomedical benchmarking in the cloud: Lessons from crowd-sourced data challenges

Crowdsourced Perceptual Ratings of Voice Quality in People With Parkinson's Disease Before and After Intensive Voice and Articulation Therapies: Secondary Outcome of a Randomized Controlled Trial

A survey of mobile crowdsensing and crowdsourcing strategies for smart mobile device users

The “Narratives” fMRI dataset for evaluating models of naturalistic language comprehension

PDF) Exploring Crowdsourced Reverse Engineering

PDF] Reprowd: Crowdsourced Data Processing Made Reproducible

PDF) Drug repurposing prediction for COVID-19 using probabilistic networks and crowdsourced curation