Reproducible research

From CEEMID Documentation
Jump to navigation Jump to search

Reproducible research is a scientific concept that can be applied to a wide range of professional designations, for example, reproducible finance in the investment process or reproducible impact assessment in policy consulting. Based on the computational reproducibility [1] we provide tools for music professionals and policymakers with tools that meet these requirements.

  • Reviewable findings: The descriptions of the methods can be independently assessed and the results judged credible. (This includes both traditional peer review and community review, and does not necessarily imply reproducibility.)
  • Replicable findings: Findings are presented in a way with necessary tools so that our users or auditors or external authorities can duplicate our results.
  • Confirmable findings: The main conclusions of the research can be attained independently without our software, because we describe in detail the algorithms and and methodology in supplementary materials. We believe that other organizations, analysts, statisticians must come to the same findings with their own methods and software. This avoid lock-in and allows independent cross-examination.
  • Auditable findings: Sufficient records (including data and software) have been archived so that the research can be defended later if necessary or differences between independent confirmations resolved. The archive might be private, as with traditional laboratory notebooks.

These computational requirements require a data workflow that relies on further principles.

  • Record retention: all aspects of reproducibility require a high level of standardized documentation. The standardization of documentation requires the use of standardized metadata, metadata structures, taxonomies, vocabularies.
  • Best available information / data universe: the quality of the findings, their confirmation and auditing success will improve with better data and facts used.
  • Data validations: The quality of the findings will greatly depend on the factual inputs. While the reproducible findings may have many problems, inputting erroneous data or faulty information will likely lead to wrong conclusions, and in all cases will make confirmation and auditing impossible. Especially when organizations use large and heterogenous data sources, even small errors, such as erroneous currency translations or accidental misuse of decimals, units can cause results that will not pass confirmation or auditing.

Not only scientists, but many professional designations, like lawyers or CFA charterholders must make their findings, recommendations to be auditable. Management and policy consultants always work with new challenges and continuously expand their available fact and data universe. CEEMID helps music professionals and policymakers with reproducible research tools to make these processes fast, low-cost, error-free, well-documented and archived.

Open or Reproducible Research is auditable research made openly available. CEEMID makes some of its critical data procedures open, by publishing both our software code as open-source software and data created by our software as open data to invite peer-review challenges. We make our know-how open to collaboration when we use know-how that is scarce and internal validation is not possible – for example, when working with a highly structured national accounts products or difficult regional data boundaries.

  1. Victoria Stodden and David H. Bailey and Jonathan M. Borwein and Randall J. LeVeque and William J. Rider and William Stei (2013): Setting the Default to Reproducible Reproducibility in Computational and Experimental Mathematics. Developed collaboratively by the ICERM workshop participants