mirror of
https://github.com/foo-dogsquared/wiki.git
synced 2025-02-12 03:19:01 +00:00
167 lines
58 KiB
HTML
167 lines
58 KiB
HTML
|
<!DOCTYPE html><html><head><meta name="viewport" content="width=device-width"/><meta charSet="utf-8"/><title>Reproducible research: methodological principles for transparent science</title><script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script><script id="MathJax-script" async="" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script><script type="text/x-mathjax-config">
|
||
|
|
MathJax = {
|
||
|
|
tex: {
|
||
|
|
inlineMath: [ ['$','$'], ['\(','\)'] ],
|
||
|
|
displayMath: [ ['$$','$$'], ['[',']'] ]
|
||
|
|
},
|
||
|
|
options = {
|
||
|
|
processHtmlClass = "math"
|
||
|
|
}
|
||
|
|
}
|
||
|
|
</script><meta name="next-head-count" content="6"/><link rel="preload" href="/wiki/_next/static/css/52fc2ba29703df73922c.css" as="style"/><link rel="stylesheet" href="/wiki/_next/static/css/52fc2ba29703df73922c.css" data-n-g=""/><noscript data-n-css=""></noscript><link rel="preload" href="/wiki/_next/static/chunks/main-ae4733327bd95c4ac325.js" as="script"/><link rel="preload" href="/wiki/_next/static/chunks/webpack-50bee04d1dc61f8adf5b.js" as="script"/><link rel="preload" href="/wiki/_next/static/chunks/framework.9d524150d48315f49e80.js" as="script"/><link rel="preload" href="/wiki/_next/static/chunks/commons.0e1c3f9aa780c2dfe9f0.js" as="script"/><link rel="preload" href="/wiki/_next/static/chunks/pages/_app-8e3d0c58a60ec788aa69.js" as="script"/><link rel="preload" href="/wiki/_next/static/chunks/940643274e605e7596ecea1f2ff8d83317a3fb76.4841a16762f602a59f00.js" as="script"/><link rel="preload" href="/wiki/_next/static/chunks/pages/%5B%5B...slug%5D%5D-1aa198f87ede1cd0e1dc.js" as="script"/></head><body><div id="__next"><main><h1>Reproducible research: methodological principles for transparent science</h1><section class="post-metadata"><span>Date: <!-- -->2020-04-12 10:24:25 +08:00</span><span>Date modified: <!-- -->2021-11-07 00:38:07 +08:00</span></section><nav class="toc"><ol class="toc-level toc-level-1"><li class="toc-item toc-item-h1"><a href="/wiki/literature.reproducible-research-methodological-principles-for-transparent-science#lab-notes-and-notebooks" class="toc-link toc-link-h1">Lab notes and notebooks</a><ol class="toc-level toc-level-2"><li class="toc-item toc-item-h2"><a href="/wiki/literature.reproducible-research-methodological-principles-for-transparent-science#brief-history-of-notes-from-a-western-point-of-view" class="toc-link toc-link-h2">Brief history of notes (from a Western point of view)</a></li><li class="toc-item toc-item-h2"><a href="/wiki/literature.reproducible-research-methodological-principles-for-transparent-science#plain-text-formatting-languages" class="toc-link toc-link-h2">Plain-text formatting languages</a></li><li class="toc-item toc-item-h2"><a href="/wiki/literature.reproducible-research-methodological-principles-for-transparent-science#search-tools-and-techniques" class="toc-link toc-link-h2">Search tools and techniques</a></li></ol></li><li class="toc-item toc-item-h1"><a href="/wiki/literature.reproducible-research-methodological-principles-for-transparent-science#computational-documents" class="toc-link toc-link-h1">Computational documents</a></li><li class="toc-item toc-item-h1"><a href="/wiki/literature.reproducible-research-methodological-principles-for-transparent-science#replicable-analysis" class="toc-link toc-link-h1">Replicable analysis</a></li><li class="toc-item toc-item-h1"><a href="/wiki/literature.reproducible-research-methodological-principles-for-transparent-science#real-life-reproducible-research" class="toc-link toc-link-h1">Real-life reproducible research</a><ol class="toc-level toc-level-2"><li class="toc-item toc-item-h2"><a href="/wiki/literature.reproducible-research-methodological-principles-for-transparent-science#data-hell" class="toc-link toc-link-h2">Data hell</a></li><li class="toc-item toc-item-h2"><a href="/wiki/literature.reproducible-research-methodological-principles-for-transparent-science#software-hell" class="toc-link toc-link-h2">Software hell</a></li><li class="toc-item toc-item-h2"><a href="/wiki/literature.reproducible-research-methodological-principles-for-transparent-science#numeric-hell" class="toc-link toc-link-h2">Numeric hell</a></li></ol></li><li class="toc-item toc-item-h1"><a href="/wiki/literature.reproducible-research-methodological-principles-for-transparent-science#additional-readings" class="toc-link toc-link-h1">Additional readings</a></li></ol></nav><p>I'm not a researcher but I've learned so many things from this course that are applicable to general workflows: organizing your digital files for easy retrieval, creating reproducible notes, and setting out to create reproducible studies with open for
|
||
|
|
In my desktop workflow, I've managed to create a generic desktop search for my projects which also serves as an format-agnostic option for personal braindump/wiki which I'll taking notes of it somewhere.
|
||
|
|
</p><p>This is the notes for <a href="https://www.fun-mooc.fr/courses/course-v1:inria+41016+self-paced/info">Reproducible research: methodological prinicples for transparent science</a>, an open course on how to create <a href="/wiki/2020-04-12-11-20-53">Reproducible research</a> and notes with various tools.
|
||
|
|
The course itself is divided into three paths for <a href="https://jupyter.org">Jupyter Notebooks</a>, R, and Org-Mode.
|
||
|
|
There is also a <a href="https://gitlab.inria.fr/learninglab/mooc-rr/mooc-rr-ressources">dedicated Git repo with additional resources</a> (most of them are in French but some come with English translations), be sure to check it out.
|
||
|
|
</p><p>Also take note some chapters of certain modules are skipped because I'm only targetting topics I'm not totally familiar with.
|
||
|
|
I may also extend the notes with my own research.
|
||
|
|
</p><h1 id="lab-notes-and-notebooks">Lab notes and notebooks</h1><p>This module also places importance on creating reproducible research notes that concerns everyone.
|
||
|
|
Specifically, it concerns preservation of values: labeling equipments, archiving notes with contextual data (e.g., dates), and creating a sketch or blueprints of the apparatus.
|
||
|
|
Anything to make it explicit as possible for other researchers who may want to take the mantle.
|
||
|
|
</p><h2 id="brief-history-of-notes-from-a-western-point-of-view">Brief history of notes (from a Western point of view)</h2><p>In a quick bullet form:
|
||
|
|
</p><ul><li><p>Clay tablets are the first recorded form of note-taking.
|
||
|
|
</p></li></ul><p>Then, wax tablets and scrolls superceded the clay tablets.
|
||
|
|
</p><ul><li><p>And then, codices (or books) existed and replaced the scrolls as a common option.
|
||
|
|
</p></li></ul><p>Along the fact that codices solve the common inconveniences from using a scroll (e.g., text navigation is difficult, no hard breaks for the text).
|
||
|
|
Furthermore, it enabled notes with navigational features such as adding table of contents, an index, and paragraph indications (such as rubrics).
|
||
|
|
</p><ul><li><p>Eusebius, a historian, created a document referred as the "Eusebian Canons" demonstrating the first use of cross-references.
|
||
|
|
</p></li><li><p>As the use of paper note-taking took off, more methods of note-taking has arised such as index notes, logbooks, and post-it notes each with their pros and cons.
|
||
|
|
</p></li><li><p>However, devices such as computers and digital tablets are becoming a viable option with the use of the right technology.
|
||
|
|
</p></li></ul><h2 id="plain-text-formatting-languages">Plain-text formatting languages</h2><p>Word processors such as Microsoft Word, Google Docs, and LibreOffice Writer enables you to create documents but it can only be opened by them.
|
||
|
|
</p><p>Plain-text files avert this problem that it can be opened by any text editors.
|
||
|
|
However, they lack the markup formatting (e.g., inserting text in boldface and italics, attaching files, inserting images and tables) that the formatted documents possess.
|
||
|
|
This is where markup languages such as HTML, Markdown, and PhpBB come into play giving us the ability to format our text.
|
||
|
|
However, certain markup languages like the aforementioned HTML can be hard-to-read.
|
||
|
|
</p><p>Thus, a new subclass of markup languages are born, lightweight markup languages.
|
||
|
|
It offers a content-oriented human-readable syntax in plain-text that we could open up in our text editors and format our content.
|
||
|
|
Examples include Markdown, Asciidoctor, Org-mode, and reStructuredText.
|
||
|
|
</p><h2 id="search-tools-and-techniques">Search tools and techniques</h2><p>I've created a dedicated note (see <a href="/wiki/2020-06-27-16-21-47">Apply search tools and techniques for your digital library</a>) on this as derived from this course.
|
||
|
|
</p><p>With digital files comes digital tools (and some techniques) to help us in searching for our notes.
|
||
|
|
These can vary from complex technologies to simple format-agnostic techniques that can make narrow searching an easy task.
|
||
|
|
</p><ul><li><p>Desktop search engines such as DocFetcher and Recoll are one of your most powerful options.
|
||
|
|
It can search through text and metadata quite fast.
|
||
|
|
</p></li><li><p>Simply creating a format-agnostic labelling system (e.g., delimiting your labels as <code class="inline-code">::LABEL::</code>) can tremendously help with searching through your notes.
|
||
|
|
This also helps in reducing information overload when searching with a broad term.
|
||
|
|
</p></li><li><p>For binary file formats like PNG, PDF, and MP4, you can embed file metadata instead.
|
||
|
|
There are free tools that lets you create those such as Exiftool and Exiv2.
|
||
|
|
It's an added bonus that some desktop search engines can also search metadata of binary files.
|
||
|
|
</p></li></ul><h1 id="computational-documents">Computational documents</h1><p>The researching process often face issues from its conception up to publication.
|
||
|
|
</p><p>Some controversial studies have been popping up revealing flaws in its computation, cherry-picking data, or simple bugs/error in its setup.
|
||
|
|
More factors include:
|
||
|
|
</p><ul><li><p>The usage on proprietary formats and services where the lack of promise of long-lasting storage is a valid concern.
|
||
|
|
Their free and open counterparts does not promise the same either but at least you could access the previous versions.
|
||
|
|
</p></li><li><p>Reliance on oversimplified graphical interfaces that hide computational details or rather the lack of logging explanations.
|
||
|
|
</p></li><li><p>Lack of backup systems, version control, and quality control that also degrades transparency.
|
||
|
|
</p></li><li><p>Lack of technical documentation which is against the rigorous and methodical nature of science.
|
||
|
|
</p></li></ul><h1 id="replicable-analysis">Replicable analysis</h1><ul><li><p>With reproducible documents come replicable analysis.
|
||
|
|
Obviously, the concept of research reproducibility is not valid until others can replicate the research.
|
||
|
|
</p></li><li><p>Acquired data from other sources shouldn't be edited "manually" (e.g., text editor, database editor) and everything regarding it would have to be modified with code.
|
||
|
|
This is especially important if you've spotted a missing chunk of the data.
|
||
|
|
</p></li><li><p>That said, missing and/or dubious data are normal due to lack of data collection or an error in data processing.
|
||
|
|
It should also be dealt with ourselves on how to handle missing or dubious data.
|
||
|
|
</p></li></ul><h1 id="real-life-reproducible-research">Real-life reproducible research</h1><p>With reproducible documents and replicable analysis, the rudimentary toolset needed for reproducible research is complete.
|
||
|
|
However, as always in the real world, there are challenges ahead.
|
||
|
|
</p><h2 id="data-hell">Data hell</h2><p>First, gathering data is often not of similar origins and nature.
|
||
|
|
Furthermore, the data is often not heterogenous which means we cannot easily establish the relationship among them.
|
||
|
|
</p><p>As much as text formats are an attractive option, there are some tradeoffs you need to keep in mind.
|
||
|
|
Data are often big and text generally consume more memory since text has to be converted into binary format.
|
||
|
|
If we want to take less, we can consider our data to be in binary in the first place.
|
||
|
|
</p><p>Binary formats are good for performance but there's a factor to consider it which is the <a href="/wiki/computing.endianness">Endianness</a>.
|
||
|
|
They can be read differently depending on the computer architecture so it is best practice to announce the endianness in your paper.
|
||
|
|
</p><p>Text formats, however, has the upper hand of easily adding <strong>metadata</strong> to our data which is a must for reproducible research.
|
||
|
|
To get around this solution, we could look into established binary data formats that attempts to tackle this problem.
|
||
|
|
It also has the advantage of using standardized tools that other researchers also use.
|
||
|
|
Examples include <a href="http://fits.gsfc.nasa.gov/">Flexible image transport system (FITS)</a> and <a href="https://www.hdfgroup.org/">Hierarchical data format (HDF)</a>.
|
||
|
|
</p><p>Data can be archived offline but it is hard to share and distribute the sources and the results to other researchers.
|
||
|
|
One can host the data themselves but the discoverability suffers which is not really in the spirit of research.
|
||
|
|
The professors recommend to utilize online archives such as <a href="https://zenodo.org/">Zenodo</a> and <a href="https://figshare.com/">Figshare</a> which solves the distribution and discoverability issues.
|
||
|
|
</p><h2 id="software-hell">Software hell</h2><p>Software can get complex at a fast rate when we try to scale up our data.
|
||
|
|
This is also added with the factor that software does not stand in the test of time, surprisingly.
|
||
|
|
</p><p>With our data and code becoming complex, the resources needed to calculate all of it is increasing as well.
|
||
|
|
Not to mention the longer (reproducible) notes which can make navigation a bit of a pain.
|
||
|
|
</p><p>Creating a well-structured document can help our readers but it does not avoid the problem when the document is becoming too long for an overview.
|
||
|
|
Certain notebooks like Org-mode enables folding of the document and only unfolding the sections that we want to see.
|
||
|
|
</p><p>Having a long reproducible document can also have a performance problem.
|
||
|
|
Certain options are trying to get around this:
|
||
|
|
</p><ul><li><p>Jupyter offer the option of delegating the calcuations to a supercomputer.
|
||
|
|
</p></li><li><p>As of 2021-03-31, the Emacs community is trying to bring the editor into native code which in turn improves the situation for Org-Mode. </p></li></ul><p>Another solution is introducing a <strong>workflow engine</strong> that takes a <strong>workflow</strong> as input.
|
||
|
|
A workflow is a language describes each step of the study into a digestible graph.
|
||
|
|
It lets you process data in different programming languages and execute them in a linear way to prevent side effects.
|
||
|
|
The process of creating reproducible documents can get complex therefore create a complex workflow but it has the added property of reusability for certain sections which then can be used by others.
|
||
|
|
Examples of a workflow engine include <a href="https://galaxyproject.org/">Galaxy</a> and <a href="https://cknowledge.io/">Collective Knowledge</a>.
|
||
|
|
</p><p>Lighter versions of workflow engines also exists.
|
||
|
|
Makefiles, in a way, describes the workflow so certain tools like <a href="https://dask.org/">Dask</a> and <a href="https://snakemake.readthedocs.io/en/stable/">Snakemake</a> tries to integrate with it.
|
||
|
|
</p><p>The professors recommend to think through the process before using a tool.
|
||
|
|
It is not bad to start with the notebooks first and as the study becomes increasingly complex, you can try to migrate to using a workflow.
|
||
|
|
</p><p>The problems of software doesn't end there, however.
|
||
|
|
For instance, under the popular software and libraries like R, <a href="https://www.scipy.org/">SciPy</a>, and <a href="https://matplotlib.org/">Matplotlib</a> are full of abstractions which can mean they use a lot of packages.
|
||
|
|
Most of the software should be able to let you know about the versions (and even the complete environment like the following R code block, for example).
|
||
|
|
</p><pre class="src-block"><code class="language-R">R.version
|
||
|
|
</code></pre><div class="exampe"> _
|
||
|
|
platform x86_64-pc-linux-gnu
|
||
|
|
arch x86_64
|
||
|
|
os linux-gnu
|
||
|
|
system x86_64, linux-gnu
|
||
|
|
status
|
||
|
|
major 4
|
||
|
|
minor 0.4
|
||
|
|
year 2021
|
||
|
|
month 02
|
||
|
|
day 15
|
||
|
|
svn rev 80002
|
||
|
|
language R
|
||
|
|
version.string R version 4.0.4 (2021-02-15)
|
||
|
|
nickname Lost Library Book
|
||
|
|
</div><p>Even then, specifying versions explicitly can only do so little since most of these libraries depend on another (SciPy, for example depends on C).
|
||
|
|
It could still "break" if your machine or the library has been updated which may or may not contain breaking changes.
|
||
|
|
</p><p>To get around this, we have to capture the environment of our code.
|
||
|
|
There are tools that specialize in this function.
|
||
|
|
</p><ul><li><p>Self-contained bundling tools like <a href="https://github.com/VIDA-NYU/reprozip">ReproZip</a> freezes the environment and share it with your colleagues.
|
||
|
|
However, if there is an issue in your code or document, you may have to rebuild the bundle.
|
||
|
|
</p></li><li><p>A more complete solution is a virtual machine (e.g., <a href="https://www.virtualbox.org/">VirtualBox</a>, <a href="https://www.qemu.org/">QEMU</a>) but it can be heavy in resources where certain factors are not important like the operating system or the hardware.
|
||
|
|
</p></li><li><p>Lighter alternatives to virtual machines like containers such as <a href="https://www.docker.com/">Docker</a> or <a href="https://singularity.lbl.gov/">Singularity</a> are more suitable for software environments.
|
||
|
|
They also offer mostly the same security as virtual machines in that the environment is isolated from the host meaning no system libraries or programs will be used;
|
||
|
|
you have to explicitly specify which depedencies are used.
|
||
|
|
</p></li><li><p>Certain package managers like <a href="https://nixos.org/">Nix</a> and <a href="https://guix.gnu.org/">GNU Guix</a> specialize in retrieving reproducible environments.
|
||
|
|
</p></li></ul><p>As mentioned before, software are fragile: they can easily evolve and break.
|
||
|
|
This is especially true for fast-moving software and libraries like Python 3 and even Org-mode by the fact that it goes through major changes 9 times.
|
||
|
|
These breaking changes can interrupt the workflow which is why it is important to look out for changes.
|
||
|
|
Another solution is to force some rulings such as coding exclusively in C or only use certain libraries and reimplement anything else.
|
||
|
|
Capturing the environments used for calculations is a matter of compromise and stability.
|
||
|
|
</p><p>Software can also be fragile that it can be easily deleted.
|
||
|
|
Just like how data has dedicated archives, certain platforms have dedicated to preserve software like <a href="https://www.softwareheritage.org/">Software Heritage</a>, <a href="https://hal.archives-ouvertes.fr">Hyper Articles en Ligne</a>, and <a href="https://archive.org/">Internet Archive</a>.
|
||
|
|
</p><h2 id="numeric-hell">Numeric hell</h2><p>In today's world where computers aids in research such as calculations, there are hidden factors looming in.
|
||
|
|
For example, representing floating points is particularly difficult and has resulted in certain quirks like the following code in Python.
|
||
|
|
</p><pre class="src-block"><code class="language-python">print(0.1 + 0.2)
|
||
|
|
</code></pre><pre class="fixed-width">0.30000000000000004</pre><p>
|
||
|
|
Not to mention, compilers can also affect the results by optimizing the code and may result in changing the order of the computations which is not a good thing for reproducible researches.
|
||
|
|
That said, compilers should be able to offer the option of configuring its compilation step such as disabling certain optimizations.
|
||
|
|
</p><p>Another problem arises is the parallel computation which supposed to make code execution faster.
|
||
|
|
Parallel computation mainly relies on the hardware and it can affect how things are when executed on different machines.
|
||
|
|
The study on how to minimize the impact is not yet fully realized.
|
||
|
|
</p><p>Last but not least are the problems when using a randomized number.
|
||
|
|
When it comes to generating random numbers, we are not using truly random numbers but pseudo-random numbers generated by deterministic algorithms.
|
||
|
|
One of many ways on how to generate 'random' numbers is taking an input referred as the seed.
|
||
|
|
The seed is then computated to get the first state, then the output of the first state is being computated to get the second, and so on.
|
||
|
|
</p><p>To make our research reproducible, we have to hardcode the seed or at least refer to it somewhere.
|
||
|
|
Here's an example of generating random numbers in Python with a fixed input.
|
||
|
|
</p><pre class="src-block"><code class="language-python">import random
|
||
|
|
|
||
|
|
random.seed(24)
|
||
|
|
for i in range(5):
|
||
|
|
print(random.random())
|
||
|
|
</code></pre><pre class="fixed-width">0.7123429878269185
|
||
|
|
0.8397997363118733
|
||
|
|
0.18259188695451745
|
||
|
|
0.9982826275179507
|
||
|
|
0.19409547872374744</pre><p>
|
||
|
|
If the same seed is used for pseudo-random number generating, we can then verify it.
|
||
|
|
</p><pre class="src-block"><code class="language-python">import random
|
||
|
|
|
||
|
|
random.seed(24)
|
||
|
|
assert random.random() == 0.7123429878269185
|
||
|
|
assert random.random() == 0.8397997363118733
|
||
|
|
assert random.random() == 0.18259188695451745
|
||
|
|
</code></pre><h1 id="additional-readings">Additional readings</h1><dl><dt><a href="https://www.fun-mooc.fr/courses/course-v1:inria+41016+self-paced/info">The course link</a></dt><dd><p>It is a great open course with great instructors, examples, and exercises to make the lessons stick with you.
|
||
|
|
</p></dd><dt><a href="https://news.ycombinator.com/item?id=22473209">Ask HN: how to take good notes</a></dt><dd><p>A general Q&A on how to take good notes and then some valuable insights.
|
||
|
|
</p></dd></dl></main></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"metadata":{"date":"\"2020-04-12 10:24:25 +08:00\"","date_modified":"\"2021-11-07 00:38:07 +08:00\"","language":"en","source":""},"title":"Reproducible research: methodological principles for transparent science","hast":{"type":"root","children":[{"type":"element","tagName":"nav","properties":{"className":"toc"},"children":[{"type":"element","tagName":"ol","properties":{"className":"toc-level toc-level-1"},"children":[{"type":"element","tagName":"li","data":{"hookArgs":[{"type":"element","tagName":"h1","properties":{"id":"lab-notes-and-notebooks"},"children":[{"type":"text","value":"Lab notes and notebooks"}]}]},"properties":{"className":"toc-item toc-item-h1"},"children":[{"type":"element","tagName":"a","properties":{"className":"toc-link toc-link-h1","href":"/literature.reproducible-research-methodological-principles-for-transparent-science#lab-notes-and-notebooks"},"children":[{"type":"text","value":"Lab notes and notebooks"}]},{"type":"element","tagName":"ol","properties":{"className":"toc-level toc-level-2"},"children":[{"type":"element","tagName":"li","data":{"hookArgs":[{"type":"element","tagName":"h2","properties":{"id":"brief-history-of-notes-from-a-western-point-of-view"},"children":[{"type":"text","value":"Brief history of notes (from a Western point of view)"}]}]},"properties":{"className":"toc-item toc-item-h2"},"children":[{"type":"element","tagName":"a","properties":{"className":"toc-link toc-link-h2","href":"/literature.reproducible-research-methodological-principles-for-transparent-science#brief-history-of-notes-from-a-western-point-of-view"},"children":[{"type":"text","value":"Brief history of notes (from a Western point of view)"}]}]},{"type":"element","tagName":"li","data":{"hookArgs":[{"type":"element","tagName":"h2","properties":{"id":"plain-text-formatting-languages"},"children":[{"type":"text","value":"Plain-text formatting languages"}]}]},"properties":{"className":"toc-item toc-item-h2"},"children":[{"type":"element","tagName":"a","properties":{"className":"toc-link toc-link-h2","href":"/literature.reproducible-research-methodological-principles-for-transparent-science#plain-text-formatting-languages"},"children":[{"type":"text","value":"Plain-text formatting languages"}]}]},{"type":"element","tagName":"li","data":{"hookArgs":[{"type":"element","tagName":"h2","properties":{"id":"search-tools-and-techniques"},"children":[{"type":"text","value":"Search tools and techniques"}]}]},"properties":{"className":"toc-item toc-item-h2"},"children":[{"type":"element","tagName":"a","properties":{"className":"toc-link toc-link-h2","href":"/literature.reproducible-research-methodological-principles-for-transparent-science#search-tools-and-techniques"},"children":[{"type":"text","value":"Search tools and techniques"}]}]}]}]},{"type":"element","tagName":"li","data":{"hookArgs":[{"type":"element","tagName":"h1","properties":{"id":"computational-documents"},"children":[{"type":"text","value":"Computational documents"}]}]},"properties":{"className":"toc-item toc-item-h1"},"children":[{"type":"element","tagName":"a","properties":{"className":"toc-link toc-link-h1","href":"/literature.reproducible-research-methodological-principles-for-transparent-science#computational-documents"},"children":[{"type":"text","value":"Computational documents"}]}]},{"type":"element","tagName":"li","data":{"hookArgs":[{"type":"element","tagName":"h1","properties":{"id":"replicable-analysis"},"children":[{"type":"text","value":"Replicable analysis"}]}]},"properties":{"className":"toc-item toc-item-h1"},"children":[{"type":"element","tagName":"a","properties":{"className":"toc-link toc-link-h1","href":"/literature.reproducible-research-methodological-principles-for-transparent-science#replicable-analysis"},"children":[{"type":"text","value":"Replicable analysis"}]}]},{"type":"element","tagName":"li","data":{"hookArgs":[{"type":"element","tagName":"h1","properties":{"id":"real-life-reproducible-research"},"children":[{"type":"text","value":"
|