Session 1: Detailed introduction & aims of this working group – (26th 16:30/27th 08:00)
- The chair should briefly introduce the working group
- The working group is focused on Reproducibility, in measurement and analysis
- We all (hopefully) believe that reproducibility is important, however it is frequently overlooked
- We want to develop strategies to ensure that our measurements are conducted, reduced, and analysed in a reproducible fashion
- We have a presence on GitHub (https://github.com/reflectivity/reproducibility), which includes some “Goals”
- We want to generate a resource explaining why reproducibility in NR and XRR are important (session 2)
- We have started building a folder of “examples” of reproducible analysis
- We want a list of papers that represent a high standard of reproducibility (for others to aim for)
- Does anyone have other ideas for achievable goals?
- Define a list of requirements for a reproducible measurement standard and how it would be documented
- Define a list of requirements for the reproducible analysis of data
- Identify individuals who will continue to work on these projects (item one; move towards a definition of this measurement standard, item two; develop a standardised protocol for producing such an analysis “workflow”)
Session 2: How to communicate the importance of reproducibility? – (26th 22:30/28th 10:00)
- Reproducibility is important but showing if can be difficult
- Most users are too busy chasing the “science” to worry about showing their analysis is reproducible
- Few will worry about their data reduction being reproducible
- How can we show the importance of this
- Develop a list of reasons why reproducibility is important in neutron and X-ray reflectometry
- Give reasons why these aspects are important to users
- Identify individuals who will continent this project (ideal situation is a publication showing the benefits of reproducibility to the user and presents problems from a lack of it).
Session 3: What are our expectations for reproducibility? – (27th 11:00/27th 22:30)
- If we are going to evangelise reproducibility, we need to set reasonable expectations
- We should be the bar that we expect, for example when we peer review a publication?
- Should there be a “reproducibility expectation” when beamtime is applied for?
- The bar needs to be shared openly so that people can aim for it
- How should we advertise the bar (publication?)?
- Revise the lists from session 1 to find a “reasonable expectation”
- Detail why items have been added/removed
- Identify individuals who will continue (ideally resulting in a page for the website that gives our bare minimum to be considered “reproducible”.