4. Use scipp
Status
Current
Context
We need to choose a library which helps us to transform “raw” neutron or muon data from the DAE, into processed quantities that we scan over.
Desirable features include:
Uncertainty propagation, following standard uncertainty propagation rules. While this could apply to any data in principle, it will be especially relevant for neutron/muon counts data.
Unit handling & conversions
Simple unit conversions, like microns to millimetres.
Neutron-specific unit conversions, like time-of-flight to wavelength
Ability to handle the typical types of data we would acquire from the DAE and process as part of a scan:
Histograms of neutron/muon counts
N-dimensional arrays
Event-mode data (in future)
Candidate solutions include:
mantid
scipp
uncertainties
numpy
+ home-grown uncertainty-propagation
Decision
Default to using
scipp
for most casesExplore using
mantid
via autoreduction APIs, where we need to do more complex reductions
Justification & Consequences
numpy
Using numpy
by itself is eliminated on the basis that we would need to write our own uncertainty-propagation code,
which is error prone.
numpy
by itself may still be used in places where uncertainty propagation is not needed.
uncertainties
The uncertainties
package tracks correlations so may have bad scaling on “large” arrays, where correlation matrices
can become large in some cases. Would need to be combined with another library, e.g. pint
, in order to also support
physical units. No neutron-specific functionality.
mantid
Mantid is not easily installable (e.g. via pip
at present).
While we have a way to call out to mantid via a REST API, initial tests have shown that the latency of this approach is around 15 seconds. This means it is unsuitable for many types of scans, for example alignment scans, where count times are far lower than 15 seconds.
However, for complex reductions, we should still consider the option of passing data out to mantid. This is especially true if reductions depend significantly on instrument geometry, on instrument-specific corrections, or on other details for which mantid is best-equipped to deal with.
Calling out to mantid via an API should also be considered if a reduction step may use significant compute resource.
scipp
scipp
will be our default way of taking raw data from the DAE and processing it into a scanned-over quantity.
However, in cases where the reduction is expensive (in terms of compute cost) or complex (either implementation, or in terms of required knowledge of geometry/instrument-specific corrections), then we should consider using mantid via the autoreduction API in those cases instead.