Data Streaming

The data streaming system is being built as a requirement for HRPD-X and possibly SANDALS-II, separate (and complementary) to the MNeuData project. HRPD-X, SANDALS-II and other future instruments will not have a traditional DAE2/DAE3 as they are now obsolete. It is architecturally similar to the system that the ESS uses to take data (neutron events, sample environment, and anything else that we can throw into a streaming platform) and write it to file. Previously ISIS aided development to the ESS’ streaming pipeline as part of an in-kind project. The system will replace the ICP at ISIS.

In general this works by producing both neutron events and histograms, sample environment data, and other diagnostic data into a Kafka cluster and having clients (consumers in Kafka lingo!) that either view data live and act on it or write the data to a nexus file. Additional information can be found here and here.

All data is serialised into Flatbuffers blobs using these schemas - we have a tool called saluki which can deserialise these and make them human-readable after they’ve been put into Kafka.

Overall architecture is as follows:

This comprises of a few different consumers and producers:

  • azawakh - This is a soft IOC which provides areaDetector views, spectra plots and so on by consuming events from the cluster and displaying them over EPICS CA/PVA.

  • borzoi - This is also a soft IOC which is more or less a drop-in replacement for the ISISDAE. It provides an interface that several clients (ie. genie, ibex_bluesky_core, ibex_gui) talk to to start/stop runs and configure streaming electronics. borzoi will send UDP packets to the streaming electronics to configure it.

  • BSTOKAFKA - This configures the forwarder with the blocks that are in an instrument’s current configuration, as well as other PVs which will either get written to a file or archived for e.g. the log plotter.

  • forwarder - See Forwarding Sample Environment

  • filewriter - See File writing

The Kafka Cluster

There is a (non-production!) Redpanda Kafka cluster at livedata.isis.cclrc.ac.uk:31092. A web interface is available here.

Important

It was decided that we no longer maintain the Kafka cluster, and it will be handled by the the Flexible Interactive Automation team. See \\isis\shares\ISIS_Experiment_Controls\On Call\autoreduction_livedata_support.txt for their support information.

How to/FAQs

See Data streaming how-to guide

Run starts/stops

See Data streaming: run starts/stops

SE Data

See Forwarding Sample Environment

Neutron events and histograms

See Data streaming: Neutron events and histograms

Filewriting

See File writing