Data streaming: topics

We have a number of topics per-instrument on livedata, the Kafka cluster we use.

Partition numbers are listed below. For variable partitions this will depend on the throughput requirements of the specific instrument.

_runInfo

partitions: 1

This contains run start and run stop flatbuffers blobs.

Flatbuffers schemas in this topic:

_rawEvents

Partitions: variable

This contains raw events (ev44 schema) and metadata (pu00) schema, as emitted by the streaming control board and the individual detector modules.

Flatbuffers schemas in this topic:

_events

partitions: variable

This contains aggregated event data. “Aggregated data” means that the metadata for each frame has been merged, and events concatenated from a large number of small messages to a smaller number of larger messages.

When sorted by message_id, this stream will contain, for each frame:

  • One pu00 message (containing vetos, period number, protons-per-pulse)

  • Zero or more ev44 messages containing the events

See documentation of kafka-event-aggregator for more details about precise format of the _events stream.

Flatbuffers schemas in this topic:

_sampleEnv

partitions: 1

This contains sample environment data forwarded from EPICS. In a .nxs file this should end up in raw_data_1/selog/

Flatbuffers schemas in this topic:

_runLog

partitions: 1

This contains run metadata forwarded from the ICP. In a .nxs file this should end up in raw_data_1/runlog/

Schemas in this topic match the ones in _sampleEnv

_monitorHistograms

partitions: variable

This contains monitor histograms. Flatbuffers schemas in this topic:

_detSpecMap

partitions: 1

This contains details of the detector-spectrum mapping. Flatbuffers schemas in this topic:

_areaDetector

partitions: variable

This is raw areaDetector data. It’s sent by this line in ISISDAE

_forwarderConfig

partitions: 1

This is the forwarder configuration, sent by Kafka forwarder configurer. Flatbuffers schemas in this topic:

_forwarderStatus

partitions: 1

This is the forwarder status topic which contains details about what PVs the forwarder is forwarding. Flatbuffers schemas in this topic:

_forwarderStorage

partitions: 1

This is the last known forwarder configuration, sent by Kafka forwarder configurer. This is for if the forwarder crashes, then it can quickly retrieve its last configuration. Flatbuffers schemas in this topic: