Data streaming topics

We have a number of topics per-instrument on livedata, the Kafka cluster we use.

_runInfo

This contains run start and run stop flatbuffers blobs.

Flatbuffers schemas in this topic:

• pl72 - Run start

• 6s4t - Run stop

_events

This contains data from event-mode events.

Flatbuffers schemas in this topic:

• ev44 - Events

_sampleEnv

This contains sample environment data forwarded from EPICS. In a .nxs file this should end up in raw_data_1/selog/

Flatbuffers schemas in this topic:

• f144 - All CA and Scalar PVA data

• se00 - PVA Arrays

• al00 - EPICS alarms

• ep01 - EPICS connection status

• un00 - EPICS units

_runLog

This contains run metadata forwarded from the ICP. In a .nxs file this should end up in raw_data_1/runlog/

Schemas in this topic match the ones in _sampleEnv

_monitorHistograms

This contains monitor histograms. Flatbuffers schemas in this topic:

• hs01 - Histograms

_detSpecMap

This contains details of the detector-spectrum mapping. Flatbuffers schemas in this topic:

• df12 - Detector-spectrum mapping

_areaDetector

This is raw areaDetector data. It’s sent by this line in ISISDAE

_forwarderConfig

This is the forwarder configuration, sent by BlockserverToKafka. Flatbuffers schemas in this topic:

• fc00 - Forwarder configuration

_forwarderStatus

This is the forwarder status topic which contains details about what PVs the forwarder is forwarding. Flatbuffers schemas in this topic:

• x5f2 - General status

_forwarderStorage

This is the last known forwarder configuration, sent by BlockserverToKafka. This is for if the forwarder crashes, then it can quickly retrieve its last configuration. Flatbuffers schemas in this topic:

• fc00 - Forwarder Configuration