Data streaming: topics
We have a number of topics per-instrument on livedata, the Kafka cluster we use.
Partition numbers are listed below. For variable partitions this will depend on the throughput requirements of the specific instrument.
_runInfo
partitions: 1
This contains run start and run stop flatbuffers blobs.
Flatbuffers schemas in this topic:
_events
partitions: variable
This contains data from event-mode events.
Flatbuffers schemas in this topic:
_sampleEnv
partitions: 1
This contains sample environment data forwarded from EPICS.
In a .nxs file this should end up in raw_data_1/selog/
Flatbuffers schemas in this topic:
_runLog
partitions: 1
This contains run metadata forwarded from the ICP.
In a .nxs file this should end up in raw_data_1/runlog/
Schemas in this topic match the ones in _sampleEnv
_monitorHistograms
partitions: variable
This contains monitor histograms. Flatbuffers schemas in this topic:
_detSpecMap
partitions: 1
This contains details of the detector-spectrum mapping. Flatbuffers schemas in this topic:
_areaDetector
partitions: variable
This is raw areaDetector data. It’s sent by this line in ISISDAE
_forwarderConfig
partitions: 1
This is the forwarder configuration, sent by BlockserverToKafka. Flatbuffers schemas in this topic:
_forwarderStatus
partitions: 1
This is the forwarder status topic which contains details about what PVs the forwarder is forwarding. Flatbuffers schemas in this topic:
_forwarderStorage
partitions: 1
This is the last known forwarder configuration, sent by BlockserverToKafka. This is for if the forwarder crashes, then it can quickly retrieve its last configuration. Flatbuffers schemas in this topic: