DETMON

Background & Timeline

DETMON is a support system for use by the detector teams to monitor and view the status of the HV (High Voltage) crates that are in use across ISIS. A pilot system should be created before Cycle 2019-01, the final system does not at present have a dead line.

Equipment

The only equipment to consider for this system are CAEN crates (types of HV crate), however, there are about 30+ of them to consider.

System Requirements

Monitoring

The following items need to be monitored for each part of the system:

Crates

Parameter

Variable

What to record

Type

ON/OFF available in network

ConnStatus

Any variation

String

Crate fan status

HVFanStat

Any variation

String

Crate fan speed

HVFanSpeed

Any variation >+-5%

String

Which slots are used in the crate

Any variation

Model Name

ModelName

Any variation

String

Software Release

SwRelease

Any variation

String

Power Supply Status

HvPwSM

Any Variation

String

IP address

IPAddr

Any Variation

String

Boards slots

Slots

Any Variation

Integer

CPU Load

CPULoad

Any Variation>=+-???

Integer

Commands in the queue

CmdQueueStatus

Any Variation>=+-???

Integer

Memory Status

MemoryStatus

Any Variation >=+-???

Integer

Boards

Parameter

Variable

What to record

Type

Model

Model

Any Variation

String

Firmware release

Fmw Release

Any Variation

String

Serial Number

SerNum

Any Variation

Integer

Number of Channels

NrOfCh

Any Variation

Integer

Temperature

Temp

Any variation >= +-1 deg C

Float

Max Voltage

HVMax

Any Variation

Float

Board Status

BdStatus

Any Variation

Integer

Channels

Parameter

Variable

What to record

Type

Channel Name

Name

Any Variation

String

Voltage Setting

V0Set

Any Variation

Float

Current Setting

I0Set

Any Variation

Float

Voltage1 Setting

V1Set

Any Variation

Float

Current1 Setting

I1Set

Any Variation

Float

Rump Up speed

RUp

Any Variation

Float

Rump Down speed

RDWn

Any Variation

Float

Trip time

Trip

Any Variation

Float

Max Voltage Software limit

SVMax

Any Variation

Float

Monitor Voltage

VMon

Any Variation >= +-1 V

Float

Monitor Current

IMon

Any Variation >= +-2 uA

Float

Status

Status

Any Variation

Integer

Power Down Mode

PDwn

Any Variation

Integer

Notes for the solution

  • This monitoring would be for every crate in use at ISIS, and for any offline/test crates as well. Note that a crate has many boards, and a board has many channels – in the test system available at time of writing there is 1 crate, 8 cards, and 23 channels per card

  • The IOC already deals with the channel requirements, although we may need to verify the trip

  • The boards and crates may need some work to add them

  • This monitoring also has to be independent of the instrument status or behaviour

  • The rate of capture for this information can be relatively slow (1 second per item)

  • It is quite possible to link multiple IOCs to the same crate

  • Recording data to nexus files is acceptable – as such by creating a large number of blocks we should be able to run a script that is based on time and uses dummy neutron data – the nexus files can be stored, and the data regarding behaviour used to provide long term information. The log plotter will provide short term information for the numeric values. A python script could start a run and end and restart at predetermined times/after certain durations

  • Whilst IBEX and NeXus files provide a solution, the data is being logged into a database, and longer term, either logging to a different database or forwarding the data that way may be more suitable – this should be explored mid-long term, but IBEX at least provided short term

  • The number of blocks would need to be created by scripting, and the majority should likely be hidden from use on IBEX.

  • The sheer number of blocks did raise a concern as to whether or not IBEX could deal with the numbers, as such some testing and proving will be needed

    • to get an idea of the potential numbers of blocks, assume every instrument has one fully populated CAEN crate

    • if every channel is defined as a block (so that it can be recorded in a Nexus file), that’s

      • 35 x 8 x 23 = 6440 blocks

    • clearly 6440 blocks is an over-estimate, but it illustrates the scale of the problem

  • Currently to tick over log files at midnight a cron job is used to end and begin a run

Configuration set up

DETMON has a lot of blocks to set up and the CAEN crates create have a dynamic amount of boards and channels set up. This set up is scripted with scripts from the NDADETMON config area (in Python/inst). There are two scripts:

  1. The script record_channels.py searches for the connected boards and channels for the crates (see the script help option for arguments to pass). The output is a JSON file containing information on the crate name, boards and channels.

  2. Using this JSON file you can then create the relevant blocks.xml, groups.xml, block_config.xml and gwblock.pvlist files using the create_config_files_from_recorded_channels.py. Once produced put them in the config area alongside the meta.xml and other configuration-specific files. In the meta.xml ensure that the configuresBlockGWAndArchiver tag is set to true and the blockserver will handle the use of the files you have created.

This two step process lets you edit the intermediate JSON to include channels and crates which are not currently on the network.

Notes

Known issues