Interlock System

Hazards and degrees of malfunction

The HAICU trap contains several hazards to personnel and equipment safety. The main hazards to personnel are posed by the high power magnet power supplies, while the main hazards to equipment also include leaks/faults in the water cooling system leading to magnet overheating and local flooding.

For mitigation purposes it is useful to separate three different degrees of malfunction: noticeable, concerning, and catastrophic. While this latter term is a little dramatic, the real distinction is between gradual changes in behaviour (e.g. the cooling water flow gradually decreases over time, indicating material buildup in the lines), and drastic changes (e.g. water flow on one of the output lines drops to zero, indicating a burst tube, spraying water, or significant reduction one of the small lines, indicating a local clog).

Naturally these need to be handled differently:

Type A: Catastrophic changes are simple thresholds that can be handled entirely in hardware, and trigger and emergency shutdown (crowbar). They must NEVER go unnoticed.

Type B: Concerning changes are also simple thresholds, but typically have finer granularity and adjustability and can benefit from some simple logic, (e.g. "more than 3 flowmeters read low")

Type C: Noticeable changes require some level of analysis of observables over time, and are used to trigger gentle system ramp-downs to prevent more serious damage. These are a first line of defense and thus somewhat redundant. Should one get missed, overall safety (and most importantly personnel safety) is not compromised.

The interlock system only handles type A and B events, type C is handled in MIDAS.

Sensors

The main concern for hazardous events in this setup is a malfunction or inadequacy in the water cooling system, which could lead to flooding or overheating.

The system is monitored for malfunctions or abnormalities by a variety of sensing systems targeting different observables:

1. Flowmeters: these small paddle-wheel flowmeters monitor the flow in all individual cooling water branches and provide a quantitative readout to a microcontroller and from there to MIDAS (B, C)
2. Flow Switches: these bulkier flowmeters sit in main water lines and have in-built threshold detection that can directly connect to an interlock system (A)
3. Thermistors: small thermistors are attached in key locations of the magnets and provide a quantitative readout to a microcontroller, much like the flowmeters (B, C)
4. Thermal Switches: bi-metal switches that provide no quantitative information but can directly connect to an interlock system, they can be placed in strategic locations (A)
5. Thermal Monitor Switch: thermistor- or thermocouple-reading box that provides a quantitative readout and and internal threshold, low granularity (optional, A, C)
6. Level Switches: simple float switches that trigger if the water in the leak-catching enclosure rises too high (A)
7. Leak Sensors: resistive wetness-sensing switches that can connect directly to an interlock system and can be placed on the floor in strategic locations (A)

Output/Switching

The main two things the interlock system needs to control are the magnet power supplies and the main water supply. Most interlock conditions that do not indicate a leak will simply turn off the magnet power supplies to prevent overheating, while leak detection additionally closes the main water valve to minimize flooding, and shuts down other sensitive electronics.

Technically this is typically achieved by opening or closing a switch connecting two control pins on the device in question.

Modular Interlock System

Requirements

One big requirement for an interlock system like this is, that it fails safe, i.e. a loss of power in the interlock system itself, or the cutting or disconnection of a wire, leads to the safe locked condition, rather than the all clear. Additionally the core interlock system should require no programming to minimize the possibility of bugs.

In the HAICU interlock system, the interlock logic is modeled by relays in series. In order for the all clear to be given, voltage must pass through a series of normally open relays that each actively get switched closed by one of the above sensors. Each input has a manual bypass switch do allow for the manual deactivation of individual inputs.

Finally the interlock system must latch. This means that, once triggered, the system does not go back into the all clear condition without human intervention. This is achieved by a separate output module.

Modules

Since the number of inputs and outputs to the system may change, it was decided to go with a modular approach instead of a single PCB. The following sections describe the individual modules.

Schematics and kiCad projects for these modules can be found in the gitlab repo.

Concept scheme of the modular interlock. Columns can be used in parallel (green A configuration) or in series (red B configuration). Input modules can also be put above top dist board, but this is not currently used. Bottom dist is not actually a separate board, but a feature common to both output modules

Power

The power module generates the 3.3V needed for TTL communication and 2.2V for the LEDs out of 5V supply voltage. It also supplies 5V to the signal input of the following module.

Power module pinout for the modular interlock

Top distribution

The top distribution board distributes power and signals to columns of interlock modules.

Manual switches can configure the signal path. The signal flow schematic is printed on the board.

• SW2 decides whether input comes from "global interlocks", which can come from input modules above or from top dist modules to the left, switchable by SW3.
• SW1 decides whether the next top dist module to the right gets the same input as this column, or the forwarded output of this column (requires another switch in bottom dist).

Switch Input

The switch input module (schematic input_moduleA) provides two-pin connectors that any switch-style sensors can be connected to. The understanding is that a closed connection signifies a satisfied interlock condition. The large LED below each connector shines green if the condition is met or red if it is not. Below each connector there is a bypass switch, which allows to deactivate that input. The small yellow LED next to the switch is on for bypassed condition. Note the large LED will be green if bypassed.

There is a ribbon connector reporting the state of each channel as a TTL level to the watchdog device described below.

TTL Input

The TTL input module does not connect to standard switch-style sensors, but instead expects TTL 3.3V levels, which it receives from a Teensy board measuring temperatures or flow rates. It, too, has 5 red/green LEDs indicating interlock status of each channel and 5 bypass switches with yellow LEDs. In contrast to the switch input module, the red/green LED still reflects input status, even if the bypass is active.

This module has the same watchdog connector as the other input module.

Signal booster

The inexpertly designed concept of this modular interlock results in a drop in signal level if too many boards are used in series. This module brings it back up to 5V.

Bottom distribution

This is not actually a separate module, but a feature common to both output modules below.

SW1 switches between normal mode B, where the signal from the column above gets used to switch this output module and any below it, and daisychain mode A, where the signal from the column above gets sent back to the top to serve as the input of the next column, while this output module receives its input from the output module to its right.

Note 1: this requires SW1 in this column's top dist board to be in the daisychain position.

Note 2: only the top-most output module should ever have SW1 in position A, unless you really know what you're doing.

Individual Output

This module provides two-pin connectors to connect to individual devices. They behave just like if a switching sensor was directly plugged into the device, with two key differences:

1. they latch, so even if the condition that triggered the interlock gets remedied, they need to be reset manually, using a push button above each connector
2. they can be switched between normally open (NO) mode and normally closed (NC) mode. The latter is e.g. needed for the AE Techron power supply.

All channels trigger together but need to be reset separately. The red/green LED above each connector shows the state.

DSub Output

This module is specifically designed to connect to the Magnet Control Box to provide the interlock signals for the Sorensen SGX power supplies. It has a single LED and reset button for all outputs together.