The diamond and the ring
The Diamond Light Source facility is used for protein crystallography and to see inside complex structures, and developing its systems presented complex challenges.
The Diamond Light Source, nestling in the Oxfordshire countryside, provides the UK with the brightest X-rays for world-class scientific research.
The control system challenges it presents are complex, and to understand them it is necessary to understand how Diamond operates. The process begins by producing 3GeV (three billion electron volts) electrons using a linear accelerator followed by a booster synchrotron. These electrons are then accumulated and stored in a 561m circumference Storage Ring (SR). When the stored electrons are deflected they lose energy, creating electromagnetic radiation (photons), called synchrotron light. Synchrotron light extends across the spectrum, from infrared, through visible light, to X-rays - and it is the X-rays that are most widely used in Diamond.
A structure supporting an array of magnets called an Insertion Device (ID) is used in the SR to deflect the electrons through a number of oscillations. This process is the source of the synchrotron light, which is then collimated and filtered, by a photon Beamline, as it travels to an experimental station where the science takes place.
The science performed at Diamond, which is located on the Harwell Science and Innovation Campus, is wide-ranging, covering as it does both physical and life sciences. Two important examples are protein crystallography and engineering science. In protein crystallography the X-rays are of similar wavelength to the atomic spacing of the sample so that when they pass through the sample they produce a diffraction pattern. This is then used to resolve the location of the atoms in the crystal, so revealing the structure of the protein. This is important in understanding biological processes and, in particular, as part of the drug discovery process. In engineering science one application is to use the penetrating power of the X-rays to see inside complex structures, and in particular to see systems as they operate. By using tomographic techniques 3D analysis of these structures is then possible.
The Diamond control system
The Diamond control system is responsible for the supervisory control of the accelerator and beamline equipment. It further provides local closed-loop control and protection of the equipment. This allows a centralised control room, manned by two staff to control the accelerator systems and each beamline to have independent control over its associated systems.
Given the physical size of Diamond and the large number sub-systems required, the Diamond control system is distributed around the site, with the controllers (known as IOCs, or Input/output controllers) as the interface to the technical systems. A Gigabit network is used to connect all the IOCs to operator consoles and control servers. There are over 600 IOCs in operation, 30 servers and around half a million process variables.
The IOCs are commonly VME based embedded systems supporting a diverse range of I/O modules including analogue, digital, serial communications, specialist precision motion control and high-speed data acquisition. For systems that require interlock protection, PLCs are used. The PLCs are interfaced to the IOCs, to allow status read back.
Control system software
The control system software is based on the open-source control system toolbox called EPICS (Experimental Physics and Industrial Control System). The EPICS toolkit implements a client-server architecture for distributed process control. In this methodology, each IOC acts as a server of real-time data that is described by a non-relational database. Each 'consumer' of the data acts as a client to these servers. The communications protocol, called Channel Access, uses UDP broadcasts to resolve process variables and TCP to establish connections. EPICS provides real-time visualisation, alarm handling and data archiving tools, together with interfaces to numerous programming environments. EPICS is the product of a collaboration of developers from similar projects around the world and is widely used on large physics projects and some industrial applications.
The positional stability of the electron beam determines the performance of Diamond and thus the quality of science that can be undertaken. The electron beam is typically 6.4µm high and 123µm wide, and to maintain good photon beam stability the electron beam needs to be maintained to within 10 per cent of these dimensions or 0.64µm or 12.3µm. This requires highly stable power supplies, precision positional measurements together with a beam position feedback system. The insertion devices, as the source of the photons, have demanding control requirements, and a gross misalignment of the electron beam can result in serious damage to the SR. All of these requirements are managed by the control system.
Magnet power supply control
The accurate positioning of the electron beam is dependent upon the accuracy and stability of the field generated by the SR magnets. This field in turn depends upon the stability of the 1,200 magnet power supplies. A current stability of better than +/- 10 ppm over an eight hour period is required.
These power supplies are switched-mode devices, using a DSP based digital controller to implement a PID control loop. The magnet current is measured by a DCCT (Direct Current Current Transformer) or precision shunt and a temperature stabilised 18-bit ADC. The DSP controller generates a modulation index to drive an FPGA which generates the PWM signals to drive the power stage.
The use of a digital controller means that a single solution can be used across a diverse range of power supply types, from 50W to 1MW. It ensures that new features and control techniques can be implemented as experience develops and problems are encountered. The digital controller also allows the tuning of internal control parameters via local or remote connections. In remote operation the power supplies are controlled via EPICS.
Measuring the position of the electron beam
Beam Position Monitors (BPMs) are located at 168 locations around the circumference of the SR to measure the electron beam position. The sensors are arranged as groups of four buttons, two above the electron beam and two below, and they measure the electric field produced by the electron beam as it passes by. From the relative strengths of the signals on the four buttons it is then possible to calculate the horizontal and vertical position of the electron beam with an accuracy of better than 1µm.
The circulating electrons are spaced by 2ns, so the signals produced by the sensors, have a 500MHz fundamental, but are under-sampled at 125MHz then down converted and decimated in successive stages by digital signal processing techniques in an FPGA. This produces a rich set of measurements, including information about the electron beam position on each turn around the SR through to a near DC value of the position. The turn-by-turn information is accessed as blocks of data, (of up to 200,000 points), and through a Fourier transformation gives spectral information about the beam motion, whereas the DC position is accessed as scalars and updated at 5Hz. An intermediate bandwidth data stream, at an update rate of 10kHz, is used for beam position feedback. Within each Beam Position Monitor is an embedded ARM processor board running an EPICS server, making the data available to the control system.
Electron beam position feedback
While considerable attention is put into maintaining the stability of the SR and systems that affect the position of the electron beam there remain external factors which disturb the beam position and have to be managed through a feedback system. These perturbations originate from environmental noise, thermal effects and even seasonal changes.
The feedback system performs a global orbit correction using the position from the 168 horizontal and vertical BPM positions to set 168 horizontal and 168 vertical corrector magnets at a 10kHz update rate and is an example of a cross directional control problem. A custom low-latency communication controller, implemented in VHDL, is used to move the horizontal and vertical positional data from the 168 BPMs to each of 24 VME computation nodes distributed around the SR. The network topology is structured as a two-dimensional torus, and gives a degree of resilience from failure of single or combinations of BPMs or links. Each of the computation nodes take the BPMs position values, and performs a mapping from BPM space to corrector magnet space. An Internal Model Controller regulator, together with a series of boundary checks, are then used to generate new demand values which are written to the 14 local power supply controllers. This provides around 20dB of suppression at 16Hz and 24Hz where most environment noise is coupled to the SR.
Protecting the SR from damage
Under a fault condition the electron beam can exceed its nominal orbit by over 1mm, and the photon beam could be directed onto uncooled (unprotected) surfaces where it would burn a hole through the stainless steel vessel in under 10msec. To prevent this, a Machine Protection System (MPS), provides a fast sub 1msec, electron beam 'dump'. Interlocks into the MPS include 336 invalid-orbit interlocks calculated in the BPMs and other critical interlocks when components are exposed to high photon beam intensities. A further 456 slow interlocks from water flow and temperature sensors are processed through PLC sub-systems and feed into the MPS. All interlocks originate around the 561m circumference SR and are grouped together locally, and then to a star point where a global interlock is produced to enable the SR operation. The MPS beam dump signal is also used to generate a post-mortem event, which creates a log of the orbit inside each BPM. This log is then analysed to understand the cause of the beam dump.
An insertion device (ID) consists of a pair of horizontal beams, above and below the electron beam axis, on which are mounted arrays of permanent magnets. The beams are motorised and when the distance between the two arrays is closed to a working gap of approximately 5mm around the electron beam, they create an intense magnet field and thus the required photon output. To achieve the required performance the magnet array parallelism must be maintained to around 2µm. However, when closed to the operating gap, some IDs can develop forces of up to 11t. The control system is required to maintain the gap and parallelism under these varying forces. The IDs use four servo motors, two per beam, with a VME based motion control card performing closed-loop position control. A protection mechanism is required to prevent damage under fault conditions. Each of the 15 installed IDs can be scanned through a number of gaps by its associated beamline, using the EPICS control system.