aaaffiliationtext: Institute for Astronomy, University of Hawai’i at Manoa,
640 N Aohoku Pl, Hilo, USAbbaffiliationtext: TNO, Stieltjesweg 1, NL-2628 CK
Delft, Netherlandsccaffiliationtext: Observatoire de la Cote d’Azur, 96 Bd de
l’Observatoire, 06300 Nice, Franceddaffiliationtext: UC Santa Cruz, 1156 High
St, Santa Cruz CA, USA

Further author information: (Send correspondence to R.Z.)
R.Z.: E-mail<EMAIL_ADDRESS>Telephone: +1 309 531 3014
M.C.: E-mail<EMAIL_ADDRESS>Telephone: +1 808 932 2317

# In-lab and On-sky Closed-loop Results of Adaptive Secondary Mirrors with
TNO’s Hybrid Variable Reluctance Actuators

Ruihan Zhang  Max Baeten  Mark R. Chun  Ellen Lee  Michael Connelley  Olivier
Lai  Stefan Kuiper  Alan Ryan  Arjo Bos  Rachel Bowens-Rubin  Philip M. Hinz

###### Abstract

We performed closed-loop lab testing of large-format deformable mirrors (DMs)
with hybrid variable reluctance actuators. TNO has been developing the hybrid
variable reluctance actuators in support for a new generation of adaptive
secondary mirrors (ASMs), which aim to be more robust and reliable. Compared
to the voice coil actuators, this new actuator technology has a higher current
to force efficiency, and thus can support DMs with thicker facesheets. Before
putting this new technology on-sky, it is necessary to understand how to
control it and how it behaves in closed-loop. We performed closed-loop tests
with the Shack-Hartmann wavefront sensor with three large-format deformable
mirrors that use the TNO actuators: DM3, FLASH, and IRTF-ASM-1 ASM. The
wavefront sensor and the real-time control systems were developed for the NASA
Infrared Telescope Facility (IRTF) and the UH 2.2-meter telescope ASMs. We
tested IRTF-ASM-1 on-sky and proved that it meets all of our performance
requirements. This work presents our lab setup for the experiments, the
techniques we have employed to drive these new ASMs, the results of our
closed-loop lab tests for FLASH and IRTF-ASM-1, and the on-sky closed-loop
results of IRTF-ASM-1 ASM.

###### keywords:

Adaptive optics, AO control theory, adaptive secondary mirrors

##  1 INTRODUCTION

Adaptive optics (AO) technology has benefited the field of astronomy for the
past few decades providing higher image resolutions compared to traditional
telescopes. [1] Pockets of air in the atmosphere have different temperatures
and thus varying indices of refraction. Light from astronomical objects
entering the earth’s atmosphere gets distorted by the atmosphere as wind blows
and convection takes place. An AO system uses its wavefront sensor (WFS) to
measure the distortion caused by the atmosphere, calculates a set of
corresponding commands with its real-time controller (RTC), and applies the
commands to a deformable mirror (DM) that physically bends to negate the
distortions caused by the atmosphere. This makes the incoming wavefront more
coherent, creating sharper images at the focal plane. To keep up with the
atmosphere, AO systems typically run with loop rates on the order of 1000 Hz.

An adaptive secondary mirror (ASM) can be used as the DM in an AO system. ASMs
are situated inside the telescope, replacing the conventional static secondary
mirror. The major advantages to using an ASM compared to a traditional AO
system that wholly sits behind the telescope are: when conjugated to the
ground layer, ASMs are capable of improving image quality at all wavelengths
over a large field of view based on studies done by Gemini[2] and Subaru[3].
Especially at Maunakea, the ground layer is found to be confined and is the
major contributor to optical turbulence [4, 5, 6]. In addition, the ASM is
part of the telescope and is seen by all instruments on the telescope, all the
instruments on the telescope may be benefited by the ASM. ASM-based AO systems
have low thermal emissitivity and high optical throughput because extra relay
optics are not needed to perform the AO correction.

ASMs have proven successful at various observatories, such as the MMT [7],the
Large Binocular Telescope [8, 9], the VLT [10], and the Magellan
Telescope[11]. These ASMs have realized the promises of providing correction
over a large field of view with low thermal emissivity and high throughput.
They all used the same actuator technology – the voice coil actuators. These
ASMs are complex, costly, and they have been difficult to operate and maintain
[12]. The choice of voice coil actuators drives many characteristics of these
ASMs. With a relatively small force output ($\sim$1N) output, a think and
delicate facesheet is required which is difficult to manufacture and handle.
The thin facesheets drive the ASMs to high actuator densities so the
facesheets do not sag between actuators. The weak force output also means the
actuators are working at non-linear regimes, so capacitive sensors are
required for every single actuator to know the actual shape of the ASM. These
ASMs use a lot of energy and generate a large amount of heat that requires a
complex cooling system. A new generation of ASMs is being developed using the
hybrid variable reluctance (HVR) actuators invented by TNO [13, 14, 15]. The
HVR actuators have a force output that is about an order of magnitude larger
than the voice coil actuators, allowing ASMs to have thicker facesheets and be
operate in the linear regime. Operating in the linear regimes means the ASM
can be used in open-loop mode with no sensor feedback. Furthermore, the
facesheet is directly bonded to each actuator which improves robustness and
ease of operation.

The first ASM to go on-sky using the HVR actuator technology is the IRTF-ASM-1
at the NASA Infrared Telescope Facility (IRTF). It is a 3-ringed 36 actuator
(6+12+18) ASM with a diameter of 245 mm. Its convex facesheet is 3.3 mm thick
and its radial actuator pitch is 37 mm [16, 17]. IRTF-ASM-1 uses the UH88
style actuators, which are also used in FLASH [15] (a flat DM with 19
actuators serving as a prototype for IRTF-ASM-1) and the UH88 ASM [13]. Before
IRTF-ASM-1 goes on-sky, it is imperative to characterize the ASM in the lab.
During this process, we developed and assembled the DM electronics, the WFS,
and the RTC to drive IRTF-ASM-1. We also experimented with multiple different
controllers for driving the AO loop to achieve optimal dynamic performance for
this style of ASMs. Due to the availability of the different ASMs we were
working with, we first started the WFS and RTC integration in July 2023 with
DM3. DM3 is a prototype DM using the same HVR actuator as UH88 but with a
different way of mounting. The actuators are mounted in rows where all
actuators in one row are integrated in an aluminum beam. This makes only grid
layouts possible. The UH88 style actuators are stand-alone units that can be
mounted in any desired orientation. Initial testing with DM3 was useful
because it uses the same electronics as FLASH and IRTF-ASM-1, and it is also a
flat mirror with the same diameter as FLASH. Then, using the same setup, we
performed closed-loop tests with FLASH once it arrived in Hawaii in December
2023. Finally, we closed the AO loop with IRTF-ASM-1 using the same RTC and
WFS in the lab after its arrival in February 2024. IRTF-ASM-1 saw its on-sky
first light in April 2024. In this work, we present the lab setup for FLASH
and its Shack-Harmann WFS design (Section 2.1). We also show the testing
instrument optical layout we used for IRTF-ASM-1 in-lab and on-sky (Section
2.2). In Section 3, we showcase the closed-loop performance of FLASH and IRTF-
ASM-1 using different loop controllers. Lastly, we present the on-sky
dynamical performance of IRTF-ASM-1 in Section 4.

##  2 Lab SETUP

###  2.1 Wavefront sensor design and optical setup for FLASH

Figure 1: Optical setup for testing FLASH. The major components are: A) R-cube
illuminator, B) 5-in lens/beam expander, C) FLASH, D) Single conjugate AO
(SCAO) Shack-Hartmann WFS, E) HASO WFS. Inside the SCAO WFSD, the optical
components are: a. collimating lens, b. 50:50 beamsplitter, c.d. relay lenses,
e. lenslet array, f.g. reimaging lenses.

The lab setup we used to test FLASH is shown in Figure 1. The R-cube
illuminator is a commercial product from Phasics that takes 625 nm light from
a fiber source, collimates it and sends it out with a cube beamsplitter. The
R-cube serves as the light source in all of the lab setups. The R-cube
illuminates the DM via a beam expander. The beam is expanded to a 127 mm
diameter beam by an in-hand f/5 lens. This illuminates most of FLASH (152 mm
diameter).

After the light hits FLASH, it reflects back through the system and gets
collimated again by the collimating lens at the entrance of the WFS tube. Then
half of the remaining light transmits through the beamsplitter and gets
resized by lenses (c) and (d) to illuminate a desired number of subapertures
on the lenslet array. The lenslet array sits at a plane conjugated to the DM.
The spot pattern formed by the lenslet array is then resized by lenses (f) and
(g) to get the desired number of pixels per subaperture on the detector (FLIR
Blackfly CMOS camera) that is at the end of the WFS tube. The SCAO Shack-
Hartmann WFS has 12x12 subapertures and 10 pixels per subaperture. All the
lenses in it are off the shelf achromatic doublets to mitigate chromatic
aberration since we are taking the WFS on-sky. The optics for the WFS are in
1-inch lens tubes cut to specific lengths, which makes the assembly of the WFS
very easy and precise. All parts in the WFS are on-shelf commercial
components. This demonstrates an economic route to custom built WFSs that are
suitable to a given camera/AO system that has few constraints, such as space.

Half of the light coming back from FLASH gets directed to the HASO WFS (a
commercial product from Imagine Optic). The DM is conjugated to the HASO WFS.
The HASO WFS is a high-order Shack-Hartmann system with a detector size of 5.3
mm x 7 mm using 50 x 68 subapertures. The purpose of the HASO WFS is to
monitor the higher order aberrations that are not seen by the 12x12 WFS and
not corrected with FLASH. We use the 12x12 WFS to drive the DM under test.

Other essential parts to the AO system are the driving electronics and the
RTC. The DMs are driven by custom voltage to current amplifiers made by TNO. A
commercial Linux machine is used as the RTC that communicates between the DM
and the WFS using in-house RTC code.

###  2.2 Design of the wavefront sensor package for IRTF-ASM-1

Figure 2: Scaled schematic diagram of the Test Instrument for IRTF-ASM-1.

There are more components to the wavefront sensor package for IRTF-ASM-1
(Figure 2) because it is used for both in-lab and on-sky testing. In addition
to the optical components described in Section 2.1, two major components added
are: 1) a point spread function (PSF) imager and 2) a pupil steering
mechanism.

The f/38 Cassegrain focus of IRTF is about 700 mm behind the telescope
interface plate. We designed the Test Instrument with these conditions in
mind. The dichroic behind the instrument entrance reflects visible light to
the arm with the WFSs at the bottom of Figure 2 and transmits the IR
wavelengths to the arm with the IR camera at the top left. The IR camera is a
QHY990 camera sensitive from 400 nm to 1.7 microns. A 1.62 micron narrow
bandpass is mounted in the camera to limit the light to longer wavelengths.
The IR camera serves as a scoring camera that allows us to image the PSF and
quantify the AO system performance. A neutral density (ND) filter wheel sits
in front of the IR camera making sure the IR camera does not get saturated.
The telescope focus falls on two locations in our instrument: one at the IR
camera, the other at the field stop.

A pupil steering mirror is located after the field stop in the wavefront
sensor path to actively steer/maintain the pupil image on the wavefront sensor
lenslet array. We added this mechanism to compensate for flexure between the
telescope (in particular the secondary) and our instrument that would
misregister the ASM and the WFS. The pupil steering mirror is used in closed-
loop with a pupil imaging camera.

When the wavefront sensor package is used in the lab, the R-cube is used as
the illumination source. To go from in-lab to on-sky, we only need to change
the collimating lens in the wavefront sensor tube since the returning beam in
the lab is f/13, whereas light from the telescope would be f/38. Using the
setup described above, we performed closed-loop experiments for IRTF-ASM-1.

##  3 LAB PERFORMANCE AND CONTROL SERVO TUNING

###  3.1 system identification

In order to find the optimal closed loop controller for an ASM and its
corresponding components (WFS & RTC), the frequency response of the system is
required. We tested the FLASH and the IRTF-ASM-1 system by sending white noise
actuator commands in open-loop to a single actuator while recording slope
values measured by the wavefront sensor. The slope measurements were converted
into an output actuator response for that actuator using the interaction
matrix of the system. By comparing the input noise signal and the measured
system response, we obtain the single input single output frequency response
of the system in both phase and amplitude for each actuator.

Figure 3: System frequency response for FLASH actuator 0.

The identification measurements on actuator 0 of FLASH, as shown in Figure 3
were measured with the system running at 895 Hz. The other actuators in FLASH
all show similar responses and are therefore omitted. A model has been fitted
to this frequency response function. We find it has a 2.5-sample delay that
can be attributed to 0.5 step of sampling time, 1 step for image readout, and
another step to perform calculations and apply actuator commands. Reducing
this delay or improving the loop rate would improve the closed loop
performance. The limiting factor of the loop rate is the readout time of the
SCAO WFS camera. As an initial step, we sacrificed the dynamic range of the
camera pixels going from 12-bit to 10-bit to obtain a higher loop speed of 998
Hz. Therefore the IRTF-ASM-1 system frequency response in Figure 4 is measured
at a higher loop rate of 998Hz. The open loop frequency response for 3 out of
36 actuators is shown as they are all very similar. There is a variation at
low frequencies at a level of about 15%. Furthermore the phase drop for some
actuators is higher than for others. Both of these variations are within
predicted margins and are likely caused by fluctuations in actuator efficiency
and in facesheet stiffness. The average low frequent gain per ring is 0.986,
1.00 and 1.036 respectively for the inner, middle and outer ring of actuators.
The higher gain of the outer ring is the result of the lower facesheet at the
outer edge. The faster phase drop of actuator 0 is the result of small
variations in the actuators, if we select three different actuators than an
actuator from the other two rings might show the fastest drop. All other
actuators show phase drops within the same range.

Figure 4: System frequency response for IRTF-ASM-1 actuator 0 (inner ring), 6
(middle ring) and 18 (inner ring)

###  3.2 modal vs zonal control

Figure 5: Bode diagram of IRTF-1 for Zernike mode Z3-Z37 with a verification
with single sines (red dots) for Zernike mode Z5, primary oblique astigmatism

A typical orthogonal basis set used by the community is the Zernike
polynomials basis [18]. Using influence functions of IRTF-1, we can convert
the zonal commands to modal commands and vice versa. We repeated the system
identification measurements by injecting noise sequentially on the first 36
Zernike modes, Z3 to Z37 (in Noll’s ordering [18]). As shown in Figure 5, they
exhibit very similar dynamic behaviors since they share the same style of
actuators, the same facesheet thickness, and were driven with the same WFS and
RTC. Furthermore, there are almost no resonance frequencies that can be
attributed to Zernike Modes as those resonances have higher frequencies than
what can be measured with the current loop rate. At 400 Hz there is a very
minor resonance, as measured on Z03, Tilt. To verify the frequency responses
measured by injecting noise, a few verification measurements are performed.
For primary Z5 oblique astigmatism, 6 measurements are performed where a
single frequency (1, 3, 10, 30, 100, and 200 Hz) sine wave is injected and its
response is shown in red dots.

###  3.3 control tuning

The first consideration for controlling the ASM is which basis to use. Common
control schemes are to close the loop for each actuator or for each mode. The
closed loop control scheme tested on IRTF-1 consists of 36 single input single
output feedback controllers for each actuator. Traditionally AO systems use a
simple Proportional-Integral (PI) controller. In this project we designed
controllers not limited to PID using classical control theory. The WFS slopes
are multiplied by the interaction matrix to obtain 36 actuator values. We want
to keep those values as close to zero to obtain a flat wavefront. The feedback
controller takes as input the error signal defined as the setpoint minus the
actuator value (with the setpoint set to zero).

To tune the controllers of the 36 control loops a first approach is to tune
one controller that is used for all 36 control loops. It is possible to fine
tune the controller for each actuator for optimal performance but the
performance gain is minimal because the variations between actuaotor responses
are small. To tune the controller the frequency responses of all actuators are
taken into account, although in this paper we present only 3 actuators for
clarity. In an iterative process, the parameters of the controller are changed
while the closed loop, the open loop, and the sensitivity transfer functions
are plotted predicting the performance of the closed loop controller. The
design goals are to achieve a high open loop cross over frequency and high
closed loop -3db bandwidth while meeting the robustness margins. Two
controllers are designed for which the parameters are given in Table 1.

Table 1: Zonal controller C1 and C6 parameters and performance metrics. Performance metrics are provided as a range for the 36 actuotors. |  | C1 | C4 | C6 |
---|---|---|---|---|---
Controller: |  |  |  |  |
gain |  | 0.37 | 0.3 | 0.74 | [-]
integrator | zero | 110 | 110 | 90 | [Hz]
Lead filter | pole | 44 | 100 | 120 | [Hz]
| zero | 400 | 600 | 950 | [Hz]
2nd order Low pass filter | pole | 400 | 400 | NA | [Hz]
| damping | 0.7 | 0.7 | NA | [-]
Performance: |  |  |  |  |
open loop crossover |  | 34 - 47 | 25 - 31 | 44 - 58 | [Hz]
closed loop bandwidth (-3dB) |  | 81 - 137 | 47 - 61 | 95 - 163 | [Hz]
modulus margin $<$ 6dB |  | 4.4 - 6.8 | 3.5 - 4.7 | 4.8 - 7.0 | [dB]
30°$<$ phase margin $<$ 60° |  | 53 - 65 | 45 - 58 | 38 - 50 | [°]
gain margin $>$6dB |  | 5.62 - 8.9 | 10.9 - 13.6 | 5.7 - 9.0 | [dB]

To ease the switching of designed closed loop controllers a generic approach
is taken where a state space controller is implemented that consists of four
matrices A,B,C and D. Therefore no implementation of lead filters or low pass
filters are required as all controller components together are converted to a
single state space representation. Such a representation is also discretized
for the specific loop rate.

Figure 6: Control tuning bode plots, based on the measured system response P
and the designed controller C. The bode plots are provided (with the same
color) for three actuators 0, 6 and 18. The performance metrics from Table 1
are indicated with circles.

Error transfer functions should be examined with the input power spectrum
taken into consideration since the end goal is to minimize residual error,
which is the integral of the product of the input power spectrum and the error
transfer function. The error that we are trying to correct with AO systems is
caused by the atmosphere, which has a power spectrum that can be modeled using
the Kolmogorov turbulence theory [19]. Most power in atmospheric turbulence is
in the lower frequencies, and its strength falls with a -$\frac{5}{3}$ power-
law as a function of frequency. An example of on-sky turbulence profile is
shown in Figure 7. Knowing that most power to be corrected is in the lower
frequencies and there is not much to correct for in the higher frequencies, as
long as the lower frequencies are well attenuated, there is margin for
amplification of the higher frequencies as shown in the error transfer
functions. Therefore, the error transfer function that provides optimal AO
performance should have maximum attenuation at the lower frequencies and a
high 0dB crossover frequency. There is always a part of the sensitivity that
peaks above 0dB meaning that those frequencies are amplified. The maximum
amplification should be as low as possible.

Figure 7: Example power spectrum of the atmospheric turbulence profile
measured using the IRTF-1 AO system in open-loop at IRTF on Maunakea, April
26, 2024. The overall shape resembles typical atmospheric turbulence profiles.
However, the sources that cause the spikes are not yet identified.

###  3.4 performance validation

Another common metric widely used in the AO community that characterizes the
performance of a DM is the settling time of its actuators. In other words, how
fast does it take for an actuator to go from position A to position B. This
can be shown in a step response curve of the actuator, which represents how
the actuator behaves when it is commanded to move in the system. We examined
the step response of actuator 0 in IRTF-1 in open-loop (Figure 8). To study
the open-loop response, we used the RTC to send direct actuator commands to
actuator 0, then the WFS records the optical response caused by moving the
actuator that deforms the surface of IRTF-1.

Unlike some DMs that can move to position almost instantaneously and may be
under-damped, such as the Micro-electro-mechanical (MEMS) DMs, the DMs made
with the TNO UH88-style actuators need to be pushed into place, making it an
over-damped system with a long open loop settling time. Closing the loop with
a controller preferably with a lead filter included improves the settling time
significantly. The settling time is defined here as the time used to reach and
stay within a range of 20% of the final value. In the first measurement, solid
lines, the step is directly applied to the actuator. The second measurement,
dashed lines, uses the same step but its fed through an open loop lead filter.
Knowing these UH88 style actuators have a large inertia to start moving, the
lead filter is designed to push the actuators harder initially to start the
ASM moving faster and eases off the push in later steps to prevent excessive
overshoot, thus achieving a shorter settling time. By using a lead filter, we
shortened the settling time of actuator 0 by about 60% with respect to a
direct open-loop step. This was tested with three step sizes [0.0010, 0.0033,
0.0050] where the full dynamic range is -1 to 1 in arbitrary command space.
These UH88-style actuators have a very large linear dynamic range of about 14
microns [15, 16]. Only small incremental steps would be used when using the
ASMs, thus the small magnitude of the test step sizes. The measured relative
amplitude is half of the input amplitude because the output is normalized. For
the direct step measurements, actuator 0 took [9, 10, 10] time steps to
settle, whereas when a lead filter was used, only [4,4,4] time steps were
needed. Since the system runs at 997 Hz, each time step is about 1 ms. In
other words, in small amplitudes, the settling time of actuator 0 can be
improved to about 4 ms by using lead filters. This explains why lead filters
make an important component to the state space controllers for our systems,
allowing them to outperform the PI controller. A few other actuators were also
tested and they exhibit near identical behaviors to actuator 0.

We also investigated the closed-loop response of the actuators, which also
includes the response time of the RTC and is representing the response time of
the whole AO system (Figure 9). We acquired the data by introducing slope
offsets in the WFS, which suddenly introduces a large error in the closed-loop
AO system. The system sees the error and moves the DM to compensate for the
injected error. Using the PI controller, the system is under-damped, so the
error signal oscillates for a long time before it settles near 0. On the other
hand, when using state space controllers like C1 and C4 (a controller very
similar to C6), the system is optimally damped. C1 and C4 both improve the
closed-loop settling time by about 84%. This also explains why the state space
controllers show much better performance in error transfer functions.

Figure 8: Open-loop step response of actuator 0 in IRTF-1. A lead filter
significantly fastens the open-loop settling time of the actuator by about
37%. Figure 9: closed-loop step response of actuator 0 in IRTF-1. The two
controllers, C1 and C4, both contain lead filters that improve the closed-loop
settling time of the actuator by about 84%.

We compared the performance of a typical PI controller and multiple state
space controllers on the FLASH AO system. We took AO telemetry in closed-loop
using the different controllers and compared them to the open-loop data, which
is essentially white noise in camera signals since there is little turbulence
in the lab. The ratio between the power spectra of closed-loop data and open-
loop is the error transfer function or sensitivity function of the system. It
showcases what frequencies are being attenuated by the closed-loop system
(below the 0 dB line) and which are amplified. Looking at the error transfer
functions of FLASH using the different controllers, controller C1, which is
the best performing state space controller we tested, works much better than
the conventional PI controller (Figure 10).

Figure 10: Error transfer functions of FLASH using different controllers.
Frequencies below 0 dB (the gray dotted line) are being attenuated, and
anything above it is amplified. The green dashed is the predicted performance
of the system using controller C1, which matches well with the measured
performance in the lab.

Here we conclude the dynamical testing of FLASH and IRTF-1 in the lab. The
first observation with the IRTF-1 was conducted on April 24, 2024, marking the
success of the first on-sky use of an ASM based on the TNO actuators.

##  4 ON-SKY PERFORMANCE OF IRTF-ASM-1 ON DYNAMICAL RESPONSE

Figure 11: On-sky error transfer functions of IRTF-ASM-1 using different
controllers. The data was taken on the nights of April 25/26, 2024.

On the nights of April 25 and 26, we got to test the effect that different
controllers have on the performance of the AO system (Figure 11). Similar to
the lab results, the PI controller, which we ran using a proportional gain of
0.3, performed less well compared to the state space controllers C1 and C6.
The 0 dB bandwidth measured on-sky for the PI controller, C1, and C6 are
correspondingly [25, 35, 44] Hz. On the other hand, C6 seems to perform much
better than C1 compared to their lab results. We examined the open-loop data
used to generate the error transfer functions for C1 and C6, they were very
similar and could not account for such large discrepancies. Therefore, the
cause of such improvement needs to be further investigated. We plan to take
more data with the different controllers under similar seeing conditions to
study their effects since what we are currently have might be a single-time
incident. One caveat to the on-sky error transfer functions is that unlike in
the lab, the atmospheric turbulence is ever-changing. The AO telemetry we
acquired include error signals that the wavefront sensor detects for 27000
steps running at 998 Hz, which corresponds to data of about 30 s. Each dataset
was taken about 20 s after another due to the finite data transfer rate. Even
though the open-loop telemetry and the closed-loop telemetry were taken one
after the other with little time difference between the two, the open-loop
data does not fully represent the atmospheric profile during the time that the
closed-loop data was taken. This could cause inaccuracies in the on-sky error
transfer functions and may be the reason that the error transfer functions are
not converging to 0 dB at high frequencies. We could improve the accuracy of
the on-sky error transfer functions by taking a set of closed-loop data
sandwiched by two sets of open-loop telemetry and take the average of the two
datasets as the open-loop spectrum.

Figure 12: A long exposure (300 s) image of Arcturus (in an arcsinh scaling)
at 1.62 microns with a Strehl ratio of 40% running IRTF-ASM-1 in closed-loop
(Lee et al. 2024)[16].

During the first observing run with IRTF-ASM-1, the best long-exposure we
obtained was a diffraction limited image of Arcturus (Figure12). The PSF has a
Strehl ratio of 40% at the wavelength of 1.62 $\mu$m. It is a 300 s exposure
made with 10 frames of 30 s exposures stacked together. The C6 state space
controller was used while the image was taken. We look forward to further
exploring the capabilities of ASMs with the TNO HVR actuators with future
observing runs using IRTF-ASM-1 and the deployment of the University of Hawaii
2.2-meter ASM.

##  5 DISCUSSION AND CONCLUSION

We have successfully demonstrated the closed-loop AO performance of ASMs made
with the novel HVR actuator technology both in-lab and on-sky. The IRTF-ASM-1
on NASA IRTF is capable of delivering diffraction limited images in H-band
(1620 nm) with a Strehl raio of 35-40% under sub-arcsecond seeing conditions.
An important lesson we have learned through the closed-loop dynamical testing
of these ASMs is that a slightly more sophisticated control servo than a PI
loop (e.g. a leadfilter) improves the performance of the system.

Given the current frequency response of our AO system with IRTF-ASM-1, the
RTC, and the as-built WFS, we were able to model the system as mentioned in
Section 3.1. We have identified the WFS camera (off the shelf CMOS USB3
camera) as the limiting factor in the bandwidth of the system. If we change to
a more capable camera, we predict the 0dB bandwidth will improve to 50-85Hz
(depending on a assumptions on the camera). For the UH88ASM we plan to use the
camera/wavefront sensor in the Robo-AO-2 system[20].

###### Acknowledgements.

This work is funded by an National Science Foundation (NSF-1910552) award. The
authors would like to thank TNO and the NASA IRTF for the opportunity and
assistance in making this project a success. Furthermore, we would like to
thank The University California Santa Cruz for supplying a slumped facesheet
and Huygens Optics for they spherical polishing of said mirror shell. Special
thanks to LEMO for providing us connectors within a short timeframe, and BUMAX
for helping us out solving a potential galling problem due to a manufacturing
error. The on-sky experiments were performed at the Infrared Telescope
Facility, which is operated by the University of Hawaii under contract
80HQTR19D0030 with the National Aeronautics and Space Administration. The
authors want to express our gratitude towards the engineering team and the
summit crew at IRTF for making the observing runs go smoothly. We also wish to
recognize and acknowledge the very significant cultural role and reverence
that the summit of Maunakea has always had within the Native Hawaiian
community. We are most fortunate to have the opportunity to conduct
observations from this mountain.

## References

  * [1] Beckers, J. M., “Adaptive Optics for Astronomy: Principles, Performance, and Applications,” Annual Review of Astron and Astrophysis 31, 13–62 (Jan. 1993).
  * [2] Szeto, K., Andersen, D., Crampton, D., Morris, S., Lloyd-Hart, M., Myers, R., Jensen, J. B., Fletcher, M., Gardhouse, W. R., Milton, N. M., Pazder, J., Stoesz, J., Simons, D., and Véran, J.-P., “A proposed implementation of a ground layer adaptive optics system on the Gemini Telescope,” in [Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series ], McLean, I. S. and Iye, M., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 6269, 626958 (June 2006).
  * [3] Minowa, Y., Koyama, Y., Ono, Y., Tanaka, I., Hattori, T., Clergeon, C., Akiyama, M., Kodama, T., Motohara, K., Rigaut, F., d’Orgeville, C., Wang, S.-Y., and Yoshida, M., “ULTIMATE-Subaru: enhancing the Subaru’s wide-field capability with GLAO,” in [Advances in Optical Astronomical Instrumentation 2019 ], Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 11203, 112030G (Jan. 2020).
  * [4] Roddier, F. J., Cowie, L. L., Graves, J. E., Songaila, A., McKenna, D., Vernin, J., Azouit, M., Caccia, J. L., Limburg, E. J., Roddier, C. A., Salmon, D. A., Beland, S., Cowley, D. J., and Hill, S., “Seeing at Mauna Kea: a joint UH-UN-NOAO-CFHT study,” in [Advanced Technology Optical Telescopes IV ], Barr, L. D., ed., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 1236, 485–491 (July 1990).
  * [5] Marks, R. D., Vernin, J., Azouit, M., Briggs, J. W., Burton, M. G., Ashley, M. C.B., and Manigault, J. F., “Antarctic site testing - microthermal measurements of surface-layer seeing at the south pole,” Astron. Astrophys. Suppl. Ser. 118(2), 385–390 (1996).
  * [6] Chun, M., Wilson, R., Avila, R., Butterley, T., Aviles, J.-L., Wier, D., and Benigni, S., “Mauna Kea ground-layer characterization campaign,” Monthly Notices of the Royal Astronomical Society 394, 1121–1130 (04 2009).
  * [7] Wildi, F. P., Brusa, G., Lloyd-Hart, M., Close, L. M., and Riccardi, A., “First light of the 6.5-m MMT adaptive optics system,” in [Astronomical Adaptive Optics Systems and Applications ], Tyson, R. K. and Lloyd-Hart, M., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 5169, 17–25 (Dec. 2003).
  * [8] Riccardi, A., Xompero, M., Briguglio, R., Quirós-Pacheco, F., Busoni, L., Fini, L., Puglisi, A., Esposito, S., Arcidiacono, C., Pinna, E., Ranfagni, P., Salinari, P., Brusa, G., Demers, R., Biasi, R., and Gallieni, D., “The adaptive secondary mirror for the Large Binocular Telescope: optical acceptance test and preliminary on-sky commissioning results,” in [Adaptive Optics Systems II ], Ellerbroek, B. L., Hart, M., Hubin, N., and Wizinowich, P. L., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 7736, 77362C (July 2010).
  * [9] Esposito, S., Riccardi, A., Fini, L., Puglisi, A. T., Pinna, E., Xompero, M., Briguglio, R., Quirós-Pacheco, F., Stefanini, P., Guerra, J. C., Busoni, L., Tozzi, A., Pieralli, F., Agapito, G., Brusa-Zappellini, G., Demers, R., Brynnel, J., Arcidiacono, C., and Salinari, P., “First light AO (FLAO) system for LBT: final integration, acceptance test in Europe, and preliminary on-sky commissioning results,” in [Adaptive Optics Systems II ], Ellerbroek, B. L., Hart, M., Hubin, N., and Wizinowich, P. L., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 7736, 773609 (July 2010).
  * [10] Biasi, R., Andrighettoni, M., Angerer, G., Mair, C., Pescoller, D., Lazzarini, P., Anaclerio, E., Mantegazza, M., Gallieni, D., Vernet, E., Arsenault, R., Madec, P. Y., Duhoux, P., Riccardi, A., Xompero, M., Briguglio, R., Manetti, M., and Morandini, M., “VLT deformable secondary mirror: integration and electromechanical tests results,” in [Adaptive Optics Systems III ], Ellerbroek, B. L., Marchetti, E., and Véran, J.-P., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 8447, 84472G (July 2012).
  * [11] Close, L. M., Males, J. R., Morzinski, K. M., Esposito, S., Riccardi, A., Briguglio, R., Follette, K. B., Wu, Y.-L., Pinna, E., Puglisi, A., Xompero, M., Quiros, F., and Hinz, P. M., “Status of MagAO and review of astronomical science with visible light adaptive optics,” in [Adaptive Optics Systems VI ], Close, L. M., Schreiber, L., and Schmidt, D., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 10703, 107030L (July 2018).
  * [12] Christou, J. C., Brusa, G., Guerra, J. C., Lefebvre, M., Miller, D., Rahmer, G., Sosa, R., and Wagner, M., “Living with adaptive secondary mirrors 365/7/24,” in [Adaptive Optics Systems IV ], Marchetti, E., Close, L. M., and Vran, J.-P., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 9148, 91480F (Aug. 2014).
  * [13] Chun, M. R., Ryan, A., Zhang, R., Kuiper, S., Ackaert, G., Baranec, C., Baeton, M., Bos, A., Bowens-Rubin, R., Dekker, B., Dungee, R., Gupta, T., Hinz, P., Jonker, W., Kamphues, F., Lai, O., Lu, J., Maniscalco, M., Monna, B., Nair, M., Nigenhuis, J., Priem, H., and Vogel, P.-A., “Progress on the University of Hawaii 2.2-meter adaptive secondary mirror,” Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series (2022).
  * [14] Jonker, W. A., Vogel, P.-A., Horst, R. t., Vu, A. T., Kuiper, S., and Kamphues, F., “Hot forming of a large deformable mirror facesheet,” Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series (2022).
  * [15] Bowens-Rubin, R., Hinz, P., Kuiper, S., and Dillon, D., “Performance of large-format deformable mirrors constructed with hybrid variable reluctance actuators II: initial lab results from FLASH,” in [Techniques and Instrumentation for Detection of Exoplanets X ], Shaklan, S. B. and Ruane, G. J., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 11823, 118231R (Sept. 2021).
  * [16] Lee, E., Chun, M. R., Lai, O., Zhang, R., Baeten, M., Bos, A., Kidron, M., Kamphues, F., Kuiper, S., Jonker, W., Connelley, M. S., Rayner, J., Ryan, A., Hinz, P. M., Bowens-Rubin, R., Lockhart, C., and Kelii, M., “First laboratory and on-sky results of an adaptive secondary mirror with TNO-style actuators on the NASA Infrared Telescope Facility,” SPIE, International Society for Optics and Photonics (2024).
  * [17] Bos, A., Kuiper, S., Kidron, M., Dekker, B., Baeten, M., Salcedo, E. V., Vermeulen, R., Kuijt, J., Boot, K., Venrooy, B. v., Buuren, R. v., Kamphues, F., Jonker, W. A., Maniscalco, M., Chun, M. R., Connelley, M. S., Vleggaar, J. J., and Hinz, P. M., “Status and first results of the NASA IRTF Adaptive Secondary Mirror,” SPIE, International Society for Optics and Photonics (2024).
  * [18] Noll, R. J., “Zernike polynomials and atmospheric turbulence.,” Journal of the Optical Society of America (1917-1983) 66, 207–211 (Mar. 1976).
  * [19] Kolmogorov, A. N., “The Local Structure of Turbulence in Incompressible Viscous Fluid for Very Large Reynolds Numbers,” Proceedings of the Royal Society of London Series A 434, 9–13 (July 1991).
  * [20] International Society for Optics and Photonics, [Commissioning results from the Robo-AO-2 facility for rapid visible/near-infrared AO imaging ], SPIE (2024).


11institutetext: Lister Hill National Center for Biomedical Communications
22institutetext: National Heart, Lung, and Blood Institute
U.S. National Institutes of Health (NIH), Bethesda, MD, 20892

# Trilateral Attention Network for Real-time Medical Image Segmentation

Ghada Zamzmi 11 Vandana Sachdev 22 Sameer Antani 11

###### Abstract

Accurate segmentation of medical images into anatomically meaningful regions
is critical for the extraction of quantitative indices or biomarkers. The
common pipeline for segmentation comprises regions of interest detection stage
and segmentation stage, which are independent of each other and typically
performed using separate deep learning networks. The performance of the
segmentation stage highly relies on the extracted set of spatial features and
the receptive fields. In this work, we propose an end-to-end network, called
Trilateral Attention Network (TaNet), for real-time detection and segmentation
in medical images. TaNet has a module for region localization, and three
segmentation pathways: 1) handcrafted pathway with hand-designed convolutional
kernels, 2) detail pathway with regular convolutional kernels, and 3) a global
pathway to enlarge the receptive field. The first two pathways encode rich
handcrafted and low-level features extracted by hand-designed and regular
kernels while the global pathway encodes high-level context information. By
jointly training the network for localization and segmentation using different
sets of features, TaNet achieved superior performance, in terms of accuracy
and speed, when evaluated on an echocardiography dataset for cardiac
segmentation. The code and models will be made publicly available in TaNet
Github page.

###### Keywords:

Semantic Segmentation Localization Echocardiography

##  1 Introduction

Medical image segmentation divides the images into different semantic or
anatomical regions, based on which important quantitative indices can be
extracted for disease diagnosis. It has been widely applied to various medical
imaging modalities including X-ray [8], ultrasound [19], and magnetic
resonance imaging [6], to delineate anatomical regions for further
quantitative analysis. Current medical image segmentation approaches can be
divided into global-based and region-based. Global-based approaches use the
entire image while the region-based approaches focus on a specific region of
interest (ROI). The majority of region-based approaches use two separate
networks for ROI localization stage and semantic segmentation stage. However,
combining localization and segmentation into a single network is efficient as
it prevents unnecessary repetitions of training separate networks in isolation
[30, 31, 10, 5]. Further, jointly training the network for localization and
segmentation improves the performance as it optimizes segmentation based on
localised ROIs; e.g., recent works [9, 23] demonstrated that the incorporation
of single pathway [9] and dual pathway [23, 9] attention mechanisms into
segmentation networks allowed to localize the target ROI while suppressing
irrelevant parts, leading to increased segmentation accuracy.

The spatial and global (context) information are both essential to the
semantic segmentation task [24]. Earlier methods for spatial information
extraction relied on hand-designed features generated by classical
descriptors; e.g., local binary pattern [12, 21]) and atlases [4, 7].
Different hand-designed methods can capture unique statistical, geometrical,
or textural features from the images and are optimized for performance and
power efficiency [17]. In recent years, deep learning-based methods for
segmentation (e.g., fully convolutional networks [16], U-Net [22]) have shown
superior performance as they capture rich low-level details at different
levels of abstraction. These deep learning methods can be divided, based on
their architectures, into: dilation architecture and encoder-decoder
architecture. The dilation architecture uses dilated convolutions to preserve
high-resolution feature representations. Examples of current state-of-the-art
architectures with dilated convolutions include DeepLabv3 [29] and PSPNet
[25]. The encoder-decoder architecture has downsampling and upsampling
components and uses skip connections to capture wider context and recover the
high-resolution feature representations. Examples of well-known encoder-
decoder architectures include U-Net [22] and SegNet [2].

Although these architectures achieved excellent segmentation performance, the
dilation convolutions and skip connections increase the computational
complexity and memory overhead, leading to a slow inference speed. To solve
this issue, Yu et al. [27] proposed a two-pathway architecture, known as the
Bilateral Segmentation Network (BiSeNet), to speed up the inference time. The
first pathway has wide channels and shallow layers to capture the spatial
information while the second pathway provides a large receptive field to
capture wider context. The two pathways concurrently generate the feature
representations, which significantly increases the efficiency. As demonstrated
in [28], this conceptual design is significantly faster than both dilation and
encoder-decoder architectures.

In this work, we extend BiSeNet and propose an efficient end-to-end Trilateral
Attention Network (TaNet) for real-time medical image localization and
segmentation. Our specific contributions are summarized as follows. First, we
integrate into BiSeNet a third handcrafted pathway ($HP$) with handcrafted-
encoded convolutional kernels. This pathway can be used to extract unique set
of morphological, statistical, or textural attributes. Second, we integrate a
Spatial Transformer Network (STN) as a module to direct the attention of the
segmentation task to specific ROIs while learning the context relationship
between them. Third, we present a strategy to jointly train the entire network
(localization + segmentation) for ROI-based semantic segmentation. TaNet
achieves impressive real-time results when evaluated on the echocardiography
dataset for cardiac segmentation. It outperforms BiSeNet [28], FCN [16], U-Net
[22], and recent methods for echocardiography segmentation [20, 1].

##  2 Echocardiogram Dataset

We used XXX echocardiogram dataset, which contains 300 videos (parasternal
long axis view or PLAX) recorded from 300 patients who were referred for
echocardiographic examination in the Clinical Center at XXX. The segmentation
ground truth (GT) masks, which were provided by experts, contain semantic
labels for the following cardiac regions: left ventricle (LV), left atrium
(LA), right ventricular outflow tract (RV), interventricular septum (IVS), and
posterior wall (PW). Examples from our dataset are provided in the
supplementary materials. In this work, we divided (subject-wise) the entire
dataset into training (80%) and testing set (20%). The training set is further
divided into training and validation using 10-fold cross validation. To
enlarge the training set, we applied the following operations: random rotation
($-15^{\circ}$ to $+15^{\circ}$), horizontal and vertical shift
($-0.25$,$0.25$), scale $[0.75,1,1.25]$, and horizontal and vertical flip. We
resized (bicubic interpolation, OpenCV-python) all the images into $512\times
512$.

##  3 Trilateral Attention Network (TaNet)

Figure 1 depicts our proposed network for ROI-based medical image
segmentation. TaNet integrates localization into the segmentation by adding
the STN module and uses three pathways for segmentation. The entire network is
trained end-to-end using a joint learning approach to prevent unnecessary
training repetitions, and allows the network to focus on specific ROIs (i.e.,
cardiac regions) while learning the context relationships among them. In the
traditional approach, localization and segmentation are trained separately,
which adds unnecessary training repetitions and disregards the relationship
between both stages.

Figure 1: Overview of TaNet with its components: STN for ROIs localization and
segmentation backbone with 3 pathways, spatial (detail) path ($SP$),
handcrafted path ($HP$), global path ($GP$). The numbers in the cubes are the
size ratios to the resolution of the input. In the fusion module, we aggregate
the feature maps from all pathways. Down, up, and $\otimes$ indicate
downsampling, upsampling, and element-wise product, respectively. The
localization module focuses the segmentation attention to different ROIs
($X$). The output of the network is an image with the segmented ROIs and the
background.

###  3.1 ROIs Localization

STN [13] is a differentiable module that can be inserted anywhere in
convolutional networks and used for applying spatial transformations to the
input image or feature map. It is differentiable in the sense that it computes
the derivative of the transformations in the module allowing to learn the
gradients of the loss with respect to the module parameters. This allows the
training of STN with any convolutional network end-to-end. STN consists of a
localization network ($L$), a grid generator ($G$), and a sampler. The
localization network learns the spatial transformation parameters ($\theta$).
The grid generator creates, based on $\theta$, the coordinates where the input
map should be sampled to obtain the transformed map. Finally, the sampler
produces the desired transformed map.

In medical images, it is common that the target ROI occupies a relatively
small portion of the image [11, 15, 3]. Hence, considering the entire image
for segmentation would add noise caused by irrelevant portions in the
background and lead to the segmentation network being biased toward the
background. In this work, we integrate STN and use it as a method for focusing
the attention of the segmentation on a specific ROI while suppressing the
potential irrelevant portions and learning the context relationship among
different ROIs. We hypothesize that the integration of STN would improve the
segmentation performance.

####  3.1.1 Localization Network

Prior to the use of localization network, we use FCN-8 [16, 26] model for
coarse segmentation. Providing a coarse segmentation of different ROIs allows
the localization network to 1) generate the transformation parameters
($\theta$) for these regions and 2) learn the context relationship among them.
The output of FCN-8 is an image with coarse semantic labels corresponding to
the cardiac regions (ROIs). Given an input image $X\in\mathbb{R}^{C\times
H\times W}$, where $C$, $H$, and $W$ represent the channels, width, and
height, respectively, the output of the coarse segmentation can be expressed
as:

$z=FCN(X)$ (1)

where $z\in\mathbb{R}^{H\times W}$. The coarse mask ($z$) is then sent to the
localization network ($L$) to generate the spatial transformation matrices
($\theta$) for $N$ cardiac regions (ROIs):

$\theta=L(z)$ (2)

where $\theta\in\mathbb{R}^{N\times 2\times 3}$; here $N=5$ as there are five
cardiac regions as shown in Figure 1. As for the localization network ($L$),
we used a simplified version of VGG16 [26] that has 8 convolutional layers and
a final regression layer to generate $N\times 2\times 3$ spatial
transformation matrix ($\theta$). $L$ outputs the spatial transformation
matrix ($\theta$) as shown in Figure 1. For each ROI, $\theta$ is defined as
follows:

$\theta=\begin{bmatrix}s_{x}&0&t_{x}\\\ 0&s_{y}&t_{y}\end{bmatrix}$ (3)

where $s_{x}$, $s_{y}$, $t_{x}$, and $t_{y}$ parameters, which are learned by
$L$, allows ROIs scaling and translation.

####  3.1.2 Grid Generator & Bilinear Sampler

Given $\theta\in\mathbb{R}^{N\times 2\times 3}$, the relevant parts of the
image (i.e., $ROI_{i}$, $i\in\\{1,2,..,N\\}$) are sampled into a sampling grid
$G$ of pixels $G_{i}=(x_{i}^{t},y_{i}^{t})$ to form an output feature map
$V\in\mathbb{R}^{C\times H^{\prime}\times W^{\prime}}$, where $C$,
$H^{\prime}$, and $W^{\prime}$ are the grid’s number of channels, height, and
width, which is the same in the input and output. The pointwise transformation
is computed as:

$\begin{pmatrix}x_{i}^{s}\\\
y_{i}^{s}\end{pmatrix}=\theta(G_{i})=\theta\begin{pmatrix}x_{i}^{t}\\\
y_{i}^{t}\\\ 1\end{pmatrix}$ (4)

where $(x_{i}^{s},y_{i}^{s})$ are the source coordinates in the input,
$(x_{i}^{t},y_{i}^{t})$ are target coordinates of the grid in the output
feature map, and $\theta$ is given in equation 3. To preform the spatial
transformation, the input and the sampling grid ($\theta(G)$) are sent to a
bilinear sampler to produce the output map $V\in\mathbb{R}^{C\times
H^{\prime}\times W^{\prime}}$. Since the bilinear sampling is differentiable,
it allows the gradients loss to flow back to the sampling grid coordinates,
and therefore to the transformation parameters $\theta$ and the localisation
network ($L$). This process allows the localization of relevant ROIs in the
input image. Note that although this work focuses on five ROIs ($N=5$), any
region can be easily added or removed by adjusting $\theta$.

###  3.2 ROIs Segmentation

We use three parallel pathways: handcrafted pathway ($HP$), spatial (detail)
pathway ($SP$), and global pathway ($GP$).

####  3.2.1 Handcrafted Pathway ($HP$) and Spatial Pathway ($SP$)

Depending on the medical imaging modality, the regular convolutional layers
can be replaced by handcrafted encoded convolutional layers. These handcrafted
encoded layers can extract a unique set of statistical, geometrical, or
textural features. In this work, we propose to replace the regular
convolutional layers with local binary encoded (LBP) convolutional layers. LBP
is an effective texture descriptor for summarizing the image’s local spatial
structure [18]. The traditional LBP achieved excellent performance for texture
segmentation and analysis in different medical imaging modality (e.g., [12,
21]) due to its robustness to illumination changes and ability to
differentiate small differences in texture and topography [18]. The
performance of LBP relies heavily on different parameters including the base,
pivot, and order. In this work, we propose to integrate LBP kernels to extract
rich set of textural features and learn the optimal set of LBP parameters.
Specifically, we utilized LBP encoded convolutional layers [14] to obtain
unique textural features at different levels of abstraction.

Similar to [14], the LBP encoded convolutional layer consists of a set of $m$
predefined fixed LBP filters that are used to generate $m$ difference maps.
These maps are then activated using a non-linear activation (i.e., ReLU)
function to generate the bitmaps. Finally, these $m$ bitmaps are combined
linearly by $m$ learnable weights. As demonstrated in [14], the number of
learnable parameters in LBP encoded convolutional layer is significantly
smaller (up to 169 times smaller) than the regular convolutional layer. As
shown in Figure 1, $HP$ has three LBP encoded layers. Each layer includes a
LBP convolution with a kernel size of 3 and a stride of 1. We hypothesize that
$HP$ pathway would capture unique textural characteristics of the medical
images and lead to better overall performance. In parallel to $HP$, $SP$ is
designed to capture rich low-level details from the images at different levels
of abstraction. This pathway contains three convolutional blocks, each
containing a $3\times 3$ convolutional layer with stride of $2$ followed by
batch normalization and ReLU activation. The number of filters in the first,
second, and third blocks are $64$, $64$, and $128$, respectively.

####  3.2.2 Global Pathway ($GP$)

As the receptive field has a great impact on segmentation, we used a
lightweight model (i.e., $Xception$) for fast-downsampling of the feature map
to obtain a sufficient receptive field and encode high level context
information. Then, we attached a global average pooling on the tail of the
lightweight model to provide the maximum receptive field with global context
information. Finally, the output of the global average pooling is up-sampled
as shown in Figure 1.

####  3.2.3 Fusion Module

The feature representations of the pathways are complementary and encode
different information; i.e., $HP$ (textural features), $SP$ (rich low-level
details), and $GP$ (high-level context). To combine these features, we
concatenate the output features from the pathways and then use batch
normalization to balance their different scales. We then pooled the
concatenated features into a single feature vector.

###  3.3 Training

We trained TaNet in two stages: pre-training and fine-tuning.

In the first stage, we pre-trained the coarse segmentation and the
localization network ($L$) as follows. First, we pre-trained the coarse
segmentation module (FCN-8) with a batch size equal to $32$, a learning rate
of $1\times 10^{-3}$ that is reduced when the loss plateau. We used Adam to
minimize the CrossEntropy loss between GT masks and the predicted coarse
segmentation. Then, we used the output of the coarse segmentation as inputs to
the localization network ($L$). The localization ($L$) is trained with a batch
size equal to $32$ and $1\times 10^{-3}$ learning rate. $L$ aims to optimize
the loss (Smooth L1) between $\theta$ and $\bar{\theta}$ (GT). In the second
stage, we loaded the pre-trained parameters from the previous stage and fine-
tuned the entire network. The entire network is trained for $100$ epochs with
a batch size equal to $32$. We used $1\times 10^{-3}$ learning rate and Adam
optimizer to minimize the following loss function:
$L_{TaNet}=\frac{1}{N}\sum_{i}^{N}L_{SM}(\tilde{z_{r}},\hat{J_{r}})$, where
$L_{SM}$ is the softmax loss, $\tilde{z_{r}}$ represents the pixel-wise
prediction for $ROI_{r}$, $r\in\\{1,2,..,N\\}$, $\hat{J_{r}}$ represents the
corresponding ground truth, and $N$ represents the total number of regions.

Figure 2: Segmentation of posterior and septal walls using BiSeNet and TaNet.
a) PW using BiSeNet, b) PW using TaNet, c) IVS using BiSeNet, and d) IVS using
TaNet. Red and green represent the ground truth and automated contours,
respectively.

##  4 Experiments

We conducted all experiments using Pytorch and performed training and
inference on NVIDIA GTX1080Ti GPU. In all experiments, we reported the
performance using mean IoU and F1, which are averaged over the test samples,
and measure the speed as frames per second (FPS). Talos [Talos] is used for
selecting the networks (e.g., filter size) and training (e.g., batch)
hyperparameters for all models. We performed ablation experiments to validate
the effectiveness of STN and HP, and compared TaNet to the state-of-the-arts.

To evaluate STN impacts on the segmentation performance, we integrated STN
module to Bisenet [28], FCN-8 [16], and UNET [22] and reported the results in
Table 1. The second rows of all models (i.e., FCN-8, UNET, BiSeNet) show that
integrating STN slightly decreases the inference speed. However, it improves
cardiac region segmentation in most cases. The red cells indicate that this
improvement is significant ($p<0.05$, t-test), and hence, it supports our
hypothesis. Specifically, these results demonstrate STN ability to increase
the performance by focusing the segmentation attention on specific ROI while
learning the context relationship of different ROIs. We also conducted another
ablation study to evaluate the impact of LBP-encoded kernels on segmentation
and reported the results in Table 1. The third rows of all models (i.e.,
FCN-8, UNET, BiSeNet) show that combining the textural features leads to an
improvement in inference speed and performance. The blue cell indicate that
the LBP-encoded kernels significantly ($p<0.05$, t-test) improves the
segmentation, especially in case of IVS and PW regions (see Figure 2). Due to
the similarity between the anterior and posterior walls, the accurate
segmentation of LV walls is challenging. However, combining the texture
kernels with low-details features improves the posterior and septal walls
segmentation as shown in the table and Figure 2. These results support our
hypothesis and demonstrate the benefits of using $HP$ in the segmentation
models.

The last row of Table 1 presents the performance of TaNet (STN, HP, SP, GP) as
compared to the baseline BiSeNet ($SP$ \+ $GP$) [28], FCN-8 [16] and UNET
[22]. The green cells indicate that the performance of the proposed TaNet
network is significantly ($p<0.05$, t-test) higher than all baseline models.
Note that TaNet has a slightly lower speed as compared to BiSeNet [28] due to
the integration of STN module and $HP$ pathway. However, this speed is still
efficient for real-time medical image analysis. We also compared TaNet with
recent cardiac segmentation works: EchoNet [20] (LV) and Res-U [1] (LV & LA).
TaNet outperformed both [20, 1] in LV segmentation; i.e., TaNet IoU= 0.95;
EchoNet IoU=0.87; Res-U IoU= 0.91. Also, TaNet outperformed [1] in LA
segmentation; i.e., TaNet IoU= 0.93; Res-U IoU= 0.88.

These results demonstrate the ability of TaNet to provide accurate medical
image segmentation in real time. The conceptual design of TaNet is suitable
for echo analysis as it allows to perform independent analysis for each
cardiac region as well as all regions together.

Table 1: Results of ablation experiments to evaluate the impact of STN and LBP-encoded kernels on segmentation. Red cells indicate statistical difference between models w/ and w/o STN, blue cells indicate statistical difference between models w/ and w/o LBP-encoded kernels, and green cell indicate statistical difference between TaNet and baseline models, w/o STN (✕) and LBP (✕).  Model | STN | LBP | LV | RV | LA | IVS | PW | FPS
---|---|---|---|---|---|---|---|---
|  |  | IoU | F1 | IoU | F1 | IoU | F1 | IoU | F1 | IoU | F1 |
| ✕ | ✕ | 0.87 | 0.93 | 0.82 | 0.90 | 0.86 | 0.92 | 0.86 | 0.93 | 0.76 | 0.86 | 5.5
FCN-8 | $\checkmark$ |  | 0.93 | 0.96 | 0.80 | 0.89 | 0.89 | 0.94 | 0.86 | 0.93 | 0.76 | 0.87 | 3.8
|  | $\checkmark$ | 0.88 | 0.94 | 0.83 | 0.91 | 0.86 | 0.93 | 0.90 | 0.95 | 0.80 | 0.89 | 9.3
| ✕ | ✕ | 0.85 | 0.91 | 0.81 | 0.89 | 0.88 | 0.93 | 0.87 | 0.93 | 0.79 | 0.88 | 4.8
UNET | $\checkmark$ |  | 0.90 | 0.95 | 0.86 | 0.92 | 0.91 | 0.95 | 0.90 | 0.94 | 0.81 | 0.89 | 3.1
|  | $\checkmark$ | 0.87 | 0.93 | 0.86 | 0.93 | 0.91 | 0.95 | 0.89 | 0.94 | 0.83 | 0.90 | 7.2
| ✕ | ✕ | 0.89 | 0.94 | 0.85 | 0.92 | 0.89 | 0.94 | 0.85 | 0.92 | 0.77 | 0.87 | 100.1
BiSeNet | $\checkmark$ |  | 0.94 | 0.97 | 0.90 | 0.94 | 0.92 | 0.96 | 0.90 | 0.95 | 0.78 | 0.87 | 90.5
|  | $\checkmark$ | 0.91 | 0.95 | 0.86 | 0.92 | 0.93 | 0.96 | 0.90 | 0.94 | 0.86 | 0.92 | 146.5
TaNet | $\checkmark$ | $\checkmark$ | 0.95 | 0.98 | 0.91 | 0.95 | 0.93 | 0.97 | 0.93 | 0.96 | 0.88 | 0.93 | 94.5

##  5 Conclusion

This work presents TaNet for real-time ROI-based segmentation in medical
images. TaNet achieved superior performance when evaluated on the echo dataset
for cardiac region segmentation. This superiority is attributed to STN ability
to focus the segmentation attention on specific ROIs while learning the
context relationship between them. Further, the use of $HP$ with texture
kernels along with the $SP$ allows to extract unique and rich low-level and
textural features. Although we used LBP-encoded pathway due to the superiority
of LBP in texture analysis, other handcrafted descriptors or kernels (e.g.,
Gabor) can be easily integrated to the handcrafted pathway following the same
formulation. While our empirical results are promising, we plan to further
evaluate TaNet on different settings. To benefit the research community, we
make our code available, and invite researchers to contribute to this effort.

## References

  * [1] Ali, Y., Janabi-Sharifi, F., Beheshti, S.: Echocardiographic image segmentation using deep res-u network. Biomedical Signal Processing and Control 64, 102248 (2021)
  * [2] Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence 39(12), 2481–2495 (2017)
  * [3] Badshah, N., Atta, H., Shah, S.I.A., Attaullah, S., Minallah, N., Ullah, M.: New local region based model for the segmentation of medical images. IEEE Access 8, 175035–175053 (2020)
  * [4] Bazin, P.L., Pham, D.L.: Homeomorphic brain image segmentation with topological and statistical atlases. Medical image analysis 12(5), 616–625 (2008)
  * [5] Cao, J., Pang, Y., Li, X.: Triply supervised decoder networks for joint detection and segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7392–7401 (2019)
  * [6] Cui, S., Mao, L., Jiang, J., Liu, C., Xiong, S.: Automatic semantic segmentation of brain gliomas from mri images using a deep cascaded neural network. Journal of healthcare engineering 2018 (2018)
  * [7] Daly, A., Yazid, H., Solaiman, B., Essoukri Ben Amara, N.: Multiatlas-based segmentation of female pelvic organs: Application for computer-aided diagnosis of cervical cancer. International Journal of Imaging Systems and Technology (2020)
  * [8] Gaál, G., Maga, B., Lukács, A.: Attention u-net based adversarial architectures for chest x-ray lung segmentation. arXiv preprint arXiv:2003.10304 (2020)
  * [9] Gu, R., Wang, G., Song, T., Huang, R., Aertsen, M., Deprest, J., Ourselin, S., Vercauteren, T., Zhang, S.: Ca-net: Comprehensive attention convolutional neural networks for explainable medical image segmentation. IEEE transactions on medical imaging (2020)
  * [10] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision. pp. 2961–2969 (2017)
  * [11] Hesamian, M.H., Jia, W., He, X., Kennedy, P.: Deep learning techniques for medical image segmentation: achievements and challenges. Journal of digital imaging 32(4), 582–596 (2019)
  * [12] Iakovidis, D.K., Keramidas, E.G., Maroulis, D.: Fuzzy local binary patterns for ultrasound texture characterization. In: International conference image analysis and recognition. pp. 750–759. Springer (2008)
  * [13] Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K.: Spatial transformer networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 2. pp. 2017–2025 (2015)
  * [14] Juefei-Xu, F., Naresh Boddeti, V., Savvides, M.: Local binary convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 19–28 (2017)
  * [15] Khor, H.L., Liew, S.C., Zain, J.M.: Region of interest-based tamper detection and lossless recovery watermarking scheme (roi-dr) on ultrasound medical images. Journal of digital imaging 30(3), 328–349 (2017)
  * [16] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3431–3440 (2015)
  * [17] Ma, J., Jiang, X., Fan, A., Jiang, J., Yan, J.: Image matching from handcrafted to deep features: A survey. International Journal of Computer Vision 129(1), 23–79 (2021)
  * [18] Ojala, T., Pietikäinen, M., Mäenpää, T.: Gray scale and rotation invariant texture classification with local binary patterns. In: European Conference on Computer Vision. pp. 404–420. Springer (2000)
  * [19] Ouahabi, A., Taleb-Ahmed, A.: Deep learning for real-time semantic segmentation: Application in ultrasound imaging. Pattern Recognition Letters (2021)
  * [20] Ouyang, D., He, B., Ghorbani, A., Yuan, N., Ebinger, J., Langlotz, C.P., Heidenreich, P.A., Harrington, R.A., Liang, D.H., Ashley, E.A., et al.: Video-based ai for beat-to-beat assessment of cardiac function. Nature 580(7802), 252–256 (2020)
  * [21] Pereira, P.M., Fonseca-Pinto, R., Paiva, R.P., Assuncao, P.A., Tavora, L.M., Thomaz, L.A., Faria, S.M.: Dermoscopic skin lesion image segmentation based on local binary pattern clustering: Comparative study. Biomedical Signal Processing and Control 59, 101924 (2020)
  * [22] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)
  * [23] Sinha, A., Dolz, J.: Multi-scale self-guided attention for medical image segmentation. IEEE journal of biomedical and health informatics (2020)
  * [24] Taghanaki, S.A., Abhishek, K., Cohen, J.P., Cohen-Adad, J., Hamarneh, G.: Deep semantic segmentation of natural and medical images: A review. Artificial Intelligence Review 54(1), 137–178 (2021)
  * [25] Weng, Y.T., Chan, H.W., Huang, T.Y.: Automatic segmentation of brain tumor from 3d mr images using segnet, u-net, and psp-net. In: International MICCAI Brainlesion Workshop. pp. 226–233. Springer (2019)
  * [26] Yin, Z., Yiu, V., Hu, X., Tang, L.: End-to-end face parsing via interlinked convolutional neural networks. Cognitive Neurodynamics pp. 1–11 (2020)
  * [27] Yu, C., Gao, C., Wang, J., Yu, G., Shen, C., Sang, N.: Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation. arXiv preprint arXiv:2004.02147 (2020)
  * [28] Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., Sang, N.: Bisenet: Bilateral segmentation network for real-time semantic segmentation. In: Proceedings of the European conference on computer vision (ECCV). pp. 325–341 (2018)
  * [29] Yurtkulu, S.C., Şahin, Y.H., Unal, G.: Semantic segmentation with extended deeplabv3 architecture. In: 2019 27th Signal Processing and Communications Applications Conference (SIU). pp. 1–4. IEEE (2019)
  * [30] Zamzmi, G., Rajaraman, S., Antani, S.: Accelerating super-resolution and visual task analysis in medical images. Applied Sciences 10(12), 4282 (2020)
  * [31] Zamzmi, G., Rajaraman, S., Antani, S.: Ums-rep: Unified modality-specific representation for efficient medical image analysis. Informatics in Medicine Unlocked p. 100571 (2021)