# Automatic detection of equiaxed dendrites using computer vision neural
networks

A. Viardin  K. Nöth  M. Torabi Rad  L.Sturz  ACCESS e.V., Intzestrasse 5,
D-52072 Aachen, Germany.

###### Abstract

Equaixed dendrites are frequently encountered in solidification. They
typically form in large numbers, which makes their detection, localization,
and tracking practically impossible for a human eye. In this paper, we show
how recent progress in the field of machine learning can be leveraged to
tackle this problem and we present computer vision neural network to
automatically detect equiaxed dendrites. Our network is trained using phase-
field simulation results, and proper data augmentation allows to perform the
detection task in solidification conditions entirely different from those
simulated for training. For example, here we show how they can successfully
detect dendrites of various sizes in a microgravity solidification experiment.
We discuss challenges in training such network along with our solutions for
them, and compare the performance of neural network with traditional methods
of shapes detection.

##  1 Introduction

Applying methods of artificial intelligence (AI), in particular ”deep
learning” for software-controlled automated identification, classification and
characterization of various characteristics in materials, based on two-
dimensional images coming from experimental results has shown a great interest
in recent years [1]. In case of _in situ_ experimental pictures coming from
video sequences, one can have several thousand of pictures to treat in order
to extract physical features that are relevant. For this purpose, innovative
AI tools must to be developed to overcome the limitations of classical methods
of image processing. For solification, several kind of microstructures can be
observed during experiments, many of them are dendrites with columnar or
equiaxed morphology. The equiaxed dendrites are appearing in isothermal
cooling conditions (they are growing in crystallographic directions), while
columnar dendrites are growing mainly along a temperature gradient in an
elongated form. The objective of this work is the automated detection and
characterization of equiaxed dendrites in a melt from time series using deep
learning methods. Methods for detection an evaluation for microstructures
using machine learning or deep learning techniques are already existing for
solidification [2] and solid state transformations [3]. These works shows that
neural networks for images detection is already working flawlessly. Deep
learning model based on architecture like Faster R-CNN [4], Mask R-CNN [5] or
Unet [6] have prooved their ability to detect objects with high accuracy.
Nevertheless, the good results concerning objects
detection/segmentation/instantiation are depending on the data used to train
the network and can be very tedious to generate manually. Especially if one
picture contains hundred of objects to detect, annotations of these objects
will be very time consuming. The objective of this article is to generate a
huge amount of realistic microstructures to train deep learning models. This
will be applied to experimental pictures obtained under reduced gravity with
resting, focused, growing and later overlapping or connected equiaxed
dendrites (experiment ”MEDI-2” [7]). Recently Hsu [8], used generative
adversarial network to generate 3D microstructures of solid oxide fuel cell
electrodes. Here, we have used phase field simulation to generate pictures for
training computer vision deep learning architectures in order to analyse
experimental microstructures. Phase field models became an important tool for
microstructure simulations in the field of solidification with or without
fluid flow, solid state transformations or mechanics for example.

##  2 Experimental results

To investigate equiaxed growth of dendrites with $<$100$>$ crystallographic
orientation the binary alloy Neopentylglycol-20.0 wt.% (D)Camphor (NPG-DC) was
selected. The phase diagram and relevant alloy properties are given in [9,
10]. The experiment MEDI-2 was carried out on board the sounding rocket
TEXUS-55 mission in 2018, where convection of the melt and sedimentation of
the dendrites is negligible. The experimental conditions were chosen such as
to obtain an equiaxed dendritic structure with diffusive conditions for heat
and mass transport during the microgravity period. The dendrites were observed
in-situ with two different optical systems to analyze the global and the
microscopic features of equiaxed solidification using the TEM 06-23 TEXUS
module. As a result, equiaxed dendrites with six dendrite arms growing
perpendicular to each other were detected, which is characteristic for a
$<$100$>$ crystallographic orientation. A schematic representation of the TEM
06-23 TEXUS module is shown in Figure 1. This hardware was provided by Airbus
Defense & Space. The temperature evolution in the solid or liquid alloy
material was measured by three Ni-CrNi thermocouples with diameter 0.25 mm,
which were located in the centre vertical plane inside the experimental cell.

Figure 1: Schematic representation of the TEM 06-23 TEXUS module6.

Figure 2 shows a series of 8-bit greyscale overview images taken with the CCD
macro camera. The field of view is 13.6 mm in width and 10.9 mm in height, and
the depth-of-field covers the full depth of the sample. The image acquisition
rate is 10 frames per second with an optical resolution of 10.625
$\mu$m/pixel. Also shown are the thermocouples inserted from the right side.

Figure 2: Overview images of the experiment cell showing multiple equiaxed
dendritic growth during the microgravity period. Also shown are the
thermocouples TZ1 to TZ3 inserted from the right side.

At the beginning of the low-gravity period the NPG-DC alloy is completely
liquid. Then, with the experimental parameters chosen, a region with equiaxed
dendrites, expands upwards in the experiment cell in a small thermal gradient.
The _in-situ_ and real-time observation of this process allows for the
investigation of the successive nucleation and growth of equiaxed dendrites,
remaining at stationary positions in the microgravity environment (Fig. 2).

##  3 Computer Vision Algorithm

Our objective is to use deep learning neural network to identify and localize
each dendrite on each experimental picture as shown in Fig 2. Several
architectures already exists for this purpose. Some of the most applied are
UNET [6], Mask R-CNN [5] and Faster R-CNN [4]. In this work, we will use
Faster R-CNN because of its efficiency and for simple preparation of training
data.

###  3.1 Faster R-CNN

Faster R-CNN is one of the most popular object detection methods. It is part
of the R-CNN series, developed by Girshick et al [11], enhanced with Fast
R-CNN [12] to finally obtain Faster R-CNN [4]. We only describe the
fundamentals of Faster R-CNN here. The input image is passed through a
convolutional neural network (CNN) to obtain a feature map of the objects
present in the image. This part of Faster R-CNN’s architecture is called the
”backbone” network. This feature map is then used by a region proposal network
(RPN) to generate region proposals (bounding boxes that contain the relevant
image objects) using anchors (fixed-size reference boxes placed uniformly in
the original image in order to detect the objects). These regions are then
filtered by NMS (Non-Maximum Suppression), which is a method that allows to
shift through the proposed regions and choose only those that are interesting.
The feature map extracted by the CNN and the relevant object bounding boxes
are used to generate a new feature map by pooling the regions of interest
(RoI). The grouped regions then pass through fully connected layers for object
area coordinate prediction and output classes. This part of the Faster R-CNN
architecture is called the header network.

###  3.2 Training and testing

The advantage of this method lies in the generic approach and can be extended
to other microstructures if required. Furthermore, a delimitation of the
dendrites both by a bounding box and a mask is automatically the outcome of
the simulation. In the context of AI and R-CNN, the task is not to simulate
the experimental NPG-DC dendrites as physically as possible, but to generate
objects that look as similar as possible to the dendrites.

##  4 Generation of training datasets

###  4.1 Phase field model

Simulations were performed using the multi-phase field model with obstacle
potentials implemented in the software MICRESS [13]. This model [14] is
implemented using finite-difference correction method used to improve
numerical accuracy [15], and anti-trapping and mobility correction methods
[16].

###  4.2 Simulation parameters

We chose a succinonitrile-1at%acetone alloy for our simulations. Its physical
properties are given in Table 1. Growth is initiated from a circular seed at
the origin of a 3 dimensional cubic domain. The domain has symmetric
boundaries conditions on each face for phase field and alloy concentration. In
this way we can simulate only one eight of a dendrite because of its internal
symmetry and so reducing calculation time. After the calculation the dendrite
is reconstructed by symmetry as we can see on the figure 4. The domain length
is 880 $\mu$m. The grid is dicretized on finite volume cubic element with a
grid spacing of 1$\mu$m. The interface thickness is 4 $\mu$m. For equiaxed
growth, isothermal conditions are used.

Figure 3: Simulation results at t=1s for 1/8 of a dendrite (isosurface for a
phase fraction of 0.5), reconstruction of the whole dendrite by symmetry

Figure 4: Simulated time evolution of reconstructed dendrite by symmetry

The results of one selected simulation are given on Fig. 4 where time
evolution of the dendrite morphology is shown. We have performed another
simulation with a varying different interface properties. This lead us to a
different morphological appearance of the equiaxed dendrite with less
pronounced tertiary arms, as we can see on the figures 6 and 6, the dendrite
arms are more smooth.

Acetone concentration ($C_{O}$) | 1at. %
---|---
Melting temp. of pure SCN ($T_{f}$) | 331.231 K
Liquidus slope ($m_{L}$) | -2.1 K at.%-1
Partition coefficient ($k$) | 0.1
Diffusion coefficient in liquid ($D_{l}$) | 1.27$\cdot$10-9 m2s-1
Diffusion coefficient in solid ($D_{s}$) | 3$\cdot$10-13 m2s-1
Gibbs-Thompson coefficient ($\Gamma$) | 6.525 $\cdot$10-8 Km
Interfacial energy ($\sigma_{sl}$) | 6.525 $\cdot$10-2 Jm-2
Table 1: Thermophysical properties of the SCN-acetone alloy [17]

###  4.3 Generation of training and test dataset

The results of 3D simulations (isocontour for a phase fraction of 0.5 ) are
presented on the figures 6 and 6 for respectively anisotropy of interfacial
energy value of $\epsilon$=0.105 and $\epsilon$=0.01, and isothermal
temperature of T = 328 K (undercooling of 1.13 K) and T = 328.67 K
(undercooling of 0.46 K). As we can see, the equiaxed dendrite on Fig. 6 have
dendritic arms with smaller tip radii and needle like tertiary arms compared
to 6. The objective here is to have the two types of dendrites that we see in
the experiments with differents kind of arm morphologies.

Figure 5: Artificial equiaxed dendrite generated with the phase field method.
The interfacial energy anistropy ($\epsilon_{sl}$) is 0.7% and the isothermal
temperature is 328 K.

Figure 6: Artificial equiaxed dendrite generated with the phase field method.
The interfacial energy anistropy ($\epsilon_{sl}$) is 0.07% and the isothermal
temperature is 328.67 K.

From these simulations results various 2D projections can be generated with
different camera angle (using paraview scripting) as we can see on the figure
7.

Figure 7: Different projections of the simulated 3D dendrites from Fig. 6.

The projections are additionally stretched, overlapped, resized and rotated
(augmentation) and anchored at random positions. A bounding box is generated
and the class indexed (”1”) for equiaxed dendrites. In addition, an artificial
black bar was generated to depict the thermocouples. The detection of the
thermocouples as separate objects has not yet been carried out. A result for a
training image is shown in Fig. 9. The Fig. 9 is the same as Fig. 9 but we
have plotted the bounding boxes that are saved in the annotations files.

Figure 8: Example of a training image with augmented objects.

Figure 9: Example of a training image with augmented objects and automatically
generated bounding boxes

##  5 Faster R-CNN

###  5.1 Model implementation

The Faster R-CNN architecture was implemented using the framework Detectron2,
which is an open-source environment for AI. Detectron2 [18] is a library from
Facebook© that provides state-of-the-art algorithms for object detection and
segmentation and often pre-trained general models. For our project we use a
ResNet50 network with FPN (Feature Pyramid Network) as backbone.

###  5.2 Training and testing parameters

We have generated 80000 pictures, such as Fig. 9 for training coming from
phase field results. The dataset for testing were done using 10 pictures, like
in Fig. 2 coming from experiments with annotations. The parameter for the
Faster R-CNN architecture are listed in the Tab. 2. The training was perfomed
on a NVIDIA RTX 6000 GPU and last less than 48 hours.

Image per batch | 2
---|---
Base learning rate | 0.003
Warmup iterations | 100
Checkpoint Period | 10000
Maximum iterations | 130000
Steps | (30000,50000,100000)
Gamma | 0.5
Momentum | 0.9
ROI heads/Batch size per image | 128
Detections per Image | 150
Evaluation period | 1000
Table 2: Parameter for the Faster R-CNN implemented with Detectron2 [18]

###  5.3 Results of Faster R-CNN detection

On the figure 10, we have plotted on the left column, the bounding box
surrounding dendrites detected using traditional detection method (i.e with
matlab) in red. On the right of the Fig. 10 we have plotted the bounding boxes
surounding dendrites detected using Faster R-CNN. On each bounding box, the
class probability is also given, it is in almost all of the cases close to
100%, this means that the probability that the detected object is a dendrite
is around 100% which is true in our case.

Figure 10: Comparison of classic detection, each on the left, and detection
with faster R-CNN, each on the right, with selected image times: 220 s, 320 s,
370s, 450s.

In general, the results are very promising. Nevertheless, small remarks have
to be done :

  * 1.

Very small dendrite at the beginning of experiments i.e 220 s are not well
detected by Faster R-CNN

  * 2.

Faster R-CNN detects dendrites near the columnar front and overlapping
dendrites much better than classical methods

  * 3.

There are artefacts with both methods: with the classic method, e.g. at the
bottom right and with Faster R-CNN, a false positive object is detected in the
upper right image area where the top thermocouple is located

  * 4.

When dendrites are strongly packed in the bottom of the experimental device,
it is hard to detect them for the traditional method as well for Faster R-CNN
algorithm

For a more detailed analysis, the evolution with time of the number of
dendrites detected was evaluated for the classical method and Faster R-CNN and
plotted in the figure 12 as well for the average bounding box area on figure
12. The artifacts were manually removed concerning the classical method as
well for Faster R-CNN. For the evolution of the number of dendrites with time,
one can see at early stage i.e between 150 and 300 seconds the number of
dendrites detected by the classical method is higher than Faster R-CNN because
Faster R-CNN did not detect very small dendrites in our first model, while
classical methods did. as classical method does. But at larger time i.e
superior to 300 s, it is opposite. Faster R-CNN detects more dendrites than
the classical method, at this stage Faster R-CNN detects overlapping dendrites
better than the classical method and gives a realistic description of the
number of dendrite in this experiment. The same observation can be made for
the average bounding box area in figure 12, the better detection of
packed/overlapping dendrites with Faster R-CNN method drive to higher average
bounding box at latter stage of the experiments.

Figure 11: Number of detected dendrites from of the experiment duration

Figure 12: Determined box area of detected dendrites from as a function of the
experiment duration

###  5.4 Conclusion

In this work we have presented a computer vision detection algorithm based on
the Faster R-CNN model trained using phase field model results for
experimental picture analysis. We have shown that the detection using Faster
R-CNN provides better results than a conventional method using shape/edge
detection. The Faster R-CNN was implemented using Detectron2 [18]. The
advantage is a limited number of phase field simulations or annotations of
experimental images. Just with two phase field simulations we are able to
generate a lot of data for training with realistically looking dendrites. This
method can be now applied to classify different solidification morphology like
columnar dendrites or seaweed dendrites using training with phase field model
results.

###  5.5 Acknowledgements

This work was financially supported by DLR (Project KICK-G, contract number
FKZ50WM2050). The authors want to thank Dr. Markus Apel for fruitful
discussion on phase field model.

## References

  * [1] Elizabeth A. Holm, Ryan Cohn, Nan Gao, Andrew R. Kitahara, Thomas P. Matson, Bo Lei, and Srujana Rao Yarasi.  Overview: Computer vision and machine learning for microstructural characterization and analysis.  Metallurgical and Materials Transactions A, 51:5985 – 5999, December 2020.
  * [2] Enzo Liotti, Carlos Arteta, Andrew Zisserman, Andrew Lui, Victor Lempitsky, and Patrick S. Grant.  Crystal nucleation in metallic alloys using x-ray radiography and machine learning.  Science Advances, 4(4):eaar4004, 2018.
  * [3] Reuben Agbozo and Wu-Yin Jin.  Quantitative metallographic analysis of gcr15 microstructure using mask r-cnn.  Journal of the Korean Society for Precision Engineering, 37:361–369, 05 2020.
  * [4] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun.  Faster R-CNN: towards real-time object detection with region proposal networks.  CoRR, abs/1506.01497, 2015.
  * [5] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick.  Mask R-CNN.  CoRR, abs/1703.06870, 2017.
  * [6] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  CoRR, abs/1505.04597, 2015.
  * [7] Gerhard Zimmermann, Martin Hamacher, and L. Sturz.  In-situ observation of equiaxed dendrites solidifying under reduced gravity conditions during the texus-55 mission – the medi-2 experiment.  In Proceedings of the 24rd ESA Symposium on European Rocket and Balloon Programmes and Related Research, October 2019.
  * [8] Tim Hsu, William K. Epting, Hokon Kim, Harry W. Abernathy, Gregory A. Hackett, Anthony D. Rollett, Paul A. Salvador, and Elizabeth A. Holm.  Microstructure generation via generative adversarial network for heterogeneous, topologically complex 3d materials.  Journal of Materials, 73, December 2021.
  * [9] Laszlo Sturz, Mickael Hamacher, Janin Eiken, and Gerhard Zimmermann, editors.  Multiple equiaxed dendrite interaction investigated on MASER-13, 11-15 June 2017.
  * [10] Gerhard Zimmermann, Mickael Hamacher, and Laszlo Sturz, editors.  Columnar-to-equiaxed transition in the transparent alloy system NPG-DC observed in the low-gravity and normal gravity experiment TRACE-3, 11-15 June 2017.
  * [11] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 580–587, 2014.
  * [12] Ross B. Girshick.  Fast R-CNN.  CoRR, abs/1504.08083, 2015.
  * [13] Micress 7.  http://www.micress.de.
  * [14] Janin Eiken, Bernd Böttger, and Ingo Steinbach.  Multiphase-field approach for multicomponent alloys with extrapolation scheme for numerical application.  Physical Review E, 73(6):066122, jun 2006.
  * [15] Janin Eiken.  Numerical solution of the phase-field equation with minimized discretization error.  IOP Conference Series: Materials Science and Engineering, 33:012105, jul 2012.
  * [16] Carre A., Böttger B., and Apel M.  Implementation of an antitrapping current for a multicomponent multiphase-field ansatz.  J. Cryst. Growth, 380:5–13, 2013.
  * [17] Alexandre Viardin, Miha Založnik, youssef Souhar, Markus Apel, and Hervé Combeau.  Mesoscopic modeling of spacing and grain selection in columnar dendritic solidification: Envelope versus phase-field model.  Acta Materialia, 122:386 – 399, January 2017.
  * [18] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2, 2019.


#  NCVX: A User-Friendly and Scalable Package for Nonconvex Optimization in
Machine Learning

Buyun Liang1<EMAIL_ADDRESS>
Tim Mitchell2<EMAIL_ADDRESS>
Ju Sun1<EMAIL_ADDRESS>

1 Department of Computer Science & Engineering, University of Minnesota, Twin
Cities, USA
2 Max Planck Institute for Dynamics of Complex Technical Systems, Germany

###### Abstract

Optimizing nonconvex (NCVX) problems, especially nonsmooth and constrained
ones, is an essential part of machine learning. However, it can be hard to
reliably solve such problems without optimization expertise. Existing general-
purpose NCVX optimization packages are powerful but typically cannot handle
nonsmoothness. GRANSO is among the first optimization solvers targeting
general nonsmooth NCVX problems with nonsmooth constraints, but, as it is
implemented in MATLAB and requires the user to provide analytical gradients,
GRANSO is often not a convenient choice in machine learning (especially deep
learning) applications. To greatly lower the technical barrier, we introduce a
new software package called NCVX, whose initial release contains the solver
PyGRANSO, a PyTorch-enabled port of GRANSO incorporating auto-differentiation,
GPU acceleration, tensor input, and support for new QP solvers. NCVX is built
on freely available and widely used open-source frameworks, and as a
highlight, can solve general constrained deep learning problems, the first of
its kind. NCVX is available at https://ncvx.org, with detailed documentation
and numerous examples from machine learning and other fields.

Keywords: BFGS-SQP, second-order methods, nonconvex optimization, nonsmooth
optimization, nonsmooth constraints, auto-differentiation, GPU acceleration,
PyTorch

##  1 Introduction

Mathematical optimization is an indispensable modeling and computational tool
for all science and engineering fields, especially for machine learning. To
date, researchers have developed numerous foolproof techniques and user-
friendly solvers and modeling languages for convex (CVX) problems, such as
SDPT3 (Toh et al., 1999), Gurobi (Gurobi Optimization, LLC, 2021), Cplex
(Cplex, 2009), TFOCS (Becker et al., 2011), CVX(PY) (Grant et al., 2008;
Diamond and Boyd, 2016), AMPL (Gay, 2015), YALMIP (Lofberg, 2004). These
developments have substantially lowered the barrier of CVX optimization for
non-experts. However, practical problems, especially from machine and deep
learning, are often nonconvex (NCVX), and possibly also nonsmooth (NSMT) and
constrained (CSTR).

There are methods and packages handling NCVX problems in restricted settings:
PyTorch (Paszke et al., 2019) and TensorFlow (Abadi et al., 2015) can solve
large-scale NCVX, NSMT problems without constraints. CSTR problems can be
heuristically turned into penalty forms, thus making the constraints implicit,
but this may not produce feasible solutions for the original problems. When
the constraints are simple, structured methods such as projected (sub)gradient
and Frank-Wolfe (Sra et al., 2012) can be used. When the constraints are
differentiable manifolds, one can consider manifold optimization methods and
packages, e.g., (Py)manopt (Boumal et al., 2014; Townsend et al., 2016),
Geomstats (Miolane et al., 2020), McTorch (Meghwanshi et al., 2018), and
Geoopt (Kochurov et al., 2020). For general CSTR problems, KNITRO (Pillo and
Roma, 2006) and IPOPT (Wächter and Biegler, 2006) implement interior-point
methods, while ensmallen (Curtin et al., 2021) and GENO (Laue et al., 2019)
rely on augmented Lagrangian methods. However, moving beyond smooth (SMT)
constraints, both of these families of methods, at best, handle only special
types of NSMT constraints. Finally, packages specialized for machine learning,
such as scikit-learn (Pedregosa et al., 2011), MLib (Meng et al., 2016) and
Weka (Witten et al., 2011), often use problem-specific solvers that cannot be
easily extended to new formulations.

##  2 The GRANSO and NCVX packages

GRANSO111http://www.timmitchell.com/software/GRANSO/ is among the first
optimization packages that can handle general NCVX, NSMT, CSTR problems
(Curtis et al., 2017):

$\displaystyle\begin{split}&\min_{{\bm{x}}\in\mathbb{R}^{n}}f({\bm{x}}),\;\;\text{s.t.
}c_{i}({\bm{x}})\leq
0,\;\forall\;i\in\mathcal{I};\;\;c_{i}({\bm{x}})=0,\;\forall\;i\in\mathcal{E}.\end{split}$
(1)

Here, the objective $f$ and constraint functions $c_{i}$’s are only required
to be almost everywhere continuously differentiable. GRANSO is based on quasi-
Newton updating with sequential quadratic programming (BFGS-SQP) and has the
following advantages: (1) unified treatment of NCVX problems: no need to
distinguish CVX vs NCVX and SMT vs NSMT problems, similar to typical nonlinear
programming packages; (2) reliable step-size rule: specialized methods for
NSMT problems, such as subgradient and proximal methods, often entail tricky
step-size tuning and require the expertise to recognize the structures (Sra et
al., 2012), while GRANSO chooses step sizes adaptively via a gold-standard
line search; (3) principled stopping criterion: GRANSO stops its iteration by
checking a theory-grounded stationarity condition for NMST problems, whereas
specialized methods are usually stopped when reaching ad-hoc iteration caps.

However, GRANSO users must derive gradients analytically222GRANSO is
implemented in MATLAB and does not support auto-differentiation, although
recent versions of MATLAB have included primitive auto-differentiation
functionalities.  and then provide code for these computations, a process
which may require some expert knowledge, is often error-prone, and in machine
learning, is generally impractical, e.g., for the training of large neural
networks. Furthermore, as part of the MATLAB software ecosystem, GRANSO is
generally hard for practitioners to integrate it with existing popular machine
learning frameworks—mostly in Python and R—and users’ own existing toolchains.
To overcome these issues and facilitate both high performance and ease of use
in machine and deep learning, we introduce a new software package called NCVX,
whose initial release contains the solver PyGRANSO, a PyTorch-enabled port of
GRANSO with several new key features: (1) auto-differentiation of all
gradients, which is a main feature to make PyGRANSO user-friendly; (2) support
for both CPU and GPU computations for improved hardware acceleration and
massive parallelism; (3) support for general tensor variables including
vectors and matrices, as opposed to the single vector of concatenated
optimization variables that GRANSO uses; (4) integrated support for OSQP
(Stellato et al., 2020) and other QP solvers for respectively computing search
directions and the stationarity measure on each iteration. OSQP generally
outperforms commercial QP solvers in terms of scalability and speed. All of
these enhancements are crucial for solving large-scale machine learning
problems. NCVX, licensed under the AGPL version 3, is built entirely on freely
available and widely used open-source frameworks; see https://ncvx.org for
documentation and examples.

##  3 Usage Examples: Dictionary Learning and Neural Perceptual Attack

In order to make NCVX friendly to non-experts, we strive to keep the user
input minimal. The user is only required to specify the optimization variables
(names and dimensions of variables) and define the objective and constraint
functions. Here, we briefly demonstrate the usage of PyGRANSO solver on a
couple of machine learning problems.

### Orthogonal Dictionary Learning (ODL, Bai et al. (2018))

One hopes to find a “transformation” ${\bm{q}}\in\mathbb{R}^{n}$ to sparsify a
data matrix ${\bm{Y}}\in\mathbb{R}^{n\times m}$:

$\displaystyle\min_{{\bm{q}}\in\mathbb{R}^{n}}\;f({\bm{q}})\doteq\frac{1}{m}\left\|{\bm{q}}^{\intercal}{\bm{Y}}\right\|_{1},\quad\text{s.t.}\;\left\|{\bm{q}}\right\|_{2}=1,$
(2)

where the sphere constraint $\left\|{\bm{q}}\right\|_{2}=1$ is to avoid the
trivial solution ${\bm{q}}=\bm{0}$. Problem (2) is NCVX, NSMT, and CSTR:
nonsmoothness comes from the objective, and nonconvexity comes from the
constraint. Demo 1 and Demo 2 show the implementations of ODL in GRANSO and
PyGRANSO, respectively. Note that the analytical gradients of the objective
and constraint functions are not required in PyGRANSO. Figure 1 shows that
PyGRANSO produces results that are faithful to that of GRANSO on ODL.

⬇

function[f,fg,ci,cig,ce,ceg]=comb_fn(q)

f = 1/m*norm(q’*Y, 1); % obj

fg = 1/m*Y*sign(Y’*q); % obj grad

ci = []; cig = []; % no ineq constr

ce = q’*q - 1; % eq constr

ceg = 2*q; % eq constr grad

end

soln = granso(n,comb_fn);

Demo 1: GRANSO for ODL

⬇

def comb_fn(X_struct):

q = X_struct.q

f = 1/m*norm(q.T@Y, p=1) # obj

ce = pygransoStruct()

ce.c1 = q.T@q - 1 # eq constr

return [f,None,ce]

var_in = {"q": [n,1]} # define variable

soln = pygranso(var_in,comb_fn)

Demo 2: PyGRANSO for ODL

Figure 1: Consistency of GRANSO (left) and PyGRANSO (right) on ODL

### Neural Perceptual Attack (NPA, Laidlaw et al. (2020))

The CSTR deep learning problem, NPA, is shown below:

$\displaystyle\max_{\widetilde{{\bm{x}}}}\;{\mathcal{L}}\left(f\left(\widetilde{{\bm{x}}}\right),y\right),\;\;\text{s.t.}\;\;d\left({\bm{x}},\widetilde{{\bm{x}}}\right)=\left\|\phi\left({\bm{x}}\right)-\phi\left(\widetilde{{\bm{x}}}\right)\right\|_{2}\leq\epsilon.$
(3)

Here, ${\bm{x}}$ is an input image, and the goal is to find its perturbed
version $\widetilde{{\bm{x}}}$ that is perceptually similar to ${\bm{x}}$
(encoded by the constraint) but can fool the classifier $f$ (encoded by the
objective). The loss ${\mathcal{L}}\left(\cdot,\cdot\right)$ is the margin
loss used in Laidlaw et al. (2020). Both $f$ in the objective and $\phi$ in
the constraint are deep neural networks with ReLU activations, making both the
objective and constraint functions NSMT and NCVX. The
$d\left({\bm{x}},\widetilde{{\bm{x}}}\right)$ distance is called the Learned
Perceptual Image Patch Similarity (LPIPS) (Laidlaw et al., 2020; Zhang et al.,
2018). Demo 3 is the PyGRANSO example for solving problem (3). Note that the
codes for data loading, model specification, loss function, and LPIPS distance
are not included here. It is almost impossible to derive analytical
subgradients for problem (3), and thus the auto-differentiation feature in
PyGRANSO is necessary for solving it.

⬇

def comb_fn(X_struct):

adv_inputs = X_struct.x_tilde

f = MarginLoss(model(adv_inputs),labels) # obj

ci = pygransoStruct()

ci.c1 = lpips_dists(adv_inputs) - 0.5 # ineq constr. percep bound epsilon=0.5

return [f,ci,None] # No eq constr

var_in = {"x_tilde": list(inputs.shape)} # define variable

soln = pygranso(var_in,comb_fn)

Demo 3: PyGRANSO for NPA

##  4 Roadmap

Although NCVX, with the PyGRANSO solver, already has many powerful features,
we plan to further improve it by adding several major components: (1)
symmetric rank one (SR1): SR1, another major type of quasi-Newton methods,
allows less stringent step-size search and tends to help escape from saddle
points faster by taking advantage of negative curvature directions (Dauphin et
al., 2014); (2) stochastic algorithms: in machine learning, computing with
large-scale datasets often involves finite sums with huge number of terms,
calling for stochastic algorithms for reduced per-iteration cost and better
scalability (Sun, 2019); (3) conic programming (CP): semidefinite programming
and second-order cone programming, special cases of CP, are abundant in
machine learning, e.g., kernel machines (Zhang et al., 2019); (4) minimax
optimization (MMO): MMO is an emerging modeling technique in machine learning,
e.g., generative adversarial networks (GANs) (Goodfellow et al., 2020) and
multi-agent reinforcement learning (Jin et al., 2020).

Acknowledgments

We would like to thank Frank E. Curtis and Michael L. Overton for their
involvement in creating the BFGS-SQP algorithm that is implemented in the
software package GRANSO. This work was supported by UMII Seed Grant Program
and NSF CMMI 2038403.

## References

  * Abadi et al. (2015) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.  URL https://www.tensorflow.org/.  Software available from tensorflow.org.
  * Bai et al. (2018) Yu Bai, Qijia Jiang, and Ju Sun.  Subgradient descent learns orthogonal dictionaries.  _arXiv preprint arXiv:1810.10702_ , October 2018.
  * Becker et al. (2011) Stephen R. Becker, Emmanuel J. Candès, and Michael C. Grant.  Templates for convex cone problems with applications to sparse signal recovery.  _Mathematical programming computation_ , 3(3):165, 2011.
  * Boumal et al. (2014) Nicolas Boumal, Bamdev Mishra, Pierre-Antoine Absil, and Rodolphe Sepulchre.  Manopt, a Matlab toolbox for optimization on manifolds.  _Journal of Machine Learning Research_ , 15(42):1455–1459, 2014.
  * Cplex (2009) IBM ILOG Cplex.  V12. 1: User’s manual for CPLEX.  _International Business Machines Corporation_ , 46(53):157, 2009.
  * Curtin et al. (2021) Ryan R. Curtin, Marcus Edel, Rahul Ganesh Prabhu, Suryoday Basak, Zhihao Lou, and Conrad Sanderson.  The ensmallen library for flexible numerical optimization.  _Journal of Machine Learning Research_ , 22(166):1–6, 2021.
  * Curtis et al. (2017) Frank E Curtis, Tim Mitchell, and Michael L Overton.  A BFGS-SQP method for nonsmooth, nonconvex, constrained optimization and its evaluation using relative minimization profiles.  _Optimization Methods and Software_ , 32(1):148–181, 2017.
  * Dauphin et al. (2014) Yann N. Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio.  Identifying and attacking the saddle point problem in high-dimensional non-convex optimization.  In _Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2_ , NIPS’14, page 2933–2941, Cambridge, MA, USA, 2014. MIT Press.
  * Diamond and Boyd (2016) Steven Diamond and Stephen Boyd.  CVXPY: A Python-embedded modeling language for convex optimization.  _Journal of Machine Learning Research_ , 17(83):1–5, 2016.
  * Gay (2015) David M. Gay.  The AMPL modeling language: An aid to formulating and solving optimization problems.  In Mehiddin Al-Baali, Lucio Grandinetti, and Anton Purnama, editors, _Numerical analysis and optimization_ , pages 95–116. Springer International Publishing, Cham, 2015.  ISBN 978-3-319-17689-5.
  * Goodfellow et al. (2020) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144, 2020.
  * Grant et al. (2008) Michael Grant, Stephen Boyd, and Yinyu Ye.  CVX: Matlab software for disciplined convex programming.  http://cvxr.com/cvx, 2008.
  * Gurobi Optimization, LLC (2021) Gurobi Optimization, LLC.  Gurobi Optimizer Reference Manual, 2021.  URL https://www.gurobi.com.
  * Jin et al. (2020) Chi Jin, Praneeth Netrapalli, and Michael Jordan.  What is local optimality in nonconvex-nonconcave minimax optimization?  In Hal Daumé III and Aarti Singh, editors, _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 4880–4889. PMLR, 13–18 Jul 2020.
  * Kochurov et al. (2020) Max Kochurov, Rasul Karimov, and Serge Kozlukov.  Geoopt: Riemannian optimization in PyTorch.  _arXiv preprint arXiv:2005.02819_ , May 2020.
  * Laidlaw et al. (2020) Cassidy Laidlaw, Sahil Singla, and Soheil Feizi.  Perceptual adversarial robustness: Defense against unseen threat models.  _arXiv preprint arXiv:2006.12655_ , June 2020.
  * Laue et al. (2019) Sören Laue, Matthias Mitterreiter, and Joachim Giesen.  GENO – GENeric Optimization for classical machine learning.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.
  * Lofberg (2004) Johan Lofberg.  YALMIP : a toolbox for modeling and optimization in MATLAB.  In _2004 IEEE International Conference on Robotics and Automation (IEEE Cat. No.04CH37508)_ , pages 284–289. IEEE, 2004.  doi: 10.1109/cacsd.2004.1393890.
  * Meghwanshi et al. (2018) Mayank Meghwanshi, Pratik Jawanpuria, Anoop Kunchukuttan, Hiroyuki Kasai, and Bamdev Mishra.  McTorch, a manifold optimization library for deep learning.  _arXiv preprint arXiv:1810.01811_ , October 2018.
  * Meng et al. (2016) Xiangrui Meng, Joseph Bradley, Burak Yavuz, Evan Sparks, Shivaram Venkataraman, Davies Liu, Jeremy Freeman, DB Tsai, Manish Amde, Sean Owen, Doris Xin, Reynold Xin, Michael J. Franklin, Reza Zadeh, Matei Zaharia, and Ameet Talwalkar.  MLlib: Machine learning in Apache Spark.  _Journal of Machine Learning Research_ , 17(34):1–7, 2016.
  * Miolane et al. (2020) Nina Miolane, Nicolas Guigui, Alice Le Brigant, Johan Mathe, Benjamin Hou, Yann Thanwerdas, Stefan Heyder, Olivier Peltre, Niklas Koep, Hadi Zaatiti, Hatem Hajri, Yann Cabanes, Thomas Gerald, Paul Chauchat, Christian Shewmake, Daniel Brooks, Bernhard Kainz, Claire Donnat, Susan Holmes, and Xavier Pennec.  Geomstats: A Python package for Riemannian geometry in machine learning.  _Journal of Machine Learning Research_ , 21(223):1–9, 2020.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  PyTorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32:8026–8037, 2019.
  * Pedregosa et al. (2011) Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011\.
  * Pillo and Roma (2006) Gianni Pillo and Massimo Roma.  _Large-Scale Nonlinear Optimization_ , volume 83 of _Nonconvex Optimization and Its Applications_.  Springer Science & Business Media, Boston, MA, 2006.
  * Sra et al. (2012) Suvrit Sra, Sebastian Nowozin, and Stephen J. Wright.  _Optimization for Machine Learning_.  The MIT Press, Cambridge, MA, 2012.
  * Stellato et al. (2020) Bartolomeo Stellato, Goran Banjac, Paul Goulart, Alberto Bemporad, and Stephen Boyd.  OSQP: an operator splitting solver for quadratic programs.  _Mathematical Programming Computation_ , 12(4):637–672, 2020.  doi: 10.1007/s12532-020-00179-2.
  * Sun (2019) Ruoyu Sun.  Optimization for deep learning: theory and algorithms.  _arXiv preprint arXiv:1912.08957_ , December 2019.
  * Toh et al. (1999) Kim-Chuan Toh, Michael J. Todd, and Reha H. Tütüncü.  SDPT3—a Matlab software package for semidefinite programming, Version 1.3.  _Optimization Methods and Software_ , 11(1-4):545–581, 1999.
  * Townsend et al. (2016) James Townsend, Niklas Koep, and Sebastian Weichwald.  Pymanopt: A Python toolbox for optimization on manifolds using automatic differentiation.  _Journal of Machine Learning Research_ , 17(137):1–5, 2016.
  * Witten et al. (2011) Ian H. Witten, Eibe Frank, and Mark A. Hall.  _Data Mining: Practical Machine Learning Tools and Techniques_.  Morgan Kaufmann, Burlington, MA, third edition, 2011.
  * Wächter and Biegler (2006) Andreas Wächter and Lorenz T. Biegler.  On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming.  _Mathematical Programming_ , 106(1):25–57, 2006\.  doi: 10.1007/s10107-004-0559-y.
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 586–595, 2018.
  * Zhang et al. (2019) Richard Y. Zhang, Cédric Josz, and Somayeh Sojoudi.  Conic optimization for control, energy systems, and machine learning: Applications and algorithms.  _Annual Reviews in Control_ , 47:323–340, 2019.


Further author information: (Send correspondence to A.L.)
A.L: E-mail<EMAIL_ADDRESS>

# A New Frontier for J-band Interferometry: Dual-band NIR Interferometry with
MIRC-X

Aaron Labdon  School of Physics and Astronomy, University of Exeter, Exeter,
Stocker Road, EX4 4QL, UK  John D. Monnier  Astronomy Department, University
of Michigan, Ann Arbor, MI 48109, USA  Stefan Kraus  School of Physics and
Astronomy, University of Exeter, Exeter, Stocker Road, EX4 4QL, UK  Jean-
Baptiste Le Bouquin  Institut de Planetologie et dAstrophysique de Grenoble,
Grenoble 38058, France  Benjamin R. Setterholm  Astronomy Department,
University of Michigan, Ann Arbor, MI 48109, USA  Narsireddy Anugu  Steward
Observatory, Department of Astronomy, University of Arizona, Tucson, USA  Theo
ten Brummelaar  The CHARA Array of Georgia State University, Mount Wilson
Observatory, Mount Wilson, CA 91203, USA  Cyprien Lanthermann  Instituut voor
Sterrenkunde, KU Leuven, Celestijnenlaan 200D, 3001 Leuven, Belgium  Claire L.
Davies  School of Physics and Astronomy, University of Exeter, Exeter, Stocker
Road, EX4 4QL, UK  Jacob Ennis  Astronomy Department, University of Michigan,
Ann Arbor, MI 48109, USA  Tyler Gardener  Astronomy Department, University of
Michigan, Ann Arbor, MI 48109, USA  Gail H. Schaefer  The CHARA Array of
Georgia State University, Mount Wilson Observatory, Mount Wilson, CA 91203,
USA  Lazlo Sturmann  The CHARA Array of Georgia State University, Mount Wilson
Observatory, Mount Wilson, CA 91203, USA  Judit Sturmann  The CHARA Array of
Georgia State University, Mount Wilson Observatory, Mount Wilson, CA 91203,
USA

###### Abstract

In this contribution we report on our work to increase the spectral range of
the Michigan Infrared Combiner-eXeter (MIRC-X) instrument at the CHARA array
to allow for dual H and J band interferometric observations. We comment on the
key science drivers behind this project and the methods of characterisation
and correction of instrumental birefringence and dispersion. In addition, we
report on the first results from on-sky commissioning in November 2019.

###### keywords:

interferometry, infrared, protoplanetary disks, stellar disks

##  1 INTRODUCTION

The Michigan Infrared Combiner (MIRC) was a six-telescope infrared beam
combiner at the CHARA telescope array [1], the world’s largest baseline
interferometer in the optical/infrared, located at the Mount Wilson
Observatory in California. In the summers of 2017 and 2018 MIRC underwent
significant upgrades, with the newly commissioned instrument being named
MIRC-X. The upgrade of MIRC-X occurred in two phases, the first of these
phases involved the implementation of an ultra-fast, ultra low read noise
near-infrared camera [2] and an overhaul of the MIRC control software. The
second phase involved the replacement of the optical fibres and beam
combination element in addition to the commissioning of polarisation
controllers. A detailed description of the upgraded MIRC-X instrument can be
found elsewhere[3, 4].

Over the past 15 years MIRC/MIRC-X has produced many outstanding results in a
variety of areas of astrophysics including the imaging of stellar surfaces of
rapid rotators, eclipsing binary systems and star spots on the surfaces of
highly magnetically active stars. [5, 6, 7, 8]. In more recent years, the
upgrades have allowed for observations of young stellar objects (YSO) with all
six telescopes for the first time [9, 10] and for high precision single-field
astrometry for the detection of Hot Jupiter objects in binary systems [11].

Historically MIRC-X has only operated on the H band, however, one of the
ultimate aims of the upgrades was to conduct dual H and J band observations.
The J band ($1.1-1.4\,\mu m$) is a relatively untapped resource in long-
baseline interferometry, but one with huge potential for scientific
discoveries. It allows access to the photosphere in giant and super-giant
stars relatively free from opacities of molecular bands. In addition, the J
band traces the warmest and smallest scales of protoplanetary disks, where
accretion and viscosity processes take place. Finally, the J band allows for
higher resolution observations than near and mid-IR allowing us to probe the
smallest scales of astrophysical objects.

In this paper we present a summary of the work done to commission the J band
for observations at the MIRC-X instrument. In section 2 we discuss the J band
specific hardware improvements required, while in section 3 we summarise the
corresponding software developments undertaken in this endeavour. In section 4
we discuss the first science results obtained in this new mode. Finally in
section 5 we provide our concluding remarks and provide a roadmap for bringing
the simultaneous H and J band observing mode into standard use at the CHARA
array.

##  2 Hardware requirements

Beyond the major instrument upgrades in 2017 and 2018 mentioned in the
introduction, various other hardware requirements were needed in order to
conduct J band observations. In this section we discuss this work and the
design decisions involved.

###  2.1 Filters

Operating in standard observing mode MIRC-X employs an H band filter with
central wavelength $\lambda=1.61\,\mu m$ and bandwidth
$\Delta\lambda=0.27\,\mu m$. However, in order to observe in the J band a new
filter which extends to a lower wavelength regime must be employed. However,
this is hampered owing to the presence of a metrology laser at CHARA. This
laser operates at $1.319\,\mu m$ and is responsible for providing accurate
metrology of the delay line cart position, a vital role at any optical
interferometer. This laser wavelength lies directly between the H and J bands
and will thus contaminate any stellar flux on the detector.

This was overcome with addition of a notch or band-stop filter. These filters
reject/attenuate signals in a specific wavelength band and pass the signals
above and below this band. In this way we can reject light from the CHARA
metrology laser and accept stellar contributions from the H band J bands above
and below this respectively. The theoretical and measured characteristics of
the notch filter we employed is shown in Figure 1.

---
Figure 1:  Theoretical and measured transmission of the notch filter across
the H and J bands. The dashed black curve represents the theoretical
transmission curve of the filter. The solid green area is the atmospheric
transmission. The blue line with points is the normalised flux measured using
the MIRC-X detector and the CHARA STS through the notch filter and the
PRISM50. The solid purple line with points is the normalised flux measured
using the MIRC-X detector and the CHARA STS through the H band filter and the
GRISM190, the current highest spectral mode of MIRC-X.

###  2.2 Longitudinal Dispersion Compensators

Long-baseline optical interferometry poses unique engineering challenges. One
of the most demanding aspects is the requirement for optical path length
compensation. The path length of the various light beams are not identical for
each telescope, owing to the position of the scope and position of the
astronomical object on the sky. A such, the path difference much be corrected
in order to ensure coherence for interferometric fringes. At CHARA this is
done using two methods, the first is to employ fixed amounts of delay which
can be moved in and out of the light path using retractable mirrors. These are
known as Pipes of Pan or PoPs. The second and universally used method is to
employ carts affixed to delay lines which track the geometric phase difference
as a stars moves across the sky. The difficulty with the second method is that
it very difficult to achieve in a vacuum, due to the size and number of moving
parts. This introduces a large amount of air into the system, potentially up
to $80\,m$ difference in air path between the beams. This air leads to
atmospheric, chromatic dispersion.

In standard single band observing, this does not pose a problem due to the
small wavelength range. However, in dual band observations this become a major
problem. If delay line carts are tracking H band fringes, the J band light
will be incoherent due to chromatic dispersion and fringes will be severely
reduced contrast to the point of being un-observable. Therefore a method of
correcting for atmospheric dispersion must be employed. At CHARA, this is in
the form of Longitudinal Dispersion Compensators (LDCs).

The LDCs consist of wedges of glass that can move in and out of the beam to
increase or decrease the thickness of glass in the beam path. The
characteristics of the LDCs are described in great detail in[12] when the LDCs
were commissioned in order to enable simultaneous near-infrared (NIR)
observations with the CLIMB instrument and visible observations with PAVO[13].
Despite not being designed for use between the H and J bands, the choice of
Schott-10 glass proved ideal for use with MIRC-X. Shown in Figure 2 is the
expected dispersion between the NIR wavebands and the expected linear movement
of the glass to successfully correct for this. The thickness of glass required
for a given air path can be calculated using

$\Delta L=\Delta
x\times\frac{n_{a}(\sigma_{1})-n_{a}(\sigma_{2})+\sigma_{1}n_{a}^{{}^{\prime}}(\sigma_{1})-\sigma_{2}n_{a}^{{}^{\prime}}(\sigma_{2})}{n_{g}(\sigma_{1})-n_{g}(\sigma_{2})+\sigma_{1}n_{g}^{{}^{\prime}}(\sigma_{1})-\sigma_{2}n_{g}^{{}^{\prime}}(\sigma_{2})}\,.$
(1)

---
Figure 2:  LEFT: Theoretical atmospheric dispersion between the NIR bands
covered by MIRC-X and the upcoming MYSTIC instrument. Group delay measurements
are used an analogy for chromatic dispersion. The J band is taken as $1.2\,\mu
m$, the H band as $1.55\,\mu m$ and the K band as $2.2\,\mu m$. The solid line
represents the dispersion in the J band with respect to the H, the dotted line
the dispersion in H with respect to k and the dashed the dispersion in J with
respect to K. RIGHT: The linear LDC glass movement required to correct the
dispersion between the J, H and K bands. The wavelengths adopted and line
meaning are the same as shown on the left.

Here, $\Delta L$ is the thickness of glass, $\Delta x$ is the air path in
meters, $n_{a}$ is the refractive index of air and $n_{g}$ is the refractive
index of the LDC glass. $\sigma_{1}$ and $\sigma_{2}$ refer to the wavenumbers
of the two observing wavelengths. In our model the refractive index of air was
calculated using a composite of Ciddor[14] and Mather[15] models in order to
cover the full NIR. Atmospheric conditions, such as temperature and humidity
were taken to be average values for the CHARA laboratory. More information on
the dependence of the refractive index of air on these parameters can be found
elsewhere [14, 15].

Overall the LDCs are capable of correcting for atmospheric longitudinal
dispersion between the H and J bands, making simultaneous observations
possible.

##  3 Software control and Data Reduction

A variety of software developments were required in order to conduct these
observations. In addition to changes to the real-time operating code of
MIRC-X/CHARA, the reduction pipeline had to be modified, as outlined below.
Also, we independently verified the data products by calibrating known
calibrator stars and ensuring the stability of the transfer function.

---
Figure 3:  LEFT: MIRC-X detector window showing laboratory induced fringes
along a single baseline. The wavelength increases towards the bottom, meaning
the J band is on top. The notch filter rejection window is clearly visible
between the H and J bands at $\sim 1.3\,\mu m$. RIGHT: Spectral calibration of
the MIRC-X simulated data. The inital guess of the spectral calibration is
shown in pale orange and the fitted calibration is shown in blue. The
translucent blue points represent those that were rejected by the pipeline due
to low flux.

###  3.1 LDC Control Software

The control of the LDCs is based upon that developed by Berger et al.[12] for
controlling dispersion between the visible and NIR. This software was adapted
and expanded by adopting the dispersion models described in section 2. The
basic function of the software is to receive the delay line cart positions in
order to calculate the total amount of atmosphere in the system. Using
equation 1 it calculates the thickness of glass required to correct for this
dispersion, which is then translated into the linear movement for the LDC.
This is done at a rate of one LDC per second, meaning a full movement cycle of
the LDCs is 6 seconds. This is sufficient given the slow rate of change of the
total air path length in the system.

###  3.2 Data Reduction Pipeline

The data reduction pipeline for MIRC-X is already well established and
publicly available[4]. It has a proven track record through several
publications. However, it was not designed to reduce J band data and so
several changes had to be made to the existing code.

The first change to the data reduction pipeline consisted of adjusting the
fringes and photometry windows to accurately detect and account for the J band
flux. Previously the flux window had been detected by fitting a simple
Gaussian to the detector flux, however, due to the presence of the notch
filter the simultaneous data is double-peaked. Figure 3 shows the detector
window of MIRC-X covering both the H and J bands, the spectral range of the J
band is just over half that of the H band.

The second change comes from the spectral calibration, that is the process of
determining the wavelength of each spectral channel. The wavelength is
assigned by fitting the frequency of the fringe patterns, based on an initial
guess. The inital guess is based on the known spectral dispersion of the prism
used, in this case $R=50$. Figure 3 shows the spectral calibration result
obtained from the MIRC-X data reduction pipeline for artificially produced
interference fringes. Both the inital guess and the final fitted spectral
calibration are shown.

---
Figure 4:  Calibration transfer function for 2019 November 19. The red-shaded
areas are those representing calibrators and green are those representing
science observations of FU Ori [10]. these are along a single baseline (S1-S2)
for the four J band spectral channels. The blue and brown points are the raw,
uncalibrated squared-visibilities for the calibrators and science target
respectively. The black and red triangles are the calibrated squared-
visibilities for the calibrators and science target respectively. the blue
represents the transfer function across the calibrators.

###  3.3 Data Calibration

In order to ensure the validity of the data products, careful calibration
tests were undertaken. The calibration procedure was unchanged from the
standard MIRC-X pipeline as described in great detail elsewhere[4]. Figure 4
shows the transfer function on a single baseline across the three hour
observing period on the night of 2019 November 19. The 4 wavelength channels
are those across the J band only, as the calibration of the H band channels
has already been verified in previous papers. The linearity of the transfer
function shows the stability of the observing across the 2.5 hour period.
Large jumps in the transfer function on short timescales would indicate
instability/inaccuracies in the dispersion control subsystems.

##  4 First Science Results

Following the upgrades to the hardware and software described in this paper,
we conducted the first science observations in November 2019. We adopted a
4-telescope combination using the first four fibres (lowest spatial
frequencies) of MIRC-X, this is the only possible observing mode with the J
band as the highest spatial frequencies are undersampled (sub-Nyquist). As the
first science target, we observed the young outbursting object FU Orionis. The
data was reduced following the reduction and calibration steps described in
section 3. Full details of the data analysis and conclusions can be found
in[10].

---
Figure 5:  Squared visibilites for the object FU Orionis with the MIRC-X
instrument. Colour scale represents the wavelength of the observations. Dark-
blue/purple points are across the J band, while green-yellow ponts are across
the H band.

We obtained two J+H band pointings on FU Ori with MIRC-X in 2019, using a
4-telescope configuration in CAL-SCI concatenation sequences. Of the available
baselines, a maximum physical baseline of $280\,$m was used corresponding to a
maximum resolution of $0.34\,\mathrm{mas}$. The calibrated squared
visibilities are shown in Figure 5. These observations represent the first J
band interferometric observations of a young stellar object and allowed us to
probe the innermost regions of this highly accreting disk. In addition to
constraining the 2-D geometry, these multi-wavelength observations allowed us
to probe the temperature gradient profile of the inner disk. Theoretical works
have proposed models for the expected temperature profile of accretion disks
since the late 1970’s[16, 17, 18]. These models follow the simple prescription

$T_{R}=T_{0}(R/R_{0})^{-Q}\,,$ (2)

where $T_{0}$ is the temperature at radius $R_{0}$ and $Q$ is gradient
exponent. The two models proposed in the literature provide alternate $Q$
parameters. For isothermal flared discs, heated only by reprocessed stellar
radiation, a value of $Q=0.5$ is predicted[19, 20]. Actively accreting,
steady-state disks that are viscously heated should exhibit a value of
$Q=0.75$ [18].

For FU Orionis we measured $Q=0.74\pm 0.02$, a value that indicates viscously
heated material. Our analysis also revealed that the disk inner radius is
equivalent to the stellar radius, meaning the disk extends down to the stellar
surface. This is indicative of boundary layer accretion dominating over
magnetospheric accretion scenarios. A summary of our results is shown in
Figure 6. This is among the first direct interferometric evidence for both
boundary layer accretion and viscously heated inner accretion disks.

---
Figure 6:  Temperature gradient of FU Ori over the inner disk. The temperature
and location of the inner radius of the disk are $5800\,\mathrm{K}$ and
$0.015\,\mathrm{au,}$ respectively. The dashed line represents the inner disk
edge. The vertical profile of the disk schematic is not to scale, nor does it
represent the vertical temperature structure of the disk. INSET-left: Linearly
polarized intensity, scattered light image obtained with the Gemini Planet
Imager (GPI) in the J-band [21]. The image is centered on FU Ori N within the
black circle. FU Ori S is also visible to the South-West. INSET -right:
Brightness distribution of the TGM disk model in the J-band, as derived in
this work. In both panels, we plot normalised surface brightness with a
logarithmic colour scale. The white lines in the image indicate the position
angle of the disk minor-axis derived from other observations in literature and
in this work.

##  5 Summary and Future Work

###  5.1 Conclusions

In conclusion, we present a new and unique observing mode for the MIRC-X
instrument. The simultaneous H and J band mode provides an exciting
opportunity to conduct very high resolution observations at a little explored
wavelength in interferometry. The J band wavelength regime is highly
interesting scientifically across a range of astrophysical objects from
stellar photospheres for circumstellar accretion disks.

We outlined the work to bring the J band to MIRC-X, including the inital
hardware and software development and later the work done to enable data
reduction and calibration in a reliable way.

We also present the first science results from the endeavour in the form of a
detailed study of the accretion disk around FU Orionis. We find evidence of
boundary layer accretion and viscous heating in this first-of-its-kind study.
In the process we demonstrate the effectiveness and application of this new
observing mode as it opens up a new frontier in interferometry.

For more information regarding MIRC-X observing modes and characterisation see
other works in these proceedings by Setterholm et al.[22] and Anugu et al.[4].

###  5.2 Future Work

The future of the J band at MIRC-X lies in community access. Follow-up
observations in this observing mode are expected in late 2020 where we expect
to perfect the observing routine and confirm dispersion models on sky.
Following this, we aim to release the J band to the wider community and offer
it as a standard MIRC-X mode from semester 2021-B onwards.

Furthermore, the J-band also represents a promising wavelength regime for the
VLT Interferometer, where the J-band visitor instrument BIFROST could enable
high-spectral resolution observations in spectral lines and open new science
applications on star formation, fundamental stellar astrophysics, and the
characterisation of star and planetary systems [23].

###### Acknowledgements.

We acknowledge support from an STFC studentship (No. 630008203) and an
European Research Council Starting Grant (Grant Agreement No. 639889). JDM
acknowledges funding from NASA NNX09AB87G, NSF NSF-ATI 1506540, and NASA XRP
Grant NNX16AD43G. This work is based upon observations obtained with the
Georgia State University Center for High Angular Resolution Astronomy Array at
Mount Wilson Observatory. The CHARA Array is supported by the National Science
Foundation under Grant No. AST-1636624 and AST-1715788. Institutional support
has been provided from the GSU College of Arts and Sciences and the GSU Office
of the Vice President for Research and Economic Development. MIRC-X received
funding from the European Research Council (ERC) under the European Union’s
Horizon 2020 research and innovation programme (Grant No. 639889).

## References

  * [1] Monnier, J. D., Berger, J.-P., Millan-Gabet, R., and ten Brummelaar, T. A., “The Michigan Infrared Combiner (MIRC): IR imaging with the CHARA Array,” in [New Frontiers in Stellar Interferometry ], Traub, W. A., ed., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 5491, 1370 (Oct. 2004).
  * [2] Anugu, N., Le Bouquin, J.-B., Monnier, J. D., Kraus, S., Ennis, J., Lanthermann, C., Setterholm, B. R., Davies, C. L., ten Brummelaar, T., Haidar, M., Dubravec, V., and Peters, S., “MIRC-X/CHARA: sensitivity improvements with an ultra-low noise SAPHIRA detector,” in [Optical and Infrared Interferometry and Imaging VI ], Creech-Eakman, M. J., Tuthill, P. G., and Mérand, A., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 10701, 1070124 (July 2018).
  * [3] Kraus, S., Monnier, J. D., Anugu, N., Le Bouquin, J.-B., Davies, C. L., Ennis, J., Labdon, A., Lanthermann, C., Setterholm, B., and ten Brummelaar, T., “The MIRC-X 6-telescope imager: key science drivers, instrument design and operation,” in [Optical and Infrared Interferometry and Imaging VI ], Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 10701, 1070123 (Jul 2018).
  * [4] Anugu, N., Le Bouquin, J.-B., Monnier, J. D., Kraus, S., Setterholm, B. R., Labdon, A., Davies, C. L., Lanthermann, C., Gardner, T., Ennis, J., Johnson, K. J. C., ten Brummelaar, T., Schaefer, G., and Sturmann, J., “MIRC-X: a highly-sensitive six telescope interferometric imager at the CHARA Array,” arXiv e-prints 1, arXiv:2007.12320 (July 2020).
  * [5] Monnier, J. D., Che, X., Zhao, M., Ekström, S., Maestro, V., Aufdenberg, J., Baron, F., Georgy, C., Kraus, S., McAlister, H., Pedretti, E., Ridgway, S., Sturmann, J., Sturmann, L., ten Brummelaar, T., Thureau, N., Turner, N., and Tuthill, P. G., “Resolving Vega and the Inclination Controversy with CHARA/MIRC,” ApJ Letters 761, L3 (Dec. 2012).
  * [6] Roettenbacher, R. M., Monnier, J. D., Korhonen, H., Harmon, R. O., Baron, F., Hackman, T., Henry, G. W., Schaefer, G. H., Strassmeier, K. G., Weber, M., and ten Brummelaar, T. A., “Contemporaneous Imaging Comparisons of the Spotted Giant $\sigma$ Geminorum Using Interferometric, Spectroscopic, and Photometric Data,” ApJ 849, 120 (Nov. 2017).
  * [7] Schaefer, G., Duchene, G., Farrington, C. D., Gardner, T., Gies, D. R., Kraus, A., Monnier, J. D., Prato, L., Ridgway, S., Roettenbacher, R., Stencel, R. E., ten Brummelaar, T., and van Belle, G., “Binary and Multiple Star Systems at High Angular Resolution,” Baas 51, 483 (May 2019).
  * [8] Chiavassa, A., Kravchenko, K., Millour, F., Schaefer, G., Schultheis, M., Freytag, B., Creevey, O., Hocdé, V., Morand, F., Ligi, R., Kraus, S., Monnier, J. D., Mourard, D., Nardetto, N., Anugu, N., Le Bouquin, J. B., Davies, C. L., Ennis, J., Gardner, T., Labdon, A., Lanthermann, C., Setterholm, B. R., and ten Brummelaar, T., “Optical interferometry and Gaia measurement uncertainties reveal the physics of asymptotic giant branch stars,” AAP 640, A23 (Aug. 2020).
  * [9] Kraus, S., Kreplin, A., Young, A. K., Bate, M. R., Monnier, J. D., Harries, T. J., Avenhaus, H., Kluska, J., Laws, A. S. E., Rich, E. A., Willson, M., Aarnio, A. N., Adams, F. C., Andrews, S. M., Anugu, N., Bae, J., ten Brummelaar, T., Calvet, N., Curé, M., Davies, C. L., Ennis, J., Espaillat, C., Gardner, T., Hartmann, L., Hinkley, S., Labdon, A., Lanthermann, C., LeBouquin, J.-B., Schaefer, G. H., Setterholm, B. R., Wilner, D., and Zhu, Z., “A triple-star system with a misaligned and warped circumstellar disk shaped by disk tearing,” Science 369, 1233–1238 (Sept. 2020).
  * [10] Labdon, A., Kraus, S., Davies, C. L., Kreplin, A., Monnier, J. D., Jean-Baptiste, L., Narsireddy, A., ten Brummelaar, T., Benjamin, S., Tyler, G., Jacob, E., Cyprien, L., Gail, S., and Anna, L., “Viscous Heating and Boundary Layer Accretion in the Disk of Outbursting Star FU Orionis,” AAP Submitted (2020).
  * [11] Gardner, T., “Hunting for Planets around Hot Stars: The ARMADA Survey at CHARA and VLTI,” in [American Astronomical Society Meeting Abstracts #235 ], American Astronomical Society Meeting Abstracts 235, 375.05 (Jan. 2020).
  * [12] Berger, D. H., ten Brummelaar, T. A., Bagnuolo, William G., J., and McAlister, H. A., “Preliminary results from the longitudinal dispersion compensation system for the CHARA array,” in [Optical and Infrared Interferometry ], Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 4838, 974–982 (2003).
  * [13] Ireland, M. J., Mérand, A., ten Brummelaar, T. A., Tuthill, P. G., Schaefer, G. H., Turner, N. H., Sturmann, J., Sturmann, L., and McAlister, H. A., “Sensitive visible interferometry with PAVO,” in [Optical and Infrared Interferometry ], Schöller, M., Danchi, W. C., and Delplancke, F., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 7013, 701324 (July 2008).
  * [14] Ciddor, P. E., “Refractive index of air: new equations for the visible and near infrared,” Appl. Opt. 35, 1566–1573 (Mar 1996).
  * [15] Mathar, R. J., “Refractive index of humid air in the infrared: model fits,” Journal of Optics A: Pure and Applied Optics 9, 470–476 (apr 2007).
  * [16] Shakura, N. I. and Sunyaev, R. A., “Black Holes in Binary Systems: Observational Appearances,” in [X- and Gamma-Ray Astronomy ], 55, 155 (Jan. 1973).
  * [17] Pringle, J. E. and Rees, M. J., “Accretion Disc Models for Compact X-Ray Sources,” AAP 21, 1 (Oct. 1972).
  * [18] Pringle, J. E., “Accretion discs in astrophysics,” ARAA 19, 137–162 (Jan. 1981).
  * [19] Kenyon, S. J. and Hartmann, L., “Spectral Energy Distributions of T Tauri Stars: Disk Flaring and Limits on Accretion,” ApJ 323, 714 (Dec 1987).
  * [20] Dullemond, C. P. and Dominik, C., “Flaring vs. self-shadowed disks: The SEDs of Herbig Ae/Be stars,” AAP 417, 159–168 (Apr. 2004).
  * [21] Laws, A. S. E., Harries, T. J., Setterholm, B. R., Monnier, J. D., Rich, E. A., Aarnio, A. N., Adams, F. C., Andrews, S., Bae, J., Calvet, N., Espaillat, C., Hartmann, L., Hinkley, S., Isella, A., Kraus, S., Wilner, D., and Zhu, Z., “Irregular Dust Features around Intermediate-mass Young Stars with GPI: Signs of Youth or Misaligned Disks?,” ApJ 888, 7 (Jan. 2020).
  * [22] Setterhol, B. R., Monnier, J. D., Bouquin, J.-B. L., Anugu, N., Labdon, A., Ennis, J., Johnson, K. J. C., Kraus, S., and ten Brummelaar, T., “Mirc-x polarinterferometry at chara,” in [Optical and Infrared Interferometry and Imaging VII ], Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 11446-24 (2020).
  * [23] Kraus, S., “Star Formation and Fundamental Stellar Astrophysics with VLTI at 1 micrometer,” in [The Very Large Telescope in 2030 ], 36 (July 2019).