11institutetext: Department of Mathematics and Computer Science,
University of Marburg, Marburg, Germany
11email<EMAIL_ADDRESS>

#  ElasticHash: Semantic Image Similarity Search by Deep Hashing with
Elasticsearch

Nikolaus Korfhage  Markus Mühling  Bernd Freisleben

###### Abstract

We present ElasticHash, a novel approach for high-quality, efficient, and
large-scale semantic image similarity search. It is based on a deep hashing
model to learn hash codes for fine-grained image similarity search in natural
images and a two-stage method for efficiently searching binary hash codes
using Elasticsearch (ES). In the first stage, a coarse search based on short
hash codes is performed using multi-index hashing and ES terms lookup of
neighboring hash codes. In the second stage, the list of results is re-ranked
by computing the Hamming distance on long hash codes. We evaluate the
retrieval performance of ElasticHash for more than 120,000 query images on
about 6.9 million database images of the OpenImages data set. The results show
that our approach achieves high-quality retrieval results and low search
latencies.

###### Keywords:

deep hashing similarity search Elasticsearch

##  1 Introduction

Query-by-content approaches based on feature representations that are learned
by deep convolutional neural networks (CNNs) have greatly increased the
performance of content-based image retrieval systems. However, state-of-the-
art methods in the field of semantic image similarity search suffer from
shallow network architectures and small data sets with few image classes in
the training as well as in the evaluation phases. Few image classes in the
training phase lead to poor generalizability to query images with unknown
content in the evaluation phase, i.e., a more fine-grained modeling of the
image content is required. Thus, high accuracy for arbitrary search queries,
fast response times, and scalability to millions of images are necessary to
meet many users’ needs both in scientific and commercial applications.

In this paper, we present ElasticHash, a high-quality, efficient, and scalable
approach for semantic image similarity search based on the most popular
enterprise full-text search and analytics engine
Elasticsearch111https://www.elastic.co (ES). ES processes queries very fast
due to inverted indices based on Lucene222https://lucene.apache.org, scales to
hundreds of servers, provides load balancing, and supports availability and
reliability. Apparently, the properties of ES are not only desirable for full-
text search, but also for semantic image similarity search. Furthermore,
integrating image similarity search into ES allows multi-modal queries, e.g.,
combining text and images in a single query. The contributions of the paper
are as follows:

  * •

We present ElasticHash, a novel two-stage approach for semantic image
similarity search based on multi-index hashing and integrate it via terms
lookup queries into ES.

  * •

We present experimental results to show that ElasticHash achieves fast
response times and high-quality retrieval results at the same time by
leveraging the benefits of short hash codes (better search times) and long
hash codes (higher retrieval quality). To the best of our knowledge, we
provide the first evaluation of image similarity search for more than 120,000
query images on about 6.9 million database images of the OpenImages data set.

  * •

We make our deep image similarity search model, the corresponding ES indices,
and a demo application available at http://github.com/umr-ds/ElasticHash.

The paper is organized as follows. In Section 2, we discuss related work.
Section 3 presents ElasticHash. In Section 4, we evaluate ElasticHash on the
OpenImages data set in terms of search latency and retrieval quality. Section
5 concludes the paper and outlines areas for future work.

##  2 Related Work

Deep learning, in particular deep CNNs, led to strong improvements in content-
based image similarity search. With increasing sizes of the underlying image
databases, the need for an efficient similarity search strategy arises. Since
high-dimensional CNN features are not suitable to efficiently search in very
large databases, large-scale image similarity search systems focus on binary
image codes for quantization or compact representations and fast comparisons
rather than full CNN features.

Recently, several deep hashing methods were introduced [25, 6, 23, 21, 2, 15,
4]. Many of them employ pairwise or triplet losses. While these methods often
achieve state-of-the-art performance on their test data sets, they are not
necessarily suitable for very large data sets and fine-grained image
similarity search based on thousands of classes. Existing deep hashing methods
are often trained using small CNNs that usually cannot capture the granularity
of very large image data sets. Often, CNN models like AlexNet [12] are used as
their backbones, and they are usually evaluated on a small number of image
classes [22, 4, 6, 25] (e.g., a sample of 100 ImageNet categories [4], about
80 object categories in COCO [14], NUS-WIDE [5] with 81 concepts, or even only
10 classes as in MNIST or CIFAR). Additionally, the image dimensions in CIFAR
and MNIST are very small (32x32 and 28x28, respectively) and thus not
sufficient for image similarity search in real-world applications. Many
approaches are trained on relatively small training data sets (e.g., 10,000 -
50,000 images [3, 4, 15]). In addition, there are no standardized benchmark
data sets, and each publication uses different splits of training, query, and
database images, which further complicates a comparison of the methods.
Furthermore, training from scratch can be prohibitively expensive for large
data sets. We observed that for large data sets with a high number of image
classes, a transfer learning approach that combines triplet loss and
classification loss leads to good retrieval results. To the best of our
knowledge, ElasticHash is the first work that presents a deep hashing model
trained and evaluated on a sufficiently large number of image classes.

The currently best performing approaches for learning to hash image
representations belong either to product quantization (PQ) methods [8, 9] and
methods based on deep hashing (DH) [23, 6]. Amato et al. [1] present PQ
approaches that transform neural network features into text formats suitable
for being indexed in ES. However, this approach cannot match the retrieval
performance of FAISS [9]. Therefore, we focus on deep hashing that in
combination with multi-index hashing (MIH) [17] can circumvent exhaustive
search in Hamming space and achieve low search latency while maintaining high
retrieval quality.

ElasticHash is related to other image similarity search methods integrated
into ES. For example, FENSHSES [16] integrates MIH into ES and has a search
latency comparable to FAISS. The method works efficiently for small radii of
the Hamming ball and relatively small data sets (500,000 images). Small
hamming radii, however, often produce too few neighbors for a query [17]. MIH
like FENSHSES is thus not suitable for our scenario of large-scale image
retrieval in ES with long binary codes (256 bits), where we require sub-second
search latency on a data set of about 7 million images. Furthermore, we solve
the shortcomings of FENSHSES using only a subset of bits rather than the whole
hash codes to perform our MIH-based coarse search. While other works extend ES
for image similarity search by modifying the Lucene library [7], our approach
is seamlessly integrated into ES without modifying its code base.

##  3 ElasticHash

ElasticHash consists of several components as shown in Figure 1: a deep
hashing component, an ES cluster, and a retrieval component. The deep hashing
component is realized as a web service using Tensorflow Serving where the
integrated deep hashing model is applied to images and the corresponding
binary codes are returned. In the first phase, the binary codes are extracted
from the database images in the indexing phase using the deep hashing
component and stored into the ES cluster. After initially building the index,
the retrieval component handles incoming query images and visualizes the
retrieval results. For this purpose, the binary codes are extracted from the
query images using the web service, the corresponding ES queries are assembled
and sent to the ES cluster that returns the final list of similar images. The
entire similarity search system can be easily deployed for production via
Docker.

The deep hashing model is described in more detail in Section 3.1, including
the training strategy and network architecture. In Section 3.2, the ES
integration is presented.

Figure 1: Overview of the workflows for image similarity search in ES.

###  3.1 Deep Hashing Model

We now describe our deep hashing model and how it is used to extract both
short and long hash codes.

The model training consists of two phases that both use ADAM as the
optimization method. First, an ImageNet-pretrained EfficientNetB3 [19] model
is trained on a data set with a larger number of classes in order to obtain a
more fine-grained embedding. In contrast to the original ImageNet dataset, it
contains all ImageNet classes with more than 1000 training images and all
classes of the Places2 [24] data set, which results in a total number of
classes of 5,390. The model is trained with cross-entropy loss on a Softmax
output. After two epochs of training the final layer with a learning rate of
0.01, all layers are trained for another 16 epochs with a learning rate of
0.0001.

In the second phase, the classification model’s weights are used to initialize
the deep hashing model. This model includes an additional 256-bit coding layer
before the class output layer with $tanh$ activation and 256 outputs. This
model is trained for 5 epochs with a learning rate of 0.0001. It is trained on
the same data set as before, however, by combining cross-entropy loss on the
output and hard triplet loss [18] on the coding layer.

With the classification loss

$\mathcal{L}_{c}=\sum_{i=1}^{K}y_{i}\log{p}_{i}$ (1)

for $K$ classes with labels $y_{i}$ and predictions $p_{i}$, and the triplet
loss

$\mathcal{L}_{t}=max(d(a,p)-d(a,n)+\gamma,0)$ (2)

for Euclidean distance $d$ between the 256-dimensional output of the coding
layer for anchor image $a$ and positive example $p$ and between $a$ and a
negative example $n$, respectively, the combined loss function is given by:

$\mathcal{L}=\alpha\mathcal{L_{c}}+\beta\mathcal{L_{t}},$ (3)

where we set margin $\gamma=2$ and weights $\alpha=1$ and $\beta=5$. We first
sample a batch of size $b=128$ images from a uniform distribution of the
classes. This batch is used for both computing the classification loss and
generating $b$ hard triplets. To make the similarity search more robust, we
used heavy data augmentation in both phases, which in addition to standard
augmentation methods includes inducing JPEG compression artifacts. After
training, the model generates 256-bit codes. These codes can be decomposed
into four 64-bit codes for fast computation of Hamming distance on long
integers. However, using codes of this length on a corpus of about 10 million
images is too expensive, even when using multi-index hashing. We therefore
extracted 64-bit codes from the original 256-bit codes to perform the
filtering on shorter codes and thus smaller Hamming ball radii. To extract the
64 most important bits from the 256-bit codes, we first partition the 256-bit
codes into four partitions by applying the Kernighan-Lin algorithm [10] on the
bit correlations. From each of the four decorrelated partitions, we then take
the first 16 bits to compose 64-bit codes.

###  3.2 Integration into ES

Before describing our image similarity search integration into ES, we will
shortly review MIH in Hamming space [17]. The idea of MIH is based on the
following observation: for two binary codes $h=(h^{1},...,h^{m})$ and
$g=(g^{1},...,g^{m})$ where $m$ is the number of partitions, $h^{k}$ and
$g^{k}$ are the $k^{th}$ subcodes and $H$ is the Hamming norm, the following
proposition holds:

$\left\|h-g\right\|_{H}\leq r\Rightarrow\exists
k\in\left\\{1,...,m\right\\}\>\left\|h^{k}-g^{k}\right\|_{H}\leq\left\lfloor\frac{r}{m}\right\rfloor$
(4)

For the case of 64-bit codes that are decomposed into $m=4$ subcodes, this
means that a code is in a Hamming radius $r<12$ if at least one of the
subcodes has a distance of $d\leq\lfloor\frac{r}{m}\rfloor=2$ from the query
subcode. The performance of MIH can be increased if the subcodes are maximally
independent of each other [20], especially for shorter codes [16]. Thus, after
training a deep hashing model, the bit positions should be permutated
accordingly.

The ES index used for retrieval contains four short codes (f_0 - f_3) and four
long subcodes (r_0 - r_3) for each image. The short codes are used for MIH and
efficiently utilize the reverse index structure of ES and are thus separated
into four subcodes of type ”keyword”. The long codes are also separated into
four subcodes in order to allow fast computation of Hamming distances for
values of type long.

An additional index is used for fast lookup of neighboring subcodes within the
retrieval query. The neighbors index does not change once it has been created
and merely serves as an auxiliary index for term queries. It requires pre-
computing all nearest neighbors for all possible 16-bit subcodes. Thus, the
index of neighbors contains $2^{16}$ documents. The document id corresponds to
the unsigned integer representation of a 16-bit subcode and can therefore
accessed within a term query. It contains a single field ”nbs” that is
assigned to a list of all neighboring 16-bit codes within a Hamming radius of
$d$ of the corresponding query subcode. Since this index basically works as a
lookup table, it could also be realized somewhere else, i.e., not as an ES
index. However, integrating the lookup table this way eliminates the need for
external code and enables fast deployment of the whole system. All documents
representing all possible 16-bit subcodes are inserted according to the query
in Listing 1.

⬇

POST /nbs-d2/_doc/<16 bit subcode>

{ "nbs" : [ <d2 neighbors of 16 bit subcode> ] }

Listing 1: Query for adding an entry to neighbor lookup index.

In this stage, MIH is realized by querying the additional index of neighbors
for fast neighbor lookup. Even with MIH, using the full code length of the
deep hashing model trained for 256-bit codes is too expensive for larger
databases. We therefore limit the code length for the filtering stage to
64-bit codes. To obtain a sufficiently large set of candidate hash codes in
the first stage, we need to search within a Hamming ball with a
correspondingly large radius. We set $d=2$, which will return at least all
codes within $r=11$ of a 64-bit code. In our setting with $d=2$, this results
in $137$ neighbors per subcode, i.e., $548$ neighbors in total.

In ES, we realize MIH by using a terms lookup. It fetches the field values of
an existing document and then uses these values as search terms (see Listing
1). In contrast to putting all neighbors into the query, using a dedicated
index for subcode neighbors has the advantage that the retrieval of
neighboring subcodes is carried out within ES. Thus, the query load is small,
and no external handling of neighbor lookup is necessary.

In the second stage, all codes obtained by MIH are re-ranked according to
their Hamming distance to the long code. To compute the Hamming distance of
the 256-bit code, the Painless Script in Listing 2 is applied to each of the
four subcodes.

⬇

POST _scripts/hd64

{ "script": { "lang": "painless",

"source": 64-Long.bitCount(params.subcode^doc[params.field].value) } }

Listing 2: Query for adding a Painless Script.

The query in Listing 3 combines the MIH step as a filter with a term query and
the re-ranking step as an application of the painless script from Listing 2 on
the filtered retrieval list.

⬇

GET /es-retrieval/_search

{ "query": {

"function_score": {

"boost_mode": "sum", "score_mode": "sum",

"functions": [ ..., {

"script_score": {

"script": { "id": "hd64",

"params": {

"field": "r_<i>",

"subcode": <64 bit subcode for re-ranking> } } }, "weight": 1 }, ... ],

"query": {

"constant_score": {

"boost": 0.

"filter": {

"bool": {

"minimum_should_match": 1,

"should": [ ..., {

"terms": {

"f_<j>": {

"id": "<16 bit subcode for lookup>",

"index": "nbs-d2",

"path": "nbs" } } }, ... ] } } } }, } } }

Listing 3: Query for performing two-stage similarity search.

##  4 Experimental Evaluation

To determine the search latency and retrieval quality of ElasticHash, we
evaluate three settings for using the binary hash codes generated by our deep
hashing model for large-scale image retrieval in ES: (1) short codes, i.e., 64
bits for both filtering and re-ranking, (2) long codes, i.e., 256 bits for
both filtering and re-ranking, and (3) ElasticHash, i.e., 64 bits for
filtering, 256 bits for re-ranking. Settings (1) and (2) are similar to the
MIH integration of Mu et al. [16].

To evaluate our approach, we use OpenImages [13], which is currently the
largest annotated image data set publicly available. It contains multi-label
annotations for 9.2 million Flickr images with 19,794 different labels and is
partitioned into training, validation, and test data set. On the average,
there are 2.4 positive labels for the training split, while the validation and
test splits have 8.8. As our database images we use all training images being
available when downloading the data set, i.e., 6,942,071 images in total. To
evaluate the retrieval quality, we use all downloaded images from the
OpenImages test and validation set as query images (121,588 images in total).
From these images, we draw a sample of 10,000 images to measure the search
latencies for the three different settings.

The quality of the retrieval lists is evaluated using the average precision
(AP) score, which is the most commonly used quality measure in image
retrieval. The AP score is calculated from the list of retrieved images as
follows:

$AP(\rho)=\frac{1}{\left|R\cap\rho^{N}\right|}\sum_{k=1}^{N}\frac{\left|R\cap\rho^{k}\right|}{k}\psi(i_{k}),\\\
\textrm{with}\quad\psi(i_{k})=\begin{cases}\ 1&\mbox{if }i_{k}\in R\\\\[5.0pt]
\ 0&\mbox{ otherwise }\end{cases}$ (5)

where $N$ is the length of the ranked image list,
$\rho^{k}=\\{i_{1},i_{2},\ldots,i_{k}\\}$ is the ranked image list up to rank
$k$, $R$ is the set of relevant documents, $\left|R\cap\rho^{k}\right|$ is the
number of relevant images in the top-$k$ of $\rho$ and $\psi(i_{k})$ is the
relevance function. We consider an image as relevant, if it has at least one
label in common with the query image. To evaluate the overall performance, the
mean AP score is calculated by taking the mean value of the AP scores over all
queries.

###  4.1 Results

We first evaluate the search latency for the queries. Next, we compare the
retrieval quality in terms of AP. The experiments were performed on a system
with an Intel Core i7-4771 CPU @ 3.50GHz and 32 GB RAM.

Figure 2: Top-10 retrieval results for (a) short codes, (b) long codes, and (c) ElasticHash for the same query image (first on the left); green: relevant result; red: irrelevant result. Table 1: Retrieval quality in terms of mean AP for different thresholds of $k$ on 121,588 query images. top $k$ | 10 | 25 | 50 | 100 | 250 | 500 | 1000
---|---|---|---|---|---|---|---
short | 87.94 | 86.08 | 84.44 | 82.54 | 79.41 | 76.44 | 72.86
long | 95.35 | 94.72 | 94.23 | 93.71 | 92.90 | 92.09 | 90.95
ElasticHash | 95.21 | 94.48 | 93.90 | 93.22 | 92.02 | 90.61 | 88.42
Table 2: Search latencies for ES queries (ms) with standard deviation for different thresholds of $k$ on 10,000 query images. top $k$ |  | 10 | 25 | 50 | 100 | 250 | 500 | 1000
---|---|---|---|---|---|---|---|---
short | $\mu$ | 23.09 | 23.98 | 24.45 | 25.58 | 28.38 | 33.09 | 42.20
$\sigma$ | 4.74 | 4.65 | 4.70 | 4.72 | 4.86 | 5.20 | 6.07
long | $\mu$ | 111.83 | 111.58 | 111.99 | 113.05 | 116.77 | 121.98 | 132.60
$\sigma$ | 16.50 | 16.58 | 16.72 | 16.54 | 17.04 | 17.13 | 17.99
ElasticHash | $\mu$ | 36.12 | 36.75 | 37.28 | 38.17 | 40.88 | 45.73 | 55.23
$\sigma$ | 7.80 | 7.96 | 7.81 | 7.89 | 7.93 | 8.12 | 8.64

Table 1 shows that for a $k$ up to 250 there is no notable decrease in
retrieval quality when employing ElasticHash rather than using the long codes
for both stages. Figure 2 shows examples of the top-10 retrieval results for
the three settings. It is evident that the retrieval quality of ElasticHash is
similar to using long codes, and both are superior to using short codes. On
the other hand, Table 2 indicates that the average retrieval time only
slightly increases compared to using short codes for both stages. This
suggests that ElasticHash is a good trade-off between retrieval quality and
search latency. Although our deep hashing model was trained on 5,390 classes,
but almost 20,000 classes occur in the validation data set, high AP values are
achieved for ElasticHash.

##  5 Conclusion

We presented ElasticHash, a novel two-stage approach for semantic image
similarity search based on deep multi-index hashing and integrated via terms
lookup queries into ES. Our experimental results on a large image data set
demonstrated that we achieve low search latencies and high-quality retrieval
results at the same time by leveraging the benefits of short hash codes
(better search times) and long hash codes (higher retrieval quality).

There are several areas for future work. For example, it would be interesting
to investigate how many classes are necessary to obtain a high degree of
generalizability. Furthermore, our loss function could be adapted to multi-
label image data. Finally, we plan to extend our approach to achieve
intentional image similarity search [11] using ES.

##  6 Acknowledgements

This work is financially supported by the German Research Foundation (DFG
project number 388420599) and HMWK (LOEWE research cluster Nature 4.0).

## References

  * [1] Amato, G., Bolettieri, P., Carrara, F., Falchi, F., Gennaro, C.: Large-scale image retrieval with Elasticsearch. In: The 41st International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 925–928 (2018)
  * [2] Cao, Y., Qi, H., Zhou, W., Kato, J., Li, K., Liu, X., Gui, J.: Binary hashing for approximate nearest neighbor search on big data: A survey. IEEE Access 6, 2039–2054 (2017)
  * [3] Cao, Z., Long, M., Wang, J., Yu, P.S.: Hashnet: Deep learning to hash by continuation. In: Proc. IEEE Int. Conf. on Computer Vision. pp. 5608–5617 (2017)
  * [4] Cao, Z., Sun, Z., Long, M., Wang, J., Yu, P.S.: Deep priority hashing. In: Proc. of the 26th ACM Int. Conference on Multimedia. pp. 1653–1661 (2018)
  * [5] Chua, T.S., Tang, J., Hong, R., Li, H., Luo, Z., Zheng, Y.: NUS-WIDE: a real-world web image database from national university of singapore. In: Proceedings of the ACM International Conference on Image and Video Retrieval. pp. 1–9 (2009)
  * [6] Erin Liong, V., Lu, J., Wang, G., Moulin, P., Zhou, J.: Deep hashing for compact binary codes learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2475–2483 (2015)
  * [7] Gennaro, C., Amato, G., Bolettieri, P., Savino, P.: An approach to content-based image retrieval based on the Lucene search engine library. In: Proc. 14th Eur. Conf. on Research and Advanced Technology for Digital Libraries. pp. 55–66 (2010)
  * [8] Jegou, H., Douze, M., Schmid, C.: Product quantization for nearest neighbor search. IEEE Trans. on Pattern Analysis and Machine Intell. 33(1), 117–128 (2010)
  * [9] Johnson, J., Douze, M., Jégou, H.: Billion-scale similarity search with GPUs. IEEE Transactions on Big Data (2019)
  * [10] Kernighan, B.W., Lin, S.: An efficient heuristic procedure for partitioning graphs. The Bell System Technical Journal 49(2), 291–307 (1970)
  * [11] Korfhage, N., Mühling, M., Freisleben, B.: Intentional image similarity search. In: IAPR Workshop on Artificial Neural Networks in Pattern Recognition. pp. 23–35. Springer (2020)
  * [12] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25, 1097–1105 (2012)
  * [13] Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al.: The open images dataset v4. International Journal of Computer Vision pp. 1–26 (2020)
  * [14] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: Common objects in context. In: European Conference on Computer Vision. pp. 740–755. Springer (2014)
  * [15] Liu, H., Wang, R., Shan, S., Chen, X.: Deep supervised hashing for fast image retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2064–2072 (2016)
  * [16] Mu, C.M., Zhao, J.R., Yang, G., Yang, B., Yan, Z.J.: Fast and exact nearest neighbor search in hamming space on full-text search engines. In: International Conference on Similarity Search and Applications. pp. 49–56. Springer (2019)
  * [17] Norouzi, M., Punjani, A., Fleet, D.J.: Fast search in hamming space with multi-index hashing. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3108–3115. IEEE (2012)
  * [18] Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: A unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 815–823 (2015)
  * [19] Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning. pp. 6105–6114 (2019)
  * [20] Wan, J., Tang, S., Zhang, Y., Huang, L., Li, J.: Data driven multi-index hashing. In: 2013 IEEE International Conference on Image Processing. pp. 2670–2673 (2013)
  * [21] Wang, J., Zhang, T., Song, J., Sebe, N., Shen, H.T.: A survey on learning to hash. IEEE Trans. on Pattern Analysis and Machine Intelligence 40(4), 769–790 (2018)
  * [22] Wang, J., Kumar, S., Chang, S.F.: Semi-supervised hashing for large-scale search. IEEE Trans. on Pattern Analysis and Machine Intell. 34(12), 2393–2406 (2012)
  * [23] Wang, J., Liu, W., Kumar, S., Chang, S.F.: Learning to hash for indexing big data - a survey. Proceedings of the IEEE 104(1), 34–57 (2015)
  * [24] Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 40(6), 1452–1464 (2017)
  * [25] Zhu, H., Long, M., Wang, J., Cao, Y.: Deep hashing network for efficient similarity retrieval. In: Proc. of the AAAI Conf. on Artificial Intelligence. vol. 30 (2016)


# A review on magnetic field induced spin crossover in LaCoO3 up to 600 T

Akihiko Ikeda<EMAIL_ADDRESS>University of Electro-Communications, Chofu,
Tokyo 182-8585, Japan  Yasuhiro H. Matsuda  Institute for Solid State Physics,
University of Tokyo, Kashiwa, Chiba 277-8581, Japan  Keisuke Sato  National
Institute of Technology, Ibaraki College, Hitachinaka, Ibaraki 312-0011, Japan
Joji Nasu  Department of Physics, Tohoku University, Sendai, Miyagi 980-8578,
Japan  PRESTO, Japan Science and Technology Agency, Honcho Kawaguchi, Saitama
332-0012, Japan

###### Abstract

LaCoO3 is known for its two-step spin crossover as a function of temperature.
Despite efforts spanning over half a century, the origin of this phenomenon is
still debated, particularly regarding how the microscopic spin states are
involved in the observed macroscopic two-step spin crossover. High magnetic
field studies on LaCoO3 are performed because the magnetic field-induced spin
crossover is induced, where the magnetic excited states become more stable in
high magnetic fields than the non-magnetic ground states. This review focuses
on the findings in LaCoO3 at high magnetic fields over the last decade. A
complex phase diagram has been revealed at high magnetic fields instead of
solving the conventional problem of LaCoO3. It suggests that appreciable spin
state correlations are in play in LaCoO3. The possibility of exciton
condensation is also discussed.

###### Contents

  1. I Introduction
  2. II Previous experiments up to 100 T
  3. III Experiments above 100 T
    1. III.1 Methods
    2. III.2 Results
  4. IV Ideas on the origin of $\alpha$, $\beta$, and $\gamma$ phases
    1. IV.1 Exciton degree of freedom in LaCoO3
    2. IV.2 Spin state crystal and exciton condensation
    3. IV.3 Duality of high spin and intermediate spin states
    4. IV.4 Magnetic field induced spin state order
  5. V Summary
    1. V.1 Conclusion
    2. V.2 Future prospects

##  I Introduction

Transition metal oxide with Perovskite structure is a fertile ground for
numerous many-body phenomena like ferroelectricity, metal-insulator
transition, high-temperature superconductivity, multiferroicity, Mott
transition, and magnetic transitions [1, 2]. On the other hand, the spin
crossover phenomenon is a change in the spin states of a single ion observed
in coordination complexes. Spin crossover is a single-ion physics generally
well described by considering the energy level scheme of multi-electron states
in a single ion. The level scheme results from the competition between Hund’s
coupling and crystal field splitting at the transition metal ions surrounded
by the ligands, forming high-spin and low-spin states.

Cobaltites are unique systems among transition metal oxides due to their spin
state degree of freedom. LaCoO3 is especially well-known for its two-step spin
crossover with temperature evolution. Unlike coordination complexes, ligands
are shared in Co oxides. It leads to the appreciable interaction between
neighboring Co ions and their spin states. This gives a basis for various spin
state ordered phases in cobaltites. This review focuses on the magnetic field
effect. Thus, please refer to a thorough review [3] for the long-standing
controversy over the spin state evolution in LaCoO3.

Figure 1: A schematic drawing of (a) crystal structure, (b) spin states, and
(c) magnetic field effect on LaCoO3.  Figure 2: (a) The magnetic
susceptibility and electric conductivity of LaCoO3 as a function of
temperature. (b) The optical conductivity at various temperatures that ranges
from 650 to 10 K. Reprinted from Ref. 4 with a re-arrangement, $\copyright$
2014 American Physical Society.  Figure 3: The schematic drawing of the
temperature evolution of the macroscopic properties of LaCoO3.

LaCoO3 shows the two-step spin crossover in the magnetic susceptibility and
electric conductivity as functions of temperature [4] as shown in Fig. 2. The
high-temperature phase above $\sim 500$ K is a paramagnetic metal. Upon
decreasing temperature across $\sim 500$ K, the electric conductivity becomes
semiconducting, indicating the formation of an appreciable charge gap. At the
same time, the magnetic susceptibility sustains the Curie paramagnetic
behavior with a reduction of the effective magnetic moment. Thus, the
intermediate temperature phase between $\sim 100$ K and $\sim 500$ K is a
paramagnetic insulator. With a further decrease in temperature, at $\sim 100$
K, the magnetic susceptibility starts to decrease, losing the Curie
paramagnetic moment. The magnetic susceptibility becomes tiny below $\sim 30$
K, indicating a spin-gap behavior where the ground state of LaCoO3 is a spin-
singlet state with a magnetic excited state at $\sim$100 K above the ground
state, that is, the spin-gap. The calculated spin gap from the magnetic
susceptibility is around 100 K. The charge gap calculated from the electric
conductivity is 140 meV [5]. The charge gap observed in the optical
conductivity measurement is 300 meV [Fig. 2]. The charge gap is larger than
the spin gap by an order of magnitude. The spin crossover is rather a robust
phenomenon that even under a high magnetic field of 33 T, the magnetic
susceptibility overlaps with the data at 10 T and 0.05 T [6]. The temperature
evolution of the macroscopic properties of LaCoO3 is summarized in Fig. 3.

##  II Previous experiments up to 100 T

Figure 4: (a) Electron spin resonance as a function of photon energy and
applied magnetic fields. (b) Obtained energy level scheme of LaCoO3. Reprinted
from Ref. 7 with a re-arrangement, $\copyright$ 2002 American Physical
Society.  Figure 5: (a) Result of a magnetization measurement up to a pulsed
high magnetic field of 70 T. (b) The temperature dependence of the high
magnetic field induced spin state transition. (c) The proposed energy diagram.
Reprinted from Ref. 8 with a re-arrangement, $\copyright$ 2009 The Physical
Society of Japan.  Figure 6: (a) Result of magnetization measurement and
magnetostriction measurement up to 100 T. (b) The phase diagram up to 100 T
and 30 K. (c) The proposed magnetization evolution up to 200 T with schematic
drawings of spin state crystals realized at each magnetization change.
Reprinted from Ref. 9 with a re-arrangement, $\copyright$ 2012 American
Physical Society.  Figure 7: (a) Results of transversal and (b) longitudinal
magnetostriction measurement on LaCoO3 up to 70 T. (c) The distortion and (d)
volume expansion calculated from the longitudinal and transversal
magnetostriction. Reprinted from Ref. 10 with a re-arrangement, $\copyright$
2020 Licensed under CC BY 4.0.

The first report on a pulsed high magnetic field study on LaCoO3 was the
electron spin resonance study to investigate the controversy over the
involvement of high-spin or intermediate spin states in the spin crossover at
100 K [7]. They measured electron spin resonance experiment for a single
crystalline LaCoO3 in various crystal orientations. The single crystal was
carefully de-twined to have only a single domain of the rhombohedrally
distorted sample. The results are summarized in Fig. 4. They observed
absorption peaks as a function of magnetic fields with wavelength variations.
Their results are summarized in the schematic energy diagram in Fig. 4. The
most important observation is that the excited state is a spin-triplet state
with a $g$-factor of 3.35. The crystal field and $g$-factor are isotropic with
a slight anisotropy. The interpretation is that the first excited state
originates from the high-spin state with $S=2$ and $g=2$ and not the
intermediate spin state with $S=1$ and $g=2$. The high-spin state is modified
with trigonal distortion and spin-orbit coupling, pushing the $S_{z}=\pm 1$
further upwards. As a result, the seemingly spin-triplet state is observed
with an effective $g$-factor of 3.35. This is theoretically supported by
several multiplet calculations [11] and a heat capacity measurement assuming
such an energy diagram [12]. Strikingly, they predicted by the obtained energy
diagram that the magnetic field-induced spin crossover will happen at 65.3 T
as shown in Fig. 4.

In 2009, Sato et al. discovered a sharp spin state transition at $\sim 60$ T
at 4.2 K [8]. The results are shown in Fig. 5. The transition field of $\sim
60$ T shows quite a good agreement with the prediction by Noguchi et al.[7].
However, several points are not in accordance with the simple energy diagram
presented by Noguchi et al.. The first feature is that the magnetization jump
is only $\sim 0.5$ $\mu_{\rm{B}}$/f.u. It is much smaller than the expected
saturation value of 2 $\mu_{\rm{B}}$/f.u. and 4 $\mu_{\rm{B}}$/f.u. for a high
and intermediate spin state of Co3+ with octahedral crystal field,
respectively. This indicates that LaCoO3 is far from its complete polarization
at $\sim 60$ T and that there should be more at a higher magnetic field
region. The second feature is that they clearly show the sharp transition in a
first-order manner with hysteresis with the spin gap behavior below the
transition magnetic field $\sim 60$ T. This feature shows that the cooperative
feature is present in LaCoO3, which was not evident in the temperature-induced
spin crossover. The third feature is the temperature dependence of the spin
state transition. They monotonously become smeared with increasing temperature
from 4. 2 K up to 40 K. On the other hand, the transition magnetic field is
not dependent on temperature, which does not agree with the simple energy
level scheme picture where the transition field is expected to decrease with
increasing temperature aided by the entropy effect. This study by Sato et al.
triggers all the later studies on LaCoO3 at high magnetic fields.

To extend the magnetic field range for search, Altarawneh et al. conducted a
cooperative study up to 100 T employing magnetization measurement based on
proximity detector oscillator [13], magnetostriction measurement utilizing
fiber Bragg grating (FBG) [14], and the 100 T pulsed magnetic field with
flywheel generator. They successfully observed the second transition above 70
T that followed the known transition at 60 T [9] as shown in Fig. 6. Here,
they used a single crystalline sample. They conducted a temperature dependence
study, finding that the transition fields shift to higher magnetic fields with
increasing temperatures beyond 25 K up to 30 K. However, the transition
magnetic field exceeds the available magnetic field of 100 T above 30 K. The
two-step transition at high magnetic fields motivated them to propose a
classical Ising model with the first and the second nearest neighbor
interactions. The model predicts a possible two-step transition at 160 T and
170 T, where the saturation of the magnetic moment will happen at the latter
transition, as shown in Fig. 6.

Rotter et al. further investigated the transition at 60 T using the FBG
magnetostriction method by realizing the measurement of longitudinal
($\epsilon_{l}=\Delta L/L\parallel B$) and transversal ($\epsilon_{t}=\Delta
L\perp B$) magnetostriction [10]. They deduced the volume change
$\epsilon_{v}=\epsilon_{l}+2\epsilon_{t}$ and distortion
$\epsilon_{d}=\epsilon_{l}-\epsilon_{t}$ using the simple relations. The
obtained results of $\epsilon_{l}$, $\epsilon_{t}$, $\epsilon_{v}$, and
$\epsilon_{d}$ are displayed in Fig. 7. The feature they found is that the
transition is isotropic. At the sharp transition at 60 T, only $\epsilon_{v}$
changes with little change in $\epsilon_{d}$. At above 60 T, $\epsilon_{d}$
show changes with the change of $\epsilon_{v}$. Based on this observation,
they concluded that the transition is more driven by the high spin state than
the intermediate spin state. Based on the discussion, the intermediate spin
state has the orbital degree of freedom in the $e_{g}$ orbital. In contrast,
the high spin state has the orbital degree of freedom in $t_{2g}$, which
should lead to the large anisotropy in the intermediate spin state, and the
high spin state should be isotropic.

The experiments so far only uncover a small portion of high magnetic field
phases in LaCoO3. Thus, a study above 100 T was needed.

Figure 8: (a) Schematic drawings, (b) photographs, (c) and waveforms of the
generated magnetic field and current for the primary coil and seed coils,
which are involved in the electromagnetic flux compression technique installed
in the institute for Solid State Physics, the University of Tokyo, Japan in
2014. Reprinted from Ref. 15, $\copyright$ 2020 Licensed under CC BY 4.0.
Figure 9: (a) waveforms of generated magnetic fields, (b) schematic drawing,
and (c) a series of flash x-ray radiography in the single turn coil method.
Figure 10: Schematic drawing of magnetization measurements above 100 T using
induction pick-up coil and single turn coil method. Reprinted from Ref. 17,
$\copyright$ 2012 The Physical Society of Japan.  Figure 11: (a) Schematic
drawing of magnetostriction measurements above 100 T using FBG and optical
filter method. The shift of the Bragg wavelength is converted to the change of
the intensity of the reflected light from the FBG by utilizing the optical
filter method. The light intensity change is detected by an avalanche
photodiode (APD) with 100 MHz high speed in a single shot measurement. (b)
Examples of the spectrum of optical filters used. (c) A photograph of a solid
sample glued to an optical fiber with FBG Reprinted from Ref. 18, $\copyright$
2017 American Institute of Physics.

##  III Experiments above 100 T

###  III.1 Methods

One needs a destructive pulse magnet to obtain a high magnetic field well
beyond 100 T experimentally. Electromagnetic flux compression is a method to
obtain a high magnetic field well beyond 500 T. A new instrument of
electromagnetic flux compression is installed in the Institute for Solid State
Physics, the University of Tokyo, Kashiwa, Japan, where, in 2018, we reported
the generation of the world’s highest magnetic field indoor of 1200 T [19].
The principle of electromagnetic flux compression is briefly explained. First,
a seed pulse magnet generates the seed magnetic field of $\sim 3$ T inside a
metallic cylinder of 11 cm diameter, a liner. The seed magnetic field inside
the liner is rapidly compressed to a diameter of less than 1 cm to obtain a
few hundred times larger magnetic field. This is realized using the implosion
of the metallic cylinder, which is driven by the electromagnetic repulsive
force generated by injecting the pulsed current of 2-5 MA to the primary coil.
The pulse width of the seed magnetic field and primary current are about 50 ms
and 100 $\mu$s, respectively. The final waveform of the generated magnetic
field is shown in Fig. 8. Right after the magnetic field is generated, a large
explosion occurs due to the mechanical energy of the coils. The samples,
cryostat, and probes do not survive the explosion. The pulse power for
generating the primary current of 5 MA is rated at 50 kV and 5 MJ, which
occupies the whole room of 900 m2. The explosion happens in the next room with
a special chamber equipped with a shock-absorbing balloon. In addition to the
5 MJ system, they installed the 2 MJ system in parallel to generate fields up
to 400 T.

One can obtain a pulsed high magnetic field up to 250 T using single turn coil
method in the same institute [20]. In the single turn coil method, a current
of 2-3 MA is injected into a copper coil with a single turn. The pulse
duration is about 7 $\mu$s. The destruction of the coil follows the generation
of the magnetic field. The coils deform and scatter outwards. Therefore, the
sample, cryostat, and probes are kept safe after each magnetic field
generation. One can repeat the experiment by replacing the exploded single
turn coil with a new one, which is a beauty of the single turn coil method
compared to the electromagnetic flux compression method.

Figure 12: Results of magnetization measurements on LaCoO3 up to 130 T using
single turn coil method and induction method for the magnetization
measurement. (a) $dM/dt$ as a function of time, (b) magnetization as a
function of magnetic field, (c) derivative of magnetization as a function of
magnetic field, (d) temperature-magnetic field phase diagram, (e) the change
of magnetization jump at the phase boundary, (f) the calculated change of
entropy at the phase boundary using Clausius-Clapeyron relation of the first
order phase transition. Reprinted from Ref. 21 with a re-arrangement,
$\copyright$ 2016 American Physical Society.

Magnetization measurement above 100 T is performed using an induction method
[17], which is schematically presented in Fig. 10. The magentization is
measured as the induction signal from the magnetization pick up coil which is
made of a pair of pick-up coils with a diameter of 1 mm placed in parallel.
Using this pick-up coil, the induction voltage from only the magnetization
change of the sample inserted either one of the pick-up coil is measured. The
induction voltage from the external magnetic field is canceled out because the
pair of the pick-up coils are connected in that way. Still, however, to
subtract the non-negligible background contribution, we have to measure two
successive data with and without the sample by generating magnetic field pulse
two times. This fact that the induction measurement is not a single shot
measurement is a considerable drawback of the technique because the 100 T
generation relies on non-linear phenomena of air-gap discharge which is not
easy to reproduce. The technique is successful in single turn coil method.
However, it is not readily applicable to the electromagentic flux compression
technique. One reason is that the reproducibility of the magnetic field is
poor. Another reason is that one can not use the identical pick-up coil
because it is lost in each magnetic field generation in the electromagnetic
flux compression. This aspect limits our measurement of magnetization up to
200 T or so. Further innovation in magnetization measurement is mandatory for
its realization at 1000 T.

Figure 13: Results of magnetostriction measurement up to 150 T on LaCoO3 using
FBG and optical filter method. (a) Magnetostriction as a function of magnetic
field, (b) Derivative of magnetostriction as a function of magnetic field, (c)
Magnetostriction and magnetic field as a function of time, (b) A phase diagram
summarizing the magnetostriction and magnetization data. Reprinted from Ref.
16 with a re-arrangement, $\copyright$ 2020 Licensed under CC BY 4.0.

Magnetostriction above 100 T is performed using the FBG-based optical filter
method, which we developed [18]. The method is a single-shot measurement.
Thus, it applies to the 100 T experiment in the single turn coil method and
the 1000 T experiment in the electromagnetic flux compression experiment. FBG
is an optical fiber with some region of a few millimeters where modulation of
the refractive index is present at the core. The modulation works as the Bragg
grating, reflecting only the light with the Bragg wavelength backward with
close to 100 % reflectivity. Upon the physical elongation or shrinkage of the
FBG itself, the Bragg wavelength shifts. Hence, by optically measuring the
Bragg wavelength, one can measure the length change of the optical fiber and
the sample glued to the optical fiber. Daou et al. first reported the
magnetostriction measurement using FBG with pulse magnets [14]. The obtained
resolution is about 10-7. It was developed for the measurement with
milliseconds pulsed magnetic fields with a time resolution of 47 kHz. After
the report, it is employed for various measurements up to 100 T [22, 23].
There was a technical difficulty in applying the FBG magnetostriction method
to above 100 T region because the destructively generated magnetic fields last
only a few $\mu$ seconds. The measurement speed was limited by the detection
scheme of the wavelength of the reflected light from fiber Bragg gratings. The
previous method relies on fast optical spectroscopy with 50 kHz. However, we
needed 100 MHz measurements for the 100 T and the 1000 T experiments. So, we
devised a technique called the optical filter method, as schematically shown
in Fig. 11. The technique uses an optical filter to shape the spectrum of the
incoming light to FBG with the sample so that the intensity of the reflected
light by the FBG is changed with the shift of the Bragg wavelength. In this
way, one can detect the magnetostriction of the sample with the intensity of
the light, which can be detected as fast as 100 MHz [18] with a resolution of
$2\times 10^{-5}$. The technique is applied to various samples, including
frustrated spin systems [24, 25] and even to liquid oxygen [26]. The method
can also be used for millisecond measurement with a better resolution of
$10^{-6}$ below 2 K [27], which has been successfully applied for magnetic
insulator volborthite [28], a ternary metallic sample Eu122 system [29], a
$5f$ heavy fermion system UTe2 [30], and many other system.

Figure 14: (a) Magnetostriction at 20 T and 50 T with magnetization at 45 T as
a function of temperature. (b) Magnetostriction and (c) magnetization in
magnetostriction plateau above 100 T as a function of the initial sample
temperature. (d) Magnetization and (e) magnetostriction behaviors at around 70
K and below 10 K shown for comparison. Reprinted from Ref. 16 with a re-
arrangement, $\copyright$ 2020 Licensed under CC BY 4.0.  Figure 15: Results
of magnetostriction measurement on LaCoO3 up to 600 T using electromagnetic
flux compression and FBG with optical filter method. Reprinted from Ref. 15,
$\copyright$ 2023 Licensed under CC BY 4.0.

###  III.2 Results

First, using the magnetization measurements, we have successfully observed the
magnetic field-induced phase transition up to 130 T for a collection of non-
oriented single crystalline samples [21]. The results are shown in Fig. 12. By
making the most of the magnetic field range of 130 T, more significant than
the previous studies, we have further investigated the temperature dependence
of the magnetic field-induced phase transition up to the initial temperature
of 120 K. Surprisingly, as one can see in the $dM/dB$ data as a function of
magnetic fields, the transition magnetic field suddenly jumps from 60-70 T to
more than 100 T when the initial temperature exceeds 30 K. Another noteworthy
trend is that the transition magnetic field positively depends on the initial
temperature beyond 30 K. The most critical finding is two kinds of high
magnetic field-induced phases at magnetic fields beyond 70 T and 100 T with
initial temperatures below and above 30 K, respectively, summarized and
clearly shown in the phase diagram in Fig. 12. This was a counter-intuitive
result. In a simple spin crossover model of a single ion, the magnetic field-
induced spin crossover will happen at the lower magnetic field with increasing
temperature because spin crossover to the excited state is induced
cooperatively by the temperature and magnetic field increases. Thus, the
experimental observation denies the single-ion picture. Instead, it strongly
supports the substantial contribution of inter-site interactions leading to
the many-body phases at high magnetic fields. We here term the high magnetic
field phases at low temperature and the high magnetic field phases at high
temperature as the $\alpha$ phase and the $\beta$ phase, respectively. The
result does not reproduce the previously observed two-step phase transition at
60 T and 70 T in 9. This may be because we used a collection of non-oriented
single crystals in the measurement above 100 T.

To obtain further insights into the observed high magnetic field-induced
phases above 100 T, we resorted to magnetostriction measurements up to 150 T
with a wide temperature range between 9 - 292 K [16]. As one can see by
comparing the magnetization data in Fig. 12 and in magnetostriction data in
Fig. 13, the trend of the transition magnetic fields in the magnetization
measurements are reproduced in the magnetostriction measurement, which is
further extended to higher magnetic fields up to 150 T. The striking finding
in the magnetostriction data up to 150 T is that magnetostriction $\Delta L/L$
changes at the transition are markedly distinct between the transition to the
$\beta$ phase and to the $\alpha$ phase. This is the height of $\Delta L/L$ at
the plateau region in Fig. 13(c). They are summarized in Fig. 14(b). The
distinct features are seen in the magnetostriction data. In contrast, it is
not observable in the magnetization data. Experimentally, magnetostriction
measurements and magnetization measurements are different at several points.
First, the former is a single-shot experiment, whereas the latter requires two
shots for each data point, leading to data at higher magnetic fields for
magnetostriction measurement. Secondly, magnetostriction is sensitive to
$\Delta L$ itself, whereas the magnetization measurement is sensitive to
$dM/dt$. This fact leads to better sensitivity at the top region of the pulsed
magnetic field for magnetostriction measurements than for magnetization
measurements. This fact leads to the present findings made using
magnetostriction data. The result strengthens the evidence that the $\alpha$
and the $\beta$ phases are distinct.

Figure 16: Result of a magnetization measurement using induction method and an
explosive driven flux compression technique. Reprinted from Ref. 31 with a re-
arrangement, $\copyright$ 2012 Springer.

To further extend the phase diagram, we measured magnetostriction on LaCoO3 up
to 600 T, using electromagnetic flux compression and FBG with optical filter
method. We have obtained results with 4 different initial temperatures of 5,
78, 108, and 185 K. The results are shown in Fig. 15a-15d. The magnetic fields
are generated commonly with a maximum magnetic field of about 600 T. However,
the wave profiles are not consistent with each other. For example, the time to
reach the maximum magnetic field varies from shot to shot. This results from
the fact that the total energy of the pulse power varied from shot to shot due
to the instability of the air gap switches. As seen in Fig. 15e-15h, the
behavior of magnetostriction changes with the changes in the initial
temperatures. At 5 K, magnetostriction shows an abrupt increase, followed by a
long plateau feature. Note that the plateau feature is disturbed by a decaying
oscillation, which is a shock wave generated by the abrupt magnetostriction
jump, also observed in the previous studies up to 150 T using the single turn
coil method [16]. At 78 K, we observed a very sharp feature of slopes with
negative and positive slopes. The change of the sign of the slopes from
negative to positive occurs at 380 T. At 108 K, a similar feature is observed
with a more thermally smeared nature. This feature is more smeared at a higher
temperature of 185 K. We summarize the observed feature in the temperature-
magnetic field phase diagram in Fig. 15i. The phase diagram is constructed
using the previous results up to 150 T [21, 16] in combination with the
present results up to 600 T.

Note that there is a previous report on the magnetization measurement on
LaCoO3 up to 500 T at 4.2 K [31]. The magnetic field was generated using a
flux compression method driven by explosives [32]. The magnetization data is
measured using a single shot induction method, where the background of the
magnetic fields is subtracted, assuming that the background signal is
proportional to the generated magnetic fields. As seen in Fig. 16(a), the raw
data of the $dM/dt$ signal shows several peak structures indicating some
magnetization increase. The obtained magnetization data is presented as a
function of magnetic fields in Fig. 16(b). The data does not agree with our
experimental data nor the published data below 100 T. The data may need some
calibration regarding the timing and background signal. However, the
experiment makes us believe that the magnetization measurement of LaCoO3 up to
500 T may be possible with some experimental improvement.

##  IV Ideas on the origin of $\alpha$, $\beta$, and $\gamma$ phases

As presented so far, the magnetic field-induced phases in LaCoO3 are
classified into at least 3 phases: the $\alpha$, the $\beta$, and $\gamma$
phases. We discuss the possible origins of those phases concerning the
electron correlation and spin state degree of freedom in LaCoO3. The spin
state degree of freedom is also viewed as the exciton degree of freedom. Their
origins are unclear because experimental evidence in a microscopic viewpoint
is missing. Hopefully, they will be uncovered with a development in
microscopic properties at high magnetic fields.

Figure 17: Schematic of the perovskite-type structure of LaCoO3, the
correspondence between the spin states and the exciton state. Reprinted from
Ref. 15, $\copyright$ 2023 Licensed under CC BY 4.0.  Figure 18: Schematic of
the effect of dispersion that makes the intermediate spin state ($S=1$) lowest
excited state . Reprinted from Ref. 33, $\copyright$ 2016 Licensed under CC BY
4.0.  Figure 19: Origing of the repulsive interactions between spin triplet
excitons (the intermediate spin states $S=1$), and the itineracy of the spin
triplet excitons (the intermediate spin states $S=1$). Reprinted from Ref. 34,
$\copyright$ 2012 Institute of Physics.

###  IV.1 Exciton degree of freedom in LaCoO3

As described in Fig. 17, LaCoO3 has three possible spin states from high-spin,
intermediate-spin, and low-spin states with $S=2$, 1, and 0, respectively.
These states are realized due to the competition between octahedrally
coordinated crystal electric field and Hund’s coupling in the $3d^{6}$
electrons in for Co3+. The original Tanabe-Sugano diagram with 3$d^{6}$ system
states that either low-spin or high-spin states can be a ground state with the
intermediate spin state always being higher in energy than the first excited
state. With an introduction of inter-site interaction, trigonal distortion to
the crystal field and spin-orbit couplings can modify the original Tanabe-
Sugano diagram, leading to the possible involvement of the intermediate spin
state. This is why the intermediate spin state is considered in addition to
the other two states. Kuneš et al. primarily focus on the itineracy of the
intermediate spin state ($S=1$), leading to the possible lowest excited state
[33]. This is schematically shown in Fig. 19.

###  IV.2 Spin state crystal and exciton condensation

Figure 20: Emergence of spin state ordered phase between the uniform low spin
state and the uniform HS states. Here, HS is a $S=1$ state, that can be
classified into the intermediate spin state $S=1$ in LaCoO3. Excitonic
insulator (EI) phases also appear at the intermediate region. Reprinted from
Ref. 35, $\copyright$ 2012 American Physical Society.  Figure 21: The
correspondence between spin states and Here, again, HS is a $S=1$ state that
can be classified into the intermediate spin state $S=1$ in LaCoO3.  Figure
22: Schematic of the hybridization between a pair of the high spin state
($S=2$) and the low spin state, a pair of two neighboring intermediate spin
states ($S=1$). Reprinted from Ref. 36, $\copyright$ 2020 American Physical
Society.

Nasu et al. investigated a ground state phase diagram of the two-orbital
Hubbard model by a mean-field analysis [35]. The phase diagram is constructed
as functions of crystal field splitting $\Delta$ and Hund’s coupling $J$ with
the electron hopping parameter $t_{a}$ between the $e_{g}$ orbitals and
$t_{b}$ between the $t_{2g}$ orbitals. At the limit of large $J$, it is
trivial that a polarization to the HS occurs. At the limit of large $\Delta$,
it is again natural to obtain the low spin phase. With an introduction of
inter-site interactions $t_{a}$ and $t_{b}$, non-trivial phases such as spin
state crystals and exciton condensation emerge. These two phases are
incompatible and competing with each other. A spin state ordered phase of the
intermediate spin state and low spin state emerges between the spatially
uniform low spin state and the spatially uniform intermediate spin state as
shown in Fig. 20. in Fig. 20, they term the intermediate spin state $S=1$ to
be a HS state. Microscopically, spin state crystallization occurs due to the
repulsive interactions between the intermediate spin states, which is, in
other words, the attractive interaction between the low spin state and the
intermediate spin state as is shown in Fig. 21. In addition to spin state
crystal phase, the excitonic insulator phase appears at the boundary between
the spin state crystal and low spin state, and the boundary between the spin
state crystal and the high spin state. Due to its itineracy, this phase
appears at low temperatures as a Bose-Einstein condensation of the spin
triplet exciton. The occurrence of exciton condensation in the spin crossover
systems is also reported in literature [37, 38, 34, 39]. The occurrence of
spin state crystal and exciton condensation is schematically depicted in Fig.
21.

Figure 23: A calculated temperature-magnetic field phase diagrams of LaCoO3
with low spin ground state and (a) spin state crystal and (b) exciton
condensation as the excited state. Reprinted from Ref. 33 with a re-
arrangement, $\copyright$2016 Licensed under CC BY 4.0.  Figure 24: (a) A
calculated temperature-magnetic field phase diagram for LaCoO3. It is
calculated using a two-orbital Hubbard model and a mean-field approximation.
(b) A closed-up of a region occurs at the first magnetic field-induced spin
crossover. (c) same area with slightly different parameters. Reprinted from
Ref. 40, $\copyright$ 2016 The Physical Society of Japan.  Figure 25:
Schematic drawing of (a) spin state crystal of high spin state $S=2$ and low
spin state $S=0$ (b) Exciton condensation (c) Co-occurrence of the spin state
crystal and exciton condensation (d) a phase diagram for 5 K in magnetic
fields with a parameter set 1 (e) a phase diagram for 78 K in magnetic fields
with a parameter set 2. Reprinted from Ref. 15, $\copyright$ 2023 Licensed
under CC BY 4.0.

###  IV.3 Duality of high spin and intermediate spin states

Two recent studies pointed out an interesting idea that might solve the long-
standing controversy over the high spin state and the intermediate spin
states. The high spin state and intermediate spin state can be hybridized if
we consider an electron hopping process. Hariki et al. suggested this idea
[36], which is schematically shown in Fig. 22. As can be seen, with two
electrons hopping between adjacent Co ions, a pair of intermediate spin states
can be transformed into a pair of high-spin states and low-spin states. In
more detail, an electron hopping at between $e_{g}$ orbitals of adjacent sites
and an electron hopping between $t_{2g}$ orbitals of adjacent sites will
realize this change of states. It is shown that high spin and intermediate
spin states can be hybridized. Such hybridization is supported by the
inelastic neutron scattering experiment where a ferromagnetic 7-site cluster
has been identified in the magnetic scattering component [41].

###  IV.4 Magnetic field induced spin state order

Sotnikov et al. and Tatsuno et al. independently reported studies on exciton
condensation and spin state crystallization induced by high magnetic fields in
LaCoO3 [40, 33]. They studied the emergence of spin state crystals and
excitonic condensation on the temperature-magnetic field plane. Sotnikov et
al. argued that excitonic condensation is more probable than spin state
crystal as the possible origin of $\alpha$ phase. This is based on the
calculated temperature-magnetic field phase diagram that the phase boundary
between the low spin state and the field-induced phase has a positive
dependence on temperature in the case of exciton condensation and negative
dependence on temperature in the case of spin state crystal, as shown in Fig.
23. Tatsuno et al. showed that spin state crystal and excitonic condensation
are induced at high magnetic fields with high and low temperatures,
respectively, as shown in Fig. 24. They indicate that the $\alpha$ and $\beta$
phases can be attributed to exciton condensation and spin state crystal. It
was argued that, with extensive magnetic fields, the re-entrant to exciton
condensation occurs before the complete saturation of the magnetic moment.

Motivated by the experimental results up to 600 T, further calculation has
been performed [15]. Theoretical calculations so far considered only the
intermediate spin state as an excited spin state. The present calculations
consider the high spin state of $S=2$, in addition to the low spin $S=0$ and
the intermediate spin state $S=1$. In addition to the interaction considered
in previous calculations, the hybridization between high spin and intermediate
spin states is taken into account [15]. The calculated result shows a variety
of excitonic condensation and spin state crystallization, and their co-
occurrence has been investigated as shown in Fig. 25. We found a parameter set
1 (PS1) and a parameter set 2 (PS2) that reproduce the appearance of two kinds
of exciton condensation and spin state crystallization, respectively, as shown
in Fig. 26. This suggests that the interaction parameters drastically changed
between the temperatures of 5 K and 78 K. Such lattice change between $\alpha$
and $\beta$ phases are confirmed in the previous study [16] as shown in Fig.
14.

Figure 26: Phase diagram the effetive model of LaCoO3 on the parameter space
of the hybiridization of high spin state and the intermediate spin state $V$,
and the magnetic field $h$ devided by the crystal field splitting $E_{1}$ with
a hopping parameter $t$ of the intermediate spin state. Reprinted from Ref.
15, $\copyright$ 2023 Licensed under CC BY 4.0.

##  V Summary

###  V.1 Conclusion

We have reviewed experimental studies on LaCoO3 at high magnetic fields.
Experimentally, several novel phases, $\alpha$, $\beta$, and $\gamma$ phases
have been discovered. The appearance of multiple phases strongly suggests that
inter-site interactions between spin states are in play, such as the itineracy
of the intermediate spin state and the hybridization (duality) of the
intermediate spin state and the high spin state. However, the experimental
evidence is limited to some macroscopic studies, and a concrete possibility of
the origin of those phases has not yet been obtained. We have introduced
theoretical studies on LaCoO3 at high magnetic fields. Those studies focus on
the Mott-Hubbard models with two orbitals, a minimal model for correlated
electron systems with spin state degrees of freedom. Various ordered phases,
including exciton condensation and spin state crystals, are identified.

###  V.2 Future prospects

Further exploration on LaCoO3 is needed to clarify the origin of $\alpha$,
$\beta$, and $\gamma$ phases using various experimental methods available at
high magnetic fields such as dielectric constant, ultrasound, electric
conductivity. A microscopic experiment in high magnetic fields, such as x-ray
diffraction and terahertz spectroscopy, is even more critical. X-ray
techniques at 100 T are being developed using x-ray free electron laser and
portable 100 T generators called PINK series [42, 43]. Single-shot Teraheltz
time domain spectroscopy is reported, which is also a promising application
with the PINK series for a 100 T experiment. Future studies will uncover the
origin of the high-field phases, leading to a more profound understanding of
the correlation effect in systems with a spin state degree of freedom.

###### Acknowledgements.

This work is supported by MEXT LEADER program No. JPMXS0320210021, JST FOREST
program No. JPMJFR222W, JSPS Grants-in-Aid for Scientific Research No.
23H04861, 23H04859, 23H01121.

## References

  * Imada  _et al._ [1998] M. Imada, A. Fujimori, and Y. Tokura, Metal-insulator transitions, Rev. Mod. Phys. 70, 1039 (1998).
  * Tokura [2006] Y. Tokura, Critical features of colossal magnetoresistive manganites, Rep. Prog. Phys. 69, 797 (2006).
  * Okimoto  _et al._ [2021] Y. Okimoto, T. Saitoh, Y. Kobayashi, and S. Ishihara, eds.,  _Spin-Crossover Cobaltite (Review and Outlook)_, Springer Series in Materials Science (Springer Singapore, 2021).
  * Doi  _et al._ [2014] A. Doi, J. Fujioka, T. Fukuda, S. Tsutsui, D. Okuyama, Y. Taguchi, T. Arima, A. Q. R. Baron, and Y. Tokura, Multi-spin-state dynamics during insulator-metal crossover in LaCoO3, Phys. Rev. B 90, 081109 (2014).
  * English  _et al._ [2002] S. R. English, J. Wu, and C. Leighton, Thermally excited spin-disorder contribution to the resistivity of LaCoO3, Phys. Rev. B 65, 220407 (2002).
  * Hoch  _et al._ [2009] M. J. R. Hoch, S. Nellutla, J. van Tol, E. S. Choi, J. Lu, H. Zheng, and J. F. Mitchell, Diamagnetic to paramagnetic transition in LaCoO3, Phys. Rev. B 79, 214421 (2009).
  * Noguchi  _et al._ [2002] S. Noguchi, S. Kawamata, K. Okuda, H. Nojiri, and M. Motokawa, Evidence for the excited triplet of LaCoO3, Phys. Rev. B 66, 094404 (2002).
  * Sato  _et al._ [2009] K. Sato, A. Matsuo, K. Kindo, Y. Kobayashi, and K. Asai, Field induced spin-state transition in $\rm{LaCoO}_{3}$, J. Phys. Soc. Jpn. 78, 093702 (2009).
  * Altarawneh  _et al._ [2012] M. M. Altarawneh, G. W. Chern, N. Harrison, C. D. Batista, A. Uchida, M. Jaime, D. G. Rickel, S. A. Crooker, C. H. Mielke, J. B. Betts, J. F. Mitchell, and M. J. R. Hoch, Cascade of Magnetic Field Induced Spin Transitions in LaCoO3, Phys. Rev. Lett. 109, 037201 (2012).
  * Rotter  _et al._ [2014] M. Rotter, Z. S. Wang, A. T. Boothroyd, D. Prabhakaran, A. Tanaka, and M. Doerr, Mechanism of spin crossover in LaCoO3 resolved by shape magnetostriction in pulsed magnetic fields, Sci. Rep. 4, 7003 (2014).
  * Ropka and Radwanski [2003] Z. Ropka and R. J. Radwanski, 5D term origin of the excited triplet in LaCoO3, Phys. Rev. B 67, 172401 (2003).
  * Kyômen  _et al._ [2005] T. Kyômen, Y. Asaka, and M. Itoh, Thermodynamical analysis of spin-state transitions in $\rm{LaCoO}_{3}$: Negative energy of mixing to assist thermal excitation to the high-spin excited state, Phys. Rev. B 71, 024418 (2005).
  * Ghannadzadeh  _et al._ [2011] S. Ghannadzadeh, M. Coak, I. Franke, P. A. Goddard, J. Singleton, and J. L. Manson, Measurement of magnetic susceptibility in pulsed magnetic fields using a proximity detector oscillator, Rev. Sci. Instrum. 82, 113902 (2011).
  * Daou  _et al._ [2010] R. Daou, F. Weickert, M. Nicklas, F. Steglich, A. Haase, and M. Doerr, High resolution magnetostriction measurements in pulsed magnetic fields using fiber bragg gratings, Rev. Sci. Instrum. 81, 033909 (2010).
  * Ikeda  _et al._ [2023] A. Ikeda, Y. H. Matsuda, K. Sato, Y. Ishii, H. Sawabe, D. Nakamura, S. Takeyama, and J. Nasu, Signature of spin-triplet exciton condensations in LaCoO3 at ultrahigh magnetic fields up to 600 T, Nat. Commun. 14, 1744 (2023).
  * Ikeda  _et al._ [2020a] A. Ikeda, Y. H. Matsuda, and K. Sato, Two spin-state crystallizations in LaCoO3, Phys. Rev. Lett. 125, 177202 (2020a).
  * Takeyama  _et al._ [2012] S. Takeyama, R. Sakakura, Y. H. Matsuda, A. Miyata, and M. Tokunaga, Precise Magnetization Measurements by Parallel Self-Compensated Induction Coils in a Vertical Single-Turn Coil up to 103 T, J. Phys. Soc. Jpn. 81, 014702 (2012).
  * Ikeda  _et al._ [2017] A. Ikeda, T. Nomura, Y. H. Matsuda, S. Tani, Y. Kobayashi, H. Watanabe, and K. Sato, High-speed 100 MHz strain monitor using fiber Bragg grating and optical filter for magnetostriction measurements under ultrahigh magnetic fields, Rev. Sci. Instrum. 88, 083906 (2017).
  * Nakamura  _et al._ [2018] D. Nakamura, A. Ikeda, H. Sawabe, Y. H. Matsuda, and S. Takeyama, Record indoor magnetic field of 1200 T generated by electromagnetic flux-compression, Rev. Sci. Instrum. 89, 095106 (2018).
  * Miura  _et al._ [2003] N. Miura, T. Osada, and S. Takeyama, Research in super-high pulsed magnetic fields at the megagauss laboratory of the university of tokyo, J. Low Temp. Phys. 133, 139 (2003).
  * Ikeda  _et al._ [2016] A. Ikeda, T. Nomura, Y. H. Matsuda, A. Matsuo, K. Kindo, and K. Sato, Spin state ordering of strongly correlating LaCoO3 induced at ultrahigh magnetic fields, Phys. Rev. B 93, 220401(R) (2016).
  * Jaime  _et al._ [2012] M. Jaime, R. Daou, S. A. Crooker, F. Weickert, A. Uchida, A. E. Feiguin, C. D. Batista, H. A. Dabkowska, and B. D. Gaulin, Magnetostriction and magnetic texture to 100.75 Tesla in frustrated SrCu2(BO3)2, Proc. Natl. Acad. Sci. U.S.A. 109, 12404 (2012).
  * Jaime  _et al._ [2017] M. Jaime, C. Corvalán Moya, F. Weickert, V. Zapf, F. F. Balakirev, M. Wartenbe, F. P. Rosa, B. J. Betts, G. Rodriguez, A. S. Crooker, and R. Daou, Fiber Bragg Grating Dilatometry in Extreme Magnetic Field and Cryogenic Conditions, Sensors 17, 2572 (2017).
  * Gen  _et al._ [2023] M. Gen, A. Ikeda, K. Aoyama, H. O. Jeschke, Y. Ishii, H. Ishikawa, T. Yajima, Y. Okamoto, X. Zhou, D. Nakamura, S. Takeyama, K. Kindo, Y. H. Matsuda, and Y. Kohama, Signatures of a magnetic superstructure phase induced by ultrahigh magnetic fields in a breathing pyrochlore antiferromagnet, Proc. Natl. Acad. Sci. U.S.A. 120, e2302756120 (2023).
  * Nomura  _et al._ [2023] T. Nomura, P. Corboz, A. Miyata, S. Zherlitsyn, Y. Ishii, Y. Kohama, Y. H. Matsuda, A. Ikeda, C. Zhong, H. Kageyama, and F. Mila, Unveiling new quantum phases in the Shastry-Sutherland compound SrCu2(BO3)2 up to the saturation magnetic field, Nat. Commun. 14, 3769 (2023).
  * Nomura  _et al._ [2021] T. Nomura, A. Ikeda, M. Gen, A. Matsuo, K. Kindo, Y. Kohama, Y. H. Matsuda, S. Zherlitsyn, J. Wosnitza, H. Tsuda, and T. C. Kobayashi, Physical properties of liquid oxygen under ultrahigh magnetic fields, Phys. Rev. B 104, 224423 (2021).
  * Ikeda  _et al._ [2018] A. Ikeda, Y. H. Matsuda, and H. Tsuda, Note: Optical filter method for high-resolution magnetostriction measurement using fiber Bragg grating under millisecond-pulsed high magnetic fields at cryogenic temperatures, Rev. Sci. Instrum. 89, 096103 (2018).
  * Ikeda  _et al._ [2019] A. Ikeda, S. Furukawa, O. Janson, Y. H. Matsuda, S. Takeyama, T. Yajima, Z. Hiroi, and H. Ishikawa, Magnetoelastic couplings in the deformed kagome quantum spin lattice of volborthite, Phys. Rev. B 99, 140412 (2019).
  * Nakamura  _et al._ [2023] R. Nakamura, A. Ishita, J. Nakamura, H. Ohta, Y. Haraguchi, H. A. Katori, H. Ishikawa, A. Matsuo, K. Kindo, M. Nohara, and A. Ikeda, Magnetic-field-induced valence change in Eu(Co1-xNix)2P2 up to 60 T, Phys. Rev. B 107, 235110 (2023).
  * Miyake  _et al._ [2022] A. Miyake, M. Gen, A. Ikeda, K. Miyake, Y. Shimizu, Y. J. Sato, D. Li, A. Nakamura, Y. Homma, F. Honda, J. Flouquet, M. Tokunaga, and D. Aoki, Magnetovolume Effect on the First-Order Metamagnetic Transition in UTe2, J. Phys. Soc. Jpn. 91, 063703 (2022).
  * Platonov  _et al._ [2012] V. V. Platonov, Y. B. Kudasov, M. P. Monakhov, and O. M. Tatsenko, Magnetically induced phase transitions in LaCoO3 in fields of up to 500 T, Phys. of the Solid State 54, 279 (2012).
  * Herlach [1999] F. Herlach, Pulsed magnets, Rep. Prog. Phys. 62, 859 (1999).
  * Sotnikov and Kuneš [2016] A. Sotnikov and J. Kuneš, Field-induced exciton condensation in LaCoO3, Sci. Rep. 6, 30510 (2016).
  * Kuneš [2015] J. Kuneš, Excitonic condensation in systems of strongly correlated electrons, J. Phys. Cond. Mat. 27, 333201 (2015).
  * Nasu  _et al._ [2016] J. Nasu, T. Watanabe, M. Naka, and S. Ishihara, Phase diagram and collective excitations in an excitonic insulator from an orbital physics viewpoint, Phys. Rev. B 93, 205136 (2016).
  * Hariki  _et al._ [2020] A. Hariki, R.-P. Wang, A. Sotnikov, K. Tomiyasu, D. Betto, N. B. Brookes, Y. Uemura, M. Ghiasi, F. M. F. d. Groot, and J. Kuněs, Melting of excitonic dispersion in LaCoO3: theory and experiment, Phys. Rev. B 101, 245162 (2020).
  * Kaneko  _et al._ [2012] T. Kaneko, K. Seki, and Y. Ohta, Excitonic insulator state in the two-orbital Hubbard model: Variational cluster approach, Phys. Rev. B 85, 165135 (2012).
  * Kuneš and Augustinský [2014] J. Kuneš and P. Augustinský, Excitonic instability at the spin-state transition in the two-band hubbard model, Phys. Rev. B 89, 115134 (2014).
  * Kaneko  _et al._ [2015] T. Kaneko, B. Zenker, H. Fehske, and Y. Ohta, Competition between excitonic charge and spin density waves: Influence of electron-phonon and Hund’s rule couplings, Phys. Rev. B 92, 115106 (2015).
  * Tatsuno  _et al._ [2016] T. Tatsuno, E. Mizoguchi, J. Nasu, M. Naka, and S. Ishihara, Magnetic field effects in a correlated electron system with spin-state degree of freedom — implications for an excitonic insulator —, J. Phys. Soc. Jpn. 85, 083706 (2016).
  * [41] K. Tomiyasu, T. Nomura, Y. Kobayashi, S. Ishihara, S. Ohira, and M. Kofu, Emergent units of itinerant spin-state excitations in LaCoO3, arXiv:1808.05888 .
  * Ikeda  _et al._ [2022] A. Ikeda, Y. H. Matsuda, X. Zhou, S. Peng, Y. Ishii, T. Yajima, Y. Kubota, I. Inoue, Y. Inubushi, K. Tono, and M. Yabashi, Generating 77 T using a portable pulse magnet for single-shot quantum beam experiments, Appl. Phys. Lett. 120, 142403 (2022).
  * Ikeda  _et al._ [2020b] A. Ikeda, Y. H. Matsuda, X. Zhou, T. Yajima, Y. Kubota, K. Tono, and M. Yabashi, Single shot x-ray diffractometry in SACLA with pulsed magnetic fields up to 16 T, Phys. Rev. Research 2, 043175 (2020b).