# Inferring Black Hole Properties from Astronomical Multivariate Time Series
with Bayesian Attentive Neural Processes

Ji Won Park  Ashley Villar  Yin Li  Yan-Fei Jiang  Shirley Ho  Joshua Yao-Yu
Lin  Phil Marshall  Aaron Roodman

###### Abstract

Among the most extreme objects in the Universe, active galactic nuclei (AGN)
are luminous centers of galaxies where a black hole feeds on surrounding
matter. The variability patterns of the light emitted by an AGN contain
information about the physical properties of the underlying black hole.
Upcoming telescopes will observe over 100 million AGN in multiple broadband
wavelengths, yielding a large sample of multivariate time series with long
gaps and irregular sampling. We present a method that reconstructs the AGN
time series and simultaneously infers the posterior probability density
distribution (PDF) over the physical quantities of the black hole, including
its mass and luminosity. We apply this method to a simulated dataset of 11,000
AGN and report precision and accuracy of 0.4 dex and 0.3 dex in the inferred
black hole mass. This work is the first to address probabilistic time series
reconstruction and parameter inference for AGN in an end-to-end fashion.

Machine Learning, ICML

##  1 Introduction

Supermassive black holes (BHs) reside at the centers of most galaxies, feeding
on diffuse matter around them. Radiation from matter falling into their
gravitational pull makes these galactic centers, called active galactic nuclei
(AGN), some of the most luminous in the Universe. The time variability
patterns of AGN light are correlated with the physical properties of the
underlying BH, such as the mass, rate of matter inflow, and age (Wold et al.,
2008; Simm et al., 2016; MacLeod et al., 2010; Suberlak et al., 2021).

Being so luminous, AGN can be observed out to great distances, close to the
edge of the observable Universe (e.g., Mortlock et al. 2011). Characterizing
faraway BHs gives us a glimpse into the little-known early Universe. By
inferring BH physics from AGN light, we can gain an understanding the origin
and evolution of the cosmos, including the nature of dark energy and dark
matter (Khadka & Ratra, 2020).

Upcoming large-sky telescope surveys herald an unprecedented increase in the
AGN data volume. The Vera Rubin Observatory Legacy Survey of Space and Time
(LSST) is projected to yield 100 million AGN time series in six optical
broadband filters over ten years (Abell et al., 2009). Traditional methods of
estimating BH properties, however, rely on expensive spectroscopic data, i.e.
measurements of the AGN light at continuous wavelengths. Obtaining
spectroscopy for millions of objects would be unfeasible. To take advantage of
all the new data, we require an efficient method that can estimate desired
quantities directly from the 6-filter LSST time series. The learned
relationship could improve our understanding of BH physics, particularly as a
physical model of AGN variability does not exist.

We present a method that simultaneously reconstructs the multivariate AGN time
series from limited observations and infers the full posterior probability
density distribution (PDF) over key BH properties. Our method is designed for
irregular, multivariate time series, as telescope data often suffer from long
seasonal gaps and irregular sampling as well as noise from the Earth’s
atmosphere and telescope optics. Uncertainty quantification is essential to
optimize follow up strategies.

This work is the first deep learning pipeline that simultaneously addresses
AGN light curve reconstruction and parameter inference in a probabilistic
manner. It is additionally the first designed for multivariate time series. At
the core of our pipeline is an attentive neural process (Kim et al., 2019), a
type of latent variable model, that has been modified for density estimation.
In the past, autoencoders have been applied to output point estimates of the
unobserved portions of the light curve in a single-filter setting (Tachibana
et al., 2020). Convolutional neural nets were trained to point-estimate the
redshifts based on multi-filter light curves from the Sloan Digital Sky Survey
(SDSS) (Schneider et al., 2010; Pasquet-Itam & Pasquet, 2018). Summary
statistics from the SDSS light curves were also fed into a neural net for
point-estimating BH mass and redshift (Lin et al., 2020).

##  2 Data

###  2.1 Multi-filter time series

The training set consisted of 11,000 simulated multi-filter time series and
the corresponding target labels. The validation set contained 50 held-out
examples, drawn from the same distribution as the training set. The input was
a simulated multivariate time series of the AGN flux, or brightness, in the
six bandpass filters $ugrizY$. This six-dimensional light curve followed the
Ornstein-Uhlenbeck (OU) process, a widely adopted stochastic model of AGN flux
variability (Kelly et al., 2009). The light curves were sampled at irregular
times, with long gaps, to simulate LSST-like observations (Reuter et al.,
2016). We added astrophysical noise to the light curves on the fly during
training (Kessler et al., 2019).

###  2.2 Target quantities

The target quantities were taken from the Second Data Challenge (DC2), a
simulated LSST-like catalog (LSST Dark Energy Science Collaboration et al.,
2021a, b). For each of the $ugrizY$ filters, there were three variability
parameters: the average flux ($m$), long-term amplitude ($SF_{\infty}$) of
fluctuations, and characteristic timescale ($\tau$) of fluctuations. The
$SF_{\infty}$ and $\tau$ parameters can approximately be interpreted as the
maximum amplitude of flux fluctuations and the timescale to reach such an
amplitude, respectively. We also included the BH mass, redshift (a measure of
distance), and the i-band absolute magnitude ($M_{i}$; a measure of the
intrinsic luminosity of the AGN). The 21 quantities in total shared
correlations empirically modeled after a well-known dataset (SDSS) of quasars
(Kelly et al., 2009; MacLeod et al., 2010).

Figure 1: Model architecture of the Bayesian ANP, modified from the
visualization in (Dubois et al., 2020).

##  3 Method

###  3.1 Model

See Figure 1 for a diagram of our model architecture. We used a latent
attentive neural process (ANP) (Kim et al., 2019) to model the conditional
distribution over regression functions mapping our input times,
$\bm{x}^{(c)}\in\mathbb{R}$, to our output $ugrizY$ fluxes,
$\bm{y}^{(c)}\in\mathbb{R}^{6}$. Conditioning on observed time/flux pairs,
$\left(\bm{x}_{C},\bm{y}_{C}\right):=\left(\bm{x}^{(c)},\bm{y}^{(c)}\right)_{c\in
C}$, called the context set, we query the model for the fluxes at some
unobserved times,
$\left(\bm{x}_{T},\bm{y}_{T}\right):=\left(\bm{x}^{(t)},\bm{y}^{(c)}\right)_{c\in
C}$, called the target set. A common convention, used here, is to set
$C\subset T$.

In the deterministic path of the forward model, each time/flux pair
$\bm{x}^{(c)},\bm{y}^{(c)}$ passes through an MLP to form a representation
$\bm{r}^{(c)}\in\mathbb{R}^{h}$. Self-attention is applied to the resulting
context representations and the model attends to these via cross-attention
with $\bm{x}^{(t)}$ to predict $\bm{y}^{(t)}$. The attention mechanism allows
it to attribute higher relative importance to time samples that are more
informative for prediction. This is useful for astronomical data, where
observations tend to be clustered together and some carry higher signal-to-
noise than others.

The latent path encodes a global understanding of the entire time series. The
encodings $\bm{r}_{C}:=\left(\bm{r}^{(c)}\right)_{c\in C}$ are aggregated
using an LSTM and passed through a multi-layer perceptron (MLP) to form a
latent variable, $\bm{z}$.111The ANP, as originally proposed by Kim et al.
2019, uses mean aggregation but we find that the LSTM yield better predictions
of our target quantities. We model $\bm{z}$ as a factorized Gaussian, with a
Gaussian prior. Each sample of $\bm{z}$ represents one realization of the
data-generating stochastic process, so $\bm{z}$ stores information about
uncertainty in the predictions $\bm{y}_{T}$.

As an update to the original ANP architecture, we additionally use $\bm{z}$
and the mean of the attention-reweighted vector $\bm{r}$ over the times to
infer the global target quantities $\bm{s}\in\mathbb{R}^{21}$, described in
Section 2.2. An MLP takes in $\bm{z}$ and outputs the parameters defining the
posterior PDF over $\bm{s}$.

###  3.2 Uncertainty quantification

We adapted the standard ANP into a Bayesian ANP (BANP) to enable posterior
inference over the network weights (Denker & LeCun, 1991). Our posterior on
$\bm{y}_{T},\bm{s}$ can be written as:

$\displaystyle p(\bm{y}_{T},\bm{s}|\bm{x}_{T},\bm{x}_{C},\bm{y}_{C})$
$\displaystyle=\int
p(\bm{y}_{T},\bm{s}|\bm{x}_{T},\bm{x}_{C},\bm{y}_{C},W)p(W|\bm{x}_{C},\bm{y}_{C})\
dW,$ (1)

where $W$ denotes the network weights. The likelihood
$p(\bm{y}_{T},\bm{s}|\bm{x}_{T},\bm{x}_{C},\bm{y}_{C},W)$ captures the
aleatoric uncertainty, which exists due to the intrinsic randomness in the
data-generating process. We marginalize over the weight posterior
$p(W|\bm{x}_{C},\bm{y}_{C})$ to account for the epistemic uncertainty, which
originates from incomplete knowledge, e.g. limited training data.

Aleatoric uncertainty:  For predicting $\bm{y}_{T}$, we would maximize the
latent neural process likelihood (Garnelo et al., 2018):

$\displaystyle p\left(\bm{y}_{T}|\bm{x}_{T},\bm{x}_{C},\bm{y}_{C},W\right)$
$\displaystyle=\int
p\left(\bm{y}_{T}|\bm{x}_{T},\bm{r}_{C},\bm{z},W\right)q\left(\bm{z}|\bm{x}_{C},\bm{y}_{C},W\right)d\bm{z},$
(2)

where we have explicitly conditioned on $W$. Similarly, the likelihood of
$\bm{s}$ is

$\displaystyle p\left(\bm{s}|\bm{x}_{C},\bm{y}_{C},W\right)=\int
p\left(\bm{s}|\bm{z},W\right)q\left(\bm{z}|\bm{x}_{C},\bm{y}_{C},W\right)d\bm{z}.$
(3)

We chose $p(\bm{s}|\bm{z})$ to be multivariate Gaussian with a full covariance
matrix, so as to model physical correlations between the target quantities.

The total aleatoric portion of our likelihood combines Equations 3.2 and 3:

$\displaystyle\log p(\bm{y}_{T},\bm{s}|\bm{x}_{T},\bm{x}_{C},\bm{y}_{C},W)$
$\displaystyle\propto\alpha\log
p\left(\bm{y}_{T}|\bm{x}_{T},\bm{x}_{C},\bm{y}_{C},W\right)+\log
p\left(\bm{s}|\bm{x}_{C},\bm{y}_{C},W\right),$ (4)

where $\alpha$ is a hyperparameter controlling the relative weight between
light curve reconstruction and parameter inference.

Epistemic uncertainty:  We used Monte Carlo (MC) dropout (Gal & Ghahramani,
2016; Kendall & Gal, 2017) to compute the integral in Equation 3.2. MC dropout
replaces the true weight posterior with a simple Bernoulli variational
approximation $q(W|\bm{x}_{C},\bm{y}_{C})$, which can be implemented by
setting random network weights to zero during training and testing. We treated
the dropout rate as a hyperparameter.

###  3.3 Optimization

Context-target split:  We provided as context the observed portions of the
light curve according to the 10-year LSST observing strategy, which consisted
of 500-1,000 samplings per filter. All six filters were observed at the same
times. The target was the union of the context set and the light curve at
every 10 days, i.e. 365 samplings per filter.

Loss function:  The exact form of Equation 3.2 is intractable. The evidence
lower bound (ELBO) can be optimized, however, using the reparameterization
trick (Kingma & Welling, 2013; Rezende et al., 2014):

$\displaystyle\log
p(\bm{y}_{T},\bm{s}|\bm{x}_{T},\bm{x}_{C},\bm{y}_{C},W)\geq$
$\displaystyle\mathbb{E}_{q(\bm{z}|\bm{x}_{T},\bm{y}_{T})}\left[\alpha^{\prime}\log
p(\bm{y}_{T}|\bm{x}_{T},\bm{r}_{C},\bm{z},W)+\log p(\bm{s}|\bm{z},W)\right]$
$\displaystyle-D_{\rm
KL}\left(q(\bm{z}|\bm{x}_{T},\bm{y}_{T},W)||q(\bm{z}|\bm{x}_{C},\bm{y}_{C},W)\right):=\mathcal{F}_{\rm
ELBO}(W)$ (5)

where $\alpha^{\prime}$ serves the same reweighting purpose as $\alpha$ in
Equation 3.2. We had $\alpha^{\prime}=10$. In each training iteration, given a
realization of weights from MC dropout, $\hat{W}\sim
q(\hat{W}|\bm{x}_{C},\bm{y}_{C})$, we performed gradient descent on
$\mathcal{F}_{\rm ELBO}(\hat{W})$ with a weight decay of 1e-5 using the ADAM
optimizer (Kingma & Ba, 2014). Training was done for 300 epochs in batch sizes
of 40. The learning rate began with 1e-3 and was halved whenever the
validation loss did not decrease for 20 epochs.

##  4 Results

Figure 2: Reconstructed $ugrizY$ light curve for a validation AGN. Blue stars
are observed context points, provided to the network at test time. Black dots
are query target points, for which the network produces the red posteriors.
The true observations lie within the 1-$\sigma$ region of our predicted light
curves.

###  4.1 Light curve reconstruction

Figure 2 shows the BANP posterior for a validation $ugrizY$ light curve. Our
model closely reconstructs the long-term amplitude and timescale of
fluctuations. This performance indicates a good understanding of $SF_{\infty}$
and $\tau$. While the truth is consistent with its 1-$\sigma$ credible
interval at all times, the uncertainties are overestimated and the model does
a poorer job of predicting the shorter-term fluctuations. One possibility is
that the reconstruction loss competes with the parameter inference loss, as
the model does not have to get the shorter-term fluctuations correct to
predict $SF_{\infty}$ and $\tau$.

Table 1: Validation metrics of time series reconstruction and parameter recovery. Lower is better. Error bars indicate the standard deviations over three training random seeds. Model | Reconstruction | Parameter
---|---|---
|  1-$\sigma$ | MAE |  1-$\sigma$ | MAE
BANP | $3.1\pm 0.9$ | $2.2\pm 0.8$ | $\mathbf{51\pm 2}$ | $\mathbf{46\pm 1}$
Baseline | N/A | N/A | $89\pm 3$ | $97\pm 3$

To assess the precision of reconstruction, we draw 100 light curves from our
flux posterior and approximate the 1-$\sigma$ credible width at each target
time as the standard deviation across the samples. We then take the average
across all target times and 50 validation AGN. Similarly, to assess the
accuracy, we obtain the absolute error of the central flux prediction at each
target time and take the average across all target times in the validation set
(mean absolute error; MAE). The resulting values, which carry units of
magnitude, are listed in Table 1.

###  4.2 Retrieval of target quantities

Figure 3: Network posterior for a validation AGN overlaid with the training
distribution for a representative subset of the target quantities. The
vertical/horizontal black lines show the true parameter values. In this
example, our network produces a tight and unbiased estimate for the BH and
light curve parameters.

Figure 3 shows the inferred posterior over a representative subset of the
target quantities in $\bm{s}$, for a single AGN. Our BANP can accurately
recover the true quantities within 1-$\sigma$ credible interval.

To assess the precision and accuracy of our parameter recovery, we proceed
similarly as in Section 4.1 to evaluate the 1-$\sigma$ uncertainty and MAE of
our posterior on the 21 target quantities. The resulting values are listed on
Table 1. Our baseline method was a residual MLP that took as input the summary
statistics of the light curve—the mean and standard deviation of the flux in
six filters—to yield an identically parameterized posterior. We find a twofold
improvement in target recovery, in both precision and accuracy. In particular,
the timescale-sensitive quantities $\tau$ and BH mass showed the most
improvement, of 150% and 120%, respectively. The BH mass was precise to 0.4
dex and accurate to 0.3 dex.

####  4.2.1 Calibration

Figure 4 plots the calibration metric introduced in Wagner-Carena et al.
(2021).222The supplementary material describes this metric in detail. For a
given fraction of the BANP posterior probability volume, $p_{X}$, the metric
plots the fraction of posterior samples that contain the truth within the
volume, $p_{Y}$. If the posterior is perfectly calibrated, $p_{X}$ of the
posterior samples would encompass the truth $p_{Y}=p_{X}$ of the time, for
every value of $p_{X}$. We apply this metric on the validation set as a whole
by averaging the $p_{Y}$ values across the validation AGN, to get $p_{Y}^{\rm
val}$. Regions of the curve with $p_{\rm Y}^{\rm val}<p_{X}$ indicate
overconfidence whereas the opposite indicates underconfidence. We find good
calibration for with our choice of MC dropout rate (0.05).

Figure 4: The confidence of our network is statistically consistent with the
truth. The calibration metric for our model falls on the $p_{Y}^{\rm
val}=p_{X}$ line, which indicates perfect calibration. Error bands are
estimated using jacknife sampling of the posterior samples.

##  5 Discussion and Future Work

We have adapted latent neural processes for probabilistic parameter estimation
and multivariate time series reconstruction. Our method is capable of
interpolating AGN light curves and precisely inferring BH parameters. Compared
to classical fitting methods, the BANP is flexible and does not assume a fixed
parameterization or kernel. This is important for processing real AGN data,
because there exists no physical parameterized model that can reliably
describe the temporal patterns. Our method is thus useful for extracting an
informative latent space from noisy and irregularly sampled time series in
general.

In future work, we plan to upgrade to more physical simulations and
asynchronous sampling in $ugrizY$, in preparation for real LSST AGN data.
Ultimately, we aspire to hierarchically infer the hyperparameters that govern
the AGN population, using the constraints from the individual AGN. Likelihood-
free inference methods such as normalizing flows (Rezende & Mohamed, 2015) are
interesting for a flexible posterior parameterization. We will also explore
incorporating physics priors into the network architecture; scalable variants
of Gaussian processes (e.g. Salimbeni et al. 2017, Wilson et al. 2016) and
latent stochastic differential equations (Li et al., 2020) are promising
alternatives to the ANP for modeling temporal correlations.

## Acknowledgements

## References

  * Abell et al. (2009) Abell, P. A., Burke, D. L., Hamuy, M., Nordby, M., Axelrod, T. S., Monet, D., Vrsnak, B., Thorman, P., Ballantyne, D., Simon, J. D., et al.  Lsst science book, version 2.0.  Technical report, 2009.
  * Denker & LeCun (1991) Denker, J. S. and LeCun, Y.  Transforming neural-net output levels to probability distributions.  In _NeurIPS_ , pp. 853–859, 1991.
  * Dubois et al. (2020) Dubois, Y., Gordon, J., and Foong, A. Y.  Neural process family.  http://yanndubs.github.io/Neural-Process-Family/, September 2020\.
  * Gal & Ghahramani (2016) Gal, Y. and Ghahramani, Z.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _ICML_ , pp. 1050–1059, 2016.
  * Garnelo et al. (2018) Garnelo, M., Schwarz, J., Rosenbaum, D., Viola, F., Rezende, D. J., Eslami, S., and Teh, Y. W.  Neural processes.  _arXiv preprint arXiv:1807.01622_ , 2018.
  * Kelly et al. (2009) Kelly, B. C., Bechtold, J., and Siemiginowska, A.  Are the variations in quasar optical flux driven by thermal fluctuations?  _The Astrophysical Journal_ , 698(1):895, 2009\.
  * Kendall & Gal (2017) Kendall, A. and Gal, Y.  What uncertainties do we need in bayesian deep learning for computer vision?  In _NeurIPS_ , pp. 5574–5584, 2017.
  * Kessler et al. (2019) Kessler, R., Narayan, G., Avelino, A., Bachelet, E., Biswas, R., Brown, P., Chernoff, D., Connolly, A., Dai, M., Daniel, S., et al.  Models and simulations for the photometric lsst astronomical time series classification challenge (plasticc).  _Publications of the Astronomical Society of the Pacific_ , 131(1003):094501, 2019.
  * Khadka & Ratra (2020) Khadka, N. and Ratra, B.  Using quasar x-ray and uv flux measurements to constrain cosmological model parameters.  _Monthly Notices of the Royal Astronomical Society_ , 497(1):263–278, 2020.
  * Kim et al. (2019) Kim, H., Mnih, A., Schwarz, J., Garnelo, M., Eslami, A., Rosenbaum, D., Vinyals, O., and Teh, Y. W.  Attentive neural processes.  _arXiv preprint arXiv:1901.05761_ , 2019.
  * Kingma & Ba (2014) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kingma & Welling (2013) Kingma, D. P. and Welling, M.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Li et al. (2020) Li, X., Wong, T.-K. L., Chen, R. T., and Duvenaud, D.  Scalable gradients for stochastic differential equations.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 3870–3882. PMLR, 2020.
  * Lin et al. (2020) Lin, J. Y.-Y., Pandya, S., Pratap, D., Liu, X., and Kind, M. C.  Agnet: Weighing black holes with machine learning.  _arXiv preprint arXiv:2011.15095_ , 2020.
  * LSST Dark Energy Science Collaboration et al. (2021a) LSST Dark Energy Science Collaboration, Abolfathi, B., Alonso, D., Armstrong, R., Aubourg, É., Awan, H., Babuji, Y. N., Bauer, F. E., Bean, R., Beckett, G., Biswas, R., Bogart, J. R., Boutigny, D., Chard, K., Chiang, J., Claver, C. F., Cohen-Tanugi, J., Combet, C., Connolly, A. J., Daniel, S. F., Digel, S. W., Drlica-Wagner, A., Dubois, R., Gangler, E., Gawiser, E., Glanzman, T., Gris, P., Habib, S., Hearin, A. P., Heitmann, K., Hernandez, F., Hložek, R., Hollowed, J., Ishak, M., Ivezić, Ž., Jarvis, M., Jha, S. W., Kahn, S. M., Kalmbach, J. B., Kelly, H. M., Kovacs, E., Korytov, D., Krughoff, K. S., Lage, C. S., Lanusse, F., Larsen, P., Guillou, L. L., Li, N., Longley, E. P., Lupton, R. H., Mandelbaum, R., Mao, Y.-Y., Marshall, P., Meyers, J. E., Moniez, M., Morrison, C. B., Nomerotski, A., O’Connor, P., Park, H., Park, J. W., Peloton, J., Perrefort, D., Perry, J., Plaszczynski, S., Pope, A., Rasmussen, A., Reil, K., Roodman, A. J., Rykoff, E. S., Sánchez, F. J., Schmidt, S. J., Scolnic, D., Stubbs, C. W., Tyson, J. A., Uram, T. D., Villarreal, A., Walter, C. W., Wiesner, M. P., Wood-Vasey, W. M., and Zuntz, J.  The LSST DESC DC2 simulated sky survey.  _The Astrophysical Journal Supplement Series_ , 253(1):31, mar 2021a.  doi: 10.3847/1538-4365/abd62c.  URL https://doi.org/10.3847/1538-4365/abd62c.
  * LSST Dark Energy Science Collaboration et al. (2021b) LSST Dark Energy Science Collaboration, Abolfathi, B., Armstrong, R., Awan, H., Babuji, Y. N., Bauer, F. E., Beckett, G., Biswas, R., Bogart, J. R., Boutigny, D., Chard, K., Chiang, J., Cohen-Tanugi, J., Connolly, A. J., Daniel, S. F., Digel, S. W., Drlica-Wagner, A., Dubois, R., Gawiser, E., Glanzman, T., Habib, S., Hearin, A. P., Heitmann, K., Hernandez, F., Hložek, R., Hollowed, J., Jarvis, M., Jha, S. W., Kalmbach, J. B., Kelly, H. M., Kovacs, E., Korytov, D., Krughoff, K. S., Lage, C. S., Lanusse, F., Larsen, P., Li, N., Longley, E. P., Lupton, R. H., Mandelbaum, R., Mao, Y.-Y., Marshall, P., Meyers, J. E., Park, J. W., Peloton, J., Perrefort, D., Perry, J., Plaszczynski, S., Pope, A., Rykoff, E. S., Sánchez, F. J., Schmidt, S. J., Uram, T. D., Villarreal, A., Walter, C. W., Wiesner, M. P., and Wood-Vasey, W. M.  Desc dc2 data release note.  _arXiv preprint arXiv:2101.04855_ , 2021b.
  * MacLeod et al. (2010) MacLeod, C. L., Ivezić, Ž., Kochanek, C., Kozłowski, S., Kelly, B., Bullock, E., Kimball, A., Sesar, B., Westman, D., Brooks, K., et al.  Modeling the time variability of sdss stripe 82 quasars as a damped random walk.  _The Astrophysical Journal_ , 721(2):1014, 2010\.
  * Mortlock et al. (2011) Mortlock, D. J., Warren, S. J., Venemans, B. P., Patel, M., Hewett, P. C., McMahon, R. G., Simpson, C., Theuns, T., Gonzáles-Solares, E. A., Adamson, A., et al.  A luminous quasar at a redshift of z= 7.085.  _Nature_ , 474(7353):616–619, 2011.
  * Niculescu-Mizil & Caruana (2005) Niculescu-Mizil, A. and Caruana, R.  Predicting good probabilities with supervised learning.  In _Proceedings of the 22nd international conference on Machine learning_ , pp. 625–632, 2005.
  * Park et al. (2021) Park, J. W., Wagner-Carena, S., Birrer, S., Marshall, P. J., Lin, J. Y.-Y., Roodman, A., LSST Dark Energy Science Collaboration, et al.  Large-scale gravitational lens modeling with bayesian neural networks for accurate and precise inference of the hubble constant.  _The Astrophysical Journal_ , 910(1):39, 2021\.
  * Pasquet-Itam & Pasquet (2018) Pasquet-Itam, J. and Pasquet, J.  Deep learning approach for classifying, detecting and predicting photometric redshifts of quasars in the sloan digital sky survey stripe 82.  _Astronomy & Astrophysics_, 611:A97, 2018.
  * Reuter et al. (2016) Reuter, M. A., Cook, K. H., Delgado, F., Petry, C. E., and Ridgway, S. T.  Simulating the LSST OCS for conducting survey simulations using the LSST scheduler.  9911:794 – 801, 2016.  doi: 10.1117/12.2232680.  URL https://doi.org/10.1117/12.2232680.
  * Rezende & Mohamed (2015) Rezende, D. and Mohamed, S.  Variational inference with normalizing flows.  In _International Conference on Machine Learning_ , pp. 1530–1538. PMLR, 2015.
  * Rezende et al. (2014) Rezende, D. J., Mohamed, S., and Wierstra, D.  Stochastic backpropagation and approximate inference in deep generative models.  In _International conference on machine learning_ , pp. 1278–1286. PMLR, 2014.
  * Salimbeni & Deisenroth (2017) Salimbeni, H. and Deisenroth, M.  Doubly stochastic variational inference for deep gaussian processes.  _arXiv preprint arXiv:1705.08933_ , 2017.
  * Schneider et al. (2010) Schneider, D. P., Richards, G. T., Hall, P. B., Strauss, M. A., Anderson, S. F., Boroson, T. A., Ross, N. P., Shen, Y., Brandt, W., Fan, X., et al.  The sloan digital sky survey quasar catalog. v. seventh data release.  _The Astronomical Journal_ , 139(6):2360, 2010\.
  * Simm et al. (2016) Simm, T., Salvato, M., Saglia, R., Ponti, G., Lanzuisi, G., Trakhtenbrot, B., Nandra, K., and Bender, R.  Pan-starrs1 variability of xmm-cosmos agn-ii. physical correlations and power spectrum analysis.  _Astronomy & Astrophysics_, 585:A129, 2016.
  * Suberlak et al. (2021) Suberlak, K. L., Ivezić, Ž., and MacLeod, C.  Improving damped random walk parameters for sdss stripe 82 quasars with pan-starrs1.  _The Astrophysical Journal_ , 907(2):96, 2021\.
  * Tachibana et al. (2020) Tachibana, Y., Graham, M. J., Kawai, N., Djorgovski, S., Drake, A. J., Mahabal, A. A., and Stern, D.  Deep modeling of quasar variability.  _The Astrophysical Journal_ , 903(1):54, 2020\.
  * Wagner-Carena et al. (2021) Wagner-Carena, S., Park, J. W., Birrer, S., Marshall, P. J., Roodman, A., Wechsler, R. H., Collaboration, L. D. E. S., et al.  Hierarchical inference with bayesian neural networks: An application to strong gravitational lensing.  _The Astrophysical Journal_ , 909(2):187, 2021\.
  * Wilson et al. (2016) Wilson, A. G., Hu, Z., Salakhutdinov, R., and Xing, E. P.  Stochastic variational deep kernel learning.  _arXiv preprint arXiv:1611.00336_ , 2016.
  * Wold et al. (2008) Wold, M., Brotherton, M., and Shang, Z.  Black hole mass and variability in quasars.  In _AIP Conference Proceedings_ , volume 1053, pp. 55–58. American Institute of Physics, 2008.

## Supplement

### Calibration metric

We summarize the semi-quantitative calibration metric introduced in Wagner-
Carena et al. (2021). This is a multi-dimensional generalization of commonly
used confidence-frequency tests (e.g. Niculescu-Mizil et al. 2005) for
evaluating the statistical consistency of model uncertainties.

Denote the $N$ parameter samples drawn from the Bayesian attentive neural
process (BANP) posterior for some AGN $k$ as
$\\{\bm{s}_{n}^{(k)}\\}_{n=1}^{N}$ and the true parameter value as
$\bm{s}_{\rm true}^{(k)}$. For a given fraction of the BANP posterior
probability volume, $p_{X}$, the metric queries the fraction of the samples
containing the truth within this volume, $p_{Y}$. More precisely,

$\displaystyle
p_{Y}^{(k)}(p_{X})=\mathbbm{1}\left\\{\frac{\sum_{n=1}^{N}\mathbbm{1}\left\\{d(\bm{s}_{n}^{(k)})<d(\bm{s}_{\rm
true}^{(k)})\right\\}}{N}<p_{X}\right\\}$ (6)

where $\mathbbm{1}\\{\cdot\\}$ is an indicator function that evaluates to 1
when the argument is true and 0 otherwise, and $d(\bm{s})$ is the distance
measure of a particular point $\bm{s}$ from the posterior predictive mean
given the posterior width. Following Park et al. (2021), we use the
Mahalanobis distance for $d$. If the posterior is perfectly calibrated,
$p_{X}$ of the samples would encompass the truth $p_{Y}=p_{X}$ of the time,
for every value of $p_{X}$.

This metric can be evaluated on the dataset as a whole by averaging the
$p_{Y}$ values from individual AGN. For $N^{\rm val}$ AGN in the validation
set, $p_{Y}$ and $p_{X}$ can be expressed as:

$\displaystyle p_{Y}^{\rm val}(p_{X})=\frac{1}{N^{\rm val}}\sum_{k=1}^{N^{\rm
val}}p_{Y}^{(k)}(p_{X})$ (7)

Plotting $p_{\rm Y}^{\rm val}$ for a grid of $p_{X}$ values yields the
calibration curve. Regions of the curve with $p_{\rm Y}^{\rm val}<p_{X}$
indicate overconfidence whereas the opposite indicates underconfidence.


These authors contributed equally to this work.

These authors contributed equally to this work.

These authors contributed equally to this work.

[1]Xun Qian [1,6]Chen Lin [1]Han-Sen Zhong

[1]Shanghai Artificial Intelligence Laboratory, Shanghai, 200232, China 2]
Fudan University, Shanghai, 200433, China 3] Harbin Institute of Technology,
Harbin, 150006, China 4]The Chinese University of Hong Kong, Hong Kong,
999077, China 5]Shanghai Jiaotong University, Shanghai, 200030, China
6]University of Oxford, Oxford, OX1 2JD, United Kingdom

# DocReLM: Mastering Document Retrieval with Language Model

Gengchen Wei<EMAIL_ADDRESS>Xinle Pang<EMAIL_ADDRESS>Tianning Zhang<EMAIL_ADDRESS>Yu Sun<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Wanli
Ouyang<EMAIL_ADDRESS>*  [  [  [  [  [

###### Abstract

With over 200 million published academic documents and millions of new
documents being written each year, academic researchers face the challenge of
searching for information within this vast corpus. However, existing retrieval
systems struggle to understand the semantics and domain knowledge present in
academic papers. In this work, we demonstrate that by utilizing large language
models, a document retrieval system can achieve advanced semantic
understanding capabilities, significantly outperforming existing systems. Our
approach involves training the retriever and reranker using domain-specific
data generated by large language models. Additionally, we utilize large
language models to identify candidates from the references of retrieved papers
to further enhance the performance. We use a test set annotated by academic
researchers in the fields of quantum physics and computer vision to evaluate
our system’s performance. The results show that DocReLM achieves a Top 10
accuracy of 44.12% in computer vision, compared to Google Scholar’s 15.69%,
and an increase to 36.21% in quantum physics, while that of Google Scholar is
12.96%.

###### keywords:

Document Retrieval, Large Language Model

##  1 Introduction

According to Crossref, the academic field contains over 140 million pieces of
academic literature, with millions of new papers published each year ([4]).
This exponential growth has made it increasingly challenging for researchers
to stay updated with the latest developments in their fields. Consequently,
there is a growing demand for efficient semantic-based document retrieval
systems([12]). However, developing such a system poses significant
challenges([25]). Firstly, academic papers are written in a highly specialized
language with domain-specific knowledge, necessitating a deep understanding of
the context to extract relevant information([26]). Secondly, comprehending a
paper involves not only reading its content but also understanding the
references and their relationships. This requires the system to be aware of
the interconnections among references within the paper. Lastly, users may not
have precise keywords to describe their information needs, requiring the
system to understand the user’s intent and provide the most relevant papers
accordingly. Though some tries ([2]) have been made to create a document
retrieval system with a general embedding model, there is still a gap between
the models’ training task and the real searching. Therefore the performance is
not good.

In this study, we introduce DocReLM, a Document Retrieval system enhanced with
Language Models to solve these challenges. LLMs ([5, 31], currently among the
most prominent areas of focus in natural language processing, demonstrate
robust capabilities in understanding complex linguistic constructs. Their
application across various domains has shown continuous improvement([33, 32,
24, 1]). In DocReLM, the LLM is utilized in two distinct ways. Initially, it
functions as an automatic data annotator, generating high-quality training
data to enhance retrieval models. This resulted in superior performance of our
dense retriever and reranker models compared to other competitors, as
illustrated in 2. The performance of other models is not as good as ours since
they are trained with general semantic similarity tasks. By training with data
generated by LLM, our model is adapted to the document retrieval task. And the
domain specialized knowledge is also distilled into the model in this process.

Subsequently, LLM acts as a searching agent, refining paper selection based on
textual information and reference relationships in the retrieved documents.
This process is similar to the human reasoning process, where the user reads
the references in the paper to find the most relevant ones. The LLM’s ability
to understand the context and relationships between references enables it to
identify the most relevant papers, thereby enhancing the system’s performance.
We find this especially useful in the natural science domain, where papers are
highly interconnected and understanding the references is crucial to
identifying relevant papers. This is also a novel approach to implement the
Large Language Model in the retrieval system.

The whole system of DocReLM is illustrated in Fig. 2. DocReLM can be basically
divided into three parts. The first two parts are a retriever and a reranker.
These two models can filter and sort the document by the relevance to the
user’s query. The third model is a reference extractor that reads the
retrieved documents and finds reference paper ID that can better answer the
question. The extracted references are then added to the final document list.

To evaluate DocReLM, we established a benchmark for semantic document
retrieval and conducted comparisons with several competitors. This benchmark
is structured into two distinct tracks: quantum physics and computer vision.
Each track encompasses a curated set of articles alongside a series of
questions, with each question paired with a standard answer annotated by an
academic researcher. In our experiments, DocReLM demonstrated a significant
edge over its competitors, achieving an accuracy rate of 38.73% in computer
vision and 26.91% in quantum physics on the top 5 results. The full result is
depicted in 1. This highlights the system’s potential to revolutionize
document retrieval in academic research.

(a) computer vision

(b) quantum physics

Figure 1: Accuracy of the final system

##  2 System Architecture

The workflow of training and inferring DocReLM is depicted in Fig. 2. The
system comprises three primary components: retriever, reranker, and reference
extractor. When presented with a user query, the retriever component utilizes
embedding to fast retrieve candidate passages from the corpus. Subsequently, a
reranker is employed to sort these passages more accurately. The final
component examines the content of the top $k$ results provided by the reranker
and generates appropriate references based on the contents of these documents.
In the following subsections, we elaborate on the structural intricacies of
each component in the model and how we utilize large language models to build
these components.

Figure 2: The training and inference of DocReLM. The LLMs are used in both
training and inference.

###  2.1 Retriever

The retriever is designed to efficiently extract a select set of document from
a vast corpus. It requires rapid and effective retriever capabilities.
Typically, two models are employed for this purpose: sparse retriever and
dense retriever([12]). The sparse retriever, such as the famous BM25 ([22]),
utilizes bag-of-words vector to match candidates. Conversely, dense embedding
models explicitly encode sentences into dense vectors([15]; [7]), employing
cosine similarity to evaluate the likelihood of a match between query and
passage embedding. In DocReLM, we adopt the neural dense retriever and utilize
the open-source jina-embedding-v2-base([11]) as the foundational model, which
we further train with our custom data. This retriever operates as a encoder
model([5]; [21]). Encoder has been applied to various tasks in the scientific
domain([13]; [35]; [17]; [10]). Leveraging contrastive learning, dense
retrievers generate embedding that emphasize key passage details, enabling the
linkage of semantically similar sentences even when they use dissimilar
terminology. However, pretrained embedding models lack specificity for
document retrieval tasks([29]), often presenting a domain knowledge gap with
domain-specific papers([27]). To address this, we incorporate a large language
model to automatically annotate pseudo queries with academic paper corpora,
thereby refining the embedding model to align more closely with our task. In
constructing this retrieval system using our tailored embedding model, we
process the document corpus, segment long documents into passages, and convert
them to embedding. These vectors are stored in the vector database Faiss ([14,
6]). When a user inputs a query, DocReLM employs the same retrieval model to
derive the query’s semantic embedding and selects candidate passages based on
the cosine distance between document and query embedding.

###  2.2 Reranker

Following the rapid retrieval process, the reranker is employed to enhance the
precision of the results. While it is more accurate in general, the reranker
operates at a slower pace compared to a retriever. This characteristic
underpins its typical usage post-retrieval, where the retriever effectively
narrows down the selection by eliminating mostly irrelevant passages. In this
task, we use cross-encoder([20]) which processes inputs consisting of a
concatenated sequence of thequery string and candidate passage from the
retrieval subsystem, demarcated by a [SEP] token. The reranker employs
attention mechanism to analyze the input, consequently synthesizing a
comprehensive feature for the entire input sequence. Subsequently, a linear
layer translates this feature into a scalar value, signifying the likelihood
that the given passage correctly answers the query. By integrating the query
and passage in its input, the reranker fosters increased interaction between
the two, thereby enhancing its expressive capabilities and outperforming the
retriever in effectiveness. A limitation of the cross-encoder is its inability
to preprocess the document corpus, necessitating fresh inference for each new
query. Therefore it cannot be used as a standalone system and requires a
retriever to provide a small set of relevant candidates. In DocReLM, we
utilize the open-source model XLM-RoBERTa-large([3]) as the base model, which
we further train with our custom data as the method of LCE([8]). The top 200
passages, as determined by the retriever, are subsequently fed into the
reranker to output a score for each concatenated pair. These candidate
passages are subsequently reordered based on their respective scores.

###  2.3 Reference Extraction

After dense passage retrieval and cross-encoder reranking, the resulted papers
are presented in the order of relevance with the query. We find that an
additional procedure can improve the result significantly. That’s the third
part in DocReLM, reference extractor, which reads the passages content and
extract the best references cited in the passage. This is aligned with
practical searching. A paper chosen by retriever and reranker may not be the
best paper to answer the query, but its reference lists has a large
probability to contain that paper. Motivated by this objective, our approach
involves instructing the large language model to extract the retrieved result
and identify more suitable papers from the references. To achieve this, we
manipulate the paper passage by inserting the identifier of the reference
paper into the text. When this modified passage is processed by a Large
Language Model (LLM), it extracts these identifiers based on the contextual
information. If the large language model determines that the passage itself
constitutes the answer, it outputs the identifier of the referenced paper.
Subsequently, our system conducts a search for the corresponding paper
associated with the provided identifier. This progress is illustrated in
Figure. 3.

Figure 3: An Example for the reference extraction. A query is entered by a
user. Retriever first return $n$ candidate passages from the entire corpus.
Then the reranker sorts these passages. Finally, the reference extractor read
the top 10 papers and extract three references for each of these paper, which
is inserted after that original paper.

##  3 Model and Training Details

We use a two-stage training strategy to train the dense passage retriever and
reranker in DocReLM. We first train the retriever with contrastive learning on
the generated training data. Then we use the trained retriever to select hard
negative samples for finetuning reranker with contrastive learning. In the
following section, we discuss the data generation and training details.

###  3.1 Generate Training Data with Large Language Model

To train the retriever and reranker using contrastive learning, a collection
of user queries and document passages is required. While datasets like MS
MARCO are available with human-annotated labels ([19]), the expense of human
annotation makes it challenging to acquire appropriate domain-specific
retrieval datasets ([27]). In this work, we use a Large Language Model (LLM)
to generate pseudo-queries from documents. We employ papers from unarXive
([23]) to create the dataset, processing their original data to remove
semantically irrelevant text and incorporating mathematical LaTeX and
reference identifiers. These identifiers are formatted as Ref.X of IDFp, where
IDFp is the identifier of the parent paper. We divide the entire paper into
passages and use vicuna-7b-v1.5-16k ([34]) to generate a query for each
sentence by providing the passage, title, and abstract. Inspired by the chain
of thought approach ([28]), we initially employ a zero-shot method ([16]) to
guide the LLM in creating an outline based on the title, abstract, and
content, before generating the queries.

We create two subsets for the experiment. One is the quantum physics category.
It has $56927$ papers and we collect $2.8M$ training data. The other is the
computer vision category. It has $37390$ papers and we collect $1.1M$ training
data. These data are both used to train the dense passage retriever and
reranker.

###  3.2 Contrastive learning

We follow the common strategy to training the retriever and reranker with
contrastive learning. With the data generated in Sec 3.1, we have multiple
pairs of query and passage, where the query $Q$ and passage $P$ in the same
pair pairs are labeled positive, and that in different pairs are labeled
negative. Contrastive learning can be seen as a binary classifier trained
these pairs. For each positive pair $(Q_{i},P_{i})$, we sample $N$ negative
pairs $(Q_{i},P_{i,n})$ and compute the loss using Eequation 1.

$\ell_{i}=-\log\frac{\mathrm{e}^{s(Q_{i},P_{i})/\tau}}{\mathrm{e}^{s(Q_{i},P_{i})/\tau}+\sum_{n=1}^{N}\mathrm{e}^{s(Q_{i},P_{i,n})/\tau}},$
(1)

where $s(Q,P)$ denotes the score function for $(Q,P)$. In retriever, $s(Q,P)$
is given by the cosine similarity of the embeddings of $Q$ and $P$. In
reranker, $s(Q,P)$ is directly given by the cross-encoder. By minimizing
Equation 1, the positive pair scores are optimized to be significantly larger
than the negative pair scores.

###  3.3 Negative Pairs for Retriever and Reranker

The negative pairs in contrastive training retriever and reranker are
generated differently. For retriever, these negative data are randomly
sampled. In each training batch, $N+1$ positive pairs are sampled, each of
which has $N$ negative pairs in the batch. [18] find larger batch size can
improve the retriever’s performance by increasing the chance of introducing
hard negative. We use gradient cache technique([9]) to increase batch size for
better performance in contrastive learning.

For training reranker, we select negative data with our trained retriever as
the method of [8] called LCE. For each query $Q_{i}$, the retriever return $N$
passages that have the highest scores. These passages are highly related to
the query. This means we utilized the most confusing wrong passages apart from
the correct one. Therefore, these wrong passages are used as high quality hard
negative samples for training the reranker.

##  4 Evaluation and results

In assessing our document retrieval system, we set up a benchmark with queries
taken from actual research scenarios. We invite researchers to create queries
in their research domains and provide papers as annotated labels. To ensure
the reliability of these annotations, the researchers are required to provide
claims from surveys. Subsequently, we create a subset of unarXive as the
corpus for retrieval. The final benchmark comprises two tracks: the quantum
physics track, which includes 301 queries and 56927 papers in the corpus, and
the computer vision track, which consists of 204 queries and 37390 papers in
the corpus. We compare the performance of our retrieval system with that of
Google Scholar. Furthermore, we conduct an ablation study to evaluate the
individual effectiveness of the three components in our system. We highlight
their advantages over similar models and demonstrate the superiority of our
model compared to pre-trained models.

Domain | Method | top-1 | top-5 | top-10 | top-20
---|---|---|---|---|---
computer vision | BM25 | 0.49 | 3.43 | 3.92 | 7.35
computer vision | text-embedding-ada-002 | 11.27 | 16.67 | 20.09 | 21.57
computer vision | jina-base-v2 | 8.33 | 14.71 | 18.14 | 22.55
computer vision | DocReLM-retriever | 17.16 | 30.39 | 39.22 | 43.63
quantum physics | BM25 | 0.33 | 1.33 | 1.99 | 2.99
quantum physics | text-embedding-ada-002 | 2.66 | 8.97 | 13.29 | 18.27
quantum physics | jina-base-v2 | 3.99 | 9.97 | 14.62 | 18.27
quantum physics | DocReLM-retriever | 3.32 | 11.96 | 15.95 | 21.93
Table 1: Results of the retriever.

We compare three baseline retrieval models for this task: BM25, ada-002 from
OpenAI and jina-base-v2. The accuracy is shown in Table 1. The sparse
retriever, BM25, shows poor performance compared to other models. Word
frequency is insufficient to capture semantic relation in this task. Our
DocReLM-retriever outperforms all of these models in both domains. In the
computer vision domain, DocReLM-retriever significantly outperforms the other
models. It achieves a top-1 accuracy that is 5.89% higher than the best
embedding model, and top-5 and top-10 accuracy that are 13.72% and 19.13%
higher, respectively. It is worth noting that all three models perform well in
the computer vision domain, with high average accuracy. In the quantum physics
domain, the performance of the embedding models is similar, with no
significant difference in the top-1 results. However, when considering the
top-5 results, our model improves the accuracy from 9.97% to 11.96% to the
best of other models.

Domain | Method | top-1 | top-5 | top-10 | top-20
---|---|---|---|---|---
computer vision | without rerank | 17.16 | 30.39 | 39.22 | 43.63
computer vision | bge-reranker-large | 19.61 | 35.78 | 41.67 | 44.12
computer vision | Cohere rerank-english-v2.0 | 16.66 | 34.31 | 40.20 | 43.63
computer vision | DocReLM-reranker | 20.10 | 37.75 | 44.61 | 47.06
quantum physics | without rerank | 3.32 | 11.96 | 15.95 | 21.93
quantum physics | bge-reranker-large | 5.32 | 14.29 | 20.27 | 25.25
quantum physics | Cohere rerank-english-v2.0 | 7.97 | 17.28 | 21.93 | 25.91
quantum physics | DocReLM-reranker | 7.31 | 17.28 | 19.93 | 26.58
Table 2: Results of the reranker based on the embedding of jina-base-v2-tuned.

To test our reranker’s performance, we choose three different rerankers, bge-
reranker-large([30]), Cohere, and DocReLM-reranker to compare. The rerankers
take the top 200 passages selected by DocReLM-retriever as input data. The
results are shown in Table 2. In the computer vision domain, the DocReLM-
reranker outperforms both the other models. It achieves a top-1 accuracy that
is 0.49% higher than the best reranker, and top-5 and top-10 accuracy that are
1.97% and 2.94% higher, respectively. In the quantum physics domain, the
DocReLM-reranker and Cohere achieve competing performance. Our models perform
better at top-20, while Cohere models excel at top-1 and top-10 scores. These
results demonstrate that the DocReLM-reranker is more effective than other
models in both domains. It highlights the importance of the reranker in
improving the accuracy of the retrieval system.

Domain | Method | top-1 | top-5 | top-10 | top-20
---|---|---|---|---|---
computer vision | without ref extraction | 20.10 | 37.75 | 44.61 | 47.06
computer vision | internLM | 20.10 | 38.73 | 44.12 | 50.00
quantum physics | without ref extraction | 7.31 | 17.28 | 19.93 | 26.58
quantum physics | internLM | 7.31 | 26.91 | 36.21 | 45.51
Table 3: Results of reference extraction.

Our last experiment evaluates the effectiveness of the reference extraction
model. We compare the performance of the DocReLM-reranker with and without
reference extraction. The results are shown in Table 3. In the computer vision
domain, the reference extraction model internLM improves the top-5 and top-20
accuracy by 0.98% and 2.94%, respectively. These results demonstrate that the
reference extraction model is effective in improving the accuracy of the
retrieval system. In the quantum physics domain, the reference extraction
model internLM improves the top-5 and top-10 accuracy by 9.63% and 16.28%,
respectively.

##  5 Discussion

This paper presented DocReLM, an innovative approach to document retrieval
using Large Language Models (LLM), demonstrating significant improvements over
traditional retrieval methods. Our system integrates a neural dense retriever,
a reranking mechanism, and a novel reference extraction component, all fine-
tuned and enhanced through the capabilities of LLMs. In the subsequent
subsections, we will discuss two aspects of DocReLM in greater detail.

###  5.1 Distilling of LLM

The neural dense retriever and cross-encoder reranker effectively capture
semantic relations between queries and documents, facilitating more accurate
retrieval of relevant document. We notice that compared to the base model we
used, there is a significant improvement after training the model with our
generated data. Since these data are generated with LLM, the comprehension
ability of LLM is distilled into the relative small model, which can adapt a
general model to a specific task.

###  5.2 combination of LLM and retrieval system

This study also introduces a novel approach to integrating LLMs with retrieval
systems. With the growing popularity of LLMs, there have been various efforts
to merge them with search engines. These attempts have primarily followed two
approaches: using LLMs to expand search queries or to summarize search
results. However, DocReLM pioneers a new direction by enabling an LLM to
understand search results and continue the search process, suggesting better
candidates. This concept could be expanded further. The process could be
iteratively repeated, allowing the LLM to traverse multiple nodes within a
citation network until it identifies the most relevant results. Such a task
requires the LLM to grasp the logical relationships between referenced papers
and possess sufficient knowledge of the field to accurately extract
references. By incorporating an LLM that can search multiple times and analyze
results to refine its searches, a general retrieval system could also be
significantly enhanced. This approach represents a promising direction for the
future of retrieval systems.

##  6 Conclusion

In conclusion, DocReLM appears to contribute positively to the field of
academic document retrieval. This study, by integrating Large Language Models
(LLMs) with traditional retrieval systems, seeks to address the challenges
posed by the increasing volume and complexity of academic document and
proposes a new direction in the application of LLMs for semantic-based search.
The use of LLMs in DocReLM to generate high-quality pseudo data, enhance
retriever and reranker models, and extract references from retrieved papers,
represents a promising step forward in leveraging the rich information
available in academic documents.

Our results suggest a notable improvement in retrieval accuracy compared to
existing systems. This improvement may be attributed to the retrieval data
generated by LLM for fine-tuning models. Moreover, DocReLM introduces an
innovative approach to employing LLMs in retrieval systems, demonstrating the
potential of these models to navigate through multiple citation network nodes
to locate relevant papers. This capability may signify a shift from
traditional keyword-based searches to more nuanced, context-aware methods that
reflect human reasoning processes.

As the landscape of academic research continues to evolve, the approach
adopted by DocReLM could pave the way for developing more sophisticated,
effective, and user-friendly document retrieval systems. The encouraging
results of this study imply considerable potential for future enhancements,
especially as LLMs further evolve and become increasingly capable of handling
complex tasks in specialized domains.

##  7 Acknowledgements

We sincerely thank Dongzhan Zhou, Guoqiang Jin, and Xingyu Zeng for their
invaluable discussions and constructive feedback. We also extend our gratitude
to Lei Bai, Hao Du, Tianxiang Gao, Jingwen He, Tong He, Di Huang, Xiaoshui
Huang, Meng Li, Yan Lu, Peixia Li, Wentao Qu, Shixiang Tang, Yuan Wang,
Xiaopei Wu, Honghui Yang, Hongliang Yan, Peng Ye, Weicai Ye, Sha Zhang, and
Peixin Zhuang for their diligent work in annotating the test dataset. This
work is partially supported by the National Key R&D Program of China
(NO.2022ZD0160100), and in part by Shanghai Committee of Science and
Technology (Grant No. 21DZ1100100).

## References

  *   * Achiam et al [2023] Achiam J, Adler S, Agarwal S, et al (2023) Gpt-4 technical report. arXiv preprint arXiv:230308774
  * August [2024] August (2024) Search the arxiv. Web page, URL https://searchthearxiv.com/, accessed: 2024-05-12
  * Conneau et al [2020] Conneau A, Khandelwal K, Goyal N, et al (2020) Unsupervised cross-lingual representation learning at scale. In: Jurafsky D, Chai J, Schluter N, et al (eds) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, pp 8440–8451
  * Crossref [2022] Crossref (2022) 2022 public data file of more than 134 million metadata records now available. URL https://www.crossref.org/blog/2022-public-data-file-of-more-than-134-million-metadata-records-now-available/
  * Devlin et al [2019] Devlin J, Chang MW, Lee K, et al (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. In: Burstein J, Doran C, Solorio T (eds) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, pp 4171–4186
  * Douze et al [2024] Douze M, Guzhva A, Deng C, et al (2024) The faiss library. arXiv preprint arXiv:240108281
  * Gao and Callan [2022] Gao L, Callan J (2022) Unsupervised corpus aware language model pre-training for dense passage retrieval. In: Muresan S, Nakov P, Villavicencio A (eds) Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Dublin, Ireland, pp 2843–2853
  * Gao et al [2021a] Gao L, Dai Z, Callan J (2021a) Rethink training of bert rerankers in multi-stage retrieval pipeline. In: The 43rd European Conference On Information Retrieval (ECIR)
  * Gao et al [2021b] Gao L, Zhang Y, Han J, et al (2021b) Scaling deep contrastive learning batch size under memory limited setup. In: Rogers A, Calixto I, Vulić I, et al (eds) Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021). Association for Computational Linguistics, Online, pp 316–321
  * Gong et al [2024] Gong Y, Xu J, Gao R, et al (2024) Single-cell biclustering for cell-specific transcriptomic perturbation detection in ad progression. Available at SSRN 4493250
  * Günther et al [2023] Günther M, Ong J, Mohr I, et al (2023) Jina embeddings 2: 8192-token general-purpose text embeddings for long documents. arXiv preprint arXiv:231019923
  * Guo et al [2022] Guo J, Cai Y, Fan Y, et al (2022) Semantic models for the first-stage retrieval: A comprehensive review. ACM Trans Inf Syst 40(4)
  * Ji et al [2021] Ji Y, Zhou Z, Liu H, et al (2021) DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome. Bioinformatics 37(15):2112–2120
  * Johnson et al [2019] Johnson J, Douze M, Jégou H (2019) Billion-scale similarity search with gpus. IEEE Transactions on Big Data 7(3):535–547
  * Karpukhin et al [2020] Karpukhin V, Oguz B, Min S, et al (2020) Dense passage retrieval for open-domain question answering. In: Webber B, Cohn T, He Y, et al (eds) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, pp 6769–6781
  * Kojima et al [2022] Kojima T, Gu SS, Reid M, et al (2022) Large language models are zero-shot reasoners. In: Koyejo S, Mohamed S, Agarwal A, et al (eds) Advances in Neural Information Processing Systems, vol 35. Curran Associates, Inc., pp 22199–22213
  * Liang et al [2023] Liang C, Bai W, Qiao L, et al (2023) Rethinking the bert-like pretraining for dna sequences. arXiv preprint arXiv:231007644
  * Neelakantan et al [2022] Neelakantan A, Xu T, Puri R, et al (2022) Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:220110005
  * Nguyen et al [2016] Nguyen T, Rosenberg M, Song X, et al (2016) MS MARCO: A human generated machine reading comprehension dataset. In: Besold TR, Bordes A, d’Avila Garcez AS, et al (eds) Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016, CEUR Workshop Proceedings, vol 1773. CEUR-WS.org
  * Nogueira et al [2020] Nogueira R, Jiang Z, Pradeep R, et al (2020) Document ranking with a pretrained sequence-to-sequence model. In: Cohn T, He Y, Liu Y (eds) Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, pp 708–718
  * Reimers and Gurevych [2019] Reimers N, Gurevych I (2019) Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In: Inui K, Jiang J, Ng V, et al (eds) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, pp 3982–3992
  * Robertson and Zaragoza [2009] Robertson S, Zaragoza H (2009) The probabilistic relevance framework: Bm25 and beyond. Found Trends Inf Retr 3(4):333–389
  * Saier and Färber [2020] Saier T, Färber M (2020) unarxive: a large scholarly data set with publications’ full-text, annotated in-text citations, and links to metadata. Scientometrics 125(3):3085–3108
  * Touvron et al [2023] Touvron H, Martin L, Stone K, et al (2023) Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:230709288
  * Vladika and Matthes [2023] Vladika J, Matthes F (2023) Scientific fact-checking: A survey of resources and approaches. In: Rogers A, Boyd-Graber J, Okazaki N (eds) Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics, Toronto, Canada, pp 6215–6230
  * Wadden et al [2020] Wadden D, Lin S, Lo K, et al (2020) Fact or fiction: Verifying scientific claims. In: Webber B, Cohn T, He Y, et al (eds) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, pp 7534–7550
  * Wang et al [2022] Wang K, Thakur N, Reimers N, et al (2022) GPL: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. In: Carpuat M, de Marneffe MC, Meza Ruiz IV (eds) Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Seattle, United States, pp 2345–2360
  * Wei et al [2022] Wei J, Wang X, Schuurmans D, et al (2022) Chain-of-thought prompting elicits reasoning in large language models. In: Koyejo S, Mohamed S, Agarwal A, et al (eds) Advances in Neural Information Processing Systems, vol 35. Curran Associates, Inc., pp 24824–24837
  * Wu et al [2023] Wu X, Ma G, Qian W, et al (2023) Query-as-context pre-training for dense passage retrieval. In: Bouamor H, Pino J, Bali K (eds) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Singapore, pp 1906–1916
  * Xiao et al [2023] Xiao S, Liu Z, Zhang P, et al (2023) C-pack: Packaged resources to advance general chinese embedding. arXiv preprint arXiv:230907597
  * Yenduri et al [2023] Yenduri G, M R, G CS, et al (2023) Generative pre-trained transformer: A comprehensive review on enabling technologies, potential applications, emerging challenges, and future directions. arXiv preprint arXiv:240206852 [cs.CL]
  * Ying et al [2024] Ying H, Zhang S, Li L, et al (2024) Internlm-math: Open math large language models toward verifiable reasoning. arXiv preprint arXiv:240206332
  * Zhang et al [2024] Zhang D, Liu W, Tan Q, et al (2024) Chemllm: A chemical large language model. arXiv preprint arXiv:240206852
  * Zheng et al [2023] Zheng L, Chiang WL, Sheng Y, et al (2023) Judging llm-as-a-judge with mt-bench and chatbot arena. In: Oh A, Naumann T, Globerson A, et al (eds) Advances in Neural Information Processing Systems, vol 36. Curran Associates, Inc., pp 46595–46623
  * Zhou et al [2023] Zhou Z, Ji Y, Li W, et al (2023) Dnabert-2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint arXiv:230615006


# CHARACTER DEGREE GRAPH OF SOLVABLE GROUPS WITH ODD DEGREE

G. Sivanesan Department of Mathematics, Government College of Engineering,
Salem 636011, Tamil Nadu, India, ORCID: 0000-0001-7153-960X.
<EMAIL_ADDRESS>and  C. Selvaraj Department of Mathematics, Periyar
University, Salem 636011, Tamil Nadu, India, ORCID: 0000-0002-4050-3177.
<EMAIL_ADDRESS>
###### Abstract.

Let $G$ be a finite group, let $Irr(G)$ be the set of all complex irreducible
characters of $G$ and let $cd(G)$ be the set of all degrees of characters in
$Irr(G).$ Let $\rho(G)$ be the set of primes that divide degrees in $cd(G).$
The character degree graph $\Delta(G)$ of $G$ is the simple undirected graph
with vertex set $\rho(G)$ and in which two distinct vertices $p$ and $q$ are
adjacent if there exists a character degree $r\in cd(G)$ such that $r$ is
divisible by the product $pq.$ In this paper, we obtain a necessary condition
for the character degree graph $\Delta(G)$ with all of its vertices are odd
degree of a finite solvable group $G$ .

###### Key words and phrases:

character graph, Eulerian graph, non-regular graph, finite solvable group

###### 2020 Mathematics Subject Classification:

05C45, 20F16, 20C15

##  1\. Introduction

Throughout this paper, $G$ will be a finite solvable group with identity $1.$
We denote the set of complex irreducible characters of $G$ by $Irr(G).$ Here
$cd(G)=\\{\chi(1)\mid\chi\in Irr(G)\\}$ is the set of all distinct degrees of
irreducible characters in $Irr(G).$ Let $\rho(G)$ be the set of all primes
that divide degrees in $cd(G).$ There is a vast literature devoted to study of
ways through which one can associate a graph with a group and that literature
can be used for investigating the algebraic structure of groups using graph
theoretical properties of associated graphs. One of these graphs is the
character degree graph $\Delta(G)$ of $G$. In fact, $\Delta(G)$ is an
undirected simple graph with vertex set $\rho(G)$ in which $p,q\in\rho(G)$ are
joined by an edge if there exists a character degree $\chi(1)\in cd(G)$ which
is divisible by $pq.$ This graph was first defined in [12] and studied by many
authors (see [9],[16] and[17]). When $G$ is a solvable group, some interesting
results on the character graph of $G$ have been obtained. Actually, Manz [11]
proved that $\Delta(G)$ has at most two connected components. Also, Manz et
al.[13] have proved that diameter of $\Delta(G)$ is at most $3$. Mahdi
Ebrahimi et al. [3] proved that the character graph $\Delta(G)$ of a solvable
group $G$ is Hamiltonian if and only if $\Delta(G)$ is a block with at least
$3$ vertices. [18] We obtain a necessary condition for the character degree
graph $\Delta(G)$ of a finite solvable group $G$ to be Eulerian. Motivated by
these studies on character degree graphs of solvable groups, we find a
necessary condition for the character degree graph $\Delta(G)$ of a solvable
group $G$ with odd degree vertices only.

##  2\. Preliminaries

In this section, we present some preliminary results which are used in the
paper. All graphs are assumed to be simple, undirected and finite. Let
$\Gamma$ be a graph with vertex set $V(\Gamma)$ and edge set $E(\Gamma)$. If
$\Gamma$ is connected, then the distance $d(u,v)$ between two distinct
vertices $u,v\in V(\Gamma)$ is the length of the shortest path between them.
The supremum of all distance between possible pairs of distinct vertices is
known as the diameter of the graph. A complete graph with $n$ vertices in
which any of the two distinct vertices are adjacent is denoted by $K_{n}.$ A
cycle with $n$ vertices is denoted by $C_{n}.$ The vertex connectivity
$k(\Gamma)$ of $\Gamma$ is defined to be the minimum number of vertices whose
removal from $\Gamma$ results in a disconnected subgraph of $\Gamma.$ A cut
vertex $v$ of a graph $\Gamma$ is a vertex such that the number of connected
components of $\Gamma-v$ is more than the number of connected components of
$\Gamma$. Similar definition is applicable in the case of cut edge of graph
$\Gamma.$ A maximal connected subgraph without a cut vertex is called a block.
By their maximality, different blocks of $\Gamma$ overlap in at most one
vertex, which is then a cut vertex. Thus, every edge of $\Gamma$ lies in a
unique block and $\Gamma$ is the union of its blocks. The degree of a vertex
$v$ in $\Gamma$ is the number of edges incident with $v$ and the same is
denoted by $d(v)$ or deg$v$. A graph $\Gamma$ is called $k$-regular, if the
degree of each vertex is $k$. A graph $\Gamma$ is said to be Eulerian if it
contains a cycle containing all vertices of $\Gamma.$ We will take into
account the following well known facts concerning character degree graphs and
they are needed in the next section.

######  Remark 2.1.

For results regarding $\Delta(G),$ we start with Pálfy’s three prime theorem
on the character degree graph of solvable groups. Pálfy theorem [15, Theorem,
p. 62] states that given a solvable group $G$ and any three distinct vertices
of $\Delta(G)$, there exists an edge incident with the other two vertices. On
applying Pálfy’s theorem, $\Delta(G)$ has at most two connected components.

######  Remark 2.2.

Let $G$ be a solvable group. Then $diam(\Delta(G))\leq 3$ [13, Theorem 3.2].
Assume that the diameter $diam(\Delta(G))=3$ and let $r,s\in\rho(G)$ be two
distinct vertices in $\Delta(G)$ such that distance $d(r,s)=3.$ When
$\Delta(G)$ has exactly diameter $3$ and contains at least $5$ vertices, Lewis
[6, p. 5487] proved that the vertex set $\rho(G)$ of $\Delta(G)$ can be
partitioned into $\rho_{1},\rho_{2},\rho_{3}$ and $\rho_{4}.$ Actually
$\rho_{4}$ is the set of all vertices of $\Delta(G)$ which are at distance $3$
from the vertex $r$ and so we have that $s\in\rho_{4},$ $\rho_{3}$ is the set
of all vertices of $\Delta(G)$ which are distance $2$ from the vertex $r,$
$\rho_{2}$ is the set of all vertices that are adjacent to vertex $r$ and
adjacent to some prime in $\rho_{3}$ and $\rho_{1}$ consists of $r,$ the set
of all vertices which are adjacent to $r$ and not adjacent to any vertex in
$\rho_{3}.$ This implies that, no vertex in $\rho_{1}$ is adjacent to any
vertex in $\rho_{3}\cup\rho_{4}$ and no vertex in $\rho_{4}$ is adjacent to
any vertex in $\rho_{1}\cup\rho_{2},$ every vertex in $\rho_{2}$ is adjacent
to some vertex in $\rho_{3}$ and vice versa, and $\rho_{1}\cup\rho_{2}$ and
$\rho_{3}\cup\rho_{4}$ both induce complete subgraphs of $\Delta(G).$

######  Remark 2.3.

Huppert [5, p. 25] listed all possible graphs $\Delta(G)$ for solvable groups
$G$ with at most $4$ vertices. In fact, every graph with $3$ or few vertices
that satisfies Pálfy’s condition occurs as $\Delta(G)$ for some solvable group
$G.$

######  Theorem 2.4.

[3, Lemma 2.7] Let $G$ be a group with $\mid\rho(G)\mid\geq 3.$ If $\Delta(G)$
is not a block and the diameter of $\Delta(G)$ is at most $2$, then each block
of $\Delta(G)$ is a complete graph.

######  Theorem 2.5.

[19, Theorem 5] The graph with four vertices in Figure 1 is not the character
degree graph of a solvable group.

Figure 1. Graph with four vertices

######  Theorem 2.6.

[10, Theorem 1.1] Let $G$ be a solvable group. Then $\Delta(G)$ has at most
one cut vertex.

######  Theorem 2.7.

[14, Theorem A] If $\Delta(G)$ is a non-complete and regular character degree
graph of a finite solvable group $G$ with $n$ vertices, then $\Gamma(G)$ is
$n-2$ is a regular graph.

######  Theorem 2.8.

[10, Lemma 2.1] Let $G$ be a solvable group and assume that $\Delta(G)$ has
diameter $3$. Then $G$ is $1$-connected if and only if $\mid\rho_{2}\mid=1$ in
the diameter $3$ partition of $\rho(G)$. In this case, if $p$ is the unique
prime in $\rho_{2}$, then $p$ is also the unique cut vertex for $\Delta(G)$.
In particular, $\Delta(G)$ has at most one cut vertex.

######  Theorem 2.9.

[8, Corollary B] If $G$ is a solvable group with $n=\mid\rho(G)\mid$ and
$\Delta(G)$ contains two vertices of degree less than $n-2$ that are not
adjacent, then the Fitting height of $G$ is at least $3$.

######  Theorem 2.10.

[4, Lemma 4.1] Let $G$ be a solvable group where the diameter of $\Delta(G)$
is $3$ with Lewis partition
$\rho(G)=\rho_{1}\cup\rho_{2}\cup\rho_{3}\cup\rho_{4}.$ Then $\Delta(G)$ is a
block if and only if $\mid\rho_{2}\mid,\mid\rho_{3}\mid\geq 2.$

######  Definition 2.11.

[7] Using direct product, one can find bigger groups from smaller groups. The
same may be used to construct higher order character degree graphs. Given
groups $A$ and $B,$ we have that $\rho(A\times B)=\rho(A)\cup\rho(B).$ Define
an edge between vertices $p$ and $q$ in $\rho(A\times B)$ if any of the
following is satisfied:

  * (i)

$p,q\in\rho(A)$ and there is an edge between $p$ and $q$ in $\Delta(A);$

  * (ii)

$p,q\in\rho(B)$ and there is an edge between $p$ and $q$ in $\Delta(B);$

  * (iii)

$p\in\rho(A)$ and $q\in\rho(B);$

  * (iv)

$p\in\rho(B)$ and $q\in\rho(A).$

Now we get a higher order character degree graph and it is called direct
product.

##  3\.  Character Degree Graphs with odd degree

######  Theorem 3.1.

Let $G$ be a solvable group and $\Delta(G)$ is a complete graph with $n\geq 2$
is even, then $\Delta(G)$ is a graph with odd degree.

Proof.  Every complete graph is a character degree graph [2, Lemma 2.1]. By
assumption, $n$ is even. So all the vertices of $\Delta(G)$ are odd degree.
$\square$

######  Theorem 3.2.

If $\Delta(G)$ is a non-complete and regular character degree graph of a
finite solvable group $G$ with $n$ vertices, then $\Delta(G)$ is not a
character degree graph with odd degrees .

Proof.  By [14, Theorem A], then $\Delta(G)$ is $n-2$ regular graph.
Case 1. $n$ is even.
If $n$ is even, then the regular graph is even. So $\Delta(G)$ is not a
character degree graph with odd degree
Case 2. $n$ is odd.
The number of vertices of odd degree is even. So $n$ can not be odd. $\square$

######  Theorem 3.3.

Let $G$ be a solvable group and $\Delta(G)$ is a graph with odd degree, then
$\Delta(G)$ is a block .

Proof.
Case 1. For, assume that $\Delta(G)$ has diameter $3$. By assumption
$\Delta(G)$ is not a block, and each block of $\Delta(G)$ contain even number
of vertices. $\Delta(G)$ has two blocks say $B_{1}$ and $B_{2}$ and it has a
cut vertex. This cut vertex is belongs to $\rho_{2}$. By Theorem 2.8,
$\mid\rho_{2}\mid=1$. Since $\rho_{1}\cup\rho_{2}$ and $\rho_{3}\cup\rho_{4}$
induce complete subgraphs, each vertex in $\rho_{3}\cup\rho_{4}$ is of odd
degree. Since $\mid\rho_{2}\mid=1$, and every vertex in $\rho_{3}$ is adjacent
to some vertex in $\rho_{2}$ by Remark 2.2. So each vertex in $\rho_{3}$ is of
even degree. Hence $\Delta(G)$ is not a graph with odd degree vertices.
Case 2. Suppose $\Delta(G)$ has diameter at most $2.$ By assumption
$\Delta(G)$ is not a block, and each block of $\Delta(G)$ contain even number
of vertices. $\Delta(G)$ has two blocks say $B_{1}$ and $B_{2}$ and it has a
cut vertex. By Theorem 2.4, each block of $\Delta(G)$ is a complete graph. So,
the cut vertex of $\Delta(G)$ is even degree. Therefore $\Delta(G)$ is not a
graph with odd degree vertices. $\square$

We prove that the character degree graph $\Delta(G)$ is a graph with odd
degree vertices only when the diameter of $\Delta(G)$ is $3.$

######  Theorem 3.4.

Let $G$ be a solvable group with diameter of $\Delta(G)$ is $3.$ Let the
Lewis’ partition of $G$ be
$\rho(G)=\rho_{1}\cup\rho_{2}\cup\rho_{3}\cup\rho_{4}.$ Then $\Delta(G)$ is a
graph with only odd degree vertices if and only if the following conditions
are hold:

  * (i)

$\Delta(G)$ is a block;

  * (ii)

Both $\mid\rho_{1}\cup\rho_{2}\mid$ and $\mid\rho_{3}\cup\rho_{4}\mid$ are
even;

  * (iii)

The subgraph of $\Delta(G)$ induced by $\rho_{2}\cup\rho_{3}$ is Eulerian.

Proof.  Assume that $\Delta(G)$ with odd degree vertices.

(i) If $\Delta(G)$ is not a block, by Theorem 3.3 $\Delta(G)$ is not a graph
with odd degree vertices. Hence $\Delta(G)$ is a block.

(ii) We claim that both $\mid\rho_{1}\cup\rho_{2}\mid$ and
$\mid\rho_{3}\cup\rho_{4}\mid$ are even. Suppose not, let us assume that
$\mid\rho_{1}\cup\rho_{2}\mid$ is odd. As mentioned in Remark 2.2,
$\rho_{1}\cup\rho_{2}$ is a complete graph and no prime in $\rho_{1}$ is
adjacent to any prime in $\rho_{3}\cup\rho_{4}.$ Hence the vertices in
$\rho_{1}$ are of even degree and so $\Delta(G)$ is not a graph with odd
degree vertices, which is a contradiction. Therefore
$\mid\rho_{1}\cup\rho_{2}\mid$ is even.

Suppose that $\mid\rho_{3}\cup\rho_{4}\mid$ is odd. As mentioned in Remark
2.2, $\rho_{3}\cup\rho_{4}$ is a complete graph and no prime in $\rho_{4}$ is
adjacent to any prime in $\rho_{1}\cup\rho_{2}.$ Hence the vertices in
$\rho_{4}$ are of even degree and so $\Delta(G)$is not a graph with odd degree
vertices, which is a contradiction. Therefore $|\rho_{3}\cup\rho_{4}|$ is
even.

(iii) Suppose the subgraph induced by $\rho_{2}\cup\rho_{3}$ is non Eulerian.
One can refer Remark 2.2 for the following facts. No prime in $\rho_{1}$ is
adjacent to any prime in $\rho_{3}\cup\rho_{4}.$ Similarly no prime in
$\rho_{4}$ is adjacent to any prime in $\rho_{1}\cup\rho_{2}.$ Since
$\rho_{1}\cup\rho_{2}$ and $\rho_{3}\cup\rho_{4}$ are complete graphs with
even number of vertices. Hence each vertex in $\rho_{2}$ and $\rho_{3}$ is of
odd degree in sub graphs induced by $\rho_{1}\cup\rho_{2}$ and
$\rho_{3}\cup\rho_{4}$. Therefore the vertices in $\rho_{1}$ and $\rho_{4}$
are of odd degree. But the degree of each vertex in $\rho_{2}$ is the sum of
the degree of that vertex in $\rho_{2}$ in the subgraph induced by
$\rho_{2}\cup\rho_{3}$ and degree of that vertex in $\rho_{2}$ in the subgraph
induced by $\rho_{1}\cup\rho_{2}$. Similarly degree of each vertex in
$\rho_{3}$ is the sum of degree of that vertex in $\rho_{3}$ in the subgraph
induced by $\rho_{2}\cup\rho_{3}$ and degree of that vertex in $\rho_{3}$ in
the subgraph induced by $\rho_{3}\cup\rho_{4}$. By assumption
$\rho_{2}\cup\rho_{3}$ is non Eulerian and hence there are vertices of odd
degree in $\rho_{2}$ or $\rho_{3}$ in the subgraph induced by
$\rho_{2}\cup\rho_{3}.$ Hence some vertices in $\rho_{2}$ or $\rho_{3}$ are of
even degree, which is a contradiction to $\Delta(G)$ is a graph with odd
degree vertices. Therefore the vertex induced subgraph $\rho_{2}\cup\rho_{3}$
is Eulerian.

Conversely, assume that conditions (i)-(iii) are true. According to Lewis
partition, the subgraphs induced by $\rho_{1}\cup\rho_{2}$ and
$\rho_{3}\cup\rho_{4}$ are complete subgraphs of $\Delta(G),$ no prime in
$\rho_{1}$ is adjacent to any prime in $\rho_{3}\cup\rho_{4}$ and no prime in
$\rho_{4}$ is adjacent to any prime in $\rho_{1}\cup\rho_{2}.$ Since both
$\mid\rho_{1}\cup\rho_{2}\mid$ and $\mid\rho_{3}\cup\rho_{4}\mid$ are even,
vertices in $\rho_{1}$ and $\rho_{4}$ are of odd degree. Since $\Delta(G)$ is
a block, by Theorem 2.10 we get that $\mid\rho_{2}\mid,\mid\rho_{3}\mid\geq
2.$ Since the subgraph induced by $\rho_{2}\cup\rho_{3}$ is Eulerian, we get
that vertices in $\rho_{2}$ and $\rho_{3}$ are of odd degree. According to
Lewis partition, all subsets $\rho_{1},\rho_{2},\rho_{3}$ and $\rho_{4}$ in
$\rho(G)$ are non-empty disjoint subsets. Therefore all the vertices in
$\rho(G)$ are odd degree. Hence $\Delta(G)$ is a graph with odd degree
vertices. $\square$

##  4\. Number of Character Degree Graphs with odd degree

In this section, we obtain the number of character degree graphs with odd
degree in terms of number of vertices. Actually, we obtain below that the
number of character degree graphs with $n$ vertices ($n\geq 6$ is even) which
are non regular by assuming that the diameter of $\Delta(G)$ is two.

######  Theorem 4.1.

The character degree graphs with n vertices ($n\geq 6$ is even) for some
solvable group $G$ has atleast $1$ non regular character degree graphs with
odd degree, Here $\Delta(G)$ is a block with diameter $2$.

Proof.  In this proof all the character degree graphs are constructed via
direct products only by definition 2.11.
Case i:$\mid\rho(G)\mid=6.$ Bissler et. al [1, p. 503] classified all
character degree graphs with $6$ vertices expect $9$ graphs listed in [1, p.
509]. But none of these $9$ graphs are character degree graphs with odd
degree. On the other hand, among twelve graphs, there exists only one non
regular graph with all the vertices as odd degree and the same is given in
Figure $2$.

Figure 2. Graph with six vertices

Case ii: $\mid\rho(G)\mid=8.$ Let $\Delta(G)$ be the character degree graphs
as in Figure $2.$ Let $\Delta(H)$ be $K_{2}$ and $\rho(H)$ are disjoint from
$\rho(G)$. By direct product, $\Delta(G\times H)$ have $8$ vertices in which
$4$ vertices have degree $5$ and remaining $4$ vertices have degree $7$.

The number of character degree graphs with odd degree which are not regular
graph is $1$.
Case iii: $\mid\rho(G)\mid=10.$ Let $\Delta(G)$ be $8$ vertices graph as in
case $3$ and $\Delta(H)$ be $K_{2}$ and $\rho(H)$ are disjoint from $\rho(G)$.
By direct product, $\Delta(G\times H)$ have $10$ vertices in which $4$
vertices have degree $7$ and remaining $6$ vertices have degree $9$.

The number of character degree graphs with odd degree which are not regular
graph is $1$.
By induction method, Let $\Delta(G)$ be the character degree graph with $n-2$
vertices in which $4$ vertices have degree $n-5$ and remaining $n-6$ vertices
have degree $n-3$ and $\Delta(H)$ be $K_{2}$ which are disjoint from
$\rho(G)$. By direct product, $\Delta(G\times H)$ have $n$ vertices in which
$4$ vertices have degree $n-3$ and remaining $n-4$ vertices have degree $n-1$.
$\square$

######  Remark 4.2.

By Theorem 2.9 solvable group in Theorem 4.1 has Fitting height is at least 3.

## Acknowledgment

The second author is partially supported by DST-FIST (Letter No: SR / FST /
MSI - $115$ / $2016$ dated: $10$ \- $10$ \- $2017$ ).

## References

  * [1] M. Bissler, J. Laubacher and M.L. Lewis, Classifying character degree graphs with six vertices, Beitr. Algebra Geom. 60(2019), 499–511.
  * [2] M. Bissler and J. Laubacher, Classifying family of character degree graphs of solvable groups, Int. J.Group Theory 8(4)(2019), 37–46.
  * [3] M. Ebrahimi, A. Iranmanesh and M. A. Hosseinzadeh, Hamiltonian character graphs, J. Algebra 428 (6) (2015), 54–66.
  * [4] M. Ebrahimi and A. Iranmanesh, Coloring of character graphs, Comm. Algebra 45:1(2017), 227–233.
  * [5] B. Huppert , Research in representation theory at Mainz (1984-1990), Progr. Math. 95(1991), 17–36.
  * [6] M.L. Lewis, Solvable groups with character degree graphs having 5 vertices and diameter 3, Comm. Algebra 30 (2002), 5485–5503.
  * [7] M.L. Lewis, Classifying character degree graphs with 5 vertices, in Finite Groups 2003, Walter de Gruyter GmbH and Co. KG, Berlin, (2004), 247– 265.
  * [8] M.L. Lewis, Character Degree Graphs of Solvable Groups of Fitting Height 2, Canad. Math. Bull. 49 (1), (2006), 127–133.
  * [9] M.L. Lewis, An overview of graphs associated with character degrees and conjugacy class sizes in finite groups, Rocky Mountain J. Math. 38 (1) (2008), 175–211.
  * [10] M.L. Lewis and Q. Meng, Solvable groups whose prime divisor character degree graphs are 1-connected, Monatsh. Math. 190 (2019), 541–548.
  * [11] O. Manz, Degree problems II: $\pi$-separable character degrees, Comm. Algebra 13 (1985), 2421–2431.
  * [12] O. Manz, R. Staszewski and W. Willems, On the number of components of a graph related to character degrees, Proc. AMS 103 (1) (1988), 31–37.
  * [13] O. Manz, W. Willems and T.R. Wolf, The diameter of the character degree graph, J. Reine Angew. Math. 402 (1989), 181–198.
  * [14] C.P. Morresi Zuccari, Regular character degree graphs, J. Algebra 411(2014), 215–224.
  * [15] P.P. Pálfy, On the character degree graph of solvable groups I: three primes, Period. Math. Hungar. 36 (1) (1998), 61-65.
  * [16] P.P. Pálfy, On the character degree graph of solvable groups II: Disconnected graphs, Studia Sci. Math. Hungar. 38 (2001), 339–355.
  * [17] C.B. Sass, Character degree graphs of Solvable groups with diameter three J. Algebra 19 (6) (2016), 1097–1127.
  * [18] G. Sivanesan, C. Selvaraj and T. Tamizh chelvam, Eulerian Character degree graphs of solvable groups (2023), arXiv: 2304.13472.
  * [19] J. Zhang, On a problem by Huppert, Bejing Daxue Xuebao Ziran Kexue Ban, 34 (2-3) (1998) , 143–150.