##  Appendix B Proof of Lemmas 6.3 and 6.4

###### Proof of Lemma 6.3.

Let $\epsilon>0$ and $\nu^{1},\omega\in\mathcal{A}_{m}$. We note that from
(2.5) it follows that

$Q^{1,\nu^{1}+\epsilon\omega}_{t}=Q^{1,\nu^{1}}_{t}-\epsilon\int^{t}_{0}\omega_{s}ds,\quad\textrm{for
all }0\leq t\leq T.$ (B.1)

We use the alternative representation of $H^{1}$ in (6.1) and (B.1) to get

$\displaystyle H^{1}(\nu^{1}+\epsilon\omega)-H^{1}(\nu^{1})$
$\displaystyle=\epsilon\mathbb{E}\Bigg{[}\int^{T}_{0}\omega_{t}(-2\lambda_{1}\nu_{t}^{1}+2\alpha
Q^{1,\nu^{1}}_{T}-\kappa_{1}Q^{1,\nu^{1}}_{t})dt$ (B.2)
$\displaystyle\quad+\int^{T}_{0}\left(\int^{t}_{0}\omega_{s}ds\right)(2\phi^{1}_{t}Q_{t}^{1,\nu^{1}}dt+\kappa_{0}\nu^{0}_{t}dt+\kappa_{1}\nu_{t}^{1}dt-
dA_{t})\Bigg{]}$
$\displaystyle\quad+\epsilon^{2}\mathbb{E}\Bigg{[}-\lambda_{1}\int^{T}_{0}\omega_{s}^{2}ds-\alpha\left(\int^{T}_{0}\omega_{s}ds\right)^{2}$
$\displaystyle\quad-\int^{T}_{0}\phi^{1}_{t}\left(\int^{t}_{0}\omega_{s}ds\right)^{2}dt+\kappa_{1}\int^{T}_{0}\omega_{t}\left(\int^{t}_{0}\omega_{s}ds\right)dt\Bigg{]}.$

From (6.12) and (B.2) we get

$\displaystyle\langle\mathcal{D}H^{1}(\nu^{1}),\omega\rangle$
$\displaystyle=\mathbb{E}\Bigg{[}\int^{T}_{0}\omega_{t}(-2\lambda_{1}\nu_{t}^{1}+2\alpha
Q^{1,\nu^{1}}_{T}-\kappa_{1}Q^{1,\nu^{1}}_{t})dt$ (B.3)
$\displaystyle\quad+\int^{T}_{0}\left(\int^{t}_{0}\omega_{s}ds\right)(2\phi^{1}_{t}Q_{t}^{1,\nu^{1}}dt+\kappa_{0}\nu^{0}_{t}dt+\kappa_{1}\nu_{t}^{1}dt-
dA_{t})\Bigg{]}.$

Since $\nu^{1},\omega\in\mathcal{A}_{m}$, $\nu^{0}\in\mathcal{A}_{M}^{q_{0}}$
and $\mathbb{E}[(\int^{T}_{0}|dA_{t}|)^{2}]<\infty$, then use Fubini’s theorem
in (B.3) to get

$\langle\mathcal{D}H^{1}(\nu^{1}),\omega\rangle=\mathbb{E}\Bigg{[}\int^{T}_{0}\omega_{t}\Big{(}-2\lambda_{1}\nu_{t}^{1}+2\alpha
Q^{1,\nu^{1}}_{T}-\kappa_{1}Q^{1,\nu^{1}}_{t}+A_{t}\\\
+\int^{T}_{t}\left(2\phi^{1}_{s}Q_{s}^{1,\nu^{1}}+\kappa_{0}\nu^{0}_{s}+\kappa_{1}\nu_{s}^{1}\right)ds-
A_{T}\Big{)}dt\Bigg{]},$

which concludes the result. ∎

###### Proof of Lemma 6.4.

In Lemma 6.2 we have shown that under Assumption 3.1, the functional
$H^{1}(\nu^{1})$ is strictly concave of any $\nu^{1}\in\mathcal{A}_{m}$.
Therefore, we may apply Proposition 2.1 of (Ekeland and Témam, 1999, Chapter
II) to obtain that

$\langle\mathcal{D}H^{1}(\nu^{1,*}),\omega\rangle=0\quad\textrm{for all
}\omega\in\mathcal{A}_{m}\iff\nu^{1,*}=\arg\sup_{\nu\in\mathcal{A}_{m}}H^{1}(\nu).$
(B.4)

The strict concavity of $H^{1}$ guarantees that the optimiser $\nu^{1,*}$ is
unique.

#### Necessity:

We assume that

$\nu^{1,*}=\arg\sup_{\nu\in\mathcal{A}_{m}}H^{1}(\nu).$

Then, (B.4) and (6.13) imply that for all $\omega\in\mathcal{A}_{m}$ it holds

$\displaystyle\langle\mathcal{D}H^{1}(\nu^{1,*}),\omega\rangle$
$\displaystyle=\mathbb{E}\Bigg{[}\int^{T}_{0}\omega_{t}\Big{(}-2\lambda_{1}\nu_{t}^{1,*}+2\alpha
Q^{1,\nu^{1,*}}_{T}-\kappa_{1}Q^{1,\nu^{1,*}}_{t}+A_{t}$
$\displaystyle\quad+\int^{T}_{t}\left(2\phi^{1}_{s}Q_{s}^{1,\nu^{1,*}}+\kappa_{0}\nu^{0}_{s}+\kappa_{1}\nu_{s}^{1,*}\right)ds-
A_{T}\Big{)}dt\Bigg{]}=0.$

By applying the optional projection theorem we get

$\mathbb{E}\Bigg{[}\int^{T}_{0}\omega_{t}\Bigg{(}-2\lambda_{1}\nu_{t}^{1,*}+\mathbb{E}_{t}\left[2\alpha
Q^{1,\nu^{1,*}}_{T}-A_{T}\right]-\kappa_{1}Q^{1,\nu^{1,*}}_{t}+A_{t}\\\
+\mathbb{E}_{t}\left[\int^{T}_{t}\left(2\phi^{1}_{s}Q_{s}^{1,\nu^{1,*}}+\kappa_{0}\nu^{0}_{s}+\kappa_{1}\nu_{s}^{1,*}\right)ds\right]\Bigg{)}dt\Bigg{]}=0,$
(B.5)

As (B.5) holds for all $\omega\in\mathcal{A}_{m}$ we deduce the following
first order condition holds

$-2\lambda_{1}\nu_{t}^{1,*}+\mathbb{E}_{t}\left[2\alpha
Q^{1,\nu^{1,*}}_{T}-A_{T}\right]-\kappa_{1}Q^{1,\nu^{1,*}}_{t}+A_{t}\\\
+\mathbb{E}_{t}\left[\int^{T}_{t}\left(2\phi^{1}_{s}Q_{s}^{1,\nu^{1,*}}+\kappa_{0}\nu^{0}_{s}+\kappa_{1}\nu_{s}^{1,*}\right)ds\right]=0,$
(B.6)

$d\mathbb{P}\otimes dt$-a.e. on $\Omega\times[0,T]$.

We define the following martingales

$\displaystyle\mathcal{M}_{t}$
$\displaystyle:=\mathbb{E}_{t}\left[\int^{T}_{0}\left(2\phi^{1}_{s}Q_{s}^{1,\nu^{1,*}}+\kappa_{0}\nu^{0}_{s}+\kappa_{1}\nu_{s}^{1,*}\right)ds\right],$
(B.7) $\displaystyle\mathcal{N}_{t}$
$\displaystyle:=\mathbb{E}_{t}\left[2\alpha Q^{1,\nu^{1,*}}_{T}-A_{T}\right].$

Note that $\mathcal{M}$ and $\mathcal{N}$ are square-integrable since
$\mathbb{E}[(\int^{T}_{0}|dA_{t}|)^{2}]<\infty$,
$\nu^{1,*},\omega\in\mathcal{A}_{m}$ and $\nu^{0}\in\mathcal{A}_{M}^{q_{0}}$.

We plug $\mathcal{M}$ and $\mathcal{N}$ in (B.6) to get

$\displaystyle-2\lambda_{1}\nu_{t}^{1,*}+\mathcal{N}_{t}-\kappa_{1}Q^{1,\nu^{1,*}}_{t}+A_{t}+\mathcal{M}_{t}-\int^{t}_{0}\left(2\phi^{1}_{s}Q_{s}^{1,\nu^{1,*}}+\kappa_{0}\nu^{0}_{s}+\kappa_{1}\nu_{s}^{1,*}\right)ds=0,$
(B.8)

From (2.5) and (B.8) it follows that $\nu^{1,*}$ solves the following BSDE

$\begin{cases}d\nu_{t}^{1,*}&=\frac{1}{2\lambda_{1}}d\mathcal{N}_{t}+\frac{1}{2\lambda_{1}}d\mathcal{M}_{t}-\frac{1}{\lambda_{1}}\phi^{1}_{t}Q_{t}^{1,\nu^{1,*}}dt-\frac{\kappa_{0}}{2\lambda_{1}}\nu^{0}_{t}dt+\frac{1}{2\lambda_{1}}dA_{t},\\\
\nu_{T}^{1,*}&=\frac{2\alpha-\kappa_{1}}{2\lambda_{1}}Q_{T}^{1,\nu^{1,*}},\end{cases}$

this gives (6.14).

#### Sufficiency:

Assume that $(Q^{1,\nu^{1,*}},\nu^{1,*})$ solves (6.14) $d\mathbb{P}\otimes
dt$-a.e. and that $\nu^{1,*}\in\mathcal{A}_{m}$. We will show that
$\langle\mathcal{D}H^{1}(\nu^{1,*}),\omega\rangle$ vanishes for all
$\omega\in\mathcal{A}_{m}$ which, once combined with (B.4), implies that
$\nu^{1,*}$ is the solution to the minor agent’s problem. Since
$(Q^{1,\nu^{1,*}},\nu^{1,*})$ solves (6.14) we get that

$\displaystyle 2\lambda_{1}\nu_{t}^{1,*}$
$\displaystyle=\mathbb{E}_{t}\left[\left(2\alpha-\kappa_{1}\right)Q^{1,\nu^{1,*}}_{T}\right]-\mathbb{E}_{t}\left[\int^{T}_{t}dA_{s}\right]+\mathbb{E}_{t}\left[\int^{T}_{t}\left(\kappa_{0}\nu^{0}_{s}+2\phi^{1}_{t}Q^{1,\nu^{1,*}}_{s}\right)ds\right]$
(B.9) $\displaystyle=\mathbb{E}_{t}\left[2\alpha
Q^{1,\nu^{1,*}}_{T}-A_{T}\right]-\kappa_{1}Q^{1,\nu^{1,*}}_{t}+A_{t}$
$\displaystyle\
\quad+\mathbb{E}_{t}\left[\int^{T}_{t}\left(\kappa_{0}\nu^{0}_{s}+2\phi^{1}_{t}Q^{1,\nu^{1,*}}_{s}+\kappa_{1}\nu^{1,*}_{s}\right)ds\right],\quad
d\mathbb{P}\otimes dt-\textrm{a.e.},$

where we used (2.5) in the second equality. Hence, $\nu^{1,*}_{t}$ satisfies
(B.6), therefore the left-hand side of (B.4) hold. ∎

##  Appendix C Proof of Proposition 6.5 and Lemma 6.7

Before we prove Lemma 6.7 we introduce the following lemma.

######  Lemma C.1.

Under Assumption 3.1, the Riccati equation (6.18) has a unique continuous
solution.

###### Proof.

Let $\hat{r}^{1}$ be the solution to the following equation

$\begin{cases}\partial_{t}\hat{r}^{1}_{t}&=-\frac{1}{\lambda_{1}}\phi^{1}_{t}+(\hat{r}^{1}_{t})^{2},\quad
0\leq t\leq T,\\\
\hat{r}^{1}_{T}&=\frac{2\alpha-\kappa_{1}}{2\lambda_{1}}.\end{cases}$ (C.1)

Since, $\hat{r}^{1}_{T}\geq 0$ and $\phi^{1}$ is a piecewise continuous,
locally bounded non-negative function over $[0,T]$, then by Theorem 2.1 of
Wonham (1968) there exists a unique solution $\hat{r}^{1}$, which is
absolutely continuous on $[0,T]$ (see also Theorem 3.5 in Freiling (2002) for
a more recent reference). As stated by Wonham (1968), the function
$\hat{r}^{1}$ satisfies (C.1) only $dt$ almost everywhere. Note that by taking
$r^{1}_{t}=-\hat{r}^{1}_{t}$, it follows that $r^{1}$ is an absolutely
continuous solution to (6.18). Uniqueness of the solution to (6.18) then
follows by the uniqueness for (C.1). ∎

###### Proof of Lemma 6.7.

Lemma C.1 proves that $r^{1}$ is continuous on $[0,T]$. Then, from (3.3) it
follows that the functions $\xi^{\pm}_{t}$ are continuous. Therefore, from
(3.4) we get that $\mathcal{K}$ is jointly continuous on $[0,T]^{2}$ hence it
$\mathcal{K}$ is bounded on $[0,T]^{2}$ and

$\int^{T}_{0}\int^{T}_{0}\left|\mathcal{K}(t,s)\right|^{2}dsdt<\infty,$

that is $\mathcal{K}$ is in $L^{2}([0,T]^{2})$. ∎

Now we are ready to prove Proposition 6.5.

###### Proof of Proposition 6.5.

In Lemma C.1 we have established that (6.18) has a unique continuous solution
$r^{1}$. We prove the rest of the claims in the following two steps.

Step 1. We show that $r^{0}_{t}$ given by (3.5) solves the BSDE (6.19); Note
that since $r^{1}$ is continuous, the function $\xi^{+}$ in (3.3) is the
unique solution of the ODE

$\frac{d\xi_{t}^{+}}{dt}=r^{1}_{t}\xi_{t}^{+},\quad\xi_{0}^{+}=1.$ (C.2)

Since $r^{1}_{t}$ is continuous on $[0,T]$, it holds that

$\int^{T}_{0}(\xi_{t}^{+})^{2}dt<\infty.$ (C.3)

Since $\xi_{t}^{+}$ satisfies (C.3), then the process

$\xi_{t}^{+}r^{0}_{t}:=\frac{1}{2\lambda_{1}}\mathbb{E}_{t}\left[\int^{T}_{t}\xi_{s}^{+}(dA_{s}-\kappa_{0}\nu^{0}_{s}ds)\right],\quad
0\leq t\leq T,$ (C.4)

is the unique strong solution to the following linear BSDE

$\begin{cases}d(\xi_{t}^{+}r^{0}_{t})&=\frac{\xi_{t}^{+}}{2\lambda_{1}}(\kappa_{0}\nu^{0}_{t}dt-
dA_{t})-\frac{1}{2\lambda_{1}}\xi_{t}^{+}d\mathcal{M}_{t}-\frac{1}{2\lambda_{1}}\xi_{t}^{+}d\mathcal{N}_{t},\\\
\xi_{T}^{+}r^{0}_{T}&=0.\end{cases}$ (C.5)

We multiply both sides in (C.4) by $\xi_{t}^{-}$ from (3.3) and use the
identity $\xi_{t}^{-}\xi_{t}^{+}=1$. By doing so, we can obtain an expression
for $r^{0}_{t}$, that is

$r^{0}_{t}=\frac{1}{2\lambda_{1}}\mathbb{E}_{t}\left[\int^{T}_{t}\xi_{t}^{-}\xi_{s}^{+}(dA_{s}-\kappa_{0}\nu^{0}_{s}ds)\right].$
(C.6)

We now show that $r^{0}$ from (C.6) is the solution to (6.19). From (C.5) and
Itô’s product rule we get

$d\xi_{t}^{+}r^{0}_{t}+\xi_{t}^{+}dr^{0}_{t}=\frac{\xi_{t}^{+}}{2\lambda_{1}}(\kappa_{0}\nu^{0}_{t}dt-
dA_{t})-\frac{1}{2\lambda_{1}}\xi_{t}^{+}d\mathcal{M}_{t}-\frac{1}{2\lambda_{1}}\xi_{t}^{+}d\mathcal{N}_{t}.$
(C.7)

Next, we use (C.2) and (C.7) to get

$\xi_{t}^{+}\left(r^{1}_{t}r^{0}_{t}dt+dr^{0}_{t}-\frac{1}{2\lambda_{1}}(\kappa_{0}\nu^{0}_{t}dt-
dA_{t})+\frac{1}{2\lambda_{1}}d\mathcal{M}_{t}+\frac{1}{2\lambda_{1}}d\mathcal{N}_{t}\right)=0.$
(C.8)

Since $\xi_{t}^{+}>0$ for all $t\in[0,T]$, we have

$r^{1}_{t}r^{0}_{t}dt+dr^{0}_{t}-\frac{1}{2\lambda_{1}}(\kappa_{0}\nu^{0}_{t}dt-
dA_{t})+\frac{1}{2\lambda_{1}}d\mathcal{M}_{t}+\frac{1}{2\lambda_{1}}d\mathcal{N}_{t}=0.$

with terminal condition $r^{0}_{T}=0$ which follows from (C.6). By comparing
this with (6.19), it follows that $r^{0}_{t}$ is the solution to the BSDE
(6.19). Recall the definition of $\mathcal{K}$ in (3.4). We substitute the
expression for $\mathcal{K}(t,s)$ from (3.4) into (C.6) from which it follows
that $r^{0}_{t}$ given by (3.5) solves (6.19).

Step 2. We show (6.20). From (3.4) it follows that $\mathcal{K}(t,s)>0$ for
all $t,s\in[0,T]^{2}$. Moreover by Lemma 6.7, $\mathcal{K}$ is bounded on
$[0,T]^{2}$. Since $A$ is of bounded variation and $\mathcal{K}$ is bounded,
then by using the conditional Jensen’s inequality and the tower property we
get

$\displaystyle\sup_{t\in[0,T]}\mathbb{E}\left[\left(\mathbb{E}_{t}\left[\int^{T}_{t}\mathcal{K}(t,s)dA_{s}\right]\right)^{2}\right]$
$\displaystyle\leq
C\mathbb{E}\left[\left(\int^{T}_{0}|dA_{s}|\right)^{2}\right]$ (C.9)
$\displaystyle<\infty,$

where we used (2.1) in the last inequality. Similarly, we can obtain that

$\displaystyle\sup_{t\in[0,T]}\mathbb{E}\left[\left(\mathbb{E}_{t}\left[\int^{T}_{t}\mathcal{K}(t,s)\nu^{0}_{s}ds\right]\right)^{2}\right]$
$\displaystyle\leq{C}\mathbb{E}\left[\int^{T}_{0}(\nu^{0}_{s})^{2}ds\right]$
(C.10) $\displaystyle<\infty,$

where we used the fact that $\nu^{0}\in\mathcal{A}_{M}^{q_{0}}$ and (2.2).
From (3.5), (C.9) and (C.10) we get

$\sup_{t\in[0,T]}\mathbb{E}[(r^{0}_{t})^{2}]<\infty,$

and (6.20) follows. ∎

## Acknowledgments

We are very grateful to Rama Cont whose numerous useful comments enabled us to
significantly improve this paper.

## References

  * Almgren and Chriss (2000) R. Almgren and N. Chriss.  Optimal execution of portfolio transactions.  _Journal of Risk_ , 3(2):5–39, 2000.
  * Atkinson (1997) K. E. Atkinson.  _The Numerical Solution of Integral Equations of the Second Kind_.  Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, 1997.
  * Belak et al. (2019) C. Belak, J. Muhle-Karbe, and K. Ou.  Liquidation in target zone models.  _Market Microstructure and Liquidity_ , 2019.
  * Carlin et al. (2007) B. I. Carlin, M. S. Lobo, and S. Viswanathan.  Episodic liquidity crises: cooperative and predatory trading.  _Journal of Finance_ , 65:2235–2274, 2007.
  * Cartea and Jaimungal (2016) Á. Cartea and S. Jaimungal.  Incorporating order-flow into optimal execution.  _Mathematics and Financial Economics_ , 10(3):339–364, 2016.
  * Cartea et al. (2015) A. Cartea, S. Jaimungal, and J. Penalva.  _Algorithmic and High-Frequency Trading_.  Cambridge University Press, 2015.
  * Casgrain and Jaimungal (2019) P. Casgrain and S. Jaimungal.  Mean field games with partial information for algorithmic trading.  Preprint, available online at https://arxiv.org/abs/1803.04094, 2019\.
  * Casgrain and Jaimungal (2020) P. Casgrain and S. Jaimungal.  Mean-field games with differing beliefs for algorithmic trading.  _Mathematical Finance_ , 30(3):995–1034, 2020\.
  * Cont (2011) R. Cont.  Statistical modeling of high-frequency financial data.  _IEEE Signal Processing_ , 28(5):16–25, 2011\.
  * Drapeau et al. (2019) S. Drapeau, P. Luo, A. Schied, and D. Xiong.  An FBSDE approach to market impact games with stochastic parameters.  Preprint, available online at https://arxiv.org/abs/2001.00622, 2019\.
  * Ekeland and Témam (1999) I. Ekeland and R. Témam.  _Convex Analysis and Variational Problems_.  Society for Industrial and Applied Mathematics, 1999.
  * Evangelista and Thamsten (2020) D. Evangelista and Y. Thamsten.  On finite population games of optimal trading, 2020.  Preprint, available online at https://arxiv.org/abs/2004.00790.
  * Freiling (2002) G. Freiling.  A survey of nonsymmetric Riccati equations.  _Linear Algebra and its Applications_ , 351-352:243–270, 2002.
  * Fu et al. (2020) G. Fu, P. Graewe, U. Horst, and A. Popier.  A mean field game of optimal portfolio liquidation.  _To appear in Mathematics of Operations Research_, 2020.
  * Gârleanu and Pedersen (2016) N. Gârleanu and L. H. Pedersen.  Dynamic portfolio choice with frictions.  _Journal of Economic Theory_ , 165:487 – 516, 2016.
  * Kirilenko et al. (2017) A. Kirilenko, A. S. Kyle, M. Samadi, and T. Tuzun.  The flash crash: The impact of high frequency trading on an electronic market.  _Journal of Finance_ , 72(3):967–998, June 2017\.
  * Lehalle and Neuman (2019) C. A. Lehalle and E. Neuman.  Incorporating signals into optimal trading.  _Finance and Stochastics_ , 23(2):275–311, 2019\.
  * Micheli et al. (2021) A. Micheli, J. Muhle-Karbe, and E. Neuman.  Closed-loop Nash competition for liquidity.  Preprint, available online at https://arxiv.org/abs/2112.02961, 2021\.
  * Neuman and Schied (2022) E. Neuman and A. Schied.  Protecting pegged currency markets from speculative investors.  _Mathematical Finance_ , 32(1):405–420, 2022\.
  * Neuman and Voß (2021) E. Neuman and M. Voß.  Trading with the crowd.  Preprint, available online at https://arxiv.org/abs/2106.09267, 2021\.
  * Neuman and Voß (2022) E. Neuman and M. Voß.  Optimal signal-adaptive trading with temporary and transient price impact.  _SIAM Journal on Financial Mathematics_ , 13(2):551–575, 2022.
  * Porter and Stirling (1990) D. Porter and D. S. G. Stirling.  _Integral equations: A Practical Treatment, from Spectral Theory to Applications_.  Cambridge University Press, 1990.
  * Roşu (2019) I. Roşu.  Fast and slow informed trading.  _Journal of Financial Markets_ , 43:1–30, 2019.
  * Schöneborn and Schied (2009) T. Schöneborn and A. Schied.  Liquidation in the face of adversity: stealth vs. sunshine trading, 2009\.  EFA 2008 Athens Meetings Paper, Available at SSRN: https://ssrn.com/abstract=1007014.
  * U.S. Securities and Exchange Commission (2014) U.S. Securities and Exchange Commission.  Equity market structure literature review part II: high frequency trading.  Available online at https://www.sec.gov/marketstructure/research/hft_lit_review_march_2014.pdf, 2014\.
  * Voß (2022) M. Voß.  A two-player portfolio tracking game.  _Mathematics and Financial Economics_ , 2022.
  * Wonham (1968) W. M. Wonham.  On a matrix Riccati equation of stochastic control.  _SIAM Journal on Control_ , 6(4):681–697, 1968\.


We present a Character-Word Long Short-Term Memory Language Model which both reduces the perplexity with respect to a baseline word-level language model and reduces the number of parameters of the model. Character information can reveal structural (dis)similarities between words and can even be used when a word is out-of-vocabulary, thus improving the modeling of infrequent and unknown words. By concatenating word and character embeddings, we achieve up to 2.77% relative improvement on English compared to a baseline model with a similar amount of parameters and 4.57% on Dutch. Moreover, we also outperform baseline word-level models with a larger number of parameters.

§ INTRODUCTION

Language models (LMs) play a crucial role in many speech and language processing tasks, among others speech recognition, machine translation and optical character recognition. The current state of the art are recurrent neural network (RNN) based LMs [19], and more specifically long short-term memory models (LSTM) [12] LMs [25] and their variants (e.g. gated recurrent units (GRU) [6]). LSTMs and GRUs are usually very similar in performance, with GRU models often even outperforming LSTM models despite the fact that they have less parameters to train. However, Jozefowicz et al. Joz:15 recently showed that for the task of language modeling LSTMs work better than GRUs, therefore we focus on LSTM-based LMs.

In this work, we address some of the drawbacks of NN based LMs (and many other types of LMs). A first drawback is the fact that the parameters for infrequent words are typically less accurate because the network requires a lot of training examples to optimize the parameters. The second and most important drawback addressed is the fact that the model does not make use of the internal structure of the words, given that they are encoded as one-hot vectors.
For example, `felicity' (great happiness) is a relatively infrequent word (its frequency is much lower compared to the frequency of `happiness' according to Google Ngram Viewer [18]) and will probably be an out-of-vocabulary (OOV) word in many applications, but since there are many nouns also ending on `ity' (ability, complexity, creativity …), knowledge of the surface form of the word will help in determining that `felicity' is a noun. Hence, subword information can play an important role in improving the representations for infrequent words and even OOV words.

In our character-word (CW) LSTM LM, we concatenate character and word embeddings and feed the resulting character-word embedding to the LSTM. Hence, we provide the LSTM with information about the structure of the word. By concatenating the embeddings, the individual characters (as opposed to e.g. a bag-of-characters approach) are preserved and the order of the characters is implicitly modeled. Moreover, since we keep the total embedding size constant, the `word' embedding shrinks in size and is partly replaced by character embeddings (with a much smaller vocabulary and hence a much smaller embedding matrix), which decreases the number of parameters of the model.

We investigate the influence of the number of characters added, the size of the character embeddings, weight sharing for the characters and the size of the (hidden layer of the) model. Given that common or similar character sequences do not always occur at the beginning of words (e.g. `overfitting' – `underfitting'), we also examine adding the characters in forward order, backward order or both orders.

We test our CW LMs on both English and Dutch. Since Dutch has a richer morphology than English due to among others its productive compounding (see e.g. [22]), we expect that it should benefit more from a LM augmented with formal/morphological information.

The contributions of this paper are the following:

* We present a method to combine word and subword information in an LSTM LM: concatenating word and character embeddings. As far as we know, this method has not been investigated before.

* By decreasing the size of the word-level embedding (and hence the huge word embedding matrix), we effectively reduce the number of parameters in the model (see section <ref>).

* We find that the CW model both outperforms word-level LMs with the same number of hidden units (and hence a larger number of parameters) and word-level LMs with the same number of parameters. These findings are confirmed for English and Dutch, for a small model size and a large model size. The size of the character embeddings should be proportional to the total size of the embedding (the concatenation of characters should not exceed the size of the word-level embedding), and using characters in the backward order improves the perplexity even more (see sections <ref>, <ref> and <ref>).

* The LM improves the modeling of OOV words by exploiting their surface form (see section <ref>).

The remainder of this paper is structured as follows: first, we discuss related work (section <ref>); then the CW LSTM LM is described (section <ref>) and tested (section <ref>). Finally, we give an overview of the results and an outlook to future work (section <ref>).

§ RELATED WORK

Other work that investigates the use of character information in RNN LMs either completely replaces the word-level representation by a character-level one or combines word and character information. Much research has also been done on modeling other types of subword information (e.g. morphemes, syllables), but in this discussion, we limit ourselves to characters as subword information.

Research on replacing the word embeddings entirely has been done for neural machine translation (NMT) by Ling et al. Ling and Costa-jussà and Fonollosa Costa, who replace word-level embeddings with character-level embeddings. Chung et al. Chung use a subword-level encoder and a character-level decoder for NMT. In dependency parsing, Ballesteros et al. Ball achieve improvements by generating character-level embeddings with a bidirectional LSTM. Xie et al. Xie work on natural language correction and also use an encoder-decoder, but operate for both the encoder and the decoder on the character level.

Character-level word representations can also be generated with convolutional neural networks (CNNs), as Zhang et al. Zhang and Kim et al. Kim have proven for text classification and language modeling respectively. Kim et al. Kim achieve state-of-the-art results in language modeling for several languages by combining a character-level CNN with highway [24] and LSTM layers. However, the major improvement is achieved by adding the highway layers: for a small model size, the purely character-level model without highway layers does not perform better than the word-level model (perplexity of 100.3 compared to 97.6), even though the character model has two hidden layers of 300 LSTM units each and is compared to a word model of two hidden layers of only 200 units (in order to keep the number of parameters similar). For a model of larger size, the character-level LM improves the word baseline (84.6 compared to 85.4), but the largest improvement is achieved by adding two highway layers (78.9). Finally, Jozefowicz et al. Joz:16 also describe character embeddings generated by a CNN, but they test on the 1B Word Benchmark, a data set of an entirely different scale than the one we use.

Other authors combine the word and character information (as we do in this paper) rather than doing away completely with word inputs. Chen et al. Chen and Kang et al. Kang work on models combining words and Chinese characters to learn embeddings. Note however that Chinese characters more closely match subwords or words than phonemes. Bojanowski et al. Boj operate on the character level but use knowledge about the context words in two variants of character-level RNN LMs. Dos Santos and Zadrozny Santos join word and character representations in a deep neural network for part-of-speech tagging. Finally, Miyamoto and Cho Miy describe a LM that is related to our model, although their character-level embedding is generated by a bidirectional LSTM and we do not use a gate to determine how much of the word and how much of the character embedding is used. However, they only compare to a simple baseline model of 2 LSTM layers of each 200 hidden units without dropout, resulting in a higher baseline perplexity (as mentioned in section <ref>, our CW model also achieves larger improvements than reported in this paper with respect to that baseline).

We can conclude that in various NLP tasks, characters have recently been introduced in several different manners. However, the models investigated in related work are either not tested on a competitive baseline [20] or do not perform better than our models [16]. In this paper, we introduce a new and straightforward manner to incorporate characters in a LM that (as far as we know) has not been investigated before.

§ CHARACTER-WORD LSTM LMS

A word-level LSTM LM works as follows: a word encoded as a one-hot column vector $\textbf{w}_t$ (at time step $t$) is fed to the input layer and multiplied with the embedding matrix $\textbf{W}_{w}$, resulting in a word embedding $\textbf{e}_t$:

\begin{equation}
\label{eq:baseline}
\textbf{e}_t = \textbf{W}_{w} \times \textbf{w}_t
\end{equation}

The word embedding of the current word $\textbf{e}_t$ will be the input for a series of non-linear operations in the LSTM layer (we refer to [27] for more details about the equations of the LSTM cell). In the output layer, probabilities for the next word are calculated based on a softmax function.

In our character-word LSTM LM, the only difference with the baseline LM is the computation of the `word' embedding, which is now the result of word and character input rather than word input only. We concatenate the word embedding with embeddings of the characters occurring in that word:

\begin{equation}
\label{eq:one-concat}
\begin{split}
\textbf{e}_t^\top = [(\textbf{W}_{w} \times \textbf{w}_t)^\top (\textbf{W}_{c}^1 \times \textbf{c}^1_{t})^\top \\
(\textbf{W}_{c}^2 \times \textbf{c}^2_t)^\top~\ldots~(\textbf{W}_{c}^n \times \textbf{c}^n_t)^\top]
\end{split}
\end{equation}

where $\textbf{c}^1_t$ is the one-hot encoding of the first character added, $\textbf{W}_c^1$ its embedding matrix and $n$ the total number of characters added to the model.
The word $\textbf{w}_t$ and its characters $\textbf{c}^1_t, \textbf{c}^2_t \ldots \textbf{c}^n_t$  are each projected onto their own embeddings, and the concatenation of the embeddings is the input for the LSTM layer. By concatenating the embeddings, we implicitly preserve the order of the characters: the embedding for e.g. the first character of a word will always correspond to the same portion of the input vector for the LSTM  (see figure <ref>). We also experimented with adding word and character embeddings (a method which does not preserve the order of the characters), but that did not improve the perplexity of the LM.

The number of characters added ($n$) is fixed. If a word is longer than $n$ characters, only the first (or last, depending on the order in which they are added) $n$ characters are added. If the word is shorter than $n$, it is padded with a special symbol. Because we can still model the surface form of OOV words with the help of their characters, this model reduces the number of errors made immediately after OOV words (see section <ref>).

§.§ Order of the characters

The characters can be added in the order in which they appear in the word (in the experiments this is called `forward order'), in the reversed order (`backward order') or both (`both orders'). In English and Dutch (and many other languages), suffixes can bear meaningful relations (such as plurality and verb conjugation) and compounds typically have word-final heads. Hence, putting more emphasis on the end of a word might help to better model those properties.

Concatenating word and character embeddings in an LSTM LM.

§.§ Weight sharing

Note that in equation <ref> each position in the word is associated with different weights: the weights for the first character $\textbf{c}^1_t$, $\textbf{W}_{c}^1$, are different from the weights for the character in the second position, $\textbf{W}_{c}^2$. Given that the input `vocabulary' for characters is always the same, one could argue that the same set of weights $\textbf{W}_c$ could be used for all positions in the word:

\begin{equation}
\label{eq:concat-weights}
\begin{split}
\textbf{e}_t^\top = [(\textbf{W}_{w} \times \textbf{w}_t)^\top (\textbf{W}_{c} \times \textbf{c}^1_{t})^\top \\
(\textbf{W}_{c} \times \textbf{c}^2_t)^\top~\ldots~(\textbf{W}_{c} \times \textbf{c}^n_t)^\top]
\end{split}
\end{equation}

However, one could also argue in favor of the opposite case (no shared weights between the characters): for example, an `s' at the end of a word often has a specific meaning, such as indicating a third person singular verb form of the present tense (in English), which it does not have at other positions in the word. Both models with and without weight sharing are tested (see section <ref>).

§.§ Number of parameters

Given that a portion of the total embedding is used for modeling the characters, the actual `word' embedding is smaller which reduces the number of parameters significantly. In a normal word-level LSTM LM, the number of parameters in the embedding matrix is

\begin{equation}
\label{size-word}
V \times E
\end{equation}

with $V$ the vocabulary size and $E = E_w$ the total embedding size/word embedding size. In our CW model however, the number of parameters is

\begin{equation}
\label{size-CW}
V \times (E - n \times E_c) + n \times (C \times E_c)
\end{equation}

with $n$ the number of characters, $E_c$ the size of the character embedding and $C$ the size of the character vocabulary. Since $V$ is by far the dominant factor, reducing the size of the purely word-level embedding vastly reduces the total number of parameters to train. If we share the character weights, that number becomes even smaller:

\begin{equation}
\label{size-CW-shared}
V \times (E - n \times E_c) + C \times E_c
\end{equation}

§ EXPERIMENTS

§.§ Setup

All LMs were trained and tested with TensorFlow [1]. We test the performance of the CW architectures for a small model and a large model, with hyperparameters based on Zaremba et al. Zaremba and Kim et al. Kim). The small LSTM consists of 2 layers of 200 hidden units and the large LSTM has 2 layers of 650 hidden units. The total size of the embedding layer always equals the size of the hidden layer. During the first 4/6 (small/large model) epochs, the learning rate is 1, after which we apply an exponential decay:

\begin{equation}
\eta_{i} = \alpha~\eta_{i-1}
\label{lr}
\end{equation}

where $\eta_i$ is the learning rate at epoch $i$ and $\alpha$ the learning rate decay, which is set to 0.5 for the small LSTM and to 0.8 for the large LSTM. The smaller $\alpha$, the faster the learning rate decreases. The total number of epochs is fixed to 13/39 (small/large model). During training, 25% of the neurons are dropped [23] for the small model and 50% for the large model. The weights are randomly initialized to small values (between -0.1 and 0.1 for the small model and between -0.05 and 0.05 for the large model) based on a uniform distribution. We train on mini-batches of 20 with backpropagation through time, where the network is unrolled for 20 steps for the small LSTM and 35 for the large LSTM. The norm of the gradients is clipped at 5 for both models.

For English, we test on the publicly available Penn Treebank (PTB) data set, which contains 900k word tokens for training, 70k word tokens as validation set and 80k words as test set. This data set is small but widely used in related work (among others Zaremba et al. Zaremba and Kim et al. Kim), enabling the comparison between different models. We adopt the same pre-processing as used by previous work [19] to facilitate comparison, which implies that the dataset contains only lowercase characters (the size of the character vocabulary is 48). Unknown words are mapped to $\langle$unk$\rangle$, but since we do not have the original text, we cannot use the characters of the unknown words for PTB.

The Dutch data set consists of components g, h, n and o of the Corpus of Spoken Dutch (CGN) [21], containing recordings of meetings, debates, courses, lectures and read text. Approximately 80% was chosen as training set (1.4M word tokens), 10% as validation set (180k word tokens) and 10% as test set (190k word tokens). The size of the Dutch data set is chosen to be similar to the size of the English data set. We also use the same vocabulary size as used for Penn Treebank (10k), since we want to compare the performance on different languages and exclude any effect of the vocabulary size. However, we do not convert all uppercase characters to lowercase (although the data is normalized such that sentence-initial words with a capital are converted to lowercase if necessary) because the fact that a character is uppercase is meaningful in itself. The character vocabulary size is 88 (Dutch also includes more accented characters due to French loan words, e.g. `café'). Hence, we do not only compare two different languages but also models with only lowercase characters and models with both upper- and lowercase characters. Moreover, since we have the original text at our disposal (as opposed to PTB), we can use the characters of the unknown words and still have a character-level representation.

§.§ Baseline models

In our experiments, we compare the CW model with two word-level baselines: one with the same number of hidden units in the LSTM layers (thus containing more parameters) and one with approximately the same number of parameters as the CW model (like Kim et al. Kim do), because we are interested in both reducing the number of parameters and improving the performance. For the latter baseline, this implies that we change the number of hidden units from 200 to 175 for the small model and from 650 to 475 for the large, keeping the other hyperparameters the same.

The number of parameters for those models is larger than for all CW models except when only 1 or 2 characters are added. The size difference between the CW models and the smaller word-level models becomes larger if more characters are added, if the size of the characters embeddings is larger and if the character weights are shared. The size of the embedding matrix for a word-level LSTM of size 475 is 10,000 $\times$ 475 = 475,000 ($V$ is 10k in all our experiments), whereas for a CW model with 10 character embeddings of size 25 it is of size 10,000 $\times$ (650 - 10 $\times$ 25) + 10 $\times$ (48 $\times$ 25) = 412,000 (the size of the character vocabulary for PTB is 48), following equation <ref>. If the character weights are shared, the size of the embedding matrix is only 401,200 (equation <ref>).

The baseline perplexities for the smaller word-level models are shown in table <ref>. In the remainder of this paper, `w$x$' = means word embeddings of size $x$ for a word-level model and `c$x$' means character embeddings of size $x$ for CW models.

3l   2cPerplexity

1lCorpus   2cSize   1cValidation   Test

4*PTB   2*small   w200   100.7   96.86

w175   102.62   98.82

2*large   w650   87.38   83.6

w465   88.39   84.38

4*CGN   2*small   w200   69.13   76

w175   69.6   76.78

2*large   w650   63.36   70.69

w475   63.88   70.88

Perplexities for the baseline models. Baselines w200 and w650 have the same number of hidden units as the CW models and baselines w175 and w475 approximately have the same number of parameters as the CW models.

§.§ English

In figure <ref>, the results for a small model trained on Penn Treebank are shown. Almost all CW models outperform the word-based baseline with the same number of parameters (2 LSTM layers of 175 units). Only the CW models in which the concatenated character embeddings take up the majority of the total embedding (more than 7 characters of embedding size 15) perform worse. With respect to the word-level LM with more parameters, only small improvements are obtained. The smaller the character embeddings, the better the performance of the CW model. For example, for a total embedding size of only 200, adding 8 character embeddings of size 15 results in an embedding consisting of 120 units `character embedding' and only 80 units `word embedding', which is not sufficient. The two best performing models add 3 and 7 character embeddings of size 5, giving a perplexity of 100.12 and 100.25 respectively, achieving a relative improvement of 2.44%/2.31% w.r.t. the w175 baseline and 0.58%/0.45% w.r.t. the w200 baseline. For those models, the `word embedding' consists of 185 and 165 units respectively.

Validation perplexity results on PTB, small model. Different sizes for the character embeddings are tested (`c5', `c10', `c15').

We test the performance of the CW architecture on a large model too. In figure <ref>, the results for different embedding sizes are shown. Just like we saw for the small model, the size of the character embeddings should not be too large: for embeddings of size 50 (`c50'), the performance drops when a larger number of characters is added.
The best result is obtained by adding 8 characters with embeddings of size 25 (`c25'): a perplexity of 85.97 (2.74%/1.61% relative improvement with respect to the w475/w650 baseline). For embeddings of size 10, adding more than 10 characters gives additional improvements (see figure <ref>).

Validation perplexity results on PTB, large model. Different sizes for the character embeddings are tested (`c10', `c25', `c50').

We also verify whether the order in which the characters are added is important (figure <ref>). The best result is achieved by adding the first 3 and the last 3 characters to the model (`both orders'), giving a perplexity of 85.69, 3.05%/1.87% relative improvement with respect to the w475/w650 baseline. However, adding more characters in both orders causes a decrease in performance. When only adding the characters in the forward order or the backward order, adding the characters in backward order seems to perform slightly better overall (best result: adding 9 characters in the backward order gives a perplexity of 85.7 or 3.04%/1.92% improvement with respect to the w475/w650 baseline).

Validation perplexity results on PTB, large model. Several options for the order in which the characters are added are investigated.

We can conclude that the size of the character embeddings should be proportional to the total embedding size: the word-level embedding should be larger than the concatenation of the character-level embeddings. Adding characters in the backward order is slightly better than adding them in the forward order, and the largest improvement is made for the large LSTM LM. The test perplexities for some of the best performing models (table <ref>) confirm these findings.

Small model   Perplexity

Baseline w175/w200    98.82/96.86

[16]   100.3

[20]   109.05

c5 with $n = 3$   96.21

c5 with $n = 7$   96.35

Large model   Perplexity

Baseline w475/w650   84.38/83.6

[16]   84.6

c25 with $n = 8$   82.69

c10 with $n = 9(b)$   82.68

c10 with $n = 3+3(b)$   82.04

Test perplexity results for  the best models on PTB. Baseline perplexities are for sizes w175/w200 for a small model and w475/w650 for a large model. $n$ = number of characters added, $(b)$ means backward order. Comparison with other character-level LMs [16] (we only compare to models without highway layers) and character-word models [20] (they do not use dropout and only report results for a small model).

If we compare the test perplexities with two related models that incorporate characters, we see that our models perform better. Kim et al. Kim generate character-level embeddings with a convolutional neural network and also report results for both a small and a large model. Their small character-level model has more hidden units than ours (300 compared to 200), but it does not improve with respect to the word-level baseline (since we do not use highway layers, we only compare with the results for models without highway layers). Their large model slightly improves their own baseline perplexity (85.4) by 0.94%. Compare with our results: 2.64% perplexity reduction for the best small LSTM (c5 with $n = 3$) and 2.77% for the best large LSTM (c10 with $n = 3+3(b)$). Miyamoto and Cho Miy only report results for a small model that is trained without dropout, resulting in a baseline perplexity of 115.65. If we train our small model without dropout we get a comparable baseline perplexity (116.33) and a character-word perplexity of 110.54 (compare to 109.05 reported by Miyamoto and Cho Miy). It remains to be seen whether their model performs equally well compared to better baselines. Moreover, their hybrid character-word model is more complex than ours because it uses a bidirectional LSTM to generate the character-level embedding (instead of a lookup table) and a gate to determine the mixing weights between the character- and word-level embeddings.

§.§ Dutch

As we explained in the introduction, we expect that using information about the internal structure of the word will help more for languages with a richer morphology. Although Dutch is still an analytic language  (most grammatical relations are marked with separate words or word order rather than morphemes), it has a richer morphology than English because compounding is a productive and widely used process and because it has more lexical variation due to inflection (e.g. verb conjugation, adjective inflection).
The results for the LSTM LMs trained on Dutch seem to confirm this hypothesis (see figure <ref>).

Validation perplexity results on CGN. Several options for the size and order of the character embeddings are investigated.

The CW model outperforms the baseline word-level LM both for the small model and the large model. The best result for the small model is obtained by adding 2 or 3 characters, giving a perplexity of 67.59 or 67.65 which equals a relative improvement of 2.89%/2.23% (w175/w200) and 2.80%/2.14% (w175/w200) respectively.

For the large model, we test several embedding sizes and orders for the characters. The best model is the one to which 6 characters in backward order are added, with a perplexity of 60.88 or 4.70%/3.91% (w475/w650) relative improvement. Just like for PTB, an embedding size of 25 proves to be the best compromise: if the characters are added in the normal order, 4 characters with embeddings of size 25 is the best model (perplexity 61.47 or 3.77%/2.98% (w475/w650) relative improvement).

These results are confirmed for the test set (table <ref>). The best small model has a perplexity of 75.04 which is 2.27% compared to the baseline and the best large model has a perplexity of 67.64, a relative improvement of 4.57%. The larger improvement for Dutch might be due to the fact that it has a richer morphology and/or the fact that we can use the surface form of the OOV words for the Dutch data set (see sections <ref> and <ref>).

Small model   Perplexity

Baseline w175/w200   76.78/76

c10 with $n = 2$   75.23

c10 with $n = 3$   75.04

Large model   Perplexity

Baseline w475/w650   70.88/70.69

c25 with $n = 4$   68.79

c25 with $n = 6(b)$   67.64

Test perplexity results for the best models on CGN. Baseline perplexities are for sizes w175/w200 for a small model and w475/w650 for a large model. $n$ = number of characters added, $(b)$ means backward order.

§.§ Random CW models

In order to investigate whether the improvements of the CW models are not caused by the fact that the characters add some sort of noise to the input, we experiment with adding real noise – random `character' information – rather than the real characters. Both the number of characters (the length of the random `word') and the `characters' themselves are generated based on a uniform distribution. In table <ref>, the relative change in perplexity, averaged over models to which 1 to 10 characters are added, with respect to the baseline word-level LM and the CW model with real characters is shown.

2l   2cRelative change in

2l   2cvalid perplexity w.r.t.

2l   1cBaseline   Char-Word

1l2*PTB   small c5   0.34 (0.30)   0.54 (0.46)

1l   large c15   0.00 (0.29)   0.53 (0.49)

1l2*CGN   small c10   - 0.18 (0.53)   1.79 (0.47)

1l   large c10   - 0.15 (0.26)   1.52 (1.24)

Relative change in validation perplexity for models to which random information is added, w.r.t. word-level and CW models. The improvements are averaged over the results for adding 1 to 10 characters/random information, the numbers between brackets are standard deviations. Negative numbers indicate a decrease in perplexity.

For English, adding random information had a negative impact on the performance with respect to both the baseline and the CW model. For Dutch on the other hand, adding some random noise to the word-level model gave small improvements. However, the random models perform much worse than the CW models. We can conclude that the characters provide meaningful information to the LM.

§.§ Sharing weights

We repeat certain experiments with the CW models, but with embedding matrices that are shared across all character positions (see section <ref>). Note that sharing the weights does not imply that the position information is lost, because for example the first portion of the character-level embedding always corresponds to the character on the first position. Sharing the weights ensures that a character is always mapped onto the same embedding, regardless of the position of that character in the word, e.g. both occurrences of `i' in `felicity' are represented by the same embedding. This effectively reduces the number of parameters.

We compare the performance of the CW models with weight sharing with the baseline word-level LM and the CW model without weight sharing. In table <ref>, the relative change with respect to those LMs is listed.

CW models with weight sharing generally improve with respect to a word-level baseline, except for the small English LM. For Dutch, the improvements are more pronounced. The difference with the CW model without weight sharing is small (right column), although not sharing the weights works slightly better, which suggests that characters can convey different meanings depending on the position in which they occur. Again, the results are more clear-cut for Dutch than for English.

2l   2cRelative change in

2l   2cvalid perplexity w.r.t.

2l   1cBaseline   Char-Word

1l2*PTB   small c10   0.53 (0.88)   0.19 (0.67)

1l   large c10   - 0.54 (0.37)   - 0.02 (0.22)

1l2*CGN   small c10   - 1.70 (0.34)   0.24 (0.30)

1l   large c10   - 2.10 (0.32)   0.15 (0.50)

Relative change in validation perplexity for CW models with weight sharing for the characters, w.r.t. baseline and CW models without weight sharing. The improvements are averaged over the results for adding 1 until 10 characters, the numbers between brackets are standard deviations. Negative numbers indicate a decrease in perplexity.

§.§ Dealing with out-of-vocabulary words

As we mentioned in the introduction, we expect that by providing information about the surface form of OOV words (namely, their characters), the number of errors induced by those words should decrease. We conduct the following experiment to check whether this is indeed the case: for the CGN test set, we keep track of the probabilities of each word during testing. If an OOV word is encountered, we check the probability of the target word given by a word-level LM and a CW LM. The word-level model is a large model of size 475 and the CW model is a large model in which 6 characters embeddings of size 25 in the backward order are used (the best performing CW model in our experiments).

We observe that in 17,483 of the cases, the CW model assigns a higher probability to the target word following an OOV word, whereas the opposite happens only in 10,724 cases. This is an indication that using the character information indeed helps in better modeling the OOV words.

§ CONCLUSION AND FUTURE WORK

We investigated a character-word LSTM language model, which combines character and word information by concatenating the respective embeddings. This both reduces the size of the LSTM and improves the perplexity with respect to a baseline word-level LM. The model was tested on English and Dutch, for different model sizes, several embedding sizes for the characters, different orders in which the characters are added and for weight sharing of the characters. We can conclude that for almost all setups, the CW LM outperforms the word-level model, whereas it has fewer parameters than the word-level model with the same number of LSTM units. If we compare with a word-level LM with approximately the same number of parameters, the improvement is larger.

One might argue that using a CNN or an RNN to generate character-level embeddings is a more general approach to incorporate characters in a LM, but this model is simple, easier to train and smaller. Moreover, related models using a CNN-based character embedding [16] do not perform better.

For both English and Dutch, we see that the size of the character embedding is important and should be proportional to the total embedding size: the total size of the concatenated character embeddings should not be larger than the word-level embedding.
Not using the characters in the order in which they appear in the word, but in the reversed order (and hence putting more emphasis on the end of the word), performs slightly better, although adding only a few characters both from the beginning and the end of the word achieves good performance too.

Using random inputs instead of the characters performed worse than using the characters themselves, thus refuting the hypothesis that the characters simply introduce noise. Sharing the weights/embedding matrices for the characters reduces the size of the model even more, but causes a small increase in perplexity with respect to a model without weight sharing. Finally, we observe that the CW models are better able to deal with OOV words than word-level LMs.

In future work, we will test other architectures to incorporate character information in a word-level LSTM LM, such as combining a character-level LSTM with a word-level LSTM.
Another representation that might be useful uses character co-occurrence vectors (by analogy with the acoustic co-occurrences used by Van hamme HAC,HAC2) rather than one-hot character vectors, because co-occurrences intrinsically give information about the order of the characters. Other models could be more inspired by human language processing: according to the theory of blocking, we humans have both a mental lexicon of frequent words and a morphological module that is used to process infrequent/ unknown words or to create new words (see e.g. [2]). This could correspond to a word-level LM for frequent words and a subword-level LM for infrequent words.

§ ACKNOWLEDGMENTS

This research is funded by the Flemish government agency IWT (project 130041, SCATE).

[1]
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen,
Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin,
Sanjay Ghemawat, Ian Goodfellow, Geoffrey Irving Andrew Harp, Michael Isard,
Rafal Jozefowicz, Yangqing Jia, Lukasz Kaiser, Manjunath Kudlur, Josh
Levenberg, Dan Mané, Mike Schuster, Rajat Monga, Sherry Moore, Derek Murray,
Chris Olah, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar,
Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol
Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang
Tensorflow: Large-scale machine learning on heterogeneous systems.
Software available from tensorflow.org.

[2]
Marc Aronoff and Frank Anshen.
Morphology and the lexicon: Lexicalization and productivity.
In Andrew Spencer and Arnold M. Zwicky, editors, The Handbook of
Morphology, pages 237–247. Blackwell Publishing.

[3]
Miguel Ballesteros, Chris Dyer, and Noah A. Smith.
Improved Transition-Based Parsing by Modeling characters
instead of words with LSTMs.
Conference on Empirical Methods in Natural Language Processing
(EMNLP), pages 349–359.

[4]
Piotr Bojanowski, Armand Joulin, and Tomás̆ Mikolov.
Alternative structures for character-level RNNs.

[5]
Xinxiong Chen, Lei Xu, Zhiyuan Liu, Maosong Sun, and Huanbo Luan.
Joint Learning of Character and Word Embeddings.
Conference on Artificial Intelligence (AAAI), pages 1236–1242.

[6]
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau,
Fethi Bougares, Holger Schwenk, and Yoshua Bengio.
Learning Phrase Representations using RNN Encoder–Decoder
for Statistical Machine Translation.
Conference on Empirical Methods in Natural Language Processing
(EMNLP), pages 1724–1734.

[7]
Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio.
A Character-Level Decoder without Explicit Segmentation for
Neural Machine Translation.

[8]
Marta R. Costa-jussà and José A.R. Fonollosa.
Character-based Neural Machine Translation.
Proceedings of the Association for Computational Linguistics
(ACL), 2:357–361.

[9]
Cícero N. dos Santos and Bianca Zadrozny.
Learning Character-level Representations for Part-of-Speech
Proceedings of The 31st International Conference on Machine
Learning (ICML), pages 1818–1826.

[10]
Hugo Van hamme.
HAC-models: a novel approach to continuous speech recognition.
Proceedings Interspeech, pages 2554–2557.

[11]
Hugo Van hamme.
An on-line NMF model for temporal pattern learning: Theory with
application to automatic speech recognition.
International Conference on Latent Variable Analysis and Signal
Separation (LVA/ICA), pages 306–313.

[12]
Sepp Hochreiter and Jürgen Schmidhuber.
Long short-term memory.
Neural Computation, 9(8):1735–1780.

[13]
Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever.
An Empirical Exploration of Recurrent Network
Proceedings of the 32nd International Conference on Machine
Learning (ICML), pages 2342–2350.

[14]
Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu.
Exploring the Limits of Language Modeling.

[15]
Moonyoung Kang, Tim Ng, and Long Nguyen.
Mandarin word-character hybrid-input Neural Network Language
Proceedings Interspeech, pages 1261–1264.

[16]
Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush.
Character-Aware Neural Language Models.
Proc. Conference on Artificial Intelligence (AAAI), pages

[17]
Wang Ling, Isabel Trancoso, Chris Dyer, and Alan Black.
Character-based Neural Machine Translation.

[18]
Jean-Baptiste Michel, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres,
Matthew K. Gray, William Brockman, The Google Books Team, Joseph P. Pickett,
Dale Hoiberg, Dan Clancy, Peter Norvig, Jon Orwant, Steven Pinker, Martin A.
Nowak, and Erez Lieberman Aiden.
Quantitative Analysis of Culture Using Millions of
Digitized Books.
Science, 331(6014):176–182.

[19]
Tomás̆ Mikolov, Martin Karafiát, Lukás̆ Burget, Jan
C̆ernocký, and Sanjeev Khudanpur.
Recurrent neural network based language model.
Proceedings Interspeech, pages 1045–1048.

[20]
Yasumasa Miyamoto and Kyunghyun Cho.
Gated Word-Character Recurrent Language Model.

[21]
Nelleke Oostdijk.
The Spoken Dutch Corpus. Overview and first Evaluation.
Proceedings Language Resources and Evaluation Conference
(LREC), pages 887––894.

[22]
Bert Réveil.
Optimizing the Recognition Lexicon for Automatic Speech
PhD thesis, University of Ghent, Belgium.

[23]
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan
Dropout: A Simple Way to Prevent Neural Networks from
Journal of Machine Learning Research, 15:1929–1958.

[24]
Rupesh K. Srivastava, Klaus Greff, and Jürgen Schmidhuber.
Training Very Deep Networks.
Neural Information Processing Systems Conference (NIPS), pages

[25]
Martin Sundermeyer, Ralf Schlüter, and Hermann Ney.
LSTM Neural Networks for Language Modeling.
Proceedings Interspeech, pages 1724–1734.

[26]
Ziang Xie, Anand Avati, Naveen Arivzhagan, Dan Jurafsky, and Andrew Y. Ng.
Neural Language Correction with Character-Based Attention.

[27]
Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals.
Recurrent Neural Network Regularization.

[28]
Xiang Zhang, Junbo Zhao, and Yann LeCunn.
Character-level Convolutional Networks for Text
Neural Information Processing Systems Conference (NIPS), pages


Vol.0 (200x) No.0, 000–000 page1 11institutetext: Physics Department, Yuxi
Normal University, Yuxi 653100, China22institutetext: Shandong Provincial Key
Laboratory of Optical Astronomy and Solar-Terrestrial Environment, Institute
of Space Sciences, Shandong University,Weihai, 264209, China;
<EMAIL_ADDRESS>
33institutetext: Physics Department, Yunan Normal University, Kunming 650092,
China
Received ********; accepted ********

# The Broad Band Spectral Energy Distributions of SDSS Blazars

Huaizhen Li 11 Luoen Chen 11 Yunguo Jiang and Tingfeng Yi 2233

###### Abstract

We compiled the radio, optical, and X-ray data of blazars from the Sloan
Digital Sky Survey (SDSS) database, and presented the distribution of
luminosities and broad band spectral indices. The distribution of luminosities
shows that the averaged luminosity of flat-spectral radio quasars (FSRQs) is
larger than that of BL Lacs objects. On the other hand, the broad band
spectral energy distribution reveals that FSRQs and low energy peaked BL Lac
objects (LBLs) objects have similar spectral properties, but high energy
peaked BL Lac objects (HBLs) have a distinct spectral property. This may be
due to that different subclasses of blazars have different intrinsic
environments and are at different cooling levels. Even so, a unified scheme
also is revealed from the color-color diagram, which hints that there are
similar physical processes operating in all objects under a range of intrinsic
physical conditions or beaming parameter.

###### keywords:

Galaxies: active; BL Lacertae objects:general; Galaxies: fundamental
parameters; Quasars: general

##  1 Introduction

Blazars are a subset of active galactic nuclei (AGNs) with strong emission at
all wavelengths. They are the brightest and most variable high energy sources
among AGNs, and have continuous spectral energy distributions (SEDs). The
continuum emission in balzars is thought to be from a relativistic jet
oriented close to the observer and emanating from the vicinity of a black hole
Ster08,Ghis86. The SEDs of blazars are characterized by a universal two-bump
structure: one in the IR to UV band, another in the MeV-GeV band. The
synchrotron radiation in a relativistic beamed jet is responsible for the
lower-energy peak, while the high-energy $\gamma$-ray emission is produced by
the inverse Compton mechanism (Sambruna et al., 1996). Generally, blazars can
be divided into two subclasses, the BL Lacertae (BL Lacs) object and the flat
spectrum radio quasars (FSRQs). The main difference between these two classes
is their emission lines: BL Lacs objects are characterized by the lack of
strong emission lines (equivalent width $<$ 5${\AA}$, while FSRQs have strong
broad emission lines with the similar strength of normal quasars (Scarpa &
Falomo, 1997).

Based on the synchrotron emission peak energy frequency, BL Lacs objects can
be divided into ”High energy peaked BL Lacs” (HBLs) and ”Low energy peaked BL
Lacs” (LBLs) (Padovani & Giommi, 1995; Giommi et al., 1995). Mei et al. (2002)
and Ma et al. (2007) found that the two subclasses of BL Lacs objects can be
distinguished by the peak frequencies $\nu_{peak}$: HBLs have
$log\nu_{peak}>14.7$, while LBLs have $log\nu_{peak}<14.7$. In addition,
Padovani & Giommi (1995) found that LBLs and HBLs can also be divided by using
the ratio of X-ray flux at 1 keV (in units of $erg$ $cm^{-2}$ $s^{-1}$) to 5
GHz radio flux density (in units of Janskys). The criterion is
$f_{x}/f_{r}\sim 10^{-11}$, corresponding to the broad band spectral index
(from radio 5 GHz to X-ray 1 keV) $\alpha_{rx}\simeq 0.75$. HBLs have the
broad band spectral index of $\alpha_{rx}\leq 0.75$, and LBLs have the
spectral index of $\alpha_{rx}>0.75$ (Giommi et al., 1995; Ma et al., 2007;
Mei et al., 2002; Urry & Padovani, 1995). For two different subclasses of BL
Lacs objects, the SEDs have been investigated by a number of authors (e.g.,
Bao et al., 2008; Chen et al., 2006; Padovani & Giommi, 1995; Giommi et al.,
1990; Nieppola et al., 2006). Padovani & Giommi (1995) and Giommi et al.
(1990) found that two subclasses occupy different regions on the $\alpha_{ro}$
\- $\alpha_{ox}$ plane. Padovani & Giommi (1995) also found there exist the
correlations between the minimum soft X-ray flux and the radio flux, and also
the correlations between radio and optical fluxes for the subsample of HBLs,
but not for that of LBLs. Nieppola et al. (2006) found that there is a
negative correlation between the luminosity and the synchrotron peak frequency
$\nu_{peak}$ at radio and optical band, whereas the correlation turns slightly
positive in X-ray (Nieppola et al., 2006). Fan et al. (2012) and Lyu et al.
(2014) found that HBLs have different properties from LBLs. Yan et al. (2014)
found that the one-zone synchrotron self-Compton (SSC) model can successfully
fit the SEDs of HBLs, but fails to explain the SEDs of LBLs. In addition, Bao
et al. (2008) found that two subclasses of BL Lacs objects are unified.

BL Lacs objects and FSRQs are grouped together under the denomination of
blazars, which eliminates the somewhat ambiguous issue of strength of the
emission lines as a classification criterion. However, there are some
differences in the individual emission properties among different blazar
subclasses. The relationship among different kinds of balzars can promote our
perception of the fundamental properties of blazars. Therefore, it is
imperative to investigate the connection among FSRQs, LBLs, and HBLs.

The relationship between of BL Lacs objects and FSRQs has been discussed by a
number of authors (e.g., Comastri et al., 1997; Fossati et al., 1998;
Ghisellini et al., 1998, 2009; Li et al., 2010; Sambruna et al., 1996; Xie et
al., 2001, 2003, 2004, 2006, 2007, 2008; Zheng et al., 2007), who assembled
the SED of many radio, X-ray, and $\gamma$-ray selected blazars. Fossati et
al. (1998) studied the SEDs of a combined blazar sample, and found that the
SEDs properties of these subclasses present a remarkable continuity and a
systematic trend as a function of source luminosity, which suggests that the
parameter describing the blazar continua is likely to be the source
luminosity. Based on the first Fermi sample, Ghisellini et al. (2009) found
that FSRQs and BL Lacs objects occupy separate regions, and obey a spectral
sequence. However, Antón & Browne (2005) found that there are selection
effects for the ”blazars sequence” reported by Fossati et al. (1998) and
Ghisellini et al. (1998). Some literatures show that HBLs have different
properties from FSRQs, but LBLs are similar to FSRQs (e.g. Chen et al., 2013;
Fan et al., 2012; Li et al., 2010; Lyu et al., 2014). However, Li et al.
(2010) and Chen et al. (2013) also found that their whole sample suggest the
unified scheme of blazars. Comastri et al. (1997) discovered that there is a
significant anticorrelation between X-ray and $\gamma$-ray spectral indices,
and also between the broadband spectral indices $\alpha_{ro}$ and
$\alpha_{x\gamma}$ of BL Lacs objects and FSRQs. The correlation between the
broadband spectral indices obtained by Comastri et al. (1997) implied that
there is a different shape in overall energy distributions from radio to
$\gamma$-ray energies between BL Lacs objects and FSRQs. Sambruna et al.
(1996) and Xie et al. (2003) found that three kinds of blazars have different
SEDs, but follow a continuous spectral sequence.

In this paper, we will study the distributions of luminosities and the radio-
optical-X-ray SEDs of SDSS blazars, and connections among LBLs, HBLs, and
FSRQs. A detailed explanation of the sample is given in Sec. 2. The
distributions of luminosity are presented in Sec. 3. The broad band spectral
energy distribution is given in Sec. 4. In Sec 5, discussions and conclusions
are presented.

##  2 The Sample of SDSS Blazars

The Sloan Digital Sky Survey (SDSS) is one of the most ambitious and
influential surveys in the history of astronomy. Plotkin et al. (2008) have
drawn a large sample of 501 BL Lacs objects candidates from the combination of
SDSS Data Release 5 (SDSS DR5) optical spectroscopy, and the Faint Images of
the Radio Sky at Twenty-Centimeters (FIRST) radio survey. Plotkin et al.
(2010) have presented a sample of 723 optically selected BL Lac candidates
from the SDSS Data Release 7 (SDSS DR7) spectroscopic database. Based on the
large radio (the NRAO VLA Sky Survey, ATCA catalogue of compact PMN sources),
ROSAT All Sky Survey (RASS), the SDSS Data Release 4 (SDSS DR4) and 2dF survey
data, Turriziani et al. (2007) presented a Radio-Optical-X-ray catalog built
at ASDC (ROXA) including 816 objects, among which 510 are confirmed blazars.
In addition, Chen et al. (2009) have also presented a sample including 118
Non-thermal jet-dominated FSRQs from SDSS Data Release 3 (SDSS DR3), X-ray
quasar sample with FIRST and GB6 radio catalogues. Based on the sample of
Plotkin et al. (2008, 2010), Turriziani et al. (2007) and Chen et al. (2009),
we compiled a large sample of 606 blazars, including 292 FSRQs and 314 BL
Lacs. All the objects of the sample have matches in RASS and measured
redshifts. In our sample, the three-band luminosities ($L_{r}$, $L_{o}$, and
$L_{x}$) and the broad-band spectral indices $\alpha_{ro}$, $\alpha_{rx}$, and
$\alpha_{ox}$ were given by the literatrues (Chen et al., 2009; Plotkin et
al., 2008, 2010; Turriziani et al., 2007). The luminosities $L_{r}$, $L_{o}$,
and $L_{x}$ are the specific luminosities (per unit frequency) at 5GHz,
5000${\AA}$, and 1keV, respectively. $\alpha_{ro}$ is the two-point spectral
indices between 5 GHz and 5000${\AA}$, $\alpha_{rx}$ is the two-point spectral
indices between 5 GHz and 1 keV, and $\alpha_{ox}$ is the two-point spectral
indices between 5000${\AA}$ and 1 keV.

As discussed in section 1, BL Lacs objects can be divided into HBLs and LBLs,
based on the radio-X-ray spectral index $\alpha_{rx}$ between 5 GHz and 1 keV.
According to literatures (Giommi et al., 1995; Ma et al., 2007; Mei et al.,
2002; Plotkin et al., 2008; Urry & Padovani, 1995), most BL Lacs objects with
$\alpha_{rx}\leq 0.75$ are HBLs, and most BL Lacs objects with
$\alpha_{rx}>0.75$ are LBLs. Therefore, for investigating the relation among
different blazar subclasses, we adopt $\alpha_{rx}=0.75$ as a rough value to
divise HBLs and LBLs for the SDSS BL Lacs objects (Giommi et al., 1995; Ma et
al., 2007; Mei et al., 2002; Urry & Padovani, 1995). Based on this criterion,
there are 270 HBLs and 44 LBLs in our sample.

##  3 Distributions of luminosity of blazars

Fossati et al. (1998) studied the SEDs of a combined blazar sample, and found
that the source luminosity is the characteristic parameter describing the
blazar continua. On the basis of the first Fermi sample, Ghisellini et al.
(2009) have found BL Lacs objects are harder and less luminous than FSRQs.
Ghisellini et al. (1998) found that HBLs are sources with the lowest intrinsic
power and the weakest external radiation field, LBLs are intrinsically more
powerful than HBLs, and FSRQs represent the most powerful blazars.

Thus, we computed the distributions of radio (at 5GHz), optical (at
5000${\AA}$) and X-ray (at 1 keV) luminosities for three subclasses of
blazars. Figure 1-3 give the distribution of luminositices for three kinds of
blazars, and all the luminositices are K-corrected to the source rest frame
(Chen et al., 2009; Plotkin et al., 2008, 2010; Turriziani et al., 2007). The
distributions of radio, optical and X-ray luminosities are plotted in Figure
1, 2, and 3, respectively.

From Figure 1, one can find that FSRQs have larger radio luminosities than BL
Lacs objects, while the radio luminosities of LBLs are more powerful than that
of HBLs. This suggests that the radio luminosities of the three kinds of
blazars, from FSRQs to LBLs to HBLs, are decreasing, which is consistent with
the argument reported by Ghisellini et al. (1998). In Figure 2, we can note
that the optical luminosities of FSRQs are larger than that of BL Lacs
objects, while HBLs have similar optical luminosities to that of LBLs.
Correspondingly, the X-ray luminosities of HBLs are systematically lower than
FSRQs, but larger than LBLs (see Figure 3).

The distributions presented in Figure 1, 2, and 3 show that the luminosity is
an important parameter to distinguish FSRQs and BL Lacs objects. A tendency of
luminosities from FSRQs to BL Lacs objects is revealed from the distributions
of luminosities. The distributions of luminosities for three kinds of blazars
presented from Figure 1-3 are consistent with the results reported by Fossati
et al. (1998) and Ghisellini et al. (1998). On the other hand, one can note
that all the distributions are continuous in properties between HBLs and LBLs,
and as well as between FSRQs and BL Lacs objects, which is in good agreement
with the previous arguments about the continuum of blazars (Fossati et al.,
1998; Ghisellini et al., 1998; Xie et al., 2003; Comastri et al., 1997;
Sambruna et al., 1996).

Figure 1: Distributions of radio luminosity at 5 GHz for three kinds of
blazars of our sample.  Figure 2: Distributions of optical luminosity at
5000${\AA}$ for three kinds of blazars of our sample.  Figure 3: Distributions
of X-ray luminosity at 1 keV for three kinds of blazars of our sample.

##  4 Broad band spectral energy distribution of blazars

Searching the connection among different blazar subclasses is very
significant, because it can substantially promote our understanding about the
fundamental nature of blazars. The relationship among different blazars has
been discussed in literatures with a unified scheme and a spectral sequence
for blazars (Fossati et al., 1998; Ghisellini et al., 1998; Sambruna et al.,
1996; Xie et al., 2003). For investigating the relationship among different
subclasses of blazars, we will analyze the relationship among the HBLs, LBLs,
and FSRQs on the basis of the broad band spectral index $\alpha_{ro}$,
$\alpha_{rx}$, and $\alpha_{ox}$.

###  4.1 Diagram of $\alpha_{rx}$ versus $\alpha_{ro}$

Here, we investigated the relationship between the broad band spectral indices
$\alpha_{rx}$ and $\alpha_{ro}$ for the whole sample. The plot is shown in
Figure 4. Figure 4 shows that there is a good correlation between
$\alpha_{rx}$ and $\alpha_{ro}$ for the whole sample. A linear regression
analysis equation for all sample is written as

$\alpha_{rx}=(0.64\pm 0.03)\alpha_{ro}+(0.43\pm 0.01),$ (1)

with a correlation coefficient $r=0.70$, and a chance probability $p<10^{-4}$.
Moreover, we also studied the relationship between $\alpha_{rx}$ and
$\alpha_{ro}$ for the FSRQs and LBLs sample. We obtained

$\alpha_{rx}=(0.42\pm 0.02)\alpha_{ro}+(0.59\pm 0.01),$ (2)

with a correlation coefficient $r=0.70$, and a chance probability $p<10^{-4}$.
The correlation analysis suggests that there is a linear correlation between
$\alpha_{rx}$ and $\alpha_{ro}$ for the whole sample, and as well as for the
FSRQs+LBLs sample. This provides evidence for the unified scheme reported by
Fossati et al. (1998) and Ghisellini et al. (1998).

In addition, Figure 4 shows that the majority of FSRQs and LBLs mix together,
which suggests that they have similar spectral properties. However, Figure 4
also reveals that the distribution of HBLs in $\alpha_{rx}$ versus
$\alpha_{ro}$ diagram is different from that of FSRQs and LBLs. This indicates
that HBLs have different spectral properties from FSRQs and LBLs. The results
are consistent with the reported results of Xie et al. (2003), who found that
HBLs and LBLs locate in different regions in the
$\alpha_{ox}-\alpha_{x\gamma}$ plane, but LBLs and FSRQs occupy the same
region in $\alpha_{ox}-\alpha_{x\gamma}$ plane. In addition, our results also
agree with that reported by Fan et al. (2012) and Lyu et al. (2014).

Figure 4: The relationship between the broadband spectral indices
$\alpha_{ro}$ and $\alpha_{rx}$ for the sources in our sample.

###  4.2 Diagram of $\alpha_{rx}$ versus $\alpha_{ox}$

In Figure 5, $\alpha_{rx}$ versus $\alpha_{ox}$ is plotted for our SDSS blazar
sample. We can find that the distribution of three kinds of blazars revealed
from Figure 5 is similar to that of Figure 4. For the whole sample, Figure 5
shows a significant correlation between $\alpha_{rx}$ and $\alpha_{ox}$, and
the linear regression analysis yields

$\alpha_{rx}=(0.30\pm 0.01)\alpha_{ox}+(0.36\pm 0.02),$ (3)

with a correlation coefficient $r=0.62$ and a chance probability $p<10^{-4}$.
Moreover, Figure 5 shows that there is a weak correlation between
$\alpha_{rx}$ and $\alpha_{ox}$ for the FSRQs and LBLs sample, and the linear
regression analysis equation is

$\alpha_{rx}=(0.07\pm 0.02)\alpha_{ox}+(0.71\pm 0.03),$ (4)

with a correlation coefficient $r=0.17$ and a chance probability $p=0.0016$.
In Figure 5, one can note that the majority of FSRQs and LBLs also occupy the
same region, but HBLs occupy a separate distinct region, which is also
consistent with previous results.

Figure 5: The relationship between the broadband spectral indices
$\alpha_{ox}$ and $\alpha_{rx}$ for the sources in our sample.

###  4.3 Diagram of $\alpha_{ox}$ versus $\alpha_{ro}$

Based on the broad band spectral index $\alpha_{ox}$ versus $\alpha_{ro}$, we
investigated the relationships between $\alpha_{ox}$ and $\alpha_{ro}$. Figure
6 plots $\alpha_{ox}$ versus $\alpha_{ro}$. A linear regression analysis shows
that there is a weak or even no correlation between $\alpha_{ox}$ and
$\alpha_{ro}$ ($r=-0.11$ and $p=0.054$) for whole sample. This independent
relation is obviously inconsistent with the foregoing correlation revealed
from Figure 4 and Figure 5. However, the correlation is significant when
considering FSRQs and LBLs sample and the the linear regression analysis
equation is

$\alpha_{ox}=(0.87\pm 0.07)\alpha_{ro}+(1.75\pm 0.04),$ (5)

where the correlation coefficient is $r=-0.56$ and the chance probability is
$p<10^{-4}$. This suggests that HBLs are different from FSRQs, but LBLs are
similar to FSRQs. In addition, Figure 6 also shows that most of the FSRQs and
LBLs locate the same region in $\alpha_{ox}$ -$\alpha_{ro}$ plot, but HBLs
occupy a separate distinct region in $\alpha_{ox}$ versus $\alpha_{ro}$ plane.
This is consistent with the distribution shown in Figure 4 and Figure 5. This
supports the foregoing results: FSRQs and LBLs have similar spectral
properties, but HBLs have distinct spectral properties.

Figure 6: The relationship between the broadband spectral indices
$\alpha_{ro}$ and $\alpha_{ox}$ for the sources in our sample.

###  4.4 Summary

As noted above, from Figure 4 and 5, there is a strong correlation between
$\alpha_{rx}$ and $\alpha_{ro}$, and as well as between $\alpha_{rx}$ versus
$\alpha_{ox}$ for the whole sample, which provides some more evidence for the
conclusion reported by Fossati et al. (1998) and Ghisellini et al. (1998).
Namely, there is a unified scheme for blazars. On the other hand, from Figure
4, 5, and 6, one can note that there are also some different distributions
from the blazar sequence reported by Fossati et al. (1998) and Ghisellini et
al. (1998). In the color-color diagram, HBLs and FSRQs occupy seperated
regions, while the LBLs and FSRQs mix together, which is consistent with that
reported in some literatures (e.g. Chen et al., 2013; Fan et al., 2012; Li et
al., 2010; Lyu et al., 2014; Xie et al., 2003). This suggests that FSRQs and
LBLs have a similar property, but HBLs have a distinct property.

##  5 Discussion and Conclusions

Based on the Slew survey, the 1-Jy samples of BL Lacs and the 2-Jy sample of
FSRQs, Fossati et al. (1998) studied the systematics of the SEDs of blazars
using data from the radio to the $\gamma$-ray band, and found that three
different kinds of blazars follow an almost continuous spectral sequence: from
FSRQs through LBLs to HBLs. Ghisellini & Tavecchio (2008) revisited the so
called ”blazar sequence”, and proposed that the power of the jet and the SED
of its emission are linked to the two main parameters of the accretion
process. This similar trend was also obtained by other authors (e.g. Xie et
al., 2003; Sambruna et al., 1996; Ghisellini et al., 1998; Böttcher & Dermer,
2002; Maraschi et al., 2008) who found that similar physical processes operate
in three kinds of blazars. However, Antón & Browne (2005) found that there are
selection effects for the ”blazars sequence”. Moreover, some authors found
that HBLs do not follow the blazars sequence (e.g. Chen et al., 2013; Fan et
al., 2012; Giommi et al., 2005; Li et al., 2010; Padovani et al., 2003;
Padovani, 2007) .

In this paper, we computed the distributions of the radio (at 5 GHz), optical
(at 5000 ${\AA}$), and X-ray (at 1 keV) luminosities. The luminosities
distributions reveal that it is an important parameter to distinguish FSRQs
and BL Lacs objects, and the distributions are continuous for three kinds of
blazars. The luminosities of FSRQs are lager than that of BL Lacs objects,
which is in good agreement with the arguments reported by other authors (Abdo
et al., 2009; Fossati et al., 1998; Ghisellini et al., 1998; Sambruna et al.,
1996). The distributions of radio luminosity support the blazars sequence
reported by Fossati et al. (1998) and Ghisellini et al. (1998). However, the
distributions of optical and X-ray luminosities do not support the sequence.

The broadband energy distribution shows that three kinds of blazars have
different spectral properties. It also shows that most FSRQs and LBLs mix
together in the color-color diagram (see Figure 4, 5, and 6), which is
consistent with previous results (Fan et al., 2012; Li et al., 2010; Sambruna
et al., 1996; Xie et al., 2003; Zheng et al., 2007). This suggests that they
have similar spectral properties, which provides some more evidence for the
conclusion of unified scheme. However, the location of HBLs is separate with
that of FSRQs and LBLs in the color-color diagram, which reveals that HBLs
have different SEDs from FSRQs and LBLs. This suggests that the results from
SDSS sample do not support the so called ”blazar sequence”, which is
consistent with the results reported by the other authors (Antón & Browne,
2005; Fan et al., 2012; Chen et al., 2013; Li et al., 2010; Padovani et al.,
2003; Padovani, 2007; Zhang et al., 2013). The spectral sequence obtained by
Fossati et al. (1998) may be related to the selection effects, because the
sample used by Fossati et al. (1998) are classic, high flux limit surveys in
the radio and X-ray (Antón & Browne, 2005). Our sample used in the paper is a
large sample including 606 blazars, which would result in a unbiased view of
blazar spectral properties. Moreover, Figure 7 gives the relations between the
redshift and spectral indices. Figure 7 shows that the spectral indices is
independent of redshift, which suggests that the selection effects of our
sample are weak.

Ghisellini et al. (1998) suggested the level of cooling is different for
different subclasses of blazars. FSRQs suffer stronger cooling, and
synchrotron emission peaks at much lower frequency. However, the cooling is
less important for HBLs, and the energetic particles can produce synchrotron
and inverse Compton (IC) emission up to high frequency. The level of cooling
of FSRQs stronger than the one of HBLs may be due to the external radiation
field (Ghisellini et al., 1998). Georganopoulos et al. (2001) suggested that
the radiating jet plasma in weak sources is outside the broad line scattering
region (BLR), while it is within in the power source. This implies that the
location of emitting region between HBLs and FSRQs might be very different
(Costamante, 2009). The jet energy of FSRQs would dissipate within the BLR,
leading that the high energy electrons in the jet will suffer greater cooling
(Chen & Bai, 2011). Ghisellini et al. (2010) suggested that the $\gamma$-ray
emission from FSRQs is likely from the Compton scattering of an external
radiation source, while for HBLs SSC is able to provide a good fit to the
$\gamma$-ray emission. In addition, based on the physical properties of
relativistic jets, Yan et al. (2014) found that the one-zone SSC model can
successfully fit the SEDs of HBLs, but fails to explain the SEDs of LBLs.
Moreover, they also suggest that the ratios of the Compton to the synchrotron
peak energy fluxes of LBLs are greater than those of HBLs and IBLs, and then
LBLs are Compton dominated (Yan et al., 2014). This suggests that there is an
external radiation field for LBLs. Therefore, the levels of cooling of FSRQs
and LBLs is stronger than HBLs, which lead that the synchrotron emission peaks
of FSRQs and LBLs is lower than ones of HBLs. Abdo et al. (2010) found FSRQs
and LBLs are the low synchrotron peaked blazars, while HBLs are high
synchrotron peaked blazars. Fan et al. (2012) suggest that if the synchrotron
peak frequency moves to the lower frequency, then the IC peak frequency may
also move to the lower frequency. Thus, the X-ray of LBLs and FSRQs are from
the combination of synchrotron emission and the IC emission, while the X-ray
of HBLs are from the synchrotron emission of very high energy electrons (Abdo
et al., 2010; Fan et al., 2012). In addition, the different SEDs between HBLs,
FSRQs, and LBLs may be related to the different intrinsical environments
around the blazar’s nucleus. The intrinsical environments of FSRQs and HBLs
have a clear, physics difference: the environment of HBLs is ”cleaner” than
that of FSRQs (Costamante, 2009). The central regions of FSRQs are rich in gas
and dust, which would lead to a high accretion rate onto the central
supermassive black hole (Böttcher & Dermer, 2002). Moreover, the material
would efficiently reprocess and scatter the accretion disk radiation. This
would lead to the observed strong optical emission lines in the BLR and to a
high energy density of the external soft-photon field in the jet (Böttcher &
Dermer, 2002).

Although HBLs have different SEDs from FSRQs and LBLs, the significant
correlation revealed from Figure 4 and 5 suggests that there is a unified
scheme for whole sample, which is consistent with the previous conclusion
reported by other authors (Comastri et al., 1997; Fossati et al., 1998;
Ghisellini et al., 1998; Li et al., 2010; Sambruna et al., 1996; Xie et al.,
2001, 2003). This hints that there is a similar physical processes operating
in all objects. In the case of the blazar-type sources where the emission is
usually associated with a stream of relativistic jet, the overall spectrum is
determined by the energy spectrum of the electrons as well as by the variation
of the physics quantities along the jet (Begelman et al., 1984). HBLs, LBLs,
and FSRQs have a significant correlation in the color-color diagram (see
Figure 4 and 5), which implies that similar physical processes operate in all
objects under a range of intrinsic physics conditions or beaming parameter. On
the other hand, the difference among three subclasses of blazars, revealed
from color-color diagram (see Figure 4, 5, and 6), should be attributed to the
different level of cooling and the intrinsical different environments around
the blazars nucleus for different subclasses blazars, which lead to different
optical and X-ray spectra for different kinds of blazars.

Figure 7: Relations between redshift and spectral indices.

###### Acknowledgements.

We are grateful to the anonymous referee for useful comments. We are grateful
for the help from Liang Chen. This research has made use of the SDSS database.
This work is supported by the National Natural Science Foundation of China
(10878013), and the Natural Science Foundation of Yunnan Province (2011FZ081,
2012FD055, 2013FB063), and the Program for Innovative Research Team (in
Science and Technology) in University of Yunnan Province (IRTSTYN), and
Science Research Foundation of Yunnan Education Department of China
(2012Y316), and the Young Teachers Program of Yuxi Nurmal University. In
addition, the work of Yunguo Jiang is supported by the NNSFC with Number
11403015.

## References

  * Abdo et al. (2009) Abdo, A. A., Ackermann, M., Ajello, M. et al., 2009, ApJ, 700, 597
  * Abdo et al. (2010) Abdo, A. A., Ackermann, M., Ajello, M. et al., 2010, ApJ, 716, 30
  * Antón & Browne (2005) Antón, S. & Browne, I. W. A., 2005, MNRAS, 356, 225
  * Bao et al. (2008) Bao, Y. Y., Chen, L. E., Li, H. Z. et al., 2008, Ap&SS, 318, 169
  * Begelman et al. (1984) Begelman, Mitchell C., Blandford, Roger D., Rees, Martin J., 1984, Rev. Mod. Phys., 56, 255
  * Böttcher & Dermer (2002) Böttcher, M. & Dermer, C. D., 2002, ApJ, 564, 86
  * Chen & Bai (2011) Chen, L. & Bai, J. M., 2011, ApJ, 735, 108
  * Chen et al. (2013) Chen, L. E., Li, H. Z., Yi, T. F. et al., 2013, , 13, 5
  * Chen et al. (2006) Chen, L. E., Xie, G. Z., Ren, J. Y. et al., 2006, ApJ, 637, 711
  * Chen et al. (2009) Chen, Z. Y., Gu, M. F. Cao, X. W., 2009, MNRAS,397, 1713
  * Comastri et al. (1997) Comastri, A., Fossati, G., Ghisellini, G. et al.,1997, ApJ, 480, 534
  * Costamante (2009) Costamante, Luigi, 2009, Int. J. Mod. Phys. D., 18, 1483
  * Fan et al. (2012) Fan, J. H., Yang, J. H., Yuan, Y. H. et al., 2012, ApJ, 761, 125
  * Fossati et al. (1998) Fossati, G., Maraschi, L., Celotti, A. et al., 1998, MNRAS, 299, 433
  * Georganopoulos et al. (2001) Georganopoulos, M., Kirk, J. G. & Mastichiadis, A., 2001, ASPC, 227, 116
  * Ghisellini et al. (1998) Ghisellini, G., Celotti, A., Fossati, G. et al., 1998, MNRAS, 301, 451
  * Ghisellini et al. (2009) Ghisellini, G., Maraschi, L., Tavecchio, F. 2009, MNRAS, 396, 105
  * Ghisellini et al. (1986) Ghisellini, G., Maraschi, L., Treves, A., Tanzi, E. G., 1986, ApJ, 310, 317
  * Ghisellini et al. (2010) Ghisellini, G., Tavecchio, F., Foschini, L. et al., 2010, MNRAS, 402, 497
  * Ghisellini & Tavecchio (2008) Ghisellini, G. & Tavecchio, F., 2008, MNRAS, 387, 1669
  * Giommi et al. (1995) Giommi, P., Ansari, S. G., Micol, A., 1995, A&AS, 109, 267
  * Giommi et al. (1990) Giommi, P., Barr, P., Pollock, A. M. T. et al., 1990, ApJ, 356, 432
  * Giommi et al. (2005) Giommi, P., Piranomonte, S., Perri, M. & Padovani, P., 2005, A&A, 434, 385
  * Li et al. (2010) Li, H. Z., Xie, G. Z., Yi, T. F. et al., 2010, ApJ, 709, 1407
  * Lyu et al. (2014) Lyu, F., Liang, E., Liang, Y. et al., 2014, ApJ, 793, 36
  * Ma et al. (2007) Ma, L., Chen, L. E., Xie, G. Z. et al., 2007, ChJAA, 7, 345
  * Maraschi et al. (2008) Maraschi, L., Foschini, L., Ghisellini, G., Tavecchio, F. & Sambruna, R. M., 2008, MNRAS, 391, 1981
  * Mei et al. (2002) Mei, D. C., Zhang, L., Jiang, Z. J., 2002, A&A, 391, 917
  * Nieppola et al. (2006) Nieppola, E., Tornikoski, M., Valtaoja, E., 2006, A&A, 445, 441
  * Padovani (2007) Padovani, P., 2007, Ap&SS, 309, 63
  * Padovani & Giommi (1995) Padovani, P. & Giommi, P., 1995, MNRAS, 277, 1477
  * Padovani et al. (2003) Padovani, P., Perlman, E. S., Landt, H., Giommi, P., & Perri, M. 2003, ApJ, 588, 128
  * Plotkin et al. (2010) Plotkin, R. M., Anderson, S. F., Brandt, W. N. et al., 2010, AJ, 139, 390
  * Plotkin et al. (2008) Plotkin, R. M., Anderson, S. F., Hall, Patrick B. et al., 2008, AJ, 135, 2453
  * Sambruna et al. (1996) Sambruna, R. M., Maraschi, L., Urry, C. M., 1996, ApJ, 463, 444
  * Scarpa & Falomo (1997) Scarpa, R. & Falomo, R., 1997, A&A, 325. 109
  * Stern & Poutanen (2008) Stern, B. E. & Poutanen, Juri, 2008, MNRAS, 383, 1695
  * Turriziani et al. (2007) Turriziani, S., Cavazzuti, E. & Giommi, P., 2007, A&A, 472, 699
  * Urry & Padovani (1995) Urry, C. M. & Padovani, P., 1995, PASP, 107, 803
  * Xie et al. (2001) Xie, G. Z., Dai, B. Z., Liang, E. w. et al., 2001, PASJ, 53, 469
  * Xie et al. (2006) Xie, G. Z., Dai, H., Mao, L. S. et al., 2006, AJ, 131, 1210
  * Xie et al. (2003) Xie, G. Z.; Ding, S. X.; Dai, H., et al., 2003, Int. J. Mod. Phys. D., 12, 781
  * Xie et al. (2004) Xie, G. Z., Zhou, S. B., Liang, E. W., 2004, AJ, 127, 53
  * Xie et al. (2008) Xie, Z. H., Hao, J. M., Du, L. M. et al., 2008, PASP, 120, 477
  * Xie et al. (2007) Xie, Z. H., Dai, H., Hao, J. M. et al., 2007, ChJAA, 7, 209
  * Xu et al. (2009) Xu, Y. D., Cao, X. W., & Wu, Q. W., 2009, ApJ, 694,107
  * Yan et al. (2014) Yan, D., Zeng, H. & Zhang, L., 2014, MNRAS, 439, 2933
  * Zhang et al. (2010) Zhang, J., Bai, J. M., Chen, L., & Liang, E. W., 2010, ApJ, 710, 1017
  * Zhang et al. (2013) Zhang, J., Zhang, S. N., & Liang, E. W., 2013, IJMPS, 23, 54
  * Zheng et al. (2007) Zheng, Y. G., Zhang, X., Hu, S. M., 2007, Ap&SS, 310, 1