# Identity-aware Facial Expression Recognition in Compressed Video

Xiaofeng Liu1,5†, Linghao Jin3†, Xu Han3, Jun Lu1∗, Jane You4, Lingsheng Kong2
1 Beth Israel Deaconess Medical Center and Harvard Medical School, Boston, MA,
USA
2 Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy
of Sciences, CAS, Changchun, China
3 Johns Hopkins University, Baltimore, MD, USA
4 Dept. of Computing, Hong Kong Polytechnic University, Hung Hom, Hong Kong
5 Fanhan Tech. Inc., Suzhou, Jiangsu, China.
†Contribute equally. *Corresponding Author.

###### Abstract

This paper targets to explore the inter-subject variations eliminated facial
expression representation in the compressed video domain. Most of the previous
methods process the RGB images of a sequence, while the off-the-shelf and
valuable expression-related muscle movement already embedded in the
compression format. In the up to two orders of magnitude compressed domain, we
can explicitly infer the expression from the residual frames and possible to
extract identity factors from the I frame with a pre-trained face recognition
network. By enforcing the marginal independent of them, the expression feature
is expected to be purer for the expression and be robust to identity shifts.
We do not need the identity label or multiple expression samples from the same
person for identity elimination. Moreover, when the apex frame is annotated in
the dataset, the complementary constraint can be further added to regularize
the feature-level game. In testing, only the compressed residual frames are
required to achieve expression prediction. Our solution can achieve comparable
or better performance than the recent decoded image based methods on the
typical FER benchmarks with about 3$\times$ faster inference with compressed
data.

##  I Introduction

Considering the natural dynamic property of the human face expression, many
works propose to explore spatio-temporal features of facial expression
recognition (FER) from the videos. Although the multi-frame sequence can
inherit richer information and the temporal-correlation between the
consecutive frames can usually be helpful for the facial expression
recognition, the video also introduced significantly redundancy. Considering
the subtle muscle movement in FER videos, the signal-to-noise ratio (SNR) can
be low [1, 2].

Figure 1: Illustration of the typical video compression and the scheme of
conventional FER methods, which first decode the video and then feed it into a
FER network.

The conventional spatio-temporal FER networks take sequentially frames to
utilize the spatial and temporal cues to represent the facial expressions [2].
The recursive neural network (RNN), 3D convolutional, and non-local network
can be used to derive information from sequences [3, 4, 5]. However, using
deep neural networks to process many consecutive frames can be very
computationally expensive when the length of video is long [6, 7, 8, 9].
Actually, many methods can achieve good performance using decision-level
fusion of image-based FER, which totally ignores the temporal dependency. This
implies the temporal cues may hard be explored in the low SNR FER videos.

We propose that the compressed domain can be suitable for FER task for four
reasons. 1) the consecutive frames in video modality have many uninformative
and repeating patterns, which may drowning the “interesting” and “true” signal
[10]. 2) the typical compression solution (e.g., MPEG-4, H.264, and HEVC)
separate the video to the I frame (intracoded frames) with the first image,
and follows several P frames (predictive frames) which encoded as the “change”
or “movement” [11], as shown in Figure. 1. The movement of face muscle is the
fundamental of expression [12]. 3) our compressed domain exploration can also
be effective since it focus on the “true” signals rather than processing the
repeatedly near-duplicates [13, 14]. 4) we do not need the decode operation in
real-world task, since the to-be processed data is usually transferred with
the compressed format.

Moreover, the high inter-subject variations caused by identity differences in
facial attributes is a long-lasting difficulty in FER community [15, 16]. We
note that the P frames can also incorporate the relative position of key
points, which are also related to the identity [17]. The typical solution for
eliminating identity from the expression representation is using the metric
learning [18, 15] or the conditional generative adversarial network (GAN)
[19]. However, these works focus on the image-based FER, and most of them
requires the identity label and multiple expressions of the same person, which
significantly limits their applications on the in-the-wild FER dataset [20].

In this work, we propose to explore the identity factor from the I frame with
the pre-trained face recognizer, e.g., FaceNet [21]. Their embeddings are
remarkably reliable, since they achieve high accuracy over millions of
identities [22], and robust to a broad range of nuisance factors such as
expression, pose, illumination and occlusion variations. Using the identity
feature as the anchor, we can explicitly enforce the marginally independence
of our identity and expression feature [23].

In summary, this paper makes the following contributions.

$\bullet$ We propose to inference expression from the residual frames, which
explores the off-the-shelf yet valuable expression-related muscle movement in
the up to two orders of magnitude compressed domain.

$\bullet$ Targeting for the identity-aware video-based FER, the independence
of expression and identity representations from P frames and I frame are
enforced.

$\bullet$ The proposed IFERCV framework achieve state-of-the-art results on
several video-based FER benchmarks with much faster inference. The promising
performance evidenced its generality and scalability.

##  II Related Works

Video-based FER. With the fast development of deep learning [24, 25, 26, 27,
28, 29, 30, 31], both the frame aggregation and spatiotemporal FER networks
are developed and outperforms the conventional methods [32, 33]. The frame
aggregation methods can utilize the image-based FER networks by making the
decision-level [34] or the feature-level frame-wise aggregation [35]. The
spatio-temporal FER networks take sequentially images to utilize both the
spatial/textural and temporal information [36]. However, the previous works
only consider the image domain, and the spatiotemporal FER does not
significantly outperforms aggregation methods [2]. To the best of our
knowledge, this is the first effort to investigate the compressed video FER,
which is orthogonal to these advantages and can be easily added to each other
[37, 38, 39, 40, 41].

Video compression Usually, the video codecs separates a video into several
Group Of Pictures (GOP). A GOP is composed of an I frame, and followed by
several P-frames or B-frames. I-frame indicates a self-contained RGB frame
with full visual representation, while the P-frame or B-frame is the inter
frames which hold motion vectors and residuals w.r.t. the previous frame [11].
[42] propose to replace the optical flow by the motion vector. However,
decoding the RGB images is still necessary in this method. More recently, in
the action recognition task, [13] propose to fuse all of the available
modalities in the compressed video, e.g., I-frame, residuals and motion
vectors, to avoid the decoding of images. Although the expression shares some
similarities with action recognition, the movement range and the utilization
of I frame are essentially different. The I frame is directly used to predict
the action and combine with the result of P frames [13], while the I frame in
FER usually be the neutral face (different from the video label). Besides, the
low resolution motion vector can not well encode the expression.

Figure 2: The illustration of IFERCV framework for identity-aware facial
expression recognition in compressed video domain.

##  III Methodology

Our goal is to design a fast and effective video-based FER framework operates
directly on the compressed domain. The overall frame work is shown in Figure.
2, which are consist of four core modules. The pre-trained identity branch and
FER branch (frame embedding network $f_{E}$, aggregation module and
Classifier) work on the I frame and undecoded P frames respectively. Then, a
disentanglement module is applied to measure the dependence of identity and
expression representations. Moreover, the complementary constraint can be
applied when the apex frame is annotated to stabilize the early stage
training.

###  III-A Modeling Compressed Representations

To illustrate the format of input video, we choose the MPEG-4 as an example
[43]. The compressed domain has two typical frames, i.e., I frame and P
frames. Specifically, the I frame $I\in\mathbb{R}^{h\times w\times 3}$ is a
complete RGB image. We use $h$ and $w$ to denote its height and width
respectively. Besides, the P frame at time $t$ $P^{t}\in\mathbb{R}^{h\times
w\times 3}$ can be reconstructed with the stored offsets, called residual
errors $\Delta^{t}\in\mathbb{R}^{h\times w\times 3}$ and motion vectors
$\mathcal{T}^{t}\in\mathbb{R}^{h\times w\times 2}$.

Noticing that the motion vectors $\mathcal{T}^{t}$ has much lower resolution,
since its values within the same macroblock are identical. Considering the
micro movements of facial expression in each frame, the coarse
$\mathcal{T}^{t}$ usually not helpful for the FER. For P frame reconstruction
$P_{i}^{t}=P_{i-\mathcal{T}_{i}^{t}}^{t-1}+\Delta_{i}^{t}$, where index all
the pixels and $P^{0}=I$. Then, $\mathcal{T}^{t}$ and $\Delta^{t}$ are
processed by discrete cosine transform and entropy-encoded.

The typical compression algorithms are only developed to compress the file
size, and the encoded format can be very different with the RGB images w.r.t.
the statistical and structural properties. Therefore, a tailored processing
network is necessary to accommodate the compressed format. Considering the
structure of residual images $\Delta^{t}$ are much simpler than the decoded
images, it is possible to utilize simpler and faster CNNs
$f_{E}:\mathbb{R}^{h\times w\times 3}\rightarrow\mathbb{R}^{512}$ to extract
the feature of each frame [3, 44]. Practically, we follow the CNN in the
typical CNN-LSTM FER structure [44, 3], but with fewer layers to explore the
information in $\Delta^{t}$. Noticing that $f_{E}$ is shared for all frames,
and only needs to store one $f_{E}$ in processing.

Besides, most existing action recognition methods with compressed video [13,
14] independently concatenate the paired $\Delta^{t}$ and $\mathcal{T}^{t}$ at
each time step and predict an action score of each P-frame. The temporal cues
and its development patterns are important for FER task [2]. We simply choose
the LSTM in [44] to model the sequential development of residual frames and
summarize the information to a expression feature $z_{E}$. Since our LSTM is
applied to 512-dim features, the computation burden is largely smaller than
work on the raw images. Noticing that more advanced RNN, 3D CNN or attention
networks can potentially be utilized to replace our LSTM model to further
boost the performance [45, 46, 47].

For the I frame with raw image format, we simply use the FaceNet [21] pre-
trained on millions of identities [22] as our identity feature extractor
$f_{I}:\mathbb{R}^{h\times w\times 3}\rightarrow z_{I},$ where
$z_{I}\in\mathbb{R}^{1024}$ denotes the identity feature. Actually, the FER
videos in many datasets start from the neutral expression which can further
facilitate identity recognition.

###  III-B Disentanglement and Complementary Constraint

Configuring a dual-branch encoder to extract the separated information has
been a standard protocol for disentanglement [23]. Several adversarial
disentanglement works demonstrate that simply separate the input may result in
the extracted feature has no meaningful information [48, 23]. The
reconstruction of input can explicitly enforce the disentangled factors to be
complementary to each other. However, reconstructing the video can be hugely
underconstrained.

Many FER datasets follow a well-defined collection protocol, which usually
starts from the neutral face and then develops to an expression. Specifically,
the video in CK+ [49, 12] consists of a sequence which shift from the neutral
expression to a apex facial expression. The last frame usually be the apex
frame, which has the most strong expression intensity. Actually, the image-
based FER methods select the last three frames to construct their training and
testing datasets. Similarly, in MMI [50], the video frames are usually start
from the neutral face and develop to the apex around the middle of video, and
returning back to the neutral in the end of video. Noticing that the apex
frame (i.e., last frame in CK+ or middle frame in MMI) can clearly incorporate
both the identity and expression information. Therefore, we are possible to
utilize the apex frame as a reference of reconstruction, and simply apply the
$\mathcal{L}_{1}$ loss.

$\mathcal{L}_{1}=||I_{Apex}-\hat{I}_{Apex}||_{2}^{2}$ (1)

where $\hat{I}_{Apex}=Dec(z_{I},z_{E})$. We note that the complementary
constraint is not a necessity in our framework, since the FER loss takes a
large weight in the FER branch. It requires to maintain sufficient information
w.r.t. expression and not easy to have nothing meaningful. But the
complementary constraint does helpful for the convergence in the early stage.
When the apex is annotated, we only need to decode the apex frame in the
decoded image domain in the start of few training epochs.

###  III-C Overall objectives

We have three to be minimized objectives, i.e., cross-entropy loss,
disentanglement loss [23] and $\mathcal{L}_{1}$ loss, which works
collaboratively to update each module. The expression classification is the
main task of the FER. We choose the typical cross-entropy loss
$\mathcal{L}_{CE}=-\sum_{c=1}^{C}y_{c}\log(Cls(z_{E})_{c})$ to ensure $z_{E}$
contains sufficient expression information and finally have a good performance
on $C$-class expression classification. We use $y_{c}$ and
$Cls(z_{E})_{c}$indicate the $c^{th}$ class probability of the label and
classifier softmax predictions respectively. Since the FER branch can be
updated with all of the losses, we assign the balance parameter
$\alpha\in[0,1]$ and $\beta\in[0,1]$ to disentanglement loss [23] and
$\mathcal{L}_{1}$ loss minimization objectives respectively. We note that only
the FER branch, i.e., $f_{E}$, LSTM and Cls, is used for testing.

##  IV Experiments

In this section, we first detail our experimental setup, present quantitative
analysis of our model, and finally compare it with state-of-the-art methods.

TABLE I: Comparison of various methods on the CK+ dataset in terms of average
recognition accuracy of seven expressions. Note that in order to make the
comparison fair, we do not consider image-based and 3D geometry based
experiment setting and models [15, 18, 1].

Method | Accuracy | Landmarks | Test speed
---|---|---|---
STM-ExpLet (2014) [51] | 94.19 | $\times$ | -
LOMo (2016) [52] | 95.10 | $\checkmark$ | -
DTAGN (2015) [53] | 97.25 | $\checkmark$ | -
PHRNN-MSCNN (2017) [54] | 98.50 | $\checkmark$ | -
(N+M)-tuplet (2018) [55] | 93.90 | $\checkmark$ | 12fps
C3D-GRU (2019) [56] | 97.25 | $\times$ | -
CTSLSTM (2019) [57] | 93.9 | $\checkmark$ | -
SC (2019) [58] | 97.60 | $\checkmark$ | -
G2-VER (2019) [59] | 97.40 | $\times$ | -
LBVCNN (2019) [47] | 97.38 | $\times$ | -
Mode variational LSTM (2019) [44] | 97.42 | $\times$ | 11fps
IFERCV | 97.44 | $\times$ | 35fps
IFERCV+Adversarial disentanglement[23] | 98.38 | $\times$ | 35fps
IFERCV-$\hat{I}$ | 97.16 | $\times$ | 35fps
IFERCV+$\mathcal{T}^{t}$ | 97.85 | $\times$ | 29fps

###  IV-A Description of the datasets

CK+ Dataset is a widely accepted FER benchmark [2, 47]. The video is collected
in a restricted environment, in which the participate subjects are facing to
the recorder with the empty background. The video in CK+ consists of a
sequence which shift from the neutral expression to a apex facial expression.
The last frame usually be the apex frame, which has the most strong expression
intensity. Following the previous works, we use subject independent 10-folds
cross validation [51, 2, 55].

Many FER datasets follow a well-defined collection protocol, which usually
starts from the neutral face and then develops to an expression. Specifically,
Actually, the image-based FER methods select the last three frames to
construct their training and testing datasets. Similarly, in MMI [50], the
video frames are usually start from the neutral face and develop to the apex
around the middle of video, and returning back to the neutral in the end of
video. Noticing that the apex frame (i.e., last frame in CK+ or middle frame
in MMI) can clearly incorporate both the identity and expression information.
Therefore, we are possible to utilize the apex frame as a reference of
reconstruction, and simply apply the $\mathcal{L}_{1}$ loss.

Figure 3: Confusion matrix of IFERCV on CK+ datasets.

AFEW Dataset is more close to the uncontrolled real-world environment. It is
consist of the video clips of movies [60]. The video in AFEW has the
spontaneous facial expression. The AFEW has seven expressions: anger, disgust,
fear, happiness, sadness, surprise and neutral. Following the evaluation
protocol in EmotiW [61], there are training, validation and testing sets.
Since its testing label is not available, we follow the previous work to use
the validation set for comparison [44]. Noticing that the validation set is
not used in training stage for parameter or hyper-parameter tuning.

###  IV-B Evaluation and ablation study

Results on CK+ dataset. The 10-fold cross-validation performance of our
proposed method is shown in the Table 1. For fair comparison, the image-based
experiment settings are not incorporated in the tables. Besides, only the
state-of-the-art (SOTA) accuracy obtained by the single-models (non-ensemble
model) are listed.

Many models, e.g., DTAGN [53], LOMo [52], PHRNN-MSCNN [54], CTSLSTM [57] and
SC [58], achieved the SOTA performance by utilizing the facial landmarks.
However, this operation highly relies on the fine-grained landmark detection,
which itself is a challenging task [2, 1], and unavoidably introduced
additional computation.

Based on the mode variational LSTM [44], our IFERCV achieves the SOTA result
without the facial landmarks, 3D face models or optical flow. It worth
noticing that the much simpler CNN encoding network makes more residual frames
can be processed parallel than [44]. Moreover, our efficiency is also benefit
from avoiding the decompress of the video. Since the videos are typically
stored and transmitted with the compressed version, and the residuals are off-
the-shelf. As a result, the proposed compressed domain IFERCV can speed up the
testing about 3 times over [44], and achieve better accuracy.

Besides, [55] is a typical metric-learning based identity removing method. Our
solution can significantly outperform it with respect to both speed and
accuracy. Actually, the sampling of tuplets usually makes the training not
scalable [1], while our identity eliminating scheme is concise and effective.

When we remove some modules from our framework, the performances have
different degrees of decline. We use -$\hat{I}$ to denote the IFERCV without
complementary constraint. The result also implies that the identity can be
well encoded by the face recognition network.

We can also follow the action recognition method [14] to concatenate the
motion vector and residual as the input, and denote as
IFERCV+$\mathcal{T}^{t}$. However, we do not achieve significant performance
on all datasets, but the inference speed in testing can be slower. This maybe
related to the coarse resolution of the motion vector can not well describe
the fine-grain muscle movement of the face.

The confusion matrix of our proposed IFERCV method on the CK+ is reported in
Figure 4 (left). The accuracy for the expression class happiness, disgust,
angry, surprise and contempt are almost perfect.

TABLE II: Comparison of various methods on the AFEW dataset in terms of
average recognition accuracy of seven expressions. *optical flow is used.

Method | Model type | Accuracy | Test speed
---|---|---|---
CNN-RNN (2016) [62] | 45.43 | Dynamic | -
Undirectional LSTM (2017) [63] | 48.60 | Dynamic | -
HoloNet (2016) [64] | 44.57 | Static | -
DSN-HoloNet (2017) [65] | 46.47 | Static | -
DenseNet-161 (2018) [66] | 51.44 | Static | -
DSN-VGGFace (2018) [67] | 48.04 | Static | -
FAN (2019) [5] | 51.18 | Static | -
CTSLSTM (2019) [57] | 51.2 | Dynamic | -
C3D-GRU (2019) [56] | 49.87 | Dynamic | -
DSTA (2019)* [68] | 42.98 | Dynamic | -
E-ConvLSTM (2019)* [69] | 45.29 | Dynamic | 4fps
Mode variational LSTM (2019) [44] | 51.44 | Dynamic | 11fps
IFERCV | 51.62 | Dynamic | 34fps
IFERCV+Adv Disentanglement[23] | 52.01 | Dynamic | 34fps
IFERCV+$\mathcal{T}^{t}$ | 51.86 | Dynamic | 30fps

Results on AFEW dataset. The evaluation of the proposed IFERCV on AFEW dataset
is shown in Table 4. We note that only the SOTA accuracy obtained by the
single-models (non-ensemble model) are listed for fair comparison. Besides,
the audio modality in AFEW can be used to boost the recognition performance
[70, 62, 63]. We note that we only focus on the image compression in this
paper, and the audio/video data are stored in separate tracks, but the
additional modality can also potentially to be add on our framework following
the multi-modal methods[70, 62].

With the simplified Mode variational LSTM-based [44] backbone, the exploration
in the compressed domain can achieve comparable or even better recognition
performance. More promisingly, our IFERCV can also achieve real-time
processing for the uncontrolled environment, which evidenced its generality.
We note that the typical time resolution in FER is 24fps [3].

Figure 4: Confusion matrix of IFERCV on AFEW datasets.

Some of the works propose to improve the image-based FER networks and combine
the frame-wise scores for video-based FER [64, 65, 66, 67, 5]. [64, 65] input
both the LBP maps and the image to the CNNs. [65] and [67] utilize the
additional supervision on intermediate layers. The image-based FER methods
[66] achieves high performance, but [66] uses a very deep network DenseNet-161
and pretrains it on the private Situ dataset. Moreover, [66] utilize the
sophisticated post-processing. Actually, an intuition of statistic based
solution is to avoid LSTM and speed up the processing. However, with the super
deep and complicated structure, their processing can be much slower than our
solution.

Both [62] and [63] use VGGFace as the backbone of $f_{E}$ and a RNN model with
LSTM units. They target to capture the temporal dynamic cues of the videos.
Moreover, [57, 56, 68] also propose to modify the LSTM model for the spatial-
temporal modeling. However, all of the above solutions are applied to the
decoded space, which requires decoding processing and needs to handle much
more complicated data. Overall, the proposed IFERCV can improve the testing
speed by a large margin and can achieves the SOTA accuracy as the previous
models.

##  V Conclusion

In this work, we propose to explore the facial expression cues directly on the
compressed video domain. We are motivated by our practical observation that
the facial muscle movements can be well encoded in the residual frames which
can be informative and free of cost. Besides, the video compression can reduce
the repeating boring patterns in the videos, which rendering the
representation to be robust. The increased relevance and reduced complexity or
redundancy in FER videos make computation much more effective. We extract the
identity and expression factor from the I frame and P frame respectively, and
explicitly enforce their independence with disentanglement regularization
[23]. When the apex frame label is available in training, the complementary
constraint can further stabilize the training. In three video-based FER
benchmarks, our IFERCV can improve the performance without the additional
identity, face model or facial landmarks labels. The processing speed of the
test stage is promising for real-time FER.

##  VI Acknowledgements

This work was supported by the Jangsu Youth Programme [BK20200238], National
Natural Science Foundation of China, Younth Programme [grant number 61705221],
NIH [NS061841, NS095986], Fanhan Technology, and Hong Kong Government General
Research Fund GRF (Ref. No.152202/14E) are greatly appreciated.

## References

  * [1] X. Liu, B. V. Kumar, P. Jia, and J. You, “Hard negative generation for identity-disentangled facial expression recognition,” _Pattern Recognition_ , vol. 88, pp. 1–12, 2019.
  * [2] S. Li and W. Deng, “Deep facial expression recognition: A survey,” _IEEE Transactions on Affective Computing_ , 2020.
  * [3] D. H. Kim, W. J. Baddar, J. Jang, and Y. M. Ro, “Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition,” _IEEE Transactions on Affective Computing_ , vol. 10, no. 2, pp. 223–236, 2017.
  * [4] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3d convolutional networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 4489–4497.
  * [5] D. Meng, X. Peng, K. Wang, and Y. Qiao, “frame attention networks for facial expression recognition in videos,” in _2019 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2019, pp. 3866–3870.
  * [6] X. Liu, B. Vijaya Kumar, C. Yang, Q. Tang, and J. You, “Dependency-aware attention control for unconstrained face recognition with image sets,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 548–565.
  * [7] X. Liu, Z. Guo, J. You, and B. Kumar, “Attention control with metric learning alignment for image set-based recognition,” _arXiv preprint arXiv:1908.01872_ , 2019.
  * [8] X. Liu, Z. Guo, S. Li, L. Kong, P. Jia, J. You, and B. Kumar, “Permutation-invariant feature restructuring for correlation-aware image set-based recognition,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 4986–4996.
  * [9] X. Liu, Z. Guo, J. You, and B. V. Kumar, “Dependency-aware attention control for image set-based face recognition,” _IEEE Transactions on Information Forensics and Security_ , vol. 15, pp. 1501–1512, 2019.
  * [10] C. Yeo, P. Ahammad, K. Ramchandran, and S. S. Sastry, “Compressed domain real-time action recognition,” in _2006 IEEE Workshop on Multimedia Signal Processing_. IEEE, 2006, pp. 33–36.
  * [11] D. Le Gall, “Mpeg: A video compression standard for multimedia applications,” _Communications of the ACM_ , vol. 34, no. 4, pp. 46–58, 1991.
  * [12] P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, “The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression,” in _2010 ieee computer society conference on computer vision and pattern recognition-workshops_. IEEE, 2010, pp. 94–101.
  * [13] C.-Y. Wu, M. Zaheer, H. Hu, R. Manmatha, A. J. Smola, and P. Krähenbühl, “Compressed video action recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6026–6035.
  * [14] Z. Shou, X. Lin, Y. Kalantidis, L. Sevilla-Lara, M. Rohrbach, S.-F. Chang, and Z. Yan, “Dmc-net: Generating discriminative motion cues for fast compressed video action recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1268–1277.
  * [15] X. Liu, B. Vijaya Kumar, J. You, and P. Jia, “Adaptive deep metric learning for identity-aware facial expression recognition,” in _CVPRW_ , 2017, pp. 522–531.
  * [16] X. Liu, B. V. Kumar, Y. Ge, C. Yang, J. You, and P. Jia, “Normalized face image generation with perceptron generative adversarial networks,” in _2018 IEEE 4th International Conference on Identity, Security, and Behavior Analysis (ISBA)_. IEEE, 2018, pp. 1–8.
  * [17] Y. Tian, T. Kanade, and J. F. Cohn, “Facial expression recognition,” in _Handbook of face recognition_. Springer, 2011, pp. 487–519.
  * [18] Z. Meng, P. Liu, J. Cai, S. Han, and Y. Tong, “Identity-aware convolutional neural network for facial expression recognition,” in _2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017)_. IEEE, 2017, pp. 558–565.
  * [19] K. Ali and C. E. Hughes, “All-in-one: Facial expression transfer, editing and recognition using a single network,” _arXiv preprint arXiv:1911.07050_ , 2019\.
  * [20] A. Dhall, R. Goecke, J. Joshi, K. Sikka, and T. Gedeon, “Emotion recognition in the wild challenge 2014: Baseline, data and protocol,” in _Proceedings of the 16th international conference on multimodal interaction_ , 2014, pp. 461–466.
  * [21] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 815–823.
  * [22] I. Kemelmacher-Shlizerman, S. M. Seitz, D. Miller, and E. Brossard, “The megaface benchmark: 1 million faces for recognition at scale,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 4873–4882.
  * [23] X. Liu, S. Li, L. Kong, W. Xie, P. Jia, J. You, and B. Vijaya Kumar, “Feature-level frankenstein: Eliminating variations for discriminative recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 637–646.
  * [24] T. Che, X. Liu, S. Li, Y. Ge, R. Zhang, C. Xiong, and Y. Bengio, “Deep verifier networks: Verification of deep discriminative models with deep generative models,” _arXiv preprint arXiv:1911.07421_ , 2019.
  * [25] X. Liu, Y. Lu, X. Liu, S. Bai, S. Li, and J. You, “Wasserstein loss with alternative reinforcement learning for severity-aware semantic segmentation,” _IEEE Transactions on Intelligent Transportation Systems_ , 2020.
  * [26] X. Liu, F. Fan, L. Kong, Z. Diao, W. Xie, J. Lu, and J. You, “Unimodal regularized neuron stick-breaking for ordinal classification,” _Neurocomputing_ , 2020.
  * [27] X. Liu, Y. Han, S. Bai, Y. Ge, T. Wang, X. Han, S. Li, J. You, and J. Lu, “Importance-aware semantic segmentation in self-driving with discrete wasserstein training.” in _AAAI_ , 2020, pp. 11 629–11 636.
  * [28] X. Liu, W. Ji, J. You, G. E. Fakhri, and J. Woo, “Severity-aware semantic segmentation with reinforced wasserstein training,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 12 566–12 575.
  * [29] G. He, X. Liu, F. Fan, and J. You, “Image2audio: Facilitating semi-supervised audio emotion recognition with facial expression image,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 912–913.
  * [30] Y. Han, X. Liu, Z. Sheng, Y. Ren, X. Han, J. You, R. Liu, and Z. Luo, “Wasserstein loss-based deep object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 998–999.
  * [31] X. Liu, T. Che, Y. Lu, C. Yang, S. Li, and J. You, “Auto3d: Novel view synthesis through unsupervisely learned variational viewpoint and global 3d representation,” _arXiv preprint arXiv:2007.06620_ , 2020.
  * [32] X. Liu, L. Kong, Z. Diao, and P. Jia, “Line-scan system for continuous hand authentication,” _Optical Engineering_ , vol. 56, no. 3, p. 033106, 2017\.
  * [33] X. Liu, Z. Li, L. Kong, Z. Diao, J. Yan, Y. Zou, C. Yang, P. Jia, and J. You, “A joint optimization framework of low-dimensional projection and collaborative representation for discriminative classification,” in _2018 24th International Conference on Pattern Recognition (ICPR)_. IEEE, 2018, pp. 1493–1498.
  * [34] S. E. Kahou, X. Bouthillier, P. Lamblin, C. Gulcehre, V. Michalski, K. Konda, S. Jean, P. Froumenty, Y. Dauphin, N. Boulanger-Lewandowski _et al._ , “Emonets: Multimodal deep learning approaches for emotion recognition in video,” _Journal on Multimodal User Interfaces_ , vol. 10, no. 2, pp. 99–111, 2016.
  * [35] B. Xu, Y. Fu, Y.-G. Jiang, B. Li, and L. Sigal, “Video emotion recognition with transferred deep feature encodings,” in _Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval_ , 2016, pp. 15–22.
  * [36] D. A. AL CHANTI and A. Caplier, “Deep learning for spatio-temporal modeling of dynamic spontaneous emotions,” _IEEE Transactions on Affective Computing_ , 2018.
  * [37] G. He, X. Liu, F. Fan, and J. You, “Classification-aware semi-supervised domain adaptation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 964–965.
  * [38] X. Liu, X. Han, Y. Qiao, Y. Ge, S. Li, and J. Lu, “Unimodal-uniform constrained wasserstein training for medical diagnosis,” in _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2019, pp. 0–0.
  * [39] X. Liu, Y. Zou, T. Che, P. Ding, P. Jia, J. You, and B. Kumar, “Conservative wasserstein training for pose estimation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 8262–8272.
  * [40] X. Liu, Y. Zou, Y. Song, C. Yang, J. You, and B. K Vijaya Kumar, “Ordinal regression with neuron stick-breaking for medical diagnosis,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 0–0.
  * [41] X. Liu, Y. Zou, L. Kong, Z. Diao, J. Yan, J. Wang, S. Li, P. Jia, and J. You, “Data augmentation via latent space interpolation for image classification,” in _2018 24th International Conference on Pattern Recognition (ICPR)_. IEEE, 2018, pp. 728–733.
  * [42] B. Zhang, L. Wang, Z. Wang, Y. Qiao, and H. Wang, “Real-time action recognition with enhanced motion vector cnns,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2718–2726.
  * [43] I. E. Richardson, “H264/mpeg-4 part 10 white paper-prediction of intra macroblocks,” _Internet Citation, Apr_ , vol. 30, 2003.
  * [44] W. J. Baddar and Y. M. Ro, “Mode variational lstm robust to unseen modes of variation: Application to facial expression recognition,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 3215–3223.
  * [45] P. Barros and S. Wermter, “Developing crossmodal expression recognition based on a deep neural model,” _Adaptive behavior_ , vol. 24, no. 5, pp. 373–396, 2016.
  * [46] J. Zhao, X. Mao, and J. Zhang, “Learning deep facial expression features from image and optical flow sequences using 3d cnn,” _The Visual Computer_ , vol. 34, no. 10, pp. 1461–1475, 2018.
  * [47] S. Kumawat, M. Verma, and S. Raman, “Lbvcnn: Local binary volume convolutional neural network for facial expression recognition from image sequences,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2019, pp. 0–0.
  * [48] M. F. Mathieu, J. J. Zhao, J. Zhao, A. Ramesh, P. Sprechmann, and Y. LeCun, “Disentangling factors of variation in deep representation using adversarial training,” in _NIPS_ , 2016, pp. 5040–5048.
  * [49] T. Kanade, J. F. Cohn, and Y. Tian, “Comprehensive database for facial expression analysis,” in _Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580)_. IEEE, 2000, pp. 46–53.
  * [50] M. Pantic, M. Valstar, R. Rademaker, and L. Maat, “Web-based database for facial expression analysis,” in _2005 IEEE international conference on multimedia and Expo_. IEEE, 2005, pp. 5–pp.
  * [51] M. Liu, S. Shan, R. Wang, and X. Chen, “Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2014, pp. 1749–1756.
  * [52] K. Sikka, G. Sharma, and M. Bartlett, “Lomo: Latent ordinal model for facial analysis in videos,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 5580–5589.
  * [53] H. Jung, S. Lee, J. Yim, S. Park, and J. Kim, “Joint fine-tuning in deep neural networks for facial expression recognition,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 2983–2991.
  * [54] K. Zhang, Y. Huang, Y. Du, and L. Wang, “Facial expression recognition based on deep evolutional spatial-temporal networks,” _IEEE Transactions on Image Processing_ , vol. 26, no. 9, pp. 4193–4203, 2017.
  * [55] X. Liu, Y. Ge, C. Yang, and P. Jia, “Adaptive metric learning with deep neural networks for video-based facial expression recognition,” _Journal of Electronic Imaging_ , vol. 27, no. 1, p. 013022, 2018.
  * [56] M. K. Lee, D. Y. Choi, D. H. Kim, and B. C. Song, “Visual scene-aware hybrid neural network architecture for video-based facial expression recognition,” in _2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019)_. IEEE, 2019, pp. 1–8.
  * [57] M. Hu, H. Wang, X. Wang, J. Yang, and R. Wang, “Video facial emotion recognition based on local enhanced motion history image and cnn-ctslstm networks,” _Journal of Visual Communication and Image Representation_ , vol. 59, pp. 176–185, 2019.
  * [58] M. Verma, H. Kobori, Y. Nakashima, N. Takemura, and H. Nagahara, “Facial expression recognition with skip-connection to leverage low-level features,” in _2019 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2019, pp. 51–55.
  * [59] T. Albrici, M. Fasounaki, S. B. Salimi, G. Vray, B. Bozorgtabar, H. K. Ekenel, and J.-P. Thiran, “G2-ver: Geometry guided model ensemble for video-based facial expression recognition,” in _2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019)_. IEEE, 2019, pp. 1–6.
  * [60] N. Perveen, D. Roy, and C. K. Mohan, “Spontaneous expression recognition using universal attribute model,” _IEEE Transactions on Image Processing_ , vol. 27, no. 11, pp. 5575–5584, 2018.
  * [61] A. Dhall, R. Goecke, S. Ghosh, J. Joshi, J. Hoey, and T. Gedeon, “From individual to group-level emotion recognition: Emotiw 5.0,” in _Proceedings of the 19th ACM international conference on multimodal interaction_ , 2017, pp. 524–528.
  * [62] Y. Fan, X. Lu, D. Li, and Y. Liu, “Video-based emotion recognition using cnn-rnn and c3d hybrid networks,” in _Proceedings of the 18th ACM International Conference on Multimodal Interaction_ , 2016, pp. 445–450.
  * [63] V. Vielzeuf, S. Pateux, and F. Jurie, “Temporal multimodal fusion for video emotion classification in the wild,” in _Proceedings of the 19th ACM International Conference on Multimodal Interaction_ , 2017, pp. 569–576.
  * [64] A. Yao, D. Cai, P. Hu, S. Wang, L. Sha, and Y. Chen, “Holonet: towards robust emotion recognition in the wild,” in _Proceedings of the 18th ACM International Conference on Multimodal Interaction_ , 2016, pp. 472–478.
  * [65] P. Hu, D. Cai, S. Wang, A. Yao, and Y. Chen, “Learning supervised scoring ensemble for emotion recognition in the wild,” in _Proceedings of the 19th ACM international conference on multimodal interaction_ , 2017, pp. 553–560.
  * [66] C. Liu, T. Tang, K. Lv, and M. Wang, “Multi-feature based emotion recognition for video clips,” in _Proceedings of the 20th ACM International Conference on Multimodal Interaction_ , 2018, pp. 630–634.
  * [67] Y. Fan, J. C. Lam, and V. O. Li, “Video-based emotion recognition using deeply-supervised neural networks,” in _Proceedings of the 20th ACM International Conference on Multimodal Interaction_ , 2018, pp. 584–588.
  * [68] X. Pan, G. Ying, G. Chen, H. Li, and W. Li, “A deep spatial and temporal aggregation framework for video-based facial expression recognition,” _IEEE Access_ , vol. 7, pp. 48 807–48 815, 2019.
  * [69] R. Miyoshi, N. Nagata, and M. Hashimoto, “Facial-expression recognition from video using enhanced convolutional lstm,” in _2019 Digital Image Computing: Techniques and Applications (DICTA)_. IEEE, 2019, pp. 1–6.
  * [70] X. Fan and T. Tjahjadi, “A dynamic framework based on local zernike moment and motion history image for facial expression recognition,” _Pattern recognition_ , vol. 64, pp. 399–406, 2017.


11institutetext: Applied and Computational Mathematics Division, NIST,
Gaithersburg, Maryland, U.S.A., 11email:
<EMAIL_ADDRESS>22institutetext: Dept.
of Computer and Information Science, University of Konstanz, Konstanz,
Germany, 22email<EMAIL_ADDRESS>
33institutetext: Dept. of Computer Science, GWU, Washington DC, U.S.A.,
33email<EMAIL_ADDRESS>
44institutetext:  DSIMG, Technische Universität, Berlin, Germany, 44email:
<EMAIL_ADDRESS>
55institutetext: Maplesoft, Waterloo, Ontario, Canada, 55email:
<EMAIL_ADDRESS>
66institutetext: Poolesville High School, Poolesville, Maryland, U.S.A.,
66email<EMAIL_ADDRESS>

# Semantic Preserving Bijective Mappings of Mathematical Formulae between
Document Preparation Systems and Computer Algebra Systems

Howard S. Cohl 11 Moritz Schubotz 22 Abdou Youssef 33 André Greiner-Petter 44
Jürgen Gerhard 55 Bonita V. Saunders 11 Marjorie A. McClain 11

###### Abstract

Document preparation systems like LaTeX offer the ability to render
mathematical expressions as one would write these on paper. Using LaTeX,
LaTeXML, and tools generated for use in the National Institute of Standards
(NIST) Digital Library of Mathematical Functions, semantically enhanced
mathematical LaTeX markup (semantic LaTeX) is achieved by using a semantic
macro set. Computer algebra systems (CAS) such as Maple and Mathematica use
alternative markup to represent mathematical expressions. By taking advantage
of Youssef’s Part-of-Math tagger and CAS internal representations, we develop
algorithms to translate mathematical expressions represented in semantic LaTeX
to corresponding CAS representations and vice versa. We have also developed
tools for translating the entire Wolfram Encoding Continued Fraction Knowledge
and University of Antwerp Continued Fractions for Special Functions datasets,
for use in the NIST Digital Repository of Mathematical Formulae. The overall
goal of these efforts is to provide semantically enriched standard conforming
MathML representations to the public for formulae in digital mathematics
libraries. These representations include presentation MathML, content MathML,
generic LaTeX, semantic LaTeX, and now CAS representations as well.

##  1 Problem and Current State

Scientists often use document preparation systems (DPS) to write scientific
papers. The well-known DPS LaTeX has become a de-facto standard for writing
mathematics papers. On the other hand, scientists working with formulae which
occur in their research often need to evaluate special or numerical values,
create figures, diagrams and tables. One often uses computer algebra systems
(CAS), programs which provide tools for symbolic and numerical computation of
mathematical expressions. DPS such as LaTeX, try to render mathematical
expressions as accurately as possible and give the opportunity for
customization of the layout of mathematical expressions. Alternatively, CAS
represent expressions for use in symbolic computation with secondary focus on
the layout of the expressions. This difference in format is a common obstacle
for scientific workflows.

For example, consider the Euler-Mascheroni (Euler) constant represented by
$\gamma$. Since generic LaTeX [1] does not provide any semantic information,
the LaTeX representation of this mathematical constant is just the command for
the Greek letter `\gamma`. Maple and Mathematica, well-known CAS, represent
the Euler constant $\gamma$ with `gamma` and `EulerGamma` respectively.
Scientists writing scientific papers, who use CAS often need to be aware of
representations in both DPS and CAS. Often different CAS have different
capabilities, which implies that scientists might need to know several CAS
representations for mathematical symbols, functions, operators, etc. One also
needs to be aware when CAS do not support direct translation. We refer to CAS
translation as either the forward or backward translation respectively as DPS
source to CAS source or vise-versa. For instance, the CAS representation of
the number $e\approx 2.71828$ (the base of the natural logarithm) in
Mathematica is `E`, whereas in Maple there is no directly translated symbol.
In Maple, one needs to evaluate the exponential function at one via `exp(1)`
to reproduce its value.

For a scientist, $\gamma$ and $e$ might represent something altogether
different from these constants, such as a variable, function, distribution,
vector, etc. In these cases, it would need to be translated in a different
way. In order to avoid these kinds of semantic ambiguities (as well as for
other reasons), Bruce Miller at NIST, developing for the Digital Library of
Mathematical Functions (DLMF) (special functions and orthogonal polynomials of
classical analysis) project, has created a set of semantic LaTeX macros [11,
9]. Extensions and ‘simplifications’ have been provided by the Digital
Repository of Mathematical Formulae (DRMF) project. We refer to this extended
set of semantic LaTeX macros as the DLMF/DRMF macro set, and the mathematical
LaTeX which uses this semantic macro set as semantic LaTeX.

Existing tools which attempt to achieve CAS translations include import/export
for LaTeX expressions (such as [8, 13]), as well as for MathML. CAS functions
such as these, mostly provide only presentation translation in LaTeX and do
not provide semantic solutions or workarounds to hidden problems such as
subtle differences in CAS function definitions. These differences may also
include differences in domains or complex branch cuts of multivalued
functions. To fill this lack of knowledge in the CAS translation process, one
needs to provide additional information in the DPS source itself and to create
interactive documents with references to definitions, theorems and other
representations of mathematical expressions. Our approach in this paper, is to
develop independent tools for translation between different CAS and semantic
LaTeX representations for mathematical expressions. We provide detailed
information about CAS translation and warn about the existence of known
differences in definitions, domains and branch cuts. For the DRMF, we have
decided to focus on CAS translation between the semantic LaTeX representations
of classical analysis and internal CAS representations for Maple and
Mathematica.

###  1.1 A CAS, generic and semantic LaTeX representation example

An example of a mathematical expression is
$P_{n}^{(\alpha,\beta)}(\cos(a\Theta))$ where $P_{n}^{(\alpha,\beta)}$ is the
Jacobi polynomial [3, (18.5.7)]. Table 1 illustrates several DPS and CAS
representations for this mathematical expression.

Table 1: DPS and CAS representations for Jacobi polynomial expression Different Systems | Different Representations
---|---
Generic LaTeX | `P_n^{(\alpha,\beta)}(\cos(a\Theta))`
semantic LaTeX | `\JacobiP{\alpha}{\beta}{n}@{\cos@{a\Theta}}`
Maple | `JacobiP(n,alpha,beta,cos(a*Theta))`
Mathematica | `JacobiP[n,\[Alpha],\[Beta],Cos[a \[CapitalTheta]]]`

Translating the generic LaTeX representation is difficult (see [1]) since the
semantic context of the $P$ is obscured. If it represents a special function,
one needs to ascertain which function it represents, because there are many
examples of standard functions in classical analysis which are given by a $P$.
The semantic LaTeX representation of this mathematical expression encapsulates
the mostly-unambiguous semantic meaning of the mathematical expression. This
facilitates translation between it and CAS representations. We use the first
scan of the Part-of-Math (POM) tagger [16] to facilitate translation between
semantic LaTeX and CAS representations.

##  2 The Part-of-Math tagger

There are different approaches for interpreting LaTeX. There exist several
parsers for LaTeX, for instance texvcjs, which is a part of Mathoid [14].
There is also LaTeXML [10, 11] which processes LaTeX. There is also an
alternative grammar developed by Ginev [5]. A new approach has been developed
[16] which is not a fully fledged grammar but only extracts POM from math
LaTeX. The purpose of the POM is to extract semantic information from
mathematics in LaTeX. The tagger works in several stages (termed scans) and
interacts with several machine learning (ML) based algorithms.

Given an input LaTeX math document, the first scan of the tagger examines
terms and groups them into sub-expressions when indicated. For instance
`\frac{1}{2}` is a sub-expression of numerator and denominator. A term is, in
the sense of Backus-Naur form, a pre-defined non-terminal expression and can
represent LaTeX macros, environments, reserved symbols (such as the LaTeX line
break command `\\`) or numerical or alphanumerical expressions. Sub-
expressions and terms get tagged due the first scan of the tagger, with two
separate tag categories: (1) definite tags (such as operation, function,
exponent, etc.) that the tagger is certain of; and tags which consist of
alternative and tentative features which include alternative roles and
meanings. These second category of tags are drawn from a specific knowledge
base which has been collected for the tagger. Tagged terms are called math
terms. Math terms are rarely distinct at this stage and often have multiple
features.

Scans 2 and 3 are expected to be completed in the next 2 years. These involve
some natural language processing (NLP) algorithms as well as ML-based
algorithms [15, 12]. Those scans will: (1) select the right features from
among the alternative features identified in the first scan; (2) disambiguate
the terms; and (3) group subsequences of terms into unambiguous sub-
expressions and tag them, thus deriving definite mostly-unambiguous semantics
of math terms and expressions. The NLP/ML algorithms include math topic
modeling, math context modeling, math document classification (into various
standard areas of math), and definition-harvesting algorithms.

Specifically, to narrow down the role/meaning of a math term, it helps to know
which area of mathematics the input document is in. This calls for a math-
document classifier. Furthermore, knowing the topic, which is more specific
than the area of the document, will shed even more light on the math terms.
Even more targeted is the notion of context which, if properly formulated,
will take the POM tagger a long way in narrowing down the tag choices.

In [16], Youssef defines a new notion of a math-term’s context, which involves
several components, such as (1) the area and topic of the term’s document; (2)
the document-provided definitions; (3) the topic model and theme class of the
term’s neighborhood in the document; (4) the actual mathematical expression
containing the term; as well as (5) a small number of natural language
sentences surrounding the mathematical expression. Parts of this context are
the textual definitions and explanations of terms and notations which can be
present or absent from the input document. These can also be near the target
terms or far and distributed from them. The NLP/ML-based algorithms for the
2nd and 3rd scans of the tagger will model and track the term’s contexts, and
will harvest definitions and explanations and associate them with the target
terms.

##  3 Semantic LaTeX to CAS translation

We have used a mathematical language parser (MLP) as an interface for the
above-described first scan of the POM tagger to build syntax trees of
mathematical expressions in LaTeX and provide CAS translations from semantic
LaTeX to CAS representations. The MLP provides all functionality to interact
with the results of the POM tagger. We extended the general information of
each term to its CAS representation, links to definitions on the DLMF/DRMF
websites, as well as the corresponding CAS websites. We also add information
about domains, position of branch cuts and further explanations if necessary.
Since the multiple scans of the POM tagger are still a work in progress, our
CAS translation is based on the first scan (see §2). Fig. 1 shows the syntax
tree corresponding to the LaTeX expression `\sqrt[3]{x^3} + \frac{y}{2}`; note
that ‘`x`’ and ‘`3`’ in ‘`x^3`’ are not treated (in Fig. 1) as siblings (i.e.,
children of ‘`^`’) because the first scan of the tagger does not recognize
this hierarchy (but it will be rectified in POM Scans 2 and 3). The general
CAS translation process translates each node without changing the hierarchy of
the tree recursively. With this approach, we are able to translate nested
function calls.

Figure 1: Syntax tree of $\sqrt[3]{x^{3}}+\frac{y}{2}$ produced by the first
scan of the POM tagger.

The syntax tree obtained by the first POM scan depends on the known terms of
the tagger. Although the tagger’s first scan tags macros if those macros’
definition are provided to it, it is currently agnostic of the DLMF/DRMF
macros. Therefore, as it currently stands, the first scan of the tagger
extracts, but does not recognize/tag DLMF/DRMF macros as hierarchical
structures, but rather treats those macros as sequences of terms. The syntax
tree in Fig. 2 was created by the tagger for our Jacobi polynomial example in
§1.1. The tagger extracts expressions enclosed between open and closed curly
braces,{…} which we refer to as delimited balanced expressions. The given
argument is a sub-expression and produces another hierarchical tree structure.

Figure 2: Syntax tree for Jacobi polynomial expression generated by the first
POM scan.

###  3.1 Implementation

CAS translations for DLMF/DRMF macros are stored in CSV files, to make them
easy to edit. Besides that, CAS translations for Greek letters and
mathematical constants are stored separately in JSON files. In addition to the
DLMF/DRMF macro set, generic LaTeX also provides built-in commands for
mathematical functions, such as `\frac` or `\sqrt`. CAS translations for these
macros are defined in another JSON file.

Since the POM tagger assumes the existence of special formatted lexicon files
to extract information for unknown commands, the CSV files containing CAS
translation information has to be converted into lexicon files. Table 2 shows
a part of the lexicon entry for the DLMF/DRMF macro `\sin@@{z}`111The usage of
multiple @ symbols in Miller’s LaTeX macro set provides capability for
alternative presentations, such as $\sin(z)$ and $\sin\;z$ for one and two @
symbols respectively.. Translations to CAS are realized by patterns with
placeholders. The symbol `$i` indicates the $i$-th variable or parameter of
the macro.

Table 2: A lexicon entry. DLMF | `\sin@@{z}`
---|---
DLMF-Link | dlmf.nist.gov/4.14#E1
Maple | `sin($0)`
Mathematica | `Sin[$0]`

Our CAS translation process is structured recursively. A CAS translation of a
node will be delegated to a specialized class for certain kinds of nodes. Even
though our CAS translation process assumes semantic LaTeX with DLMF/DRMF
macros, we sometimes allow for extra information obtained from generic LaTeX
expressions. For instance, we distinguish between the following cases: (1) a
Latin letter is used for an elementary constant; (2) a generic LaTeX command
(such as the LaTeX command for a Greek letter) is used for an elementary
constant. In both cases, the program checks if there are known DLMF/DRMF
macros to represent the constant in semantic LaTeX. If so, we inform the user
of the DLMF/DRMF macro for the constant, but the Latin letter or LaTeX command
is not translated.

There are currently only three known Latin letters where this occurs, the
imaginary unit $i$, Euler’s number $e$, and Catalan’s constant $C$. If one
wants to translate the Latin letter to the constant, then one needs to use the
designated macro. In these three cases they are `\iunit`, `\expe` and
`\CatalansConstant`. Examples of LaTeX commands which may represent elementary
constants are $\pi$ and $\alpha$ which are often used to represent the ratio
of a circle’s circumference to its diameter, and the fine-structure constant
respectively which are `\cpi` and `\finestructure`. Hence, Latin and Greek
letters will be always translated as Latin and Greek letters respectively.

Figure 3: Flow diagram for translation between semantic LaTeX and a CAS
representations. The MLP is the only interface to the POM tagger and provides
all functionality for interaction with the results of the POM tagger (such as
analyzing the syntax tree and extracting information from the lexicon.)

The program consists of two executable JAR files. One organizes the
transformation from CSV files to lexicon files, while the other translates the
generated syntax tree to a CAS representation. Fig. 3 describes the CAS
translation process. The program currently supports forward CAS translations
for Maple and Mathematica.

##  4 Maple to semantic LaTeX translation

Maple has its own syntax and programming language, and users interact with
Maple by entering commands and expressions in Maple syntax. For example, the
mathematical expression
${\displaystyle\int_{0}^{\infty}(\pi+\sin(2x))/x^{2}dx,}$ would be entered in
Maple as

$\texttt{int((Pi+sin(2*x))/x\^{}2, x=0..infinity)}.$ (1)

In the sequel, we will refer to Maple syntax such as the syntactically correct
format (1) as (i) the 1D Maple representation. Maple also provides a (ii) 2D
representation (whose internal format is similar to MathML), and its display
is similar to the LaTeX rendering of the mathematical expression. In addition,
Maple uses two internal representations (iii) `Maple_DAG`, and (iv)
`Inert_Form` representation. Note that, even though DAG commonly refers to the
general graph theoretic/generic data structure, directed acyclic graph, in
Maple it has become synonymous with “Maple internal data structure,” whether
it actually represents a DAG or not.

Figure 4: Example Maple_DAG for (1).

In our translation from Maple to semantic LaTeX, only the Maple 1D and
`Inert_Fo`\-
`rm` representations are used. Programmatic access to the Maple kernel (its
internal data structures/commands) from other programming languages such as
Java or C is possible through a published application programming interface
(API) called OpenMaple [6, §14.3]. The OpenMaple Java API is used in this
project. Some of the functionality used includes (1) parsing a string in 1D
representation and converting it to its `Maple_DAG` and `Inert_Form`
representations (see below); (2) accessing elements of Maple’s internal data
structures; (3) performing manipulations on Maple data structures in the Maple
kernel.

Mathematical expressions in Maple are internally represented as `Maple_DAG`
representations. Fig. 4 illustrates the `Maple_DAG` representation of the 1D
Maple expression (1). The variable $x$ is stored only once in memory, and all
three occurrences of it refer to the same Maple object. This type of common
subexpression reuse is the reason why Maple data structures are organized as
`DAG`s and not as trees. In addition to mathematical expressions, Maple also
has a variety of other data structures (e.g., sets, lists, arrays, vectors,
matrices, tables, procedures, modules). The structure of a `Maple_DAG` is in
the form HeaderData1$\cdots$Datan . Header encodes both the type and the
length $n$ of the `Maple_DAG` and $Data_{1},$ $\dots,$ $Data_{n}$ are
`Maple_DAG`s (see [6, Appendix A.3]).

For this project, another tree-like representation that closely mirrors the
internal `Maple_DAG` representation (and can be accessed more easily through
the OpenMaple Java API) was chosen, the `Inert_Form`. The `Inert_Form` is
given by nested function calls of the form `_Inert_XXX`$\,\,(Data_{1},\ ...,\
Data_{n})$, where XXX is a type tag (see [6, Appendix A.3]), and $Data_{1},\
\dots,\ Data_{n}$ can themselves be `Inert_Form`s. In Maple, the `Inert_Form`
representation can be obtained via the command ToInert. For example, the
`Inert_Form` representation of the Maple expression (1) is

_Inert_FUNCTION( _Inert_NAME("Int"), _Inert_EXPSEQ( _Inert_PROD( _Inert_SUM(
_Inert_NAME("Pi"),

_Inert_FUNCTION( _Inert_NAME("sin"), _Inert_EXPSEQ( _Inert_PROD(
_Inert_NAME("x"),

_Inert_INTPOS(2))))), _Inert_POWER( _Inert_NAME("x"), _Inert_INTNEG(2)))

_Inert_EQUATION( _Inert_NAME("x"), _Inert_RANGE( _Inert_INTPOS(0),
_Inert_NAME("infinity"))))).

In order to facilitate access to the `Inert_Form` from the OpenMaple Java API,
the `Inert_Form` is converted to a nested list representation, where the first
element of each (sub)-list is an _Inert_XXX tag. For example, the Maple
equation x=0..infinity which contains the integration bounds (which is a
sub-`Maple_DAG` of Maple expression (1)), is as follows in the nested list
representation of the `Inert_Form`:
| [_Inert_EQUATION, [_Inert_NAME, "x"], [_Inert_RANGE, [_Inert_INTPOS, 0],
[_Inert_NAME, "infinity"]]].

###  4.1 Implementation

Figure 5: The program flow diagram explains the translation from Maple to
semantic LaTeX. The input string is parsed into a Maple object and Maple
procedures create a new internal form of the object and builds a nested list
from this new form. The CAS translation process assembles the semantic LaTeX
expression by translating each element recursively.

Our CAS translation engine enters the 1D Maple representation via the
OpenMaple API for Java [7] and converts the previously described `Inert_Form`
to a nested list representation. For Maple expressions, the nested list has a
tree structure. We have organized the backward translation in a similar
fashion to the forward translation (see §3).

Since Maple automatically tries to simplify input expressions, we implemented
some additional changes to prevent such simplifications and changes to the
input expression. We would prefer that the representation of a translated
expression remain as similar as possible to the input expression. This
facilitates user comprehension, as well as the debugging process, of the CAS
translation. Maple’s internal representation presents obstacles when trying to
keep an internal expression in the syntactical form of the input expression.
For instance, Maple performs automatic (1) simplification of input
expressions; (2) representation of radicals as powers with fractional
exponents (e.g., `\sqrt[5]{x^3}` represented as `x^{3/5}`); (3) representation
of negative terms as positive terms multiplied by `-1` (since Maple’s internal
structure has no primitives for negation or subtraction); and (4)
representation of division by a term as a multiplication of that term raised
to a negative power (since Maple’s internal structure has no primitives for
division).

To prevent automatic simplifications in Maple, one can enclose input
expressions between single quotes '…', also known as unevaluation quotes. This
does not prevent arithmetic simplifications but does prevent all other
simplifications to the input expression. For instance, if we have input
`sin(Pi)+2-1`, then the output is `1`; and if we have input '`sin(Pi)+2-1`',
then the output is `sin(Pi)+1`. By using unevaluation quotes, Maple does not
convert a radical to a power with fractional exponents, and the internal
representation remains an unevaluated `sqrt` (for square roots) or `root` (for
higher order radicals). Maple automatically represents a negative term such as
`-a` by a product `a*(-1)`. To resolve this we first switch the order of the
terms so that constants are in front, e.g., `(-1)*a`, and then check if the
leading constant is positive or negative. If it is negative, we remove the
multiplication and insert a negative sign in front of the term.

Maple’s rendering engine only changes negative powers to fractions if the
power is a ratio of integers, otherwise it keeps the exponent representation.
We only translate terms with negative integer exponents to fractions, and
otherwise retain the internal exponent representation. For this purpose, we
perform a preprocessing step (in Maple) that introduces a new DIVIDE element
in the tree representation. For instance, without the DIVIDE element the input
`(1/(x+3))^(-I)` produces `\left((3+x)^{-1}\right)^{-\iunit}`, and with the
DIVIDE element it produces `\left(\frac{1}{3+x}\right)^{-\iunit}`.

Using the above described manipulations, a typical translated expression is
very similar to the input expression. As an example, without any of the
techniques above, the input expression `cos(Pi*2)/sqrt((3*beta)/4-3*I)` would
be automatically simplified and changed internally, and the resulting semantic
LaTeX would be `2\idt(3\idt\beta+12\idt\iunit\idt(-1))^{-\frac{1}{2}}`.111\idt
is our semantic LaTeX macro which represents multiplication without any
corresponding presentation appearance. With unevaluation quotes, the CAS
translation produces
—`\cos@{\cpi\idt2}\idt\left(\sqrt{\beta\idt\frac34+\iunit\idt(-3)}\right)^{-1}`.
Furthermore, with our improvements for subtractions, we translate the radicand
to
`\frac{3}{4}\idt\beta-3\idt\iunit`, and with the DIVIDE element, we translate
the base with exponent `-1` as a fraction, and our translated expression is
—`\frac{\cos@{2\idt\cpi}}{\sqrt{\frac{3}{4}\idt\beta-3\idt\iunit}}`,
which is very similar to the input expression.

##  5 Evaluation

Here, we describe our approach for validating the correctness of our mappings,
as well as discuss the performance of our system obtained on a hand crafted
test set.

One validation approach is to take advantage of numerical evaluation using
software tools such as the DLMF Standard Reference Tables (DLMF Tables) [4],
CAS, and software libraries222See for instance:
http://dlmf.nist.gov/software.. These tools provide numerical evaluation for
special functions with their own unique features. One can validate forward CAS
translations by comparing numerical values in CAS to ground truth values.

Another validation approach is to use mathematical relations between different
functions. For instance, if we forward translate two functions separately, one
could determine if the relation between the two translated functions remains
valid. One example relation is for the Jacobi elliptic functions ${\rm sn}$,
${\rm cn}$, ${\rm dn}$, and the complete elliptic integral $K$ [3, Table
22.4.3], namely ${\rm sn}(z+K(k),k)={\rm cn}(z,k)/{\rm dn}(z,k),$ where
$z\in{\mathbb{C}},$ and $k\in(0,1)$. In the limit as $k\to 0$, this relation
produces $\sin@{z+\frac{\pi}{2}}=\cos(z),$ where $z\in{\mathbb{C}}$. The DLMF
provides relations such as these for many special functions. An alternative
relation is particularly helpful to validate CAS translations with different
positions of branch cuts, namely the relation between the parabolic cylinder
function $U$ and the modified Bessel function of the second kind [3,
(12.7.10)] $U(0,z)=\sqrt{z/(2\pi)}K_{1/4}(\tfrac{1}{4}z^{2}),$ where
$z\in{\mathbb{C}}$. Note that $z^{2}$ is no longer on the principal branch of
the modified Bessel function of the second kind when
${\mathrm{ph}}(z)\in(\frac{\pi}{2},\pi)$, but a CAS would still compute values
on the principal branch. Therefore, a CAS translation from
`\BesselK{\frac{1}{4}}@{\frac{1}{4}z^2}` to `BesselK(1/4,(1/4)*z^2)` is
incorrect if ${\mathrm{ph}}(z)\in(\frac{\pi}{2},\pi)$, even though the
equation is true in that domain. In order for the CAS to verify the formula in
that domain, it must use [3, (10.34.4)] for the function on the right-hand
side. Other validation tests may not be able to identify a problem with this
CAS translation.

One obstacle for such relations are the limitations of ever-improving CAS
simplification functions. Define the formula difference, as the difference
between the left- and right-hand sides of a mathematical formula. CAS simplify
for the Jacobi elliptic/trigonometric relation should produce `0`, but might
have more difficulties with the parabolic cylinder function relation. However,
CAS simplify functions work more effectively on round trip tests.

###  5.1 Round trip tests

One of the main techniques we use to validate CAS translations are round trip
tests which take advantage of CAS simplification functions. Since we have
developed CAS translations between semantic LaTeX $\leftrightarrow$ Maple,
round trip tests are evaluated in Maple. Maple’s simplification function is
called `simplify`. Two expressions are symbolically equivalent, if `simplify`
returns zero for the formula difference. On the other hand, it is not possible
to disprove the equivalence of the expressions when the function returns
something different to zero.

Our round trip tests start either from a valid semantic LaTeX expression or
from a valid Maple expression. A CAS translation from the start representation
to the other representation and back again is called one cycle. A round trip
reaches a fixed point, when the string representation is identical to its
previous string representation. The round trip test concludes when it reaches
a fixed point in both representations. Additionally, we test if the fixed
point representation in Maple is symbolically equivalent to the input
representation by simplifying the differences between both of these with the
Maple simplify function. Since there is no mathematical equivalence tester for
LaTeX expressions (neither generic nor semantic LaTeX), we manually verify
LaTeX representations for our test cases by rendering the LaTeX.

Table 3: A round trip test reach a fixed point. step | semantic LaTeX/Maple representations
---|---
0 |  `\frac{\cos@{a\Theta}}{2}`
1 | `(cos(a*Theta))/(2)`
2 | `\frac{1}{2}\idt\cos@{a\idt\Theta}`
3 | `(1)/(2)*cos(a*Theta)`

As shown in §4.1, prior to backward translation, in round trip testing, there
will be differences between input and output Maple representations. After
adapting these changes, and assuming the functions exist in both semantic
LaTeX and CAS, the round trip test should reach a fixed point. In fact, we
reached a fixed point in semantic LaTeX after one cycle and in Maple after
$1\tfrac{1}{2}$ cycles (see Table 3 for an example) for most of the cases we
tried. If the input representation is already identical to Maple’s
representation, then the fixed point will be reached after at most a half
cycle.

One example exception is for CAS translations which introduce additional
function compositions on arguments. For instance, Legendre’s incomplete
elliptic integrals [3, (19.2.4-7)] are defined with the amplitude $\phi$ in
the first argument, while Maple’s implementation takes the trigonometric sine
of the amplitude as the first argument. For instance, one has the CAS
translations `\EllIntF@{\phi}{k}` $\mapsto$ `EllipticF(sin(phi),k)`, and
`\EllIntF@{\asin@{z}}{k}` $\mapsfrom$ `EllipticF(z,k)`. These CAS translations
produce an infinite chain of sine and inverse sine function calls. Because
round trip tests prevent simplification during the translation process (see
§4.1), Maple is not used to simplify the chain until the round trip test is
concluded.

###  5.2 Summary of evaluation techniques

Equivalence tests for special function relations are able to verify relations
in CAS as well as identify hidden problems such as differences in branch cuts
and CAS limitations. We use the simplify method to test equivalences. For the
relations in §5, CAS simplify for the Jacobi elliptic function example yields
`0`. Furthermore, a spectrum of real, complex, and complex conjugate numerical
values for $z$ and $k\in(0,1)$ the formula difference converges to zero for an
increasing precision. If simplification returns something other than zero, we
can test the equivalence for specific values. For the Bessel function
relation, the formula difference for $z=1+i$ converges to zero for increasing
precision, but does not converge to zero if $z=-1+i$. However, using analytic
continuation [3, (10.34.4)], it does converges to zero. Clearly, the numerical
evaluation test is also able to locate branch cut issues in the CAS
translation. Furthermore, this provides a very powerful debugging method for
our translation as well as for CAS functionality. This was demonstrated by
discovering an overall sign error in DLMF equation [3, (14.5.14)].

Round trip tests are also useful for identifying syntax errors in the semantic
LaTeX since the CAS translation then fails. The simplification procedure is
improved for round trip tests, because it only needs to simplify similar
expressions with identical function calls. However, this approach is not able
to identify hidden problems that a CAS translation might need to resolve in
order to be correct, if the round trip test has not reached a fixed point.
Other than with the round trip test approach, we have not discovered any
automated tests for backward CAS translations. We have evaluated 37 round trip
test cases which produce a fixed point, similar to that given in Table 3.
These use formulae from the DLMF/DRMF and produce a difference of the left-
and right-hand sides equaling `0`.

We have created a test dataset333We are planning to make the dataset available
from http://drmf.wmflabs.org. of 4,165 semantic LaTeX formulae, extracted from
the DLMF. We translated each test case to a representation in Maple and used
Maple’s simplify function on the formula difference to verify that the
translated formulae remain valid. Our forward translation tool (§3) was able
to translate 2,232 (approx. 53.59%) test cases and verify 477 of these. Pre-
conversion improved the effectiveness of simplify and were used to convert the
translated expression to a different form before simplification of the formula
difference. We used conversions to exponential and hypergeometric form and
expanded the translated expression. Pre-conversion increased the number of
formulae verified to 662 and 1,570 test cases were translated but not
verified. The remaining 1,933 test cases were not translated, because they
contain DLMF/DRMF macros without a known translation to Maple (987 cases),
such as the $q$-hypergeometric function [3, (17.4.1)] (in 58 cases), or an
error appeared during the translation or verification process (639 cases).
Furthermore, 316 cases were ignored, because they did not contain enough
semantic information to provide a translation or the test case was not a
relation. It is interesting to note that we were able to enhance the semantics
of 74 Wronskian relations by rewriting the macro so that it included the
variable that derivatives are taken with respect to as a parameter. A similar
semantic enhancement is possible for another 186 formulae where the
potentially ambiguous prime notation ‘ ' ’ is used for derivatives.
 Acknowledgements 444The mention of specific products, trademarks, or brand
names is for purposes of identification only. Such mention is not to be
interpreted in any way as an endorsement or certification of such products or
brands by the National Institute of Standards and Technology, nor does it
imply that the products so identified are necessarily the best available for
the purpose. All trademarks mentioned herein belong to their respective
owners. We are indebted to Wikimedia Labs, the XSEDE project,
Springer-Verlag, the California Institute of Technology, and Maplesoft for
their contributions and continued support. We would also like to thank Eric
Weisstein for supplying the Wolfram eCF dataset, Annie Cuyt, Franky
Backeljauw, and Stefan Becuwe for supplying the University of Antwerp CFSF
Maple dataset, and Adri Olde Daalhuis for discussions related to complex
multivalued functions.

## References

  * [1] H.S. Cohl et al.  “Growing the Digital Repository of Mathematical Formulae with Generic LaTeX Sources”  In _Intelligent Computer Mathematics, Lecture Notes in Artificial Intelligence 9150_ Springer, 2015, pp. 280–287
  * [2] Howard S. Cohl et al.  “Semantic Preserving Bijective Mappings of Mathematical Formulae between Document Preparation Systems and Computer Algebra Systems”  In _10th Conference on Intelligent Computer Mathematics_ , 2017
  * [3] “NIST Digital Library of Mathematical Functions” F.W.J. Olver, A.B. Olde Daalhuis, D.W. Lozier, B.I. Schneider, R.F. Boisvert, C.W. Clark, B.R. Miller and B.V. Saunders, eds., http://dlmf.nist.gov/, Release 1.0.14 of 2017-12-21, 2017  URL: %5Chref%7Bhttp://dlmf.nist.gov/%7D%7Bhttp://dlmf.nist.gov%7D
  * [4] “DLMF Standard Reference Tables” Joint project of NIST ACMD and U. Antwerp’s CMA Group, seen 3/2017, http://dlmftables.uantwerpen.be/, 2017
  * [5] D. Ginev  “LaTeXML-Plugin-MathSyntax” seen 3/2017, https://github.com/dginev/LaTeXML-Plugin-MathSyntax/ URL: https://github.com/dginev/LaTeXML-Plugin-MathSyntax/
  * [6] L. Bernardin, P. Chin, P. DeMarco, K.O. Geddes, D.E.G. Hare, K.M. Heal, G. Labahn, J.P. May, J. McCarron, M.B. Monagan, D. Ohashi, S.M. Vorkoetter  “Maple Programming Guide”, 2016  Maplesoft, a division of Waterloo Maple Inc.
  * [7] Maplesoft  “OpenMaple API” Since Maple 9, seen 3/2017, https://www.maplesoft.com/support/help/Maple/view.aspx?path=OpenMaple
  * [8] Maplesoft  “Produce output suitable for LaTeX 2e” Since version Maple 18, http://www.maplesoft.com/support/help/Maple/view.aspx?path=latex
  * [9] B.R. Miller  “Drafting DLMF Content Dictionaries”, Talk presented at the OpenMath workshop of the 9th Conference on Intelligent Computer Mathematics CICM 2016., 2016
  * [10] B.R. Miller  “LaTeXML: A LaTeX to XML Converter” seen 3/2017, http://dlmf.nist.gov/LaTeXML/ URL: http://dlmf.nist.gov/LaTeXML/
  * [11] B.R. Miller and A. Youssef  “Technical Aspects of the Digital Library of Mathematical Functions”  In _Annals of Mathematics and Artificial Intelligence_ 38.1-3 Kluwer Academic Publishers, 2003, pp. 121–136
  * [12] R. and M.  “Mathematical Language Processing Project”  In _CICM Workshops_ 1186, CEUR Workshop Proceedings  CEUR-WS.org, 2014
  * [13] Wolfram Research  “Generating and Importing TeX”, https://reference.wolfram.com/language/tutorial/GeneratingAndImportingTeX.html
  * [14] M. Schubotz and G. Wicke  “Mathoid: Robust, Scalable, Fast and Accessible Math Rendering for Wikipedia”  In _Intelligent Computer Mathematics_ 8543, Lecture Notes in Computer Science, 2014, pp. 224–235
  * [15] M. Schubotz et al.  “Semantification of Identifiers in Mathematics for Better Math Information Retrieval”  In _The 39th Annual ACM Special Interest Group on Information Retrieval, Pisa, Tuscany, Italy_ , 2016
  * [16] A. Youssef  “Part-of-Math Tagging and Applications”  In _Submitted to the 10th Conference on Intelligent Computer Mathematics (CICM 2017), Edinburgh, Scotland, July 2017_ , 2017

Listing 1: Use the following BibTeX code to cite this article