# Toward Developing a Multimodal Multi-party Hindi Humorous Dataset for Humor
Recognition in Conversations

Dushyant Singh Chauhan†, Gopendra Vikram Singh†, Navonil Majumder+, Amir
Zadeh∓,
Asif Ekbal†, Pushpak Bhattacharyya†, Louis-philippe Morency∓, and Soujanya
Poria+ † Department of Computer Science & Engineering
Indian Institute of Technology Patna India
{1821CS17, gopendra_1921CS15,asif<EMAIL_ADDRESS>
∓ Language Technology Institute, Carnegie Mellon University, Pittsburgh USA
{abagherz<EMAIL_ADDRESS>
+ ISTD, Singapore University of Technology and Design Singapore
{navonil_majumder<EMAIL_ADDRESS>
(2021)

# M2H2: A Multimodal Multiparty Hindi Dataset For Humor Recognition in
Conversations

Dushyant Singh Chauhan†, Gopendra Vikram Singh†, Navonil Majumder+, Amir
Zadeh∓,
Asif Ekbal†, Pushpak Bhattacharyya†, Louis-philippe Morency∓, and Soujanya
Poria+ † Department of Computer Science & Engineering
Indian Institute of Technology Patna India
{1821CS17, gopendra_1921CS15,asif<EMAIL_ADDRESS>
∓ Language Technology Institute, Carnegie Mellon University, Pittsburgh USA
{abagherz<EMAIL_ADDRESS>
+ ISTD, Singapore University of Technology and Design Singapore
{navonil_majumder<EMAIL_ADDRESS>
(2021)

###### Abstract.

Humor recognition in conversations is a challenging task that has recently
gained popularity due to its importance in dialogue understanding, including
in multimodal settings (i.e., text, acoustics, and visual). The few existing
datasets for humor are mostly in English. However, due to the tremendous
growth in multilingual content, there is a great demand to build models and
systems that support multilingual information access. To this end, we propose
a dataset for Multimodal Multiparty Hindi Humor (M2H2) recognition in
conversations containing 6,191 utterances from 13 episodes of a very popular
TV series ”Shrimaan Shrimati Phir Se”. Each utterance is annotated with
humor/non-humor labels and encompasses acoustic, visual, and textual
modalities. We propose several strong multimodal baselines and show the
importance of contextual and multimodal information for humor recognition in
conversations. The empirical results on M2H2 dataset demonstrate that
multimodal information complements unimodal information for humor recognition.
The dataset and the baselines are available at http://www.iitp.ac.in/~ai-nlp-
ml/resources.html and https://github.com/declare-lab/M2H2-dataset.

Deep learning, Humor, Hindi dataset, Multimodal multiparty dataset

††copyright: acmcopyright††journalyear: 2021††doi:
10.1145/1122445.1122456††conference: ICMI ’21: 23rd ACM International
Conference on Multimodal Interaction; October 18–22, 2021; Montreal,
Canada††booktitle: ICMI ’21: 23rd ACM International Conference on Multimodal
Interaction, Montreal, Canada††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06††ccs: Computing methodologies††ccs: Machine
learning††ccs: Machine learning approaches††ccs: Neural networks

##  1\. Introduction

Humor (Warren et al., 2018) is defined as the nature of experiences to induce
laughter and provide amusement. it can be implicit as well as explicit.
Implicit humor may be expressed with neutral facial expressions and a polite
tone, while explicit humor can be conveyed with laughter or funny facial
expressions. Explicit humor may be easy to detect in comparison to implicit
humor because of laughter or facial expression. So, while text (Bolkan et al.,
2018; Scheel, 2017; Boykoff and Osnes, 2019; Khandelwal et al., 2018; Sane et
al., 2019; Gal, 2019; Weller and Seppi, 2019; Blinov et al., 2019; Castro et
al., 2017; Zhao et al., 2019; Garimella et al., 2020) alone may not always be
enough to understand humor but if the utterance is multimodal in nature (Hasan
et al., 2019; Fallianda et al., 2018; Ritschel et al., 2019, 2020; Mirnig et
al., 2017; Piata, 2020; Song et al., 2021; Vasquez and Aslan, 2021; Sabur et
al., 2020; Veronika, 2020; Yang et al., 2019) and is accompanied with a video
of the facial expressions and tone of the speaker, it can be easily understood
that the utterance contains humor.

Humor recognition is especially challenging in many Indian languages because
of the following reasons: unavailability of resources and tools; morphological
richness; free word order etc. In our current work, we focus on Hindi, one of
the most popularly spoken languages in India. In terms of the number of
speakers, Hindi ranks third111https://currentaffairs.adda247.com/hindi-
ranks-3rd-most-spoken-language-in-the-world/ all over in the world. In
comparison to English, Hindi is much richer in terms of morphology. In Hindi,
all the nouns and adjectives have genders, and the verb agrees in number and
gender with them. While, English is free of that confounding factor of the
gender of a common noun, and is also free of the further complication that the
verb has to agree with the gender of the noun or pronoun governing it. These
challenges motivated us to do some qualitative research work for Humor
detection in Hindi.

In Hindi, there is no available dataset for humour detection (neither unimodal
nor multimodal). In our work, we introduce a new dataset, named as M2H2, which
includes not only textual dialogues but also their corresponding visual and
audio counterparts. The main contributions of our proposed research are as
follows:

  * •

We propose a dataset for Multimodal Multi-party Hindi Humor recognition in
conversations. There are 6,191 utterances in the M2H2 dataset;

  * •

We set up two strong baselines, viz., DialogueRNN (Majumder et al., 2019) and
bcLSTM (Poria et al., 2017) with MISA (Hazarika et al., 2020) for multimodal
fusion. Both DialogueRNN and bcLSTM are contextual models.

  * •

Empirical results on the M2H2 dataset show the efficacy of multimodal
information over only textual information for humor detection.

##  2\. Dataset

Figure 1. Some samples from dataset.

We gathered samples from a famous TV show ”Shrimaan Shrimati Phir Se”222Total
of 4.46 hours in length and annotated them manually. We make groups of these
samples (utterances) based on their context into scenes. Each utterance in
each scene consists of a label indicating humor of that utterance i.e.,
humor333There are 1/$3^{rd}$ utterances which are humorous. or non-humor.
Besides, each utterance is also annotated with its speaker and listener
information. In multiparty conversation, listener identification poses a great
challenge. In our dataset, we define the listener as the party in the
conversation to whom the speaker is replying. Each utterance in each scene is
coupled with its context utterances, which are preceding turns by the speakers
participating in the conversation. It also contains multi-party conversations
that are more challenging to classify than dyadic variants. We show the
dataset statistics in Table 1.

Description | Statistic
---|---
#Scenes | 136
#Utterances | 6191
Min #Utterances per Scene | 8
Max #Utterances per Scene | 155
#Utterances per Scene | 45.52
#Words | 34472
#Unique words | 5454
#Humor class | 2089
#Non-humor class | 4102
#Speakers | 41
Total duration | 4.46 Hours

Table 1. Dataset statistics.

###  2.1. Challenges

We crawled the video file and transcription file using YouTube for the TV show
”Shrimaan Shrimati Phir Se”. The downloaded Hindi transcription file had some
errors. So, we hired transcribers, who were experts in the Hindi language.
They wrote Hindi utterances based on the audio files.

As the context is based on the scenes, we divide the whole dataset into scenes
based on its context. Each utterance in each scene consists of a humor label
(i.e., humor or non-humor) and speaker and listener information.

###  2.2. Annotation Guidelines

We employ three Ph.D. students with high proficiency in Hindi and with prior
experience in labeling Humor and Non-humor in conversational settings.

The guidelines for annotation, along with some examples (c.f. Figure 1), were
explained to the annotators before starting the annotation process. The
annotators were asked to annotate every utterance with humor/non-humor and
corresponding speaker and listener information. We also annotated 20
annotations by ourselves and set it as a quality checker to evaluate the
quality of the annotators. A majority voting scheme was used for selecting the
final humor/non-humor. We achieve an overall Fleiss’ (Fleiss, 1971) kappa
score of 0.84, which is considered to be reliable.

##  3\. Baseline Methods

In this section, we provide strong benchmarks for M2H2 dataset. We employ
three strong baselines frameworks for humor detection.

###  3.1. Strong Baseline for Multimodal Fusion: MISA

MISA (Modality-Invariant and-Specific Representations for Multimodal Sentiment
Analysis) (Hazarika et al., 2020) has two main stages: modality representation
learning and modality fusion. It divides each modality into two sub-spaces.
The first subspace is modality-invariant, in which representations from
different modalities learn to share similarities and minimize the distance
between them. The second subspace is modality-specific, which is unique to
every modality and contains its distinguishing characteristics. These
representations give a comprehensive perspective of multimodal data, which is
utilized for fusion and task prediction. The model shows the state-of-the-art
(SOTA) for MOSI (Zadeh et al., 2016) and MOSEI (Zadeh et al., 2018) datasets.

###  3.2. Strong Conversation Classification Baseline #1: DialogueRNN

DialogueRNN (Majumder et al., 2019) is a multi-party framework tailored for
modelling emotions and sentiments in conversations. In DialogueRNN, they
explained a novel recurrent neural network-based system that tracks of each
party state during a discussion and uses this knowledge to classify emotions.
It employs three levels of gated recurrent units (GRU) to represent the
conversational context in order to accurately recognize emotions, intensity,
and attitudes in a discussion. They showed the SOTA performance on two
different datasets i.e., IEMOCAP (Busso et al., 2008) and AVEC (Schuller et
al., 2012).

###  3.3. Strong Conversation Classification Baseline #2: bcLSTM

bcLstm (Poria et al., 2017) is a bidirectional contextual LSTM. Bi-directional
LSTMs are formed by stacking two uni-directional LSTMs with opposing
directions. As a result, an utterance can learn from utterances that come
before and after it in the video which is of course context.

Leveraging the above three strong baselines, we formulate two setups, viz.
MISA with DialogueRNN (MISA+DialogueRNN) and MISA with bcLSTM (MISA+bcLSTM).
Here, MISA acts as a fusion model while DialogueRNN and bcLSTM are contextual
models for conversation classification. DialogueRNN and bcLSTM have shown
excellent performance in different conversation classification tasks such as
Emotion Recognition Conversation. For MISA+DialogueRNN, we first pass
multimodal inputs through MISA and obtain the fused features. These fused
features are then passed through DialogueRNN for humor classification.
Similarly, for MISA+bcLSTM, We first pass multimodal inputs through MISA and
obtain the fused features. Then, these fused features are passed through
bcLSTM for humor classification.

###  3.4. Feature Extraction

For textual features, we take the pre-trained 300-dimensional Hindi fastText
embeddings (Joulin et al., 2016). For visual feature extraction, we use
3D-ResNeXt-101444https://github.com/kaiqiangh/extracting-video-features-
ResNeXt (Xie et al., 2017) which is pre-trained on Kinetics at a rate of 1.5
features per second and a resolution of 112. While, we use openSMILE (Eyben et
al., 2010) for acoustic feature extraction.
openSMILE555https://github.com/audeering/opensmile can extract Low-Level
Descriptors (LLD) and change them using different filters, functions, and
transformations. We use a tonal low-level features group of openSMILE to
extract the features.

##  4\. Experiments and Analysis

###  4.1. Experimental Setup

We evaluate our proposed model on the M2H2 dataset. We perform five-fold
cross-validation for experiments. Empirically, We take five666Baseline models
give the best result at five. utterances as context for a particular
utterance.

We implement our proposed model on the Python-based
PyTorch777https://pytorch.org/ deep learning library. As the evaluation
metric, we employ precision (P), recall (R), and F1-score (F1) for humor
recognition. We use Adam as an optimizer, Softmax as a classifier for humor
detection, and the categorical cross-entropy as a loss function.

###  4.2. Results and Analysis

We evaluate our proposed architecture with all the possible input combinations
(c.f. Table 2) i.e. unimodal (T, A, V), bimodal (T+V, T+A, A+V) and trimodal
(T+V+A).

For MISA+DialogueRNN, trimodal achieves the best precision of 71.21%
($16.05$888maximum improvement over unimodal (V) points $\uparrow$ and
$9.51$999maximum improvement over bimodal (A+V) points $\uparrow$), recall of
72.11% ($14.79$ points $\uparrow$ and $8.9$ points $\uparrow$) and F1-score of
71.67% ($14.93$ points $\uparrow$ and $9.23$ points $\uparrow$). We observe
that trimodal performs better than the unimodal and bimodal. We show the
results in Table 2.

| MISA+DialogueRNN | MISA+bcLSTM
---|---|---
Labels | P | R | F1 | P | R | F1
T | 67.11 | 68.21 | 67.65 | 66.27 | 66.51 | 66.38
A | 57.91 | 59.10 | 58.52 | 57.51 | 58.92 | 58.21
V | 55.16 | 57.32 | 56.74 | 53.13 | 55.21 | 54.19
T+V | 70.03 | 70.61 | 70.31 | 67.74 | 68.89 | 68.31
T+A | 69.70 | 69.90 | 69.63 | 67.41 | 68.12 | 67.75
A+V | 61.70 | 63.21 | 62.44 | 59.61 | 61.23 | 60.40
T+V+A | 71.21 | 72.11 | 71.67 | 69.04 | 69.83 | 69.43

Table 2. Experiment results for MISA+DialogueRNN and MISA+bcLSTM.

Similarly, for MISA+bcLSTM, trimodal achieves the best precision of 69.04%
($15.91$101010maximum improvement over unimodal (V) points $\uparrow$ and
$9.43$111111maximum improvement over bimodal (A+V) points $\uparrow$), recall
of 69.83% ($14.62$ points $\uparrow$ and $8.6$ points $\uparrow$) and F1-score
of 69.43% ($15.24$ points $\uparrow$ and $9.03$ points $\uparrow$). We observe
that trimodal performs better than the unimodal and bimodal.

Correct Prediction
---
|  |  |  | Predicted
|  |  |  | Unimodal | Multimodal
|  Hindi Utterances |  English Utterances | Actual | (T) | (T+V+A)
1 |  ar bYAp m m tMh Kd l k jAtA. |  Oh in old age I would have taken you myself. | humor | non-humor | humor
2 |  y EdlzbA khtA h nA Ek EpCl j06wm m vo CoVml moVml kroXpEt kA eklOtA bVA TA, rAiV? |  This Dilruba says that he was the only son of Chhotumal Motumal Crorepati in his previous life, right? | non-humor | humor | non-humor
3 |  -VAP vAlo! mrA mtlb |  Staff guys! I mean | humor | non-humor | humor
4 |  kC yAd aAyA? kC yAd aAyA? |  Remember anything? remember something? | non-humor | humor | non-humor
5 |  jAt vC4w esF afB bAt nhF krt. |  Do not do such inauspicious things while leaving. | humor | non-humor | humor
Incorrect Prediction
1 |  y log mJ bht mAr rh h. |  These guys are beating me hard | humor | non-humor | non-humor
2 |  hAy, VoVo! |  hey, Toto! | non-humor | non-humor | humor
3 |  y EpkF lAI. |  Pinky brought this | humor | humor | non-humor
4 |  ar BAI, mrF aslF vAlF gn yhF CV gi. |  Hey brother, my original gun is left here | non-humor | humor | humor
5 |  kOn sF aslF h kOn sF nklF h. |  which is real which is fake | non-humor | non-humor | humor

Table 3. Error analysis: Some correct and incorrect predicted samples.

###  4.3. Ablation Study

We also perform an ablation study (c.f. Table 4) to show the efficacy of
contextual models. We perform experiments on MISA (i.e., without contextual
information). As per the result, we can see when MISA is used with contextual
models (bcLSTM or DialogueRNN) it shows significant improvement rather than
when it uses alone.

Setup | P | R | F1
---|---|---|---
MISA+DialogueRNN | 71.21 | 72.11 | 71.67
MISA+bcLSTM | 69.04 | 69.83 | 69.43
MISA | 67.61 | 70.26 | 68.90

Table 4. Ablation study

###  4.4. Error Analysis

In this section, we perform error analysis for our baseline system i.e,
MISA+DialogueRNN. We take some samples which are correctly and incorrectly
predicted by the baseline model to analyze the model’s strengths and
weaknesses.

We show a few samples121212For the global audience, we also translate these
Hindi utterances into English. (c.f. Table 4.2) which are correctly predicted
by the multimodal setup, but incorrectly predicted by the unimodal setup. For
example, second utterance of the corrected prediction in Table 4.2, y EdlzbA
khtA h nA Ek EpCl j06wm m vo CoVml moVml kroXpEt kA eklOtA bVA TA, rAiV? (This
Dilruba says that he was the only son of Chhotumal Motumal Crorepati in his
previous life, right?) Gokhale is giving idea to Keshav with a polite tone and
neutral facial expression. The actual label is non-humor but the unimodal
model (text) predicts it as humorous instance while the multimodal model
predicts the label correctly because it is clear from the acoustic and visual
features that there were no humorous punches but only normal talk. We show
some visual frames in Figure 2 which clearly show the neutral facial
expression.

Figure 2. Some visual frames of second utterance of correct prediction in
Table 4.2.

On the other hand, we also show some samples, where the multimodal model fails
to predict the correct label. For example, first utterance of incorrect
prediction in Table 4.2, y log mJ bht mAr rh h. (these guys are beating me
hard) has the actual label as humor. But, the multimodal fails to predict the
correct label because of the sad tone and the sad visual expression of Dilruba
(EdlzbA). We show some visual frames in Figure 3 which clearly shows the sad
facial expression of Dilruba.

Figure 3. Some visual frames of first utterance of incorrect prediction in
Table 4.2.

##  5\. Conclusion

In this paper, we have presented a novel Multimodal Multi-party Hindi Humor
(M2H2) recognition dataset for conversations (4.46 hours in length). We
employed two strong baseline setups, viz. MISA with DialogueRNN and MISA with
bcLSTM. MISA is a fusion model while DialogueRNN and bcLSTM are contextual
models. We believe this dataset will also be useful as a useful resource for
both conversational humor recognition and multimodal artificial intelligence.
Empirical results on the M2H2 dataset demonstrate that the multimodal
baselines yield better performance over the unimodal framework.

In the future, we would like to extend our work towards the multi-party
dialogue generation in Hindi with the help of humor and speaker information.

## Acknowledgement

Dushyant Singh Chauhan acknowledges the support of Prime Minister Research
Fellowship (PMRF), Govt. of India. Asif Ekbal acknowledges the Young Faculty
Research Fellowship (YFRF), supported by Visvesvaraya PhD scheme for
Electronics and IT, Ministry of Electronics and Information Technology
(Meit/8Y), Government of India, being implemented by Digital India Corporation
(formerly Media Lab Asia). Soujanya Poria is supported by the SUTD SRG grant
no. T1SRIS19149.

## References

  * (1)
  * Blinov et al. (2019) Vladislav Blinov, Valeria Bolotova-Baranova, and Pavel Braslavski. 2019\.  Large dataset and language model fun-tuning for humor recognition. In _Proceedings of the 57th annual meeting of the association for computational linguistics_. 4027–4032.
  * Bolkan et al. (2018) San Bolkan, Darrin J Griffin, and Alan K Goodboy. 2018\.  Humor in the classroom: the effects of integrated humor on student learning.  _Communication Education_ 67, 2 (2018), 144–164.
  * Boykoff and Osnes (2019) Maxwell Boykoff and Beth Osnes. 2019.  A laughing matter? Confronting climate change through humor.  _Political Geography_ 68 (2019), 154–163.
  * Busso et al. (2008) Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008.  IEMOCAP: Interactive emotional dyadic motion capture database.  _Language resources and evaluation_ 42, 4 (2008), 335\.
  * Castro et al. (2017) Santiago Castro, Luis Chiruzzo, Aiala Rosá, Diego Garat, and Guillermo Moncecchi. 2017\.  A crowd-annotated spanish corpus for humor analysis.  _arXiv preprint arXiv:1710.00477_ (2017).
  * Eyben et al. (2010) Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010\.  Opensmile: the munich versatile and fast open-source audio feature extractor. In _Proceedings of the 18th ACM international conference on Multimedia_. ACM, 1459–1462.
  * Fallianda et al. (2018) Fallianda Fallianda, Rani Yuni Astiti, and Zulvy Alivia Hanim. 2018.  Analyzing Humor in Newspaper Comic Strips Using Verbal-Visual Analysis.  _Lingua Cultura_ 12, 4 (2018), 383–388.
  * Fleiss (1971) Joseph L. Fleiss. 1971\.  Measuring nominal scale agreement among many rater.  _Psychological Bulletin_ 76 (1971), 378–382.
  * Gal (2019) Noam Gal. 2019.  Ironic humor on social media as participatory boundary work.  _New Media & Society_ 21, 3 (2019), 729–749.
  * Garimella et al. (2020) Aparna Garimella, Carmen Banea, Nabil Hossain, and Rada Mihalcea. 2020.  ” Judge me by my size (noun), do you?”YodaLib: A Demographic-Aware Humor Generation Framework.  _arXiv preprint arXiv:2006.00578_ (2020).
  * Hasan et al. (2019) Md Kamrul Hasan, Wasifur Rahman, Amir Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency, et al. 2019\.  Ur-funny: A multimodal language dataset for understanding humor.  _arXiv preprint arXiv:1904.06618_ (2019).
  * Hazarika et al. (2020) Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. 2020\.  MISA: Modality-Invariant and-Specific Representations for Multimodal Sentiment Analysis.  _arXiv preprint arXiv:2005.03545_ (2020).
  * Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016.  FastText.zip: Compressing text classification models.  _arXiv preprint arXiv:1612.03651_ (2016).
  * Khandelwal et al. (2018) Ankush Khandelwal, Sahil Swami, Syed S Akhtar, and Manish Shrivastava. 2018.  Humor detection in english-hindi code-mixed social media content: Corpus and baseline system.  _arXiv preprint arXiv:1806.05513_ (2018).
  * Majumder et al. (2019) Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2019.  DialogueRNN: An Attentive RNN for Emotion Detection in Conversations. In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , Vol. 33. 6818–6825.
  * Mirnig et al. (2017) Nicole Mirnig, Gerald Stollnberger, Manuel Giuliani, and Manfred Tscheligi. 2017.  Elements of humor: How humans perceive verbal and non-verbal aspects of humorous robot behavior. In _Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction_. 211–212.
  * Piata (2020) Anna Piata. 2020\.  Stylistic humor across modalities: The case of Classical Art Memes.  _Internet Pragmatics_ 3, 2 (2020), 174–201.
  * Poria et al. (2017) Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. 2017.  Context-dependent sentiment analysis in user-generated videos. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Vol. 1. 873–883.
  * Ritschel et al. (2019) Hannes Ritschel, Ilhan Aslan, David Sedlbauer, and Elisabeth André. 2019.  Irony man: augmenting a social robot with the ability to use irony in multimodal communication with humans.  (2019).
  * Ritschel et al. (2020) Hannes Ritschel, Thomas Kiderle, Klaus Weber, Florian Lingenfelser, Tobias Baur, and Elisabeth André. 2020.  Multimodal joke generation and paralinguistic personalization for a socially-aware robot. In _International Conference on Practical Applications of Agents and Multi-Agent Systems_. Springer, 278–290.
  * Sabur et al. (2020) Andy Jefferson Sabur, Retno Purwani Sari, and Tatan Tawami. 2020\.  Identification Of The Multimodal Structure Of Humor In An Animated Superhero Film. In _International Conference on Language, Linguistics, and Literature (COLALITE) 2020_.
  * Sane et al. (2019) Sushmitha Reddy Sane, Suraj Tripathi, Koushik Reddy Sane, and Radhika Mamidi. 2019.  Deep learning techniques for humor detection in Hindi-English code-mixed tweets. In _Proceedings of the Tenth Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis_. 57–61.
  * Scheel (2017) Tabea Scheel. 2017\.  Definitions, theories, and measurement of humor.  In _Humor at work in teams, leadership, negotiations, learning and health_. Springer, 9–29.
  * Schuller et al. (2012) Björn Schuller, Michel Valster, Florian Eyben, Roddy Cowie, and Maja Pantic. 2012.  Avec 2012: the continuous audio/visual emotion challenge. In _Proceedings of the 14th ACM international conference on Multimodal interaction_. 449–456.
  * Song et al. (2021) Kwangok Song, Kyle M Williams, Diane L Schallert, and Alina Adonyi Pruitt. 2021.  Humor in multimodal language use: Students’ Response to a dialogic, social-networking online assignment.  _Linguistics and Education_ 63 (2021), 100903.
  * Vasquez and Aslan (2021) Camilla Vasquez and Erhan Aslan. 2021.  “Cats be outside, how about meow”: multimodal humor and creativity in an internet meme.  _Journal of Pragmatics_ 171 (2021), 101–117.
  * Veronika (2020) Zhigailova Veronika. 2020\.  Multimodal Discourse Analysis of Humor in Picture Books for Children.  (2020).
  * Warren et al. (2018) Caleb Warren, Adam Barsky, and A Peter McGraw. 2018\.  Humor, comedy, and consumer behavior.  _Journal of Consumer Research_ 45, 3 (2018), 529–552.
  * Weller and Seppi (2019) Orion Weller and Kevin Seppi. 2019.  Humor detection: A transformer gets the last laugh.  _arXiv preprint arXiv:1909.00252_ (2019).
  * Xie et al. (2017) Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. 2017.  Aggregated residual transformations for deep neural networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 1492–1500.
  * Yang et al. (2019) Zixiaofan Yang, Lin Ai, and Julia Hirschberg. 2019.  Multimodal Indicators of Humor in Videos. In _2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR)_. IEEE, 538–543.
  * Zadeh et al. (2018) Amir Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018\.  Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ (Melbourne, Australia). Association for Computational Linguistics, 2236–2246.  http://aclweb.org/anthology/P18-1208
  * Zadeh et al. (2016) A. Zadeh, R. Zellers, E. Pincus, and L. P. Morency. 2016\.  Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages.  _IEEE Intelligent Systems_ 31, 6 (Nov 2016), 82–88.  https://doi.org/10.1109/MIS.2016.94
  * Zhao et al. (2019) Zhenjie Zhao, Andrew Cattle, Evangelos Papalexakis, and Xiaojuan Ma. 2019.  Embedding lexical features via tensor decomposition for small sample humor recognition. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.


# Investigating polaron transitions with polar molecules

Felipe Herrera  Department of Chemistry, University of British Columbia,
Vancouver, B.C., V6T 1Z1, Canada  Department of Chemistry, Purdue University,
West Lafayette, IN 47907, USA  Department of Chemistry and Chemical Biology,
Harvard University, 12 Oxford St., Cambridge, MA 02138, USA  Kirk W. Madison
Department of Physics and Astronomy, University of British Columbia,
Vancouver, B.C., V6T 1Z1, Canada  Roman V. Krems  Department of Chemistry,
University of British Columbia, Vancouver, B.C., V6T 1Z1, Canada  Mona Berciu
Department of Physics and Astronomy, University of British Columbia,
Vancouver, B.C., V6T 1Z1, Canada

###### Abstract

We determine the phase diagram of a polaron model with mixed breathing-mode
and Su-Schrieffer-Heeger couplings and show that it has two sharp transitions,
in contrast to pure models which exhibit one (for Su-Schrieffer-Heeger
coupling) or no (for breathing-mode coupling) transition. Our results indicate
that the physics of realistic mixed polaron models is much richer than that of
simplified models. We then show that ultracold molecules trapped in optical
lattices can be used to study precisely this mixed Hamiltonian, and that the
relative contributions of the two couplings can be tuned with external
electric fields. The parameters of current experimental set-ups place them in
the region where one of the transitions occurs. We propose a scheme to measure
the polaron dispersion using stimulated Raman spectroscopy.

###### pacs:

34.50.Cx, 67.85.-d, 37.10.Gh, 37.10.De, 34.20.Cf, 52.55.Jd, 52.55.Lf, 37.10.Jk

Introduction: Polarons, which are the low-energy dressed quasiparticles
appearing in the spectrum of particles coupled to bosonic fields, have been of
broad interest in physics ever since their first study by Landau Landau . The
generic Hamiltonian for a single polaron problem:

${\cal
H}=\sum_{k}\epsilon_{k}c_{k}^{\dagger}c_{k}+\sum_{q}\hbar\Omega_{q}b^{\dagger}_{q}b_{q}+\sum_{k,q}g_{k,q}c^{\dagger}_{k+q}c_{k}\left(b^{\dagger}_{-q}+b_{q}\right)$
(1)

includes the kinetic energy of the bare particle (first term), the energy of
the bosonic mode (second term) and their interaction (third term), where the
particle scatters by absorbing or emitting bosons.

There are two mechanisms for the particle-boson coupling, as the presence of
bosons can change (i) the potential or (ii) the kinetic energy of the bare
particle. The former always leads to a vertex $g_{q}$ independent of the
particle’s momentum $k$, while in the latter case the vertex $g_{k,q}$ depends
explicitly on both momenta. As an example, consider interactions between
electrons and phonons. Vibrations of nearby atoms modulate the potential
energy of an electron. Much studied examples of such type (i) interactions are
the Holstein Holstein:1959  and Fröhlich Frohlich  models. At the same time,
by modulating the distance between lattice sites, lattice vibrations also
affect the hopping integrals. Such effects are described by type (ii) models
like the Su-Schrieffer-Heeger (SSH) model SSH:1988 .

Most of the early polaron studies focused on type (i) models, in particular on
the search for a self-trapping transition whereby at strong coupling the
bosons create a potential well so deep that it traps the polaron. In the
absence of impurities, however, this idea has proved to be wrong: type (i)
polarons have a finite effective mass for any finite coupling strength. The
absence of transitions for type (i) models has been demonstrated analytically
Gerlach:1991 . These models have a smooth crossover from light, highly mobile
polarons at weak coupling to heavy, small polarons at strong coupling.

This standard view of the polaron as a quasiparticle that becomes heavier with
increased coupling is now strongly challenged by results for type (ii) models.
Recent work has shown that in such models, the polaron can be lighter than the
bare particle, since the bosons affect the particle’s hopping so it may move
more easily in their presence Holger ; Berciu:2010 ; Marchand:2010 . The
boson-mediated dispersion can be quite different from that of the bare
particle, for example it may have a ground state with a different momentum
Berciu:2010 . If this happens, a sharp transition should occur when the boson-
mediated contribution to the dispersion becomes dominant. Indeed, such a
transition was recently predicted for the single SSH polaron Stojanovic:2008 ;
Marchand:2010 .

These results are interesting not just for challenging a long-established
paradigm, but also for raising the question of what happens in realistic
systems, where a mix of both types of coupling is generally expected. For
example, what is the fate of the transition in a mixed model if the coupling
is varied smoothly from $g_{k,q}$ to $g_{q}$? This also makes it highly
desirable to find systems described by such mixed Hamiltonians but where,
unlike in solid-state systems, the various parameters can be tuned
continuously so that the resulting polaron behaviour can be systematically
investigated.

In this Letter we elucidate the evolution of this transition as the coupling
interpolates between type (ii) SSH and type (i) breathing mode (BM)
Goodvin:2008 . Surprisingly, we find that the phase diagram has two sharp
transitions, and that these may occur even when the type (i) coupling is
dominant. This shows that polaron physics is much richer than generally
assumed. We then show that this mixed Hamiltonian describes polar molecules
trapped in an optical lattice, and moreover, that the parameters of current
experimental set-ups place them in the region where a transition is expected
to occur. Thus, experimental confirmation of these transitions is within
reach. Furthermore, we propose a detection scheme equivalent to Angle-Resolved
Photoemission Spectroscopy (ARPES) in solid-state systems Andrea , which
directly measures the polaron dispersion and can therefore pinpoint the
transition. Finally, as we discuss in conclusion, such experiments hold the
promise not only to clarify many other effects of interplay between type (i)
and type (ii) couplings on single polarons, but also to investigate the finite
concentration regime to look for quantum phase transitions.

Phase diagram: Consider the particle-boson coupling

$g_{k,q}=\frac{2i}{\sqrt{N}}\left\\{\alpha\left[\sin(k+q)-\sin(k)\right]+\beta\sin(q)\right\\},$
(2)

where $N$ is the number of lattice sites in a one-dimensional chain. The
$(k,q)$-dependent part, with energy scale $\alpha$, describes SSH coupling
Marchand:2010  while the $k$-independent part, with energy scale $\beta$,
describes BM type modulations of the site energy by bosons Goodvin:2008 . We
assume Einstein bosons $\Omega_{q}=\Omega$ and a free particle dispersion
$\epsilon_{k}=+2t\cos(k)$ with $t>0$ note2 . Following Ref. Marchand:2010 , we
define the effective SSH coupling strength
$\lambda=2\alpha^{2}/(t\hbar\Omega)$ and the adiabaticity ratio
$A=\hbar\Omega/t$. In addition, we introduce $R\equiv\beta/\alpha$ to
characterize the relative strength of the two types of coupling.

Figure 1: (color online). Phase diagram $\lambda$ vs. $A$ at fixed $R$.
Symbols show the location of the transition if $R=0$ (blue circles) and
$R=-0.5$ (red circles). The three shaded regions represent molecules with
large (a, LiCs), intermediate (b, RbCs) and small (c, KRb) dipoles, and
dressing schemes 1 ($R=-0.5$ for a) and 2 ($R=0$ for b and c), respectively.
For each region, the three lines correspond to lattice constants of 256 nm
(lower bound), 532 nm (dashed line) and 775 nm (upper bound, not shown for
region (c) because it is off-scale). For each curve $\hbar\Omega$ varies
between 1 and 100 kHz.

The SSH polaron ($R=0$) was predicted to undergo a sharp transition at a value
$\lambda^{*}$ Marchand:2010 . Its physical origin is simple to understand in
the anti-adiabatic limit $A\gg 1$ where the SSH coupling leads to an effective
next-nearest neighbor hopping $i\leftrightarrow i+2$ of the particle, by first
creating and then removing a boson at site $i+1$ Marchand:2010 . Its amplitude
is $t_{2}=-\alpha^{2}/(\hbar\Omega)=-\lambda t/2<0$, so its contribution
$-2t_{2}\cos(2k)$ to the total dispersion has a minimum at $\pi/2$, unlike the
bare dispersion which has a minimum at $\pi$. If $4|t_{2}|=t$, corresponding
to $\lambda^{*}={1\over 2}$ in the limit of $A\gg 1$, a sharp transition marks
the switch from a non-degenerate ground state with momentum $k_{gs}=\pi$ (for
$\lambda<\lambda^{*}$) to a doubly-degenerate one with
$|k_{gs}|\rightarrow{\pi\over 2}$ (for $\lambda>\lambda^{*}$). As $A$
decreases the number of phonons in the polaron cloud increases. This
renormalizes both hoppings $t\rightarrow t^{*}$, $t_{2}\rightarrow t_{2}^{*}$,
so $\lambda^{*}$ changes smoothly with $A$ as shown by the blue circles in
Fig. 1 (see also Fig. 4 of Ref. Marchand:2010 ). We also plot $\lambda^{*}$
for $R=-0.5$ (red circles), showing that the sharp polaron transition persists
for coexisting type (i) and type (ii) couplings. These results were generated
with the Momentum Average (MA) approximation, specifically its variational
flavor where the polaron cloud is allowed to extend over any three consecutive
sites Berciu:2010 ; Marchand:2010 . For $A\geq 0.3$, MA was shown to be very
accurate for both SSH and BM couplings Marchand:2010 ; Goodvin:2008 .

Figure 2: (color online). Phase diagram $\lambda$ vs. $R$ at fixed $A$,
showing two sharp transitions: one from a non-degenerate ground state with
$k_{\text{gs}}=\pi$ to a doubly-degenerate ground state with
$0<|k_{\text{gs}}|<\pi$, and the second back to a non-degenerate ground state
with $k_{\text{gs}}=0$. The results are for $A=\hbar\Omega/t\rightarrow\infty$
(red lines) and for $A=5$ (blue circles).

To understand the evolution of $\lambda^{*}$ with $R$, consider again the
limit $A\gg 1$. In addition to the second-nearest neighbor hopping $t_{2}$,
there is now also a dynamically generated nearest-neighbor hopping
$t_{1}=2\alpha\beta/(\hbar\Omega)=R\lambda t$. This describes processes where
the particle hops from site $i$ to $i+1$ leaving behind a boson at $i$ (SSH
coupling) followed by absorption of the boson while the particle stays at
$i+1$ (BM coupling); or vice versa, hence the factor of 2. The total nearest-
neighbor hopping is thus $t^{*}=t-t_{1}$, and the transition now occurs when
$4|t_{2}|=t^{*}\rightarrow\lambda^{*}=1/(2+R)$. Thus, for $R<0$, the
interference between the SSH and the BM couplings results in a larger
effective $t^{*}$ leading to a larger $\lambda^{*}$. In particular,
$\lambda^{*}\rightarrow\infty$ for $R\leq-2$, i.e. no transition occurs here.
The lack of a transition is not surprising when $R\rightarrow-\infty$, since
here the BM coupling is dominant and pure $g_{q}$ models do not have
transitions Gerlach:1991 . Our results show that for mixed SSH+BM coupling,
the switch from having to not having a transition occurs abruptly at $R=-2$ if
$A\gg 1$. This value must change continuously with $A$, therefore we expect
this switch to always occur at a finite $R$.

This is confirmed in Fig. 2, where we plot $\lambda^{*}$ vs. $R$ for
$A\rightarrow\infty$ and $A=5$. Surprisingly, we find not just the transition
at $\lambda^{*}\sim 1/(R+2)$, but also a second one which marks the crossing
to a ground state with $k_{gs}=0$. Its origin is also easy to understand in
the anti-adiabatic limit: if $R\lambda>1$, $t^{*}$ is negative and favors a
ground state at $k_{\text{gs}}=0$ instead of $k_{\text{gs}}=\pi$. For $A\gg 1$
this second transition is at $\lambda^{*}=1/(R-2)$ if $R>2$. At finite $A$, it
moves towards smaller $(R,\lambda)$ values, see Fig. 2.

Interestingly, this shows that for $R\rightarrow+\infty$ there are two nearby
transitions for the shift $k_{gs}=\pi$ to $k_{\text{gs}}=0$. This seems to
contradict the proof that a type-(i) Hamiltonian cannot have transitions
Gerlach:1991 , however, even for $R\rightarrow\infty$ this is a mixed
Hamiltonian if $\lambda\neq 0$. The transition is indeed absent if $\alpha=0$.
This is an example of the rare occurrence where a perturbatively small term
has a large effect on the behavior of the system.

Cold molecule implementation: Polar molecules in optical lattices can be used
to implement Hamiltonian (1) in a wide region of the parameter space.
Specifically, we consider molecules prepared in the ro-vibrational ground
state of the spinless electronic state ${}^{1}\Sigma$ and trapped on an
optical lattice in the Mott insulator phase, as recently demonstrated
experimentally Ospelkaus:2006 ; Chotia:2012 . We assume that there is at most
one molecule per lattice site.

The dipole-dipole interaction between molecules in different sites can be
modified by applying a DC electric field
$\mathbf{E}=E_{\text{DC}}\hat{\mathbf{z}}$ Micheli:2006 ; Micheli:2007 ;
Rabl:2007 ; Herrera:2010 ; Jesus:2010 ; Gorshkov:2011 . Here we consider two
schemes for dressing the rotational states of molecules with electric fields
that are relevant for polaron observation, scheme 1 involving a DC electric
field only, and scheme 2 involving combined optical and DC electric fields.
For the former, we define the two-state subspace
$|g\rangle=|\tilde{0},0\rangle$ and $|e\rangle=|\tilde{1},0\rangle$, where
$|\tilde{N},M_{N}\rangle$ denotes the field-dressed state that correlates
adiabatically with the field-free rotational state $|N,M_{N}\rangle$. $N$ is
the rotational angular momentum and $M_{N}$ is the projection of $N$ along the
electric field vector. In this basis we define the pseudospin operator
$\hat{c}^{\dagger}_{i}\equiv|e_{i}\rangle\langle g_{i}|$ that creates a
rotational excitation at site $i$. This excitation (the “bare particle”) can
be transferred between molecules in different lattice sites with an amplitude
$t_{ij}=\gamma\,U_{ij}\,(1-3\cos^{2}\Theta)$, where
$U_{ij}=d^{2}/|\mathbf{r}_{i}-\mathbf{r}_{j}|^{3}$, $d$ is the permanent
dipole moment, $\mathbf{r}_{i}$ is the position of molecule in site $i$,
$\Theta$ is the polar angle of the intermolecular separation vector,
$\gamma=\mu_{eg}^{2}/d^{2}\leq 1$ is the dimensionless transition dipole
moment that depends on the strength of the DC electric field. The excitation
hopping amplitude is finite even for vanishing field strengths. The field-
induced dipole-dipole interaction shifts the energy of the state
$|e_{i}\rangle$ by $D_{i}=\sum_{j}D_{ij}$. Here
$D_{ij}=-\kappa\,U_{ij}\,(1-3\cos^{2}\Theta)$, where
$\kappa=|\mu_{g}(\mu_{e}-\mu_{g})|/d^{2}$ and $\mu_{g}$($\mu_{e}$) is the
induced dipole of the ground(excited) state. Dipolar couplings outside this
two-level subspace are suppressed when the electric field separates state
$|e\rangle$ from other excited states.

The free quasiparticle dispersion is $\epsilon_{k}=\varepsilon_{0}+2t\cos(k)$
where the site energy is $\varepsilon_{0}=\hbar\omega_{eg}+D_{0}$, with the
single-molecule rotational excitation energy $\hbar\omega_{eg}\sim 10$ GHz and
$t\equiv t_{12}$. The center-of-mass vibration of molecules in the optical
lattice potential is coupled to their internal rotation through the dependence
of $U_{ij}$ on $\mathbf{r}_{i}-\mathbf{r}_{j}$. For harmonic vibrations with
linear coupling between internal and external degrees of freedom Rabl:2007 ;
Herrera:2011 , the boson term in Eq. (1) describes lattice phonons whose
spectrum depends on the trapping laser intensity and the DC electric field
Herrera:2011 . Here we consider weak DC fields and moderate trapping
frequencies which give a gapped and nearly dispersionless (Einstein) phonon
spectrum with frequency $\Omega$.

With these definitions, the system is described by SSH and BM-like couplings
with the energy scales $\alpha=-3(t_{12}/a_{L})\sqrt{\hbar/2m\Omega}$ and
$\beta=-3(D_{12}/a_{L})\sqrt{\hbar/2m\Omega}$, respectively, where $m$ is the
mass of the molecule and $a_{L}$ is the lattice constant. The ratio
$R=\beta/\alpha=\mu_{g}(\mu_{e}-\mu_{g})/\mu_{eg}^{2}$ is independent of the
intensity of the trapping laser or of the orientation of the array with
respect to the DC field. In the field dressing scheme 1 we have $|R|<1/2$ for
$dE_{\text{DC}}/B_{e}\leq 1$. In the combined AC-DC dressing scheme 2, the
same DC field strength and orientation is used as above, but an additional
two-color Raman coupling redefines the two-level subspace as
$|g\rangle=\sqrt{a}\;|\tilde{0},0\rangle+\sqrt{1-a}\;|\tilde{2},0\rangle$ and
$|e\rangle=\sqrt{b}\;|\tilde{1},0\rangle+\sqrt{1-b}\;|\tilde{3},0\rangle$
(details in the Supplementary Information). This dressing scheme can be used
to effectively enhance the hopping amplitude by a factor $f>1$ yielding
$\alpha\rightarrow f\alpha$, without changing the value of $\beta$, nor the
phonon dispersion. When using this dressing scheme, any point in the phase
diagram transforms as $\lambda\rightarrow f\lambda$ and $A\rightarrow A/f$,
thus shifting the system towards stronger SSH couplings.

The frequency of lattice phonons in a 1D array is
$\Omega=(2/\hbar)\sqrt{V_{0}E_{R}}$ where $V_{0}$ is the lattice depth and
$E_{R}=\hbar^{2}\pi^{2}/2ma_{L}^{2}$ is the recoil energy. The particle-boson
coupling can thus be written as $\lambda=18(E_{R}/t)(\pi A)^{-2}$. The shaded
regions in Fig. 1 show accessible points in the polaron phase diagram
$(\lambda,A)$ for LiCs, RbCs and KRb molecules, illustrating the flexibility
in varying the Hamiltonian parameters when using the two field dressing
scenarios and different experimental settings. Figure 1 shows that the
transition characterized by the shift from a non-degenerate ground state
$k_{\text{gs}}=\pi$ to a degenerate ground state $0<|k_{\text{gs}}|<\pi$ can
be studied using molecular species with moderate dipole moments such as RbCs,
in lattices with a site separation $a_{L}\approx 500$ nm. However, the
transition is easier to observe for molecules with large dipole moments such
as LiRb and LiCs. For weakly dipolar molecules such as KRb, the strong
coupling region can be achieved using $a_{L}<500$ nm.

The most direct way to detect the transition is to measure the polaron
dispersion. We propose the stimulated Raman spectroscopic scheme illustrated
in Fig. 3 to achieve this goal. We consider a one-dimensional array initially
prepared in the absolute ground state
$|g\rangle=|g_{1},\ldots,g_{N}\rangle|\\{0\\}\rangle$, where $|\\{0\\}\rangle$
is the phonon vacuum. We consider two linearly-polarized laser beams with
wavevectors arranged such that $\mathbf{k}_{1}-\mathbf{k}_{2}$ is parallel to
the molecular array. If the laser beams are far-detuned from any vibronic
resonance the effective light-matter interaction operator can be written as
$\hat{V}(t)=-g_{N}[\hat{c}^{\dagger}_{q}\text{e}^{-i\omega
t}+\hat{c}_{q}\text{e}^{i\omega t}]$, where $g_{N}$ is a size-dependent
coupling energy proportional to the amplitudes of both laser beams,
$q=|\mathbf{k}_{1}-\mathbf{k}_{2}|$ and $\omega=\omega_{1}-\omega_{2}$ are the
net momentum and energy transferred from the fields to the molecules. For
short interaction times (linear response), the system is excited from
$|g\rangle$ into the one-particle sector with a probability proportional to
the spectral function $\mathcal{A}(q,\omega)=-\text{Im}[G(q,\omega)]/\pi$
where $G(q,\omega)=\langle g|\hat{c}_{q}(\hbar\omega-{\cal
H}+i\eta)^{-1}\hat{c}^{\dagger}_{q}|g\rangle$ is the one-particle Green’s
function. As a result, for any $q$ the polaron energy $E_{q}$ equals the
energy $\hbar\omega$ of the lowest-energy peak in $\mathcal{A}(q,\omega)$, in
analogy with ARPES measurements Andrea .

To measure $\mathcal{A}(q,\omega)$, the stimulated Raman excitation rate can
be determined using state selective resonance enhanced multi-photon ionization
(REMPI), see Fig. 3(b). With some probability this converts the rotational
excitation (the “particle”) into a molecular ion which can be extracted from
the chain and detected by a multi-channel plate ion detector. Ionization and
subsequent detection efficiencies for a 2 step REMPI processes can easily
exceed 20% Stwalley:2011 , and a properly gated integrator can resolve the
arrival of a single molecular ion. Using a 3D lattice, a set of uncoupled
parallel 1D arrays can be realized and excited simultaneously, increasing the
signal to noise ratio of the detection step.

Figure 3: (Color online) (a) A two-photon stimulated Raman transition creates
a polaron state with a well defined momentum $q$ and energy
$\omega=\omega_{1}-\omega_{2}$. (b) The presence of the quasiparticle is
subsequently detected using resonantly-enhanced multi-photon ionization
(REMPI).

In summary, we presented the first (to our knowledge) phase diagram for a
mixed type polaron Hamitonian note3 , which showed that polaron physics is
much richer than previously thought and that sharp transitions may occur even
for dominantly type-(i) Hamiltonians ($R\gg 1$). We showed that polar
molecules trapped in optical lattices can be used to study this physics, and
proposed an ARPES-like detection scheme to directly measure the polaron
dispersion and thus identify the transitions expected to occur in such
systems.

Polarons with type (i) coupling have been observed for a single atomic
impurity immersed in a Fermi gas Schirotzek:2009 ; Nascimbene:2009 ; Will:2011
, and proposed for realization in lattice setups using atom-molecule systems
Ortner:2009 , self-assembled crystals in strong DC fields Rabl:2007 , and
recently trapped ions Stojanovic:2012 . Realization of type (ii) coupling has
been considered using Rydberg atoms Hague:2012 . Using these systems it would
be possible to explore the region of the phase diagram with $k_{\text{gs}}=0$
for $t<0$ (or equivalently $k_{\text{gs}}=\pi$ for $t>0$). Here we showed that
using trapped molecules in weak DC electric fields it is possible to explore
transitions into the phase with $0<|k_{\text{gs}}|<\pi$, where novel polaron
physics is expected to occur Marchand:2010 ; Stojanovic:2008 .

Many other aspects of single polaron physics can be investigated with trapped
polar molecules. Examples include investigating the effects of dispersive
phonons (most theoretical work assumes Einstein bosons), or novel effects
resulting from quadratic particle-boson coupling in the strong coupling
regime. Studying polaron phase diagrams in higher dimensions is easily
achieved with the same experimental scheme. Generalizations to studies of bi-
polarons are also of significant interest, to understand the pairing mechanism
for dominantly type-(ii) models (most bi-polaron studies are for type-(i)
Holstein and Fröhlich models). Finally, one may also be able to adapt the
polar molecules systems to study finite polaron concentrations and search for
quantum phase transitions QPT .

Acknowledgements: Work supported by NSERC and CIFAR. FH would also like to
thank NSF CCI center “Quantum Information for Quantum Chemistry (QIQC)”, Award
number CHE-1037992.

## References

  * (1) L. D. Landau, Phys. Z. Sowjetunion 3, 664 (1933).
  * (2) T. Holstein, Ann. Phys. (N.Y.) 8, 325 (1959).
  * (3) H. Fröhlich, Adv. Phys. 3, 325 (1954).
  * (4) A. Heeger, S. Kivelson, J. R. Schrieffer and W. -P. Su, Rev. Mod. Phys. 60, 781 (1988).
  * (5) B. Gerlach and H. Löwen, Rev. Mod. Phys. 63, 63 (1991).
  * (6) A. Alvermann, D. M. Edwards, and H. Fehske, Phys. Rev. Lett. 98, 056602 (2007).
  * (7) M. Berciu and H. Fehske, Phys. Rev. B 82, 085116 (2010).
  * (8) D. J. J. Marchand et al., Phys. Rev. Lett. 105, 266605 (2010).
  * (9) K. Hannewald et al., Phys. Rev. B. 7, 075211 (2004).
  * (10) V. M. Stojanović and M. Vanević, Phys. Rev. B 78, 214301 (2008).
  * (11) G. L. Goodvin and M. Berciu, Phys. Rev. B 23, 235120 (2008).
  * (12) A. Damascelli, Z. Hussain, and Z.-X. Shen, Rev. Mod. Phys. 75, 473 (2003).
  * (13) The results in Ref. Marchand:2010  are for $t<0$ in our sign convention. They can be mapped to $t>0$ results by a unitary transformation that changes $k\rightarrow k+\pi$ and $R\rightarrow-R$. Length is measured in units of the lattice constant.
  * (14) C. Ospelkaus et al., Phys. Rev. Lett. 97, 120402 (2006).
  * (15) A. Chotia et al., Phys. Rev. Lett. 108, 080405 (2012).
  * (16) A. Micheli, G. K. Brennen and P. Zoller, Nat. Phys. 2, 341 (2006).
  * (17) A. Micheli, G. Pupillo, H. P. Büchler and P. Zoller, Phys. Rev. A 76, 043604 (2007).
  * (18) P. Rabl and P. Zoller, Phys. Rev. A. 76, 042308 (2007).
  * (19) A. V. Gorshkov et al., Phys. Rev. A. 84, 033619 (2011).
  * (20) F. Herrera, M. Litinskaya and R. V. Krems, Phys. Rev. A 82, 033428 (2010).
  * (21) J. P. Ríos, F. Herrera and R. V. Krems, New J. Phys. 12, 103007 (2010).
  * (22) F. Herrera and R. V. Krems, Phys Rev. A. 84, 051401(R) (2011).
  * (23) M. A. Bellos et al., Phys. Chem. Chem. Phys. 13, 18880 (2011).
  * (24) A. Schirotzek, C. -H. Wu, A. Sommer and M. W. Zwierlein, Phys. Rev. Lett 102, 230402 (2009).
  * (25) S. Nascimbène et al., Phys. Rev. Lett 103, 170402 (2009).
  * (26) S. Will et al., Phys. Rev. Lett. 106, 115305 (2011).
  * (27) M. Ortner et al., New J. Phys. 11, 055045 (2009)
  * (28) V. M. Stojanovic, T. Shi, C. Bruder and J. I. Cirac, Phys. Rev. Lett. 109, 250501 (2012).
  * (29) J. P. Hague and C. MacCormick, New J. Phys. 14, 033019 (2012).
  * (30) Variational results for mixed Holstein + SSH coupling were presented in J. Sun, Y. Zhao, and W. Liang, Phys. Rev. B 79, 155112 (2009), however the complete phase diagram or the existence of a transition is not discussed.
  * (31) For a phase-diagram at half-filling in systems with type-(i) couplings, see M. Hohenadler, F. F. Assaad and H. Fehske, Phys. Rev. Lett. 109, 116407 (2012) and references therein. For SSH coupling, a Peierls instability occurs at half-filling SSH:1988 . We are not aware of studies investigating mixed models at finite polaron concentrations.

## Supplementary Material

### Rotational state dressing schemes

Static field dressing.- This dressing scheme involves an homogeneous DC
electric field acting on a one-dimensional array of $\mathcal{N}$ polar
molecules. The field can have an arbitrary orientation with respect to the
array axis, described by the polar angle $\Theta$. We consider ${}^{1}\Sigma$
molecules each prepared in its lower vibrational and electronic state. The
rotational state of the molecules is described by the pendular state
$|\tilde{N},M\rangle$, which is an eigenstate of $\hat{N}_{z}$ and
$\hat{H}_{\text{R}}=B_{e}\hat{N}^{2}-dE_{\text{DC}}\cos\theta$, where
$\hat{N}$ is the rotational angular momentum operator, $B_{e}$ is the
rotational constant, $d$ is magnitude of the permanent dipole moment, and
$\theta$ is the polar angle of the internuclear axis Carrington:2003. We use
the two-level subspace $|g\rangle=|\tilde{0},0\rangle$ and
$|e\rangle=|\tilde{1},0\rangle$ to define the particle creation operator
$\hat{c}^{\dagger}_{{}_{i}}=|e_{i}\rangle\langle g_{i}|$. If the state
$|\tilde{1},0\rangle$ is energetically separated from the degenerate states
$|\tilde{1},\pm 1\rangle$ by an amount larger than the dipole-dipole
interaction energy between neighbouring molecules in an array, the two-level
approximation is valid and the dipole-dipole interaction operator reduces to
$\hat{V}_{\text{dd}}=|\mathbf{r}_{ij}|^{-3}\left(1-3\cos^{2}\Theta\right)\hat{d}_{0}(i)\hat{d}_{0}(j)$,
where $\mathbf{r}_{ij}$ is the intermolecular separation vector, $\Theta$ its
polar angle with respect to the dc field, and $\hat{d}_{p}$ is the
$p$-component of the space-fixed dipole operator in spherical coordinates.
This simplification of the dipole operator is accurate for electric field
strengths $E_{\text{DC}}>1$ V/cm, regardless of its orientation with respect
to the molecular array. The dipolar energies $t_{12}=\langle
e_{i}g_{j}|\hat{V}_{\text{dd}}|g_{i}e_{j}\rangle$ and $D_{12}=\langle
e_{1}g_{2}|\hat{V}_{\text{dd}}|e_{1}g_{2}\rangle-\langle
g_{1}g_{2}|\hat{V}_{\text{dd}}|g_{1}g_{2}\rangle$ are evaluated by
diagonalizing $\hat{H}_{\text{R}}$ numerically and using the eigenvectors to
compute the matrix elements of $\hat{V}_{\text{dd}}$ for a given value of the
field strength $E_{\text{DC}}$.

Combined static and infrared dressing.- This schme involves a DC electric
field as before and an additional infrared laser fields that induce Raman
couplings between rotational states of the same parity. The Raman coupling
scheme is illustrated in Fig. 1. It consists of two uncoupled lambda systems
$\Lambda_{1}=\\{|^{1}\Sigma,v=0\rangle|\tilde{0},0\rangle,|^{1}\Sigma,v=1\rangle|\tilde{1},0\rangle,|^{1}\Sigma,v=0\rangle|\tilde{2},0\rangle\\}$
and
$\Lambda_{2}=\\{|^{1}\Sigma,v=0\rangle|\tilde{1},0\rangle,|^{1}\Sigma,v=1\rangle|\tilde{2},0\rangle,|^{1}\Sigma,v=0\rangle|\tilde{3},0\rangle\\}$.
Here $|^{1}\Sigma,v\rangle$ denotes the electronic and vibrational quantum
numbers. Four linearly polarized infrared fields are needed to implement the
scheme. The corresponding transitions are far detuned by $\delta\omega>10^{2}$
MHz from other rotational lines in the presence of a DC field
$E_{\text{DC}}\sim 1$ kV/cm.

Let $\Omega=|\Omega|\text{e}^{i\phi}$ be the Rabi frequency associated with a
dipole-allowed transition. For each lambda system the adiabatic state with
zero eigenvalue (dark state) can be written as [2]

$|\lambda_{0}\rangle=\cos\alpha(t)|\epsilon_{1}\rangle-\sin\alpha(t)\exp(i\beta)|\epsilon_{3}\rangle,$
(3)

where the mixing angle $\alpha(t)$ is defined by
$\tan\alpha(t)=|\Omega_{P}(t)/\Omega_{S}(t)|$, and the relative phase of the
fields is $\beta=\phi_{P}-\phi_{S}$. The RWA holds when
$\text{max}\left\\{\Delta,\Omega_{P},\Omega_{S}\right\\}\ll B_{e}$. The two
lambda systems in Fig. 1 are uncoupled due to the anharmonicity of the
rotational spectrum for small detunings $\Delta_{1}$ and $\Delta_{2}$.
Selection rules restrict the coupling between rovibrational states with
different angular momentum projection $M$.

Figure 4: (a) Infrared dressing of rotational states in the presence of a dc
electric field. Two independent lambda systems are used to construct the dark
state superpositions
$|g\rangle=\sqrt{a}|\tilde{0},0\rangle+\sqrt{1-a}|\tilde{2},0\rangle$ and
$|e\rangle=\sqrt{b}|\tilde{1},0\rangle+\sqrt{1-b}|\tilde{3},0\rangle$. (b)
Dipole-dipole energy ratio $R=D_{12}/t_{12}$ as a function of the DC fields
strength $E_{\text{DC}}$ for DC field rotational dressing ($a=b=1$) and
combined AC-DC dressing with $a=b=1/2$.

We want to use the dark states of the lambda systems $\Lambda_{1}$ and
$\Lambda_{2}$ to define the two-level subspace $|g\rangle$ and $|e\rangle$,
respectively. It is in this field-dressed space where excitation hopping
processes take place in a molecular array. Let the molecules be initially
prepared in the rovibrational ground state
$|g\rangle=|^{1}\Sigma,v=0\rangle|\tilde{0},0\rangle$. We assume that only the
Stokes fields $\Omega_{S_{1}}(t)$ and $\Omega_{S_{2}}(t)$ are present. In this
case the mixing angles $\alpha_{1}(t)$ and $\alpha_{2}(t)$ vanish and the dark
states are the same as the DC dressing scheme
$|g\rangle=|^{1}\Sigma,v=0\rangle|\tilde{0},0\rangle$ and
$|e\rangle=|^{1}\Sigma,v=0\rangle|\tilde{1},0\rangle$. By slowly turning on
the pump fields $\Omega_{P_{1}}(t)$ and $\Omega_{P_{2}}(t)$ the dark states
become

$\displaystyle|g\rangle$ $\displaystyle=$
$\displaystyle\cos\alpha_{1}(t)|^{1}\Sigma,v=0\rangle|\tilde{0},0\rangle+\sin\alpha_{1}(t)|^{1}\Sigma,v=0\rangle|\tilde{2},0\rangle,$
$\displaystyle|e\rangle$ $\displaystyle=$
$\displaystyle\cos\alpha_{2}(t)|^{1}\Sigma,v=0\rangle|\tilde{1},0\rangle+\sin\alpha_{2}(t)|^{1}\Sigma,v=0\rangle|\tilde{3},0\rangle.$

The Raman lasers have the relative phases $\beta_{1}=\beta_{2}=\pi$, which can
be achieved experimentally. The states $|g\rangle$ and $|e\rangle$ can be
manipulated independently by adiabatically tuning the pump laser intensities
relative to the Stokes beams. We consider the specific situation where
$\alpha_{1}=\alpha_{2}=\pi/2$. Omitting the labels $v$ and $M$, the excitation
hopping amplitude $t_{ij}$ is then given by

$\displaystyle t_{ij}$ $\displaystyle\approx$
$\displaystyle\frac{1-3\cos^{2}\Theta}{|\mathbf{r}_{ij}|^{3}}\left(\frac{1}{4}\right)\left\\{d_{10}^{2}+d_{21}^{2}+d_{32}^{2}+2d_{21}d_{10}+2d_{32}d_{10}+2d_{21}d_{32}\right\\}$
(4)

where
$d_{NN^{\prime}}=\langle\tilde{N}|\hat{d}_{0}|\tilde{N}^{\prime}\rangle=d_{N^{\prime}N}$.
We have ignored couplings with $\Delta\tilde{N}>1$, which are suppressed in
the presence of weak dc electric fields $E_{\text{DC}}\leq B_{e}/d$. Equation
4 becomes exact in the limit $E_{\text{DC}}\rightarrow 0$ where the transition
dipole moments can be evaluated analytically using angular momentum algebra
[1].The transition dipole moments in Eq. (4) have the same sign and therefore
contribute to the enhancement of the hopping amplitude
$t_{12}^{\prime}=f\,t_{12}$ with respect to the pure dc dressing scheme. The
diagonal energy $D_{12}$ has a similar behaviour as in the type-I coupling
only for weak dc electric fields $dE_{\text{DC}}/B_{e}\ll 1$. For non-
perturbative fields, contributions from higher rotational states
$\tilde{N}\geq 2$ modify significantly its static field dependence. In Fig.
1(b) we show the achievable values of the ratio $R=D_{12}/t_{12}$ using the
combined AC-DC dressing scheme described above, as function of the DC field
strength. The hopping amplitude is enhanced by a factor $f\approx 2$ with
respect to the static scheme. The ratio $R$ has the same sign as the static
dressing case ($R<0$) for weak fields $E_{\text{DC}}\leq 2B_{e}/d$, but is up
to an order of magnitude smaller. The ratio changes sign at
$E_{\text{DC}}\approx 2.8B_{e}/d$ and has a local maximum
$R_{\text{max}}\approx 0.32$ at $E_{\text{DC}}=7B_{e}/d$.

Phonon dispersion.- In this work the dispersion of the phonon spectrum is
ignored due to the current limitations of the Momentum Average (MA)
approximation we use in the main text to solve the polaron problem [3]. For a
1D array the phonon frequency can be written as
$\omega_{q}=\omega_{0}\sqrt{1+12\rho\gamma(q)}$, where
$\rho=V^{gg}_{12}/V_{0}$ is the ratio between the ground state dipolar energy
$V_{12}^{gg}=\langle g_{1}g_{2}|\hat{V}_{\text{dd}}|g_{1}g_{2}\rangle$ and the
optical lattice depth $V_{0}$, and $\gamma(q)=2\sin^{2}(q)$ [4]. The
dispersion is determined by the parameter $\rho$, which can be kept small for
shallow lattices $V_{0}\approx 18-20E_{R}$ in the static dressing scheme only
for weak dc fields $E_{\text{DC}}\leq B_{e}/d$. For higher fields we have
$|V_{12}|^{gg}>|t_{12}|$, thus the phonon dispersion becomes an important
energy scale. For combined AC-DC dressing, the phonon dispersion can be safely
ignored since $|V^{gg}_{12}|\ll|t_{12}|$ over the entire range of electric
fields in Fig. 1.

  1. (1)

J. Brown and A. Carrington, Rotational Spectroscopy of Diatomic Molecules,
(Cambridge University Press, 2003).

  2. (2)

K. Bergmann, H. Theuer and B. W. Shore, Rev. Mod. Phys. 70, 1003 (1998).

  3. (3)

G. L. Goodvin and M. Berciu, Phys. Rev. B 23, 235120 (2008).

  4. (4)

F. Herrera and R. V. Krems, Phys Rev. A. 84, 051401(R) (2011).