spanning the area of entity resolution. Upon a thorough investigation of
entity resolution research there have been three main areas of debate in this
field: the evaluation metrics used, the scope of the datasets used and the
idea of commonsense knowledge induction in entity resolution. We will be
providing an overview of these issues and the progress made in addressing
them.

The issues with the evaluation metrics to be used for CR have been delineated
by many prominent researchers (Ng, 2010; Mitkov, 2001a). We have now
progresses from evaluation using simple metrics like “Hobb’s Accuracy Metric”
to much more advanced metrics like MUC (Vilain et al, 1995), $B^{3}$ (Bagga
and Baldwin, 1998) and CEAF (Luo, 2005) which capture the entirety of the CR
task. In spite of the progress made over the years, as pointed out by many
researchers, the metric currently used for CR (Pradhan et al, 2012) is still
fundamentally the average of three faulty metrics (i.e., MUC, $B^{3}$ and
CEAF). Recently, there have been metrics proposed to circumvent the issues
faced by the earlier metrics. Some of these include modifications on existing
metrics (Cai and Strube, 2010) and the other the new LEA metric (Moosavi and
Strube, 2016). We encourage researchers to evaluate their models on these
recently proposed metrics in addition to the earlier standard metrics.
Another area pertaining to CR research is whether the standard datasets for
the task address different types of references that exist in natural language.
As discussed earlier the field of entity resolution is composed of different
types of references. Some of these references are rare and some types are not
labelled by the current CR datasets (Zeldes and Zhang, 2016). This has led to
proliferation of research targeting specific types of references like multi-
antecedent references (Vala et al, 2016), Abstract Anaphora (Marasović et al,
2017) and One Anaphora (Goldberg and Michaelis, 2017). We suggest that to
address this issue future datasets released clearly specify the types of
references considered for labelling and the ones not. We also encourage future
CR models to carry out cross-domain evaluations on other datasets which are
also annotated in CoNLL format like the Character Identification dataset
(Moosavi and Strube, 2016). This will mainly aid in the process of identifying
the types of references which still pose a challenge to the state-of-the-art
CR algorithms.

Since the inception of the field of CR it has been known that some type of
references are extremely hard to resolve for a machine mainly because it
requires some amount of external world knowledge. Though the usefulness of
world knowledge for a coreference system has been known since the late
nineties, early mention pair models (Soon et al, 2001; Ng and Cardie, 2002b;
Yang et al, 2003) did not incorporate any form of world knowledge into the
system. As knowledge resources became less noisy and exhaustive, some CR
researchers started deploying world knowledge to CR. The two main questions to
be answered were whether world knowledge offered complementary benefits and
whether the noisy nature of world knowledge would effect the performance of
the model negatively. Several researchers have deployed world knowledge in the
form of web-based encyclopaedias (Uryupina et al, 2012), un-annotated data
(Daumé III and Marcu, 2005), coreference annotated data (Bengtson and Roth,
2008), and knowledge bases like YAGO, Framenet (Rahman and Ng, 2011) and
Wordnet (Durrett and Klein, 2013). World knowledge was mainly incorporated as
features into the mention pair models and cluster ranking models. These
features were often defined over NPs and verbs. Some initial algorithms like
Ng reported an increase in performance up to 4.8% on inducing world-knowledge
features from YAGO and FrameNet on the CR task. While some others (Durrett and
Klein, 2013) reported only minor performance gains using world-knowledge on
system mentions extracted by coreference systems. Some models instead of
representing commonsense knowledge as features, also used predicates to encode
the commonsense relations (Peng et al, 2015). They evaluated their model on
hard CR problems that fit the definition of the Winograd Schema Challenge. As
posited by Durret and Klein (Durrett and Klein, 2013), the task of modeling
complex linguistic constraints into a coreference system remains an uphill
battle.

##  13 Conclusion

Our survey presents an exhaustive overview of the entity resolution field,
which forms a core component of natural language processing research. In this
survey, we put forth a detailed account of the types of references and the
important constraints for entity resolution with the aim of establishing the
bread scope of the task. We also clarify the boundaries between the tasks of
coreference resolution and anaphora resolution for more focussed research
progress in the future. In addition, we also attempt to compare the
predominantly used evaluation metrics. We observe that though there are
multiple datasets available, the state-of-the-art methods have not been
evaluated on them. With the spirit of encouraging more exhaustive evaluations,
we also provide an account on the datasets released for the task.

Entity resolution research has seen a shift from rule-based methods to deep
learning methods. To this end, we provide an analysis of the types of
algorithms used with special focus on recent deep learning methods. Anaphora
resolution is a very important component of the suitcase research problem of
sentiment analysis. As the research in the intersection of these two fields is
scarce, we also establish a background for the inter-dependency between the
two tasks. Finally, we also state the outstanding issues in this task
requiring attention, thus laying a firm cornerstone for future researchers to
build on.

## References

  * Aone and Bennett (1995) Aone C, Bennett SW (1995) Evaluating automated and manual acquisition of anaphora resolution strategies. In: Proceedings of the 33rd annual meeting on Association for Computational Linguistics, Association for Computational Linguistics, pp 122–129
  * Atlas and Levinson (1981) Atlas JD, Levinson SC (1981) It-clefts, informativeness and logical form: Radical pragmatics (revised standard version). In: Radical pragmatics, Academic Press, pp 1–62
  * Bagga and Baldwin (1998) Bagga A, Baldwin B (1998) Algorithms for scoring coreference chains. In: The first international conference on language resources and evaluation workshop on linguistics coreference, Granada, vol 1, pp 563–566
  * Baldwin (1997) Baldwin B (1997) Cogniac: high precision coreference with limited knowledge and linguistic resources. In: Proceedings of a Workshop on Operational Factors in Practical, Robust Anaphora Resolution for Unrestricted Texts, Association for Computational Linguistics, pp 38–45
  * Batista-Navarro and Ananiadou (2011) Batista-Navarro RT, Ananiadou S (2011) Building a coreference-annotated corpus from the domain of biochemistry. In: Proceedings of BioNLP 2011 Workshop, Association for Computational Linguistics, pp 83–91
  * Bawden et al (2017) Bawden R, Sennrich R, Birch A, Haddow B (2017) Evaluating discourse phenomena in neural machine translation. arXiv preprint arXiv:171100513
  * Bean and Riloff (2004) Bean D, Riloff E (2004) Unsupervised learning of contextual role knowledge for coreference resolution. In: Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004
  * Bengtson and Roth (2008) Bengtson E, Roth D (2008) Understanding the value of features for coreference resolution. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, pp 294–303
  * Berger et al (1996) Berger AL, Pietra VJD, Pietra SAD (1996) A maximum entropy approach to natural language processing. Computational linguistics 22(1):39–71
  * Bergler et al (2003) Bergler S, Witte R, Khalife M, Li Z, Rudzicz F (2003) Using knowledge-poor coreference resolution for text summarization. In: Proceedings of DUC, vol 3
  * Björkelund and Farkas (2012) Björkelund A, Farkas R (2012) Data-driven multilingual coreference resolution using resolver stacking. In: Joint Conference on EMNLP and CoNLL-Shared Task, Association for Computational Linguistics, pp 49–55
  * Brennan et al (1987) Brennan SE, Friedman MW, Pollard CJ (1987) A centering approach to pronouns. In: Proceedings of the 25th annual meeting on Association for Computational Linguistics, Association for Computational Linguistics, pp 155–162
  * Cai and Strube (2010) Cai J, Strube M (2010) Evaluation metrics for end-to-end coreference resolution systems. In: Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Association for Computational Linguistics, pp 28–36
  * Cambria (2016) Cambria E (2016) Affective computing and sentiment analysis. IEEE Intelligent Systems 31(2):102–107
  * Cambria et al (2017) Cambria E, Poria S, Gelbukh A, Thelwall M (2017) Sentiment analysis is a big suitcase. IEEE Intelligent Systems 32(6):74–80
  * Cambria et al (2018) Cambria E, Poria S, Hazarika D, Kwok K (2018) SenticNet 5: Discovering conceptual primitives for sentiment analysis by means of context embeddings. In: AAAI, pp 1795–1802
  * Carbonell and Brown (1988) Carbonell JG, Brown RD (1988) Anaphora resolution: a multi-strategy approach. In: Proceedings of the 12th conference on Computational linguistics-Volume 1, Association for Computational Linguistics, pp 96–101
  * Carlson et al (2003) Carlson L, Marcu D, Okurowski ME (2003) Building a discourse-tagged corpus in the framework of rhetorical structure theory. In: Current and new directions in discourse and dialogue, Springer, pp 85–112
  * Castagnola (2002) Castagnola L (2002) Anaphora resolution for question answering. PhD thesis, Massachusetts Institute of Technology
  * Chang et al (2013) Chang KW, Samdani R, Roth D (2013) A constrained latent variable model for coreference resolution. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp 601–612
  * Chaturvedi et al (2018) Chaturvedi I, Cambria E, Welsch R, Herrera F (2018) Distinguishing between facts and opinions for sentiment analysis: Survey and challenges. Information Fusion 44:65–77
  * Chen and Manning (2014) Chen D, Manning C (2014) A fast and accurate dependency parser using neural networks. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 740–750
  * Chen and Choi (2016) Chen YH, Choi JD (2016) Character identification on multiparty conversation: Identifying mentions of characters in tv shows. In: Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp 90–100
  * Chinchor (1998) Chinchor NA (1998) Overview of muc-7/met-2. Tech. rep., SCIENCE APPLICATIONS INTERNATIONAL CORP SAN DIEGO CA
  * Clark and Manning (2015) Clark K, Manning CD (2015) Entity-centric coreference resolution with model stacking. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), vol 1, pp 1405–1415
  * Clark and Manning (2016a) Clark K, Manning CD (2016a) Deep reinforcement learning for mention-ranking coreference models. arXiv preprint arXiv:160908667
  * Clark and Manning (2016b) Clark K, Manning CD (2016b) Improving coreference resolution by learning entity-level distributed representations. arXiv preprint arXiv:160601323
  * Cohen et al (2010) Cohen KB, Johnson HL, Verspoor K, Roeder C, Hunter LE (2010) The structural and content aspects of abstracts versus bodies of full text journal articles are different. BMC bioinformatics 11(1):492
  * Cohen and Singer (1999) Cohen WW, Singer Y (1999) A simple, fast, and effective rule learner. AAAI/IAAI 99:335–342
  * Collobert and Weston (2008) Collobert R, Weston J (2008) A unified architecture for natural language processing: Deep neural networks with multitask learning. In: Proceedings of the 25th international conference on Machine learning, ACM, pp 160–167
  * Culotta et al (2007) Culotta A, Wick M, McCallum A (2007) First-order probabilistic models for coreference resolution. In: Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pp 81–88
  * Cybulska and Vossen (2014) Cybulska A, Vossen P (2014) Guidelines for ecb+ annotation of events and their coreference. Tech. rep., Technical Report NWR-2014-1, VU University Amsterdam
  * Daelemans et al (2004) Daelemans W, Zavrel J, Van Der Sloot K, Van den Bosch A (2004) Timbl: Tilburg memory-based learner. Tilburg University
  * Daumé III and Marcu (2005) Daumé III H, Marcu D (2005) A large-scale exploration of effective global features for a joint entity detection and tracking model. In: Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, Association for Computational Linguistics, pp 97–104
  * Denis and Baldridge (2007) Denis P, Baldridge J (2007) Joint determination of anaphoricity and coreference resolution using integer programming. In: Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pp 236–243
  * Denis and Baldridge (2008) Denis P, Baldridge J (2008) Specialized models and ranking for coreference resolution. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, pp 660–669
  * Denis and Baldridge (2009) Denis P, Baldridge J (2009) Global joint models for coreference resolution and named entity classification. Procesamiento del Lenguaje Natural 42
  * Ding and Liu (2010) Ding X, Liu B (2010) Resolving object and attribute coreference in opinion mining. In: Proceedings of the 23rd International Conference on Computational Linguistics, Association for Computational Linguistics, pp 268–276
  * Dixon (2003) Dixon RM (2003) Demonstratives: A cross-linguistic typology. Studies in Language International Journal sponsored by the Foundation “Foundations of Language” 27(1):61–112
  * Doddington et al (2004) Doddington GR, Mitchell A, Przybocki MA, Ramshaw LA, Strassel S, Weischedel RM (2004) The automatic content extraction (ace) program-tasks, data, and evaluation. In: LREC, vol 2, p 1
  * Durrett and Klein (2013) Durrett G, Klein D (2013) Easy victories and uphill battles in coreference resolution. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp 1971–1982
  * Durrett and Klein (2014) Durrett G, Klein D (2014) A joint model for entity analysis: Coreference, typing, and linking. Transactions of the Association for Computational Linguistics 2:477–490
  * Fernandes et al (2012) Fernandes ER, Dos Santos CN, Milidiú RL (2012) Latent structure perceptron with feature induction for unrestricted coreference resolution. In: Joint Conference on EMNLP and CoNLL-Shared Task, Association for Computational Linguistics, pp 41–48
  * Fillmore (1986) Fillmore CJ (1986) Pragmatically controlled zero anaphora. In: Annual Meeting of the Berkeley Linguistics Society, vol 12, pp 95–107
  * Finkel and Manning (2008) Finkel JR, Manning CD (2008) Enforcing transitivity in coreference resolution. In: Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers, Association for Computational Linguistics, pp 45–48
  * Gasperin and Briscoe (2008) Gasperin C, Briscoe T (2008) Statistical anaphora resolution in biomedical texts. In: Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1, Association for Computational Linguistics, pp 257–264
  * Ge et al (1998) Ge N, Hale J, Charniak E (1998) A statistical approach to anaphora resolution. In: Sixth Workshop on Very Large Corpora
  * Ghaddar and Langlais (2016) Ghaddar A, Langlais P (2016) Wikicoref: An english coreference-annotated corpus of wikipedia articles. In: LREC
  * Goldberg and Michaelis (2017) Goldberg AE, Michaelis LA (2017) One among many: Anaphoric one and its relationship with numeral one. Cognitive science 41(S2):233–258
  * Grishman and Sundheim (1996) Grishman R, Sundheim B (1996) Message understanding conference-6: A brief history. In: COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics, vol 1
  * Gross et al (1993) Gross D, Allen J, Traum D (1993) The trains 91 dialogues (trains tech. note 92–1). Rochester, NY: University of Rochester, Department of Computer Science
  * Grosz et al (1995) Grosz BJ, Weinstein S, Joshi AK (1995) Centering: A framework for modeling the local coherence of discourse. Computational linguistics 21(2):203–225
  * Guillou et al (2014) Guillou L, Hardmeier C, Smith A, Tiedemann J, Webber B (2014) Parcor 1.0: A parallel pronoun-coreference corpus to support statistical mt. In: 9th International Conference on Language Resources and Evaluation (LREC), MAY 26-31, 2014, Reykjavik, ICELAND, European Language Resources Association, pp 3191–3198
  * Gundel et al (2005) Gundel J, Hedberg N, Zacharski R (2005) Pronouns without np antecedents: How do we know when a pronoun is referential. Anaphora processing: linguistic, cognitive and computational modelling pp 351–364
  * Haghighi and Klein (2007) Haghighi A, Klein D (2007) Unsupervised coreference resolution in a nonparametric bayesian model. In: Proceedings of the 45th annual meeting of the association of computational linguistics, pp 848–855
  * Haghighi and Klein (2009) Haghighi A, Klein D (2009) Simple coreference resolution with rich syntactic and semantic features. In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 3-Volume 3, Association for Computational Linguistics, pp 1152–1161
  * Haghighi and Klein (2010) Haghighi A, Klein D (2010) Coreference resolution in a modular, entity-centered model. In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Association for Computational Linguistics, pp 385–393
  * Harabagiu and Maiorano (1999) Harabagiu SM, Maiorano SJ (1999) Knowledge-lean coreference resolution and its relation to textual cohesion and coherence. The Relation of Discourse/Dialogue Structure and Reference
  * Harabagiu et al (2001) Harabagiu SM, Bunescu RC, Maiorano SJ (2001) Text and knowledge mining for coreference resolution. In: Proceedings of the second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies, Association for Computational Linguistics, pp 1–8
  * Hardmeier and Federico (2010) Hardmeier C, Federico M (2010) Modelling pronominal anaphora in statistical machine translation. In: IWSLT (International Workshop on Spoken Language Translation); Paris, France; December 2nd and 3rd, 2010., pp 283–289
  * Hasler and Orasan (2009) Hasler L, Orasan C (2009) Do coreferential arguments make event mentions coreferential. In: Proc. the 7th Discourse Anaphora and Anaphor Resolution Colloquium (DAARC 2009), Citeseer
  * Heeman and Allen (1995) Heeman PA, Allen JF (1995) The trains 93 dialogues. Tech. rep., ROCHESTER UNIV NY DEPT OF COMPUTER SCIENCE
  * Heim (1982) Heim I (1982) The semantics of definite and indefinite nps. University of Massachusetts at Amherst dissertation
  * Hobbs (1978) Hobbs JR (1978) Resolving pronoun references. Lingua 44(4):311–338
  * Hou et al (2013) Hou Y, Markert K, Strube M (2013) Global inference for bridging anaphora resolution. In: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp 907–917
  * Jakob and Gurevych (2010) Jakob N, Gurevych I (2010) Using anaphora resolution to improve opinion target identification in movie reviews. In: Proceedings of the ACL 2010 Conference Short Papers, Association for Computational Linguistics, pp 263–268
  * Kennedy and Boguraev (1996) Kennedy C, Boguraev B (1996) Anaphora for everyone: pronominal anaphora resoluation without a parser. In: Proceedings of the 16th conference on Computational linguistics-Volume 1, Association for Computational Linguistics, pp 113–118
  * Kibble (2001) Kibble R (2001) A reformulation of rule 2 of centering theory. Computational Linguistics 27(4):579–587
  * Kim et al (2003) Kim JD, Ohta T, Tateisi Y, Tsujii J (2003) Genia corpus—a semantically annotated corpus for bio-textmining. Bioinformatics 19(suppl_1):i180–i182
  * Kim et al (2008) Kim JD, Ohta T, Tsujii J (2008) Corpus annotation for mining biomedical events from literature. BMC bioinformatics 9(1):10
  * Kim et al (2011) Kim JD, Pyysalo S, Ohta T, Bossy R, Nguyen N, Tsujii J (2011) Overview of bionlp shared task 2011. In: Proceedings of the BioNLP shared task 2011 workshop, Association for Computational Linguistics, pp 1–6
  * Lappin and Leass (1994) Lappin S, Leass HJ (1994) An algorithm for pronominal anaphora resolution. Computational linguistics 20(4):535–561
  * Le et al (2016) Le TT, Vo TH, Mai DT, Quan TT, Phan TT (2016) Sentiment analysis using anaphoric coreference resolution and ontology inference. In: International Workshop on Multi-disciplinary Trends in Artificial Intelligence, Springer, pp 297–303
  * Lee et al (2011) Lee H, Peirsman Y, Chang A, Chambers N, Surdeanu M, Jurafsky D (2011) Stanford’s multi-pass sieve coreference resolution system at the conll-2011 shared task. In: Proceedings of the fifteenth conference on computational natural language learning: Shared task, Association for Computational Linguistics, pp 28–34
  * Lee et al (2013) Lee H, Chang A, Peirsman Y, Chambers N, Surdeanu M, Jurafsky D (2013) Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics 39(4):885–916
  * Lee et al (2017a) Lee H, Surdeanu M, Jurafsky D (2017a) A scaffolding approach to coreference resolution integrating statistical and rule-based models. Natural Language Engineering 23(5):733–762
  * Lee et al (2017b) Lee K, He L, Lewis M, Zettlemoyer L (2017b) End-to-end neural coreference resolution. arXiv preprint arXiv:170707045
  * Levesque et al (2011) Levesque HJ, Davis E, Morgenstern L (2011) The winograd schema challenge. In: Aaai spring symposium: Logical formalizations of commonsense reasoning, vol 46, p 47
  * Liang and Wu (2004) Liang T, Wu DS (2004) Automatic pronominal anaphora resolution in english texts. International Journal of Computational Linguistics & Chinese Language Processing, Volume 9, Number 1, February 2004: Special Issue on Selected Papers from ROCLING XV 9(1):21–40
  * Luo (2005) Luo X (2005) On coreference resolution performance metrics. In: Proceedings of the conference on human language technology and empirical methods in natural language processing, Association for Computational Linguistics, pp 25–32
  * Ma et al (2014) Ma C, Doppa JR, Orr JW, Mannem P, Fern X, Dietterich T, Tadepalli P (2014) Prune-and-score: Learning for greedy coreference resolution. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp 2115–2126
  * Ma et al (2018) Ma Y, Peng H, Cambria E (2018) Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive LSTM. In: AAAI, pp 5876–5883
  * Marasović et al (2017) Marasović A, Born L, Opitz J, Frank A (2017) A mention-ranking model for abstract anaphora resolution. arXiv preprint arXiv:170602256
  * McCallum and Wellner (2003) McCallum A, Wellner B (2003) Object consolidation by graph partitioning with a conditionally-trained distance metric. In: KDD Workshop on Data Cleaning, Record Linkage and Object Consolidation, Citeseer
  * McCallum and Wellner (2005) McCallum A, Wellner B (2005) Conditional models of identity uncertainty with application to noun coreference. In: Advances in neural information processing systems, pp 905–912
  * McCarthy and Lehnert (1995) McCarthy JF, Lehnert WG (1995) Using decision trees for coreference resolution. arXiv preprint cmp-lg/9505043
  * Mikolov et al (2013) Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems, pp 3111–3119
  * Mitkov (1998) Mitkov R (1998) Robust pronoun resolution with limited knowledge. In: Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics-Volume 2, Association for Computational Linguistics, pp 869–875
  * Mitkov (1999) Mitkov R (1999) Anaphora resolution: the state of the art. Citeseer
  * Mitkov (2001a) Mitkov R (2001a) Outstanding issues in anaphora resolution. In: International Conference on Intelligent Text Processing and Computational Linguistics, Springer, pp 110–125
  * Mitkov (2001b) Mitkov R (2001b) Towards a more consistent and comprehensive evaluation of anaphora resolution algorithms and systems. Applied Artificial Intelligence 15(3):253–276
  * Mitkov (2014) Mitkov R (2014) Anaphora resolution. Routledge
  * Mitkov et al (2002) Mitkov R, Evans R, Orasan C (2002) A new, fully automatic version of mitkov’s knowledge-poor pronoun resolution method. In: International Conference on Intelligent Text Processing and Computational Linguistics, Springer, pp 168–186
  * Moosavi and Strube (2016) Moosavi NS, Strube M (2016) Which coreference evaluation metric do you trust? a proposal for a link-based entity aware metric. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol 1, pp 632–642
  * Moosavi and Strube (2017) Moosavi NS, Strube M (2017) Lexical features in coreference resolution: To be used with caution. arXiv preprint arXiv:170406779
  * Ng (2005) Ng V (2005) Machine learning for coreference resolution: From local classification to global ranking. In: Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, Association for Computational Linguistics, pp 157–164
  * Ng (2010) Ng V (2010) Supervised noun phrase coreference research: The first fifteen years. In: Proceedings of the 48th annual meeting of the association for computational linguistics, Association for Computational Linguistics, pp 1396–1411
  * Ng and Cardie (2002a) Ng V, Cardie C (2002a) Combining sample selection and error-driven pruning for machine learning of coreference rules. In: Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, Association for Computational Linguistics, pp 55–62
  * Ng and Cardie (2002b) Ng V, Cardie C (2002b) Improving machine learning approaches to coreference resolution. In: Proceedings of the 40th annual meeting on association for computational linguistics, Association for Computational Linguistics, pp 104–111
  * Nicolae and Nicolae (2006) Nicolae C, Nicolae G (2006) Bestcut: A graph algorithm for coreference resolution. In: Proceedings of the 2006 conference on empirical methods in natural language processing, Association for Computational Linguistics, pp 275–283
  * Nicolov et al (2008) Nicolov N, Salvetti F, Ivanova S (2008) Sentiment analysis: Does coreference matter. In: AISB 2008 Convention Communication, Interaction and Social Intelligence, vol 1, p 37
  * O’Connor and Heilman (2013) O’Connor B, Heilman M (2013) Arkref: A rule-based coreference resolution system. arXiv preprint arXiv:13101975
  * Oneto et al (2016) Oneto L, Bisio F, Cambria E, Anguita D (2016) Statistical learning theory and ELM for big social data analysis. IEEE Computational Intelligence Magazine 11(3):45–55
  * Peng et al (2015) Peng H, Khashabi D, Roth D (2015) Solving hard coreference problems. In: Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp 809–819
  * Pennington et al (2014) Pennington J, Socher R, Manning C (2014) Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 1532–1543
  * Poesio (2004) Poesio M (2004) Discourse annotation and semantic annotation in the gnome corpus. In: Proceedings of the 2004 ACL Workshop on Discourse Annotation, Association for Computational Linguistics, pp 72–79
  * Poesio and Kabadjov (2004) Poesio M, Kabadjov MA (2004) A general-purpose, off-the-shelf anaphora resolution module: Implementation and preliminary evaluation. In: LREC
  * Poesio et al (2008) Poesio M, Artstein R, et al (2008) Anaphoric annotation in the arrau corpus. In: LREC
  * Poon and Domingos (2008) Poon H, Domingos P (2008) Joint unsupervised coreference resolution with markov logic. In: Proceedings of the conference on empirical methods in natural language processing, Association for Computational Linguistics, pp 650–659
  * Pradhan et al (2011) Pradhan S, Ramshaw L, Marcus M, Palmer M, Weischedel R, Xue N (2011) Conll-2011 shared task: Modeling unrestricted coreference in ontonotes. In: Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, Association for Computational Linguistics, pp 1–27
  * Pradhan et al (2012) Pradhan S, Moschitti A, Xue N, Uryupina O, Zhang Y (2012) Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In: Joint Conference on EMNLP and CoNLL-Shared Task, Association for Computational Linguistics, pp 1–40
  * Pradhan et al (2014) Pradhan S, Luo X, Recasens M, Hovy E, Ng V, Strube M (2014) Scoring coreference partitions of predicted mentions: A reference implementation. In: Proceedings of the conference. Association for Computational Linguistics. Meeting, NIH Public Access, vol 2014, p 30
  * Preuss (1992) Preuss S (1992) Anaphora resolution in machine translation. TU, Fachbereich 20, Projektgruppe KIT
  * Pustejovsky et al (2002) Pustejovsky J, Castano J, Sauri R, Rumshinsky A, Zhang J, Luo W (2002) Medstract: creating large-scale information servers for biomedical libraries. In: Proceedings of the ACL-02 workshop on Natural language processing in the biomedical domain-Volume 3, Association for Computational Linguistics, pp 85–92
  * Quinlan (1986) Quinlan JR (1986) Induction of decision trees. Machine learning 1(1):81–106
  * Raghunathan et al (2010) Raghunathan K, Lee H, Rangarajan S, Chambers N, Surdeanu M, Jurafsky D, Manning C (2010) A multi-pass sieve for coreference resolution. In: Proceedings of 2010 Conference on Empirical Methods in NAturalLanguage Processing, Association for Computational Linguistics, pp 492–501
  * Rahman and Ng (2009) Rahman A, Ng V (2009) Supervised models for coreference resolution. In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 2-Volume 2, Association for Computational Linguistics, pp 968–977
  * Rahman and Ng (2011) Rahman A, Ng V (2011) Coreference resolution with world knowledge. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, Association for Computational Linguistics, pp 814–824
  * Rand (1971) Rand WM (1971) Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association 66(336):846–850
  * Recasens and Hovy (2011) Recasens M, Hovy E (2011) Blanc: Implementing the rand index for coreference evaluation. Natural Language Engineering 17(4):485–510
  * Recasens and Martí (2010) Recasens M, Martí MA (2010) Ancora-co: Coreferentially annotated corpora for spanish and catalan. Language resources and evaluation 44(4):315–345
  * Recasens and Vila (2010) Recasens M, Vila M (2010) On paraphrase and coreference. Computational Linguistics 36(4):639–647
  * Recasens et al (2010) Recasens M, Màrquez L, Sapena E, Martí MA, Taulé M, Hoste V, Poesio M, Versley Y (2010) Semeval-2010 task 1: Coreference resolution in multiple languages. In: Proceedings of the 5th International Workshop on Semantic Evaluation, Association for Computational Linguistics, pp 1–8
  * Recasens et al (2013) Recasens M, de Marneffe MC, Potts C (2013) The life and death of discourse entities: Identifying singleton mentions. In: Proceedings of the 2013 Conference of the North American Chapter of the Association of Computational Linguistics:Human Language Technologies, pp 627–633
  * Regneri and Wang (2012) Regneri M, Wang R (2012) Using discourse information for paraphrase extraction. In: Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Association for Computational Linguistics, pp 916–927
  * Roberts (1989) Roberts C (1989) Modal subordination and pronominal anaphora in discourse. Linguistics and philosophy 12(6):683–721
  * Van der Sandt (1992) Van der Sandt RA (1992) Presupposition projection as anaphora resolution. Journal of semantics 9(4):333–377
  * Segura-Bedmar et al (2009) Segura-Bedmar I, Crespo M, de Pablo C, Martínez P (2009) Drugnerar: linguistic rule-based anaphora resolver for drug-drug interaction extraction in pharmacological documents. In: Proceedings of the third international workshop on Data and text mining in bioinformatics, ACM, pp 19–26
  * Soon et al (2001) Soon WM, Ng HT, Lim DCY (2001) A machine learning approach to coreference resolution of noun phrases. Computational linguistics 27(4):521–544
  * Soricut and Marcu (2003) Soricut R, Marcu D (2003) Sentence level discourse parsing using syntactic and lexical information. In: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, Association for Computational Linguistics, pp 149–156
  * Steinberger et al (2007) Steinberger J, Poesio M, Kabadjov MA, Ježek K (2007) Two uses of anaphora resolution in summarization. Information Processing & Management 43(6):1663–1680
  * Stoyanov et al (2010) Stoyanov V, Cardie C, Gilbert N, Riloff E, Buttler D, Hysom D (2010) Coreference resolution with reconcile. In: Proceedings of the ACL 2010 Conference Short Papers, Association for Computational Linguistics, pp 156–161
  * Strube et al (2002) Strube M, Rapp S, Müller C (2002) The influence of minimum edit distance on reference resolution. In: Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, Association for Computational Linguistics, pp 312–319
  * Su et al (2008) Su J, Yang X, Hong H, Tateisi Y, Tsujii J (2008) Coreference resolution in biomedical texts: a machine learning approach. In: Dagstuhl Seminar Proceedings, Schloss Dagstuhl-Leibniz-Zentrum für Informatik
  * Tateisi et al (2005) Tateisi Y, Yakushiji A, Ohta T, Tsujii J (2005) Syntax annotation for the genia corpus. In: Companion Volume to the Proceedings of Conference including Posters/Demos and tutorial abstracts
  * Tetreault (1999) Tetreault JR (1999) Analysis of syntax-based pronoun resolution methods. In: Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, Association for Computational Linguistics, pp 602–605
  * Tetreault (2001) Tetreault JR (2001) A corpus-based evaluation of centering and pronoun resolution. Computational Linguistics 27(4):507–520
  * Uryupina et al (2012) Uryupina O, Poesio M, Giuliano C, Tymoshenko K (2012) Disambiguation and filtering methods in using web knowledge for coreference resolution. In: Cross-Disciplinary Advances in Applied Natural Language Processing: Issues and Approaches, IGI Global, pp 185–201
  * Vala et al (2016) Vala H, Piper A, Ruths D (2016) The more antecedents, the merrier: Resolving multi-antecedent anaphors. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol 1, pp 2287–2296
  * Valdivia et al (2018) Valdivia A, Luzón V, Cambria E, Herrera F (2018) Consensus vote models for detecting and filtering neutrality in sentiment analysis. Information Fusion 44:126–135
  * Versley et al (2008) Versley Y, Pozetto SP, Poesio M, Eidlman V, Jern A, Smith J, Yang X, Moschitti A (2008) Bart: A modular toolkit for coreference resolution. In: Proceedings of the 46th Annual MEeting of Association for Computational Linguistics on Human Language Technologies: Demo Session, Association for Computational Linguistics, pp 9–12
  * Vicedo and Ferrández (2000) Vicedo JL, Ferrández A (2000) Importance of pronominal anaphora resolution in question answering systems. In: Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, Association for Computational Linguistics, pp 555–562
  * Vieira et al (2005) Vieira R, Salmon-Alt S, Gasperin C, Schang E, Othero G (2005) Coreference and anaphoric relations of demonstrative noun phrases in multilingual corpus. Anaphora Processing: linguistic, cognitive and computational modeling pp 385–403
  * Vilain et al (1995) Vilain M, Burger J, Aberdeen J, Connolly D, Hirschman L (1995) A model-theoretic coreference scoring scheme. In: Proceedings of the 6th conference on Message understanding, Association for Computational Linguistics, pp 45–52
  * Walker (1989) Walker MA (1989) Evaluating discourse processing algorithms. In: Proceedings of the 27th annual meeting on Association for Computational Linguistics, Association for Computational Linguistics, pp 251–261
  * Walker et al (1998) Walker MA, Joshi AK, Prince EF (1998) Centering theory in discourse. Oxford University Press
  * Watson-Gegeo (1981) Watson-Gegeo KA (1981) The pear stories: Cognitive, cultural, and linguistic aspects of narrative production
  * Werlen and Popescu-Belis (2017) Werlen LM, Popescu-Belis A (2017) Using coreference links to improve spanish-to-english machine translation. In: Proceedings of the 2nd Workshop on Coreference Resolution Beyond OntoNotes (CORBON 2017), pp 30–40
  * Weston et al (2015) Weston J, Bordes A, Chopra S, Rush AM, van Merriënboer B, Joulin A, Mikolov T (2015) Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:150205698
  * Wiseman et al (2015) Wiseman S, Rush AM, Shieber S, Weston J (2015) Learning anaphoricity and antecedent ranking features for coreference resolution. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), vol 1, pp 1416–1426
  * Wiseman et al (2016) Wiseman S, Rush AM, Shieber SM (2016) Learning global features for coreference resolution. arXiv preprint arXiv:160403035
  * Witte and Bergler (2003) Witte R, Bergler S (2003) Fuzzy coreference resolution for summarization. In: Proceedings of 2003 International Symposium on Reference Resolution and Its Applications to Question Answering and Summarization (ARQAS), pp 43–50
  * Yang et al (2003) Yang X, Zhou G, Su J, Tan CL (2003) Coreference resolution using competition learning approach. In: Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1, Association for Computational Linguistics, pp 176–183
  * Yang et al (2004) Yang X, Zhou G, Su J, Tan CL (2004) Improving noun phrase coreference resolution by matching strings. In: International Conference on Natural Language Processing, Springer, pp 22–31
  * Yang et al (2008) Yang X, Su J, Tan CL (2008) A twin-candidate model for learning-based anaphora resolution. Computational Linguistics 34(3):327–356
  * Young et al (2018a) Young T, Cambria E, Chaturvedi I, Zhou H, Biswas S, Huang M (2018a) Augmenting end-to-end dialogue systems with commonsense knowledge. In: AAAI, pp 4970–4977
  * Young et al (2018b) Young T, Hazarika D, Poria S, Cambria E (2018b) Recent trends in deep learning based natural language processing. IEEE Computational Intelligence Magazine 13(3)
  * Zeldes (2017) Zeldes A (2017) The gum corpus: creating multilayer resources in the classroom. Language Resources and Evaluation 51(3):581–612
  * Zeldes and Zhang (2016) Zeldes A, Zhang S (2016) When annotation schemes change rules help: A configurable approach to coreference resolution beyond ontonotes. In: Proceedings of the Workshop on Coreference Resolution Beyond OntoNotes (CORBON 2016), pp 92–101
  * Zhuang et al (2006) Zhuang L, Jing F, Zhu XY (2006) Movie review mining and summarization. In: Proceedings of the 15th ACM international conference on Information and knowledge management, ACM, pp 43–50


# An Evaluation Dataset and Strategy for Building Robust Multi-turn Response
Selection Model

Kijong Han  Seojin Lee  Wooin Lee  Kakao Enterprise, South Korea  Joosung Lee
Kakao Enterprise, South Korea  Dong-hun Lee

###### Abstract

Multi-turn response selection models have recently shown comparable
performance to humans in several benchmark datasets. However, in the real
environment, these models often have weaknesses, such as making incorrect
predictions based heavily on superficial patterns without a comprehensive
understanding of the context. For example, these models often give a high
score to the wrong response candidate containing several keywords related to
the context but using the inconsistent tense. In this study, we analyze the
weaknesses of the open-domain Korean Multi-turn response selection models and
publish an adversarial dataset to evaluate these weaknesses. We also suggest a
strategy to build a robust model in this adversarial environment.

##  1 Introduction

Multi-turn response selection is a task that selects the best response among
given candidates for a given dialogue context. Response selection models have
recently shown comparable performance to humans (Cui et al., 2020) in the
several in-domain/held-out benchmarks (Lowe et al., 2015; Zhang et al., 2018a;
Dinan et al., 2020). However, in the actual service environment, these models
are often found to have weaknesses. For example, the model gives the highest
score to the wrong response, which has high word overlap with the context
(Yuan et al., 2019) or semantically similar to the context (Whang et al.,
2021).

Held-out evaluation often overestimates the real-world performance of the
model (Ribeiro et al., 2020), so adversarial datasets for evaluating
weaknesses have been constructed for each task, such as NLI (Naik et al.,
2018; McCoy et al., 2019), and MRC (Jia and Liang, 2017; Rajpurkar et al.,
2018). A framework for comprehensively evaluating the general linguistic
abilities of the model was also studied (Ribeiro et al., 2020).

Several works evaluated adversarial cases for the response selection task Yuan
et al. (2019); Whang et al. (2021). However, they just automatically generate
adversarial responses by copying words in the context. In this study, we
analyze the weaknesses of the various aspects of the open-domain Korean Multi-
turn Response Selection models and construct an adversarial dataset manually.
A total of 2,220 test cases are constructed, and each test case are classified
by type.

Neural networks do not generalize well to such an adversarial setting because
they tend to use superficial patterns and spurious correlation of the dataset
overly, which makes models biased(Clark et al., 2019; Nam et al., 2020). Thus,
various debiasing methods have been studied to alleviate this phenomenon (He
et al., 2019; Utama et al., 2020). In this study, we show that debasing method
is also effective in adeversarial evaluation for multi-turn response selection
task.

Type | Context | Adversarial Response | # cases
---|---|---|---
Repetition | [A] I’m hungry / [B] What do you want to eat? | I’m hungry | 400
| Negation
---
| [A] Wrap up before you go outside / [B] Why?
---
[A] It’s freezing cold.
| Yes, indeed. it’s not that cold
---
today.
454
| Tense
---
| [A] I can’t wait to watch "Joker" / [B] I watched the movie.
---
It was really impressive. / [A] Wow! I should watch it.
You really enjoyed it. | 158
| Subject-
---
Object
| [A] I’m in love with BTS / [B] Why do you like them
---
so much? / [A] $\phi$(their) Songs are great
Thanks (for complimenting me) | 374
| Lexcial
---
Contradiction
[A] It’s freezing cold today. |  | Yes, indeed. It’s way too hot
---
out today.
254
| Interrogative
---
Word
| [A] I saw Jennie today / [B] What does she look like? /
---
[A] $\phi$(she) Looks so pretty
Who’s so pretty? | 236
Topic |  | [A] Isn’t the weather nice today? / [B] Oh, is it?
---
[A] Yeah, it’s sunny and warm.
Bring your umbrella with you. | 344
Table 1: Examples of adversarial data for each type. $\phi$ denotes a zero
anaphora in Korean.

In the retrieval-based chatbot system where response selection is used,
response candidates are composed as follows. All utterances in the database
are used as response candidates (Humeau et al., 2020), or part of them
filtered through search engines are used (Zhou et al., 2020). To filter the
candidates, machine learning-based embeddings or word-level similarity
algorithms(e.g., BM25), which also have weaknesses in an adversarial setting,
are used (Zhou et al., 2020). Therefore, almost every time a response is
selected by the actual system, adversarial cases are included in the
candidates. Thus, robustness to adversarial cases is more important for
response selection task. We also construct a real environment test set and
experiment that the model robust to an adversarial case has high performance
in the real environment.

##  2 Adversarial Test Dataset

We analyze the incorrect responses in the internal service log and categorize
the types of frequent errors. There are a total of seven types, and details of
each type are as follows.

Repetition An incorrect response repeating one of the utterances in the
context.

Negation A negation is either added to or omitted from a correct response,
generating an erroneous response with reversed affirmative or negative
meaning. A test set for a negation error intentionally generates a negative
response by adding or removing ‘안’ or ‘못’, which are negative adverbs in
Korean (short-form negation) or ‘-지 않다,’ ‘-지 못하다, ’ or ‘-지 말다’ which are
negative auxiliary predicates in Korean (Long-form negation) in order to test
whether the model understands such semantic reversal.

Tense A morpheme or expression marking tense is added to or removed from a
correct response, generating an erroneous response in tense that is
inconsistent with the given context. A test set for tense errors adds or
replaces morphemes or expressions marking the future tense such as ‘-겠-,’ or
ones marking the past tense such as‘-었-’ to test whether the model fully
understands the context disconnection triggered by such tense change.

Subject-Object A test set for subject-object errors generates a response
inconsistent with the context due to confusion of the subject and object for a
certain action. In particular, since zero anaphora can be found frequently in
Korean sentences, incorrect responses are often made because of a failure in
identifying the hidden subject of the previous context. This test set uses a
subject or an object differently from the ones used in a correct response to
examine whether the model fully understands the context disconnection caused
by such errors.

Lexical Contradiction A key lexicon of a correct response is replaced with one
that holds either conflicting or opposite meaning against the said key
lexicon, generating an incorrect response. A test set for lexical
contradiction errors replaces a key lexicon in a sentence with an antonym
(e.g. hot vs cold) or a word that cannot be used instead (e.g. rain vs snow)
to check whether the model understands the precise meaning of such lexicon.

Interrogative Word A test set for interrogative word errors generates a
response in a form of 5W1H questions to ask for information that has already
been explicitly or implicitly shared in previous dialogues.

Topic A key sentence or vocabulary is replaced with another sentence or term
that does not fit in the previous context even though they frequently appear
together in the given topic. While this error is similar to the lexical
contradiction error to a certain extent, the replacement words used in this
test do not hold conflicting or opposite meanings but instead have less
semantic relevance to the context of the previous dialogue (e.g. sunny vs
umbrella). The test set assesses whether a model fully understands the fact
that while the replacement vocabulary is the one that is frequently used in
the same given topic, the response does not correctly reflect the context of
the previous dialogue.

Five annotators generate a total of 200 dialogue sessions. For each session
$i$, annotators create two correct responses and an arbitrary number($M_{i}$)
of incorrect responses based on the instruction described above. All sessions
and responses are reviewed and filtered by experts. We set up one test case to
consist of context, one correct response, and one incorrect response.
Therefore, $2*M_{i}$ test cases were extracted for each session, and a total
of 2,220 test cases are constructed. It evaluates whether the model gives the
correct answer a higher score than the incorrect one for a given context.
Statistics and examples are described in Table 1. We release this data set at
https://github.com/kakaoenterprise/KorAdvMRSTestData.

##  3 Method

Suppose that dataset is $D=\left\\{(c_{i},r_{i},y_{i})\right\\}_{i=1}^{N}$,
where $c_{i}$ denotes a dialogue context, $r_{i}$ is a response utterance, and
$y_{i}\in\left\\{0,1\right\\}$ is a label. The context
$c_{i}=\left\\{u_{i,1},u_{i,2},...,u_{i,k_{i}}\right\\}$ consists of sequence
of $k_{i}$ utterances. The label $y_{i}=1$ means that $r_{i}$ is sensible
response for context $c_{i}$.

###  3.1 Baseline: Fine-tuning BERT

We adopt fine-tuning BERT (Devlin et al., 2019) as a baseline. In this work,
similar to the previous works that fine-tuned BERT for the Multi-turn Response
Selection task (Gu et al., 2020; Whang et al., 2020; Han et al., 2021), the
input token sequence of BERT $x_{i}$ is composed as follows.

$\displaystyle x_{i}=[CLS]u_{i,1}[EOU]...u_{i,k_{i}}[EOU][SEP]$ (1)
$\displaystyle r_{i}[EOU][SEP]$

The [EOU] is a special token indicating that the utterance is over. The final
output hidden vector of the [CLS] token in BERT is fed into a fully connected
layer with softmax activation. Then, the BERT is fine-tuned to minimize cross
entropy loss between the target label and output of this layer.

###  3.2 Debiasing Strategy

In general, correct dialogue response utilizes keywords or topics in the
context. Neural networks tend to use such superficial patterns(e.g., keyword,
topic) overly, which makes models biased (Clark et al., 2019; Nam et al.,
2020). We see this bias as the main cause of the response selection model’s
vulnerability to an adversarial environment. Thus, we experimented by applying
various debiasing techniques to the response selection task, and DRiFt (He et
al., 2019) was the most effective. The main concept of the debiasing strategy
we used is to train a debiased model to fit the residual of the biased model,
focusing on examples that cannot be predicted well by biased features only (He
et al., 2019). Details of the method using DRiFt are as follows.

Figure 1: Overall architecture for training debiased model utilizing biased
model. The grey line represents that gradient is backpropagated only to the
debiased model.

First, we train an auxiliary biased model using only biased features. The
biased model is a single fully connected layer with softmax activation and
trained with cross-entropy loss. The biased feature vector used as an input
$\phi_{i}$ is as follows.

$\displaystyle\phi_{i}=[JS_{morph}(c_{i},r_{i}),JS_{morph}(u_{i,k_{i}},r_{i}),$
(2) $\displaystyle
JS_{wordpiece}(c_{i},r_{i}),JS_{wordpiece}(u_{i,k_{i}},r_{i})]$

We use the Jaccard similarity($JS$) between the whole context($c_{i}$) and
response($r_{i}$) as input features. We also use the JS between the last
utterance($u_{i,k_{i}}$) and $r_{i}$, because the last utterance is most
important (Zhang et al., 2018b; Ma et al., 2019). We use two tokenizers: the
WordPiece (Wu et al., 2016), and the morpheme analyzer. We assume that these
words overlap feature could capture keyword and topic bias.

Second, we train a debiased model utilizing a biased model, as shown in Figure
1. The overall structure of the debiased model is the same as the baseline,
but only the learning scheme is different. Let b is output hidden vector of
the biased model, d is output hidden vector of the debiased model,
$p_{b}=softmax(\textbf{b})$, and $p_{d}=softmax(\textbf{d})$. DRiFt method
minimize cross entropy loss between $p_{a}=softmax(\textbf{b}+\textbf{d})$ and
target labels. Thus, the loss function is defined as follows.

| Train | Val |  | In-domain
---
Test
| Adv
---
Test
| Real
---
Test
# pairs | 500K | 10K | 10K | 4,440 | 5,490
# cands | 2 | 10 | 10 | 2 | 10
pos:neg | 1:1 | 1:9 | 1:9 | 1:1 | 4.4:5.6
# turns | 4.6 | 4.5 | 4.6 | 2.9 | 3.3
Table 2: Statistics for each dataset.

Method | In-domain | Adversarial | Real Env.
---|---|---|---
baseline | 86.4±0.5 | 39.4±1.7 | 36.2±0.9
+deb | 85.4±0.7 | 43.5±2.3 | 36.5±0.6
+UMS | 87.5±0.7 | 42.9±2.5 | 38.8±0.6
+UMS+deb | 87.0±0.7 | 47.2±2.1 | 40.4±0.7
Table 3: Overall performance of each method.

Type | Repetition | Negation | Tense |  | Subject
---
-Object
| Lexical
---
Contradiction
| Interrogative
---
Word
Topic
baseline | 12.9±2.5 | 36.1±2.3 | 41.8±2.3 | 55.0±1.9 | 41.1±1.8 | 46.1±2.2 | 50.7±1.8
+deb | 26.2±5.1 | 40.6±2.8 | 43.0±2.6 | 56.2±1.9 | 43.9±1.6 | 45.5±3.2 | 51.8±1.9
+UMS | 26.2±6.2 | 34.5±1.8 | 45.3±2.0 | 61.2±2.9 | 41.6±3.1 | 46.4±2.0 | 51.1±1.6
+UMS+deb | 40.0±5.4 | 38.5±1.1 | 46.9±2.3 | 63.9±3.0 | 43.3±1.7 | 45.9±2.6 | 52.8±1.7
Table 4: Performance for each adversarial type.

$\displaystyle Loss=-\log{p_{a}(y_{i})}=$ (3)
$\displaystyle-\log{p_{b}(y_{i})}-\log{p_{d}(y_{i})}+\log{\sum_{l=0}^{L-1}{p_{b}(l)p_{d}(l)}}$

$L$ is the number of classification classes(2 for this task). The gradient is
backpropagated only to the debiased model. The last term encourages output
from the debiased model $p_{d}$, to have minimal projection on output from the
biased model $p_{b}$ (He et al., 2019). Derivation of equation 3 is in
Appendix A. At test time, only debiased model is used.

###  3.3 Combination with Multi-task Learning

Recently, self-supervised learning approaches have shown state-of-the-art
performance in the response selection task (Whang et al., 2021; Xu et al.,
2021). These works devise auxiliary tasks to understand the dialogue better
and train the model in a multi-task manner. The final loss function in these
methods is the weighted sum of losses of auxiliary tasks and main task (i.e.,
determine given response is a sensible response to the context). Thus,
debiasing strategy could be easily combined with these methods by replacing
the loss function of the main task with equation 3. We also experiment with
self-supervised learning approach UMS (Whang et al., 2021), and we show that
it is also effective in not only in-domain but also adversarial and real
environments.

##  4 Experiments and Results

###  4.1 Experiment Setup

We construct an experimental dataset using the corpus that we produced in-
house and the public Korean dialogue corpus111https://corpus.korean.go.kr. We
split these corpora into three, and each is for training, validation, and
test. Statistics of each dataset are described in Table 2. #pairs denote the
number of context-response pairs, #cands denotes the number of candidates per
context, pos:neg denotes the ratio of positive and negative responses in
candidates, and #turns denote the average turns per context. Details on the
construction are as follows.

Train, valid, and in-domain test The last utterance of the dialogue session is
used as a positive response and the rest as context. Negative responses are
randomly chosen from the other dialogue.

Adversarial test It is described in the Section 2.

Real environment test In a real environment, response candidates are not
sampled randomly but are sampled through a search system (Zhou et al., 2020),
or all utterances without sampling are used as candidates (Humeau et al.,
2020). There are many adversarial negatives in this situation, as described in
Section 1. We build a dataset by simulating this situation in a similar way to
the previous works (Wu et al., 2017; Zhang et al., 2018b).

We take a dialogue session from the test corpus and internal service log as
context. We trained a bi-encoder-based context and response embedding model
(Humeau et al., 2020) and indexed embeddings of all utterances in the corpus.
Then, we retrieve the top 10 utterances based on the similarity score between
context embedding as response candidates. For each response, three annotators
labeled whether it is sensible to the context. The response determined by more
than two people as sensible was selected as the positive response.

###  4.2 Results

We measure the performance ten times for each model and report the mean and
standard deviation in Table 3. See Appendix B for details of training. The
baseline is a fine-tuned BERT described in Section 3.1. "deb" denotes a
debiasing strategy described in Section 3.2. UMS denotes a self-supervised
multi-task learning method described in Section 3.3. Precision@1 is used as an
evaluation metric for all test sets.

Debiasing strategy significantly improves adversarial test performance in both
baseline and UMS model; it achieves absolute improvements of 4.1% and 4.3% on
baseline and UMS. A decline in performance is observed in the in-domain test;
-1.0% and -0.5% on baseline and UMS, as the DRiFt debiasing method (He et al.,
2019) shows a slight performance degradation in the in-domain test. However,
It improves performance in the comprehensive real environment test; +0.3% and
+1.6% on baseline and UMS. This supports our argument that robustness to
adversarial cases is important in the response selection task. Additionally,
+UMS+deb outperforms +deb in all test set. From this, it can be seen that the
debiasing strategy and UMS have a synergistic effect.

The performance of each adversarial type is reported in Table 4. Since we used
word-level Jacard Similarity as a biased feature, the debiasing strategy shows
huge performance improvement in the Repetition type, which simply uses word
sequence in context as a negative response. There is no improvement in the
Interrogative Word type. We assume that the reason for it is that this type is
difficult because it requires understanding all 5W1H from the context.

##  5 Conclusion

We analyze the weaknesses of the open-domain Korean Multi-turn Response
Selection models and publish an adversarial dataset to evaluate these
weaknesses. We suggest a strategy to build a robust model to an adversarial
and real environment with the experimental results. We expect that this work
and dataset will help improve the response selection model.

##  6 Ethical Considerations

The adversarial dataset we publish is generated manually. All sessions and
responses in the dataset are reviewed and filtered by the experts, and we also
considered ethical issues in this process. Thus, there is no hate speech or
privacy issue in our dataset.

## References

  * Clark et al. (2019) Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019.  Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4060–4073.
  * Cui et al. (2020) Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. 2020.  Mutual: A dataset for multi-turn dialogue reasoning.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1406–1416.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Dinan et al. (2020) Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2020.  The second conversational intelligence challenge (convai2).  In _The NeurIPS’18 Competition_ , pages 187–208. Springer.
  * Gu et al. (2020) Jia-Chen Gu, Tianda Li, Quan Liu, Zhen-Hua Ling, Zhiming Su, Si Wei, and Xiaodan Zhu. 2020.  Speaker-aware bert for multi-turn response selection in retrieval-based chatbots.  In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_, pages 2041–2044.
  * Han et al. (2021) Janghoon Han, Taesuk Hong, Byoungjae Kim, Youngjoong Ko, and Jungyun Seo. 2021.  Fine-grained post-training for improving retrieval-based dialogue systems.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1549–1558.
  * He et al. (2019) He He, Sheng Zha, and Haohan Wang. 2019.  Unlearn dataset bias in natural language inference by fitting the residual.  _EMNLP-IJCNLP 2019_ , page 132.
  * Humeau et al. (2020) Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020.  Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring.  In _International Conference on Learning Representations_.
  * Jia and Liang (2017) Robin Jia and Percy Liang. 2017.  Adversarial examples for evaluating reading comprehension systems.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2021–2031.
  * Lowe et al. (2015) Ryan Lowe, Nissan Pow, Iulian Vlad Serban, and Joelle Pineau. 2015.  The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems.  In _Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 285–294.
  * Ma et al. (2019) Wentao Ma, Yiming Cui, Nan Shao, Su He, Weinan Zhang, Ting Liu, Shijin Wang, and Guoping Hu. 2019.  Triplenet: Triple attention network for multi-turn response selection in retrieval-based chatbots.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 737–746.
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3428–3448.
  * Naik et al. (2018) Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018.  Stress test evaluation for natural language inference.  _arXiv preprint arXiv:1806.00692_.
  * Nam et al. (2020) Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. 2020.  Learning from failure: Training debiased classifier from biased classifier.  _NeurIPS_.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for squad.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789.
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond accuracy: Behavioral testing of nlp models with checklist.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4902–4912.
  * Utama et al. (2020) Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020.  Mind the trade-off: Debiasing nlu models without degrading the in-distribution performance.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8717–8729.
  * Whang et al. (2020) Taesun Whang, Dongyub Lee, Chanhee Lee, Kisu Yang, Dongsuk Oh, and Heuiseok Lim. 2020.  An effective domain adaptive post-training method for bert in response selection.  _Proc. Interspeech 2020_ , pages 1585–1589.
  * Whang et al. (2021) Taesun Whang, Dongyub Lee, Dongsuk Oh, Chanhee Lee, Kijong Han, Dong-hun Lee, and Saebyeok Lee. 2021.  Do response selection models really know what’s next? utterance manipulation strategies for multi-turn response selection.  _AAAI_.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_.
  * Wu et al. (2017) Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. 2017.  Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 496–505.
  * Xu et al. (2021) Ruijian Xu, Chongyang Tao, Daxin Jiang, Xueliang Zhao, Dongyan Zhao, and Rui Yan. 2021.  Learning an effective context-response matching model with self-supervised tasks for retrieval-based dialogues.  _AAAI_.
  * Yuan et al. (2019) Chunyuan Yuan, Wei Zhou, Mingming Li, Shangwen Lv, Fuqing Zhu, Jizhong Han, and Songlin Hu. 2019.  Multi-hop selector network for multi-turn response selection in retrieval-based chatbots.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 111–120.
  * Zhang et al. (2018a) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018a.  Personalizing dialogue agents: I have a dog, do you have pets too?  _arXiv preprint arXiv:1801.07243_.
  * Zhang et al. (2018b) Zhuosheng Zhang, Jiangtong Li, Pengfei Zhu, Hai Zhao, and Gongshen Liu. 2018b.  Modeling multi-turn conversation with deep utterance aggregation.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 3740–3752.
  * Zhou et al. (2020) Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum. 2020.  The design and implementation of xiaoice, an empathetic social chatbot.  _Computational Linguistics_ , 46(1):53–93.

##  Appendix A Derivation of Loss Function

Let $b$ is output hidden vector of the biased model, $d$ is output hidden
vector of the debiased model, $y_{i}\in\left\\{0,1\right\\}$ is the label
value, $p_{b}=softmax(b)$, $p_{d}=softmax(d)$, and $p_{a}=softmax(b+d)$.

$\displaystyle Loss=-\log p_{a}(y)$ $\displaystyle=-\log
e^{b_{y}+d_{y}}+\log\sum_{l}e^{b_{l}+d_{l}}$ $\displaystyle=-\log
e^{b_{y}}-\log e^{d_{y}}+\log\sum_{l}e^{b_{l}}e^{d_{l}}$ $\displaystyle=-\log
e^{b_{y}}-\log e^{d_{y}}+\log\sum_{l}e^{b_{l}}e^{d_{l}}$
$\displaystyle+\log\sum_{l}e^{b_{l}}-\log\sum_{l}e^{b_{l}}$
$\displaystyle+\log\sum_{l}e^{d_{l}}-\log\sum_{l}e^{d_{l}}$
$\displaystyle=-(\log e^{b_{y}}-\log\sum_{l}e^{b_{l}})$ $\displaystyle-(\log
e^{d_{y}}-\log\sum_{l}e^{d_{l}})$
$\displaystyle+(\log\sum_{l}e^{b_{l}}e^{d_{l}}-\log\sum_{l}e^{b_{l}}\sum_{l}e^{d_{l}})$
$\displaystyle=-\log\frac{e^{b_{y}}}{\sum_{l}e^{b_{l}}}-\log\frac{e^{d_{y}}}{\sum_{l}e^{d_{l}}}$
$\displaystyle+\log\sum_{l}\frac{e^{b_{l}}e^{d_{l}}}{\sum_{l}e^{b_{l}}\sum_{l}e^{d_{l}}}$
$\displaystyle=-\log p_{b}(y)-\log p_{d}(y)+log\sum_{l}p_{b}(l)p_{d}(l)$

##  Appendix B Training Details

The biased model, which consists of a single fully connected layer, is trained
using the AdamW optimizer with a learning rate of 5e-4 and for 3 epochs. BERT-
based models, including baseline, UMS, and debiased models, are trained using
the AdamW optimizer with a learning rate of 2.5e-5 and for 3 epochs on 4
Nvidia Volta v100 GPU. The batch size is 128 for every model. We train and
evaluate 10 times for each model and calculate mean and standard deviation.
For each model, a checkpoint that shows the best performance in the real
environment is selected for performance measure.