Approach   Labels/Aliases     Descriptions    Knowledge graph structure     Hyper-relational structure     Types     Additional Information
AGDISTIS [112]

MAG [75]                         STANDS4 <ref>

DoSeR [137]

Comparison between the utilized Wikidata characteristics of each KG-agnostic approach.

§ EL-ONLY RESULTS AND DISCUSSION

[origin=c]-66Mulang et al. [77]
[origin=c]-66LSH-ELMo model [85]
[origin=c]-663cmNED using DL on Graphs [17] tn:elonly1    [origin=c]-66Botha et al. [15]

AIDA-CoNLL [50]   0.9494 [77] tn:elonly6 , tn:elonly3   0.73 [85]   -   -

ISTEX-1000 [23]   0.9261 [77] tn:elonly4   -   -   -

Wikidata-Disamb [17]   0.9235 [77] tn:elonly5   -   0.916 [17]   -

Mewsli-9 [15]      -     -     -     0.91 [15] tn:elonly7

1  Model with best result

2  Accuracy instead of $F_1$

3  DCA-SL used

4  XLNet used

5  Roberta used

6  Recall instead of $F_1$

Results: EL-only.

The results for EL-only approaches can be found in Table <ref>.
AIDA-CoNLL results are available for three of the four approaches, but the results for one is the accuracy instead of the $F_1$-measures.

The available labels for each item and property make language-model-based approaches possible that perform quite well [77]. No approaches are available to compare to the one by Botha et al. [15], but the result demonstrates the promising performance of multilingual EL with Wikidata as the target KG.

Dataset     Links

T-REx [32]     <https://hadyelsahar.github.io/t-rex>

NYT2018 [68, 67]     not found

ISTEX-1000 [23]     <https://github.com/wetneb/opentapioca/blob/master/data>

LC-QuAD 2.0 [26]      <https://github.com/AskNowQA/LC-QuAD2.0/tree/master/dataset>

Knowledge Net [22]     <https://github.com/diffbot/knowledge-net/tree/master/dataset>

KORE50DYWC [79]     <https://www.aifb.kit.edu/web/KORE_50
Kensho Derived Wikimedia Dataset [55]      <https://www.kaggle.com/kenshoresearch/kensho-derived-wikimedia-data>

CLEF HIPE 2020 [28]     <https://github.com/impresso/CLEF-HIPE-2020/tree/master/data>

Mewsli-9 [15]     <https://metatext.io/datasets/mewsli-9->

TweekiData [45]     <https://github.com/ucinlp/tweeki/tree/main/data/Tweeki_data>

TweekiGold [45]    <https://github.com/ucinlp/tweeki/tree/main/data/Tweeki_gold>

Links to datasets

#1ISBN #1#1#1 #1#1 #1#1#1#1 #1#1 #1#1#1#1 #1#1 #1#1 #1#1 #1#1 #1#1 et al. #1#1 #1#1 #1#1 #1#1 #1#1 #1#1 #1#1 #1#1 #1#1 #1#1 #1#1 #1#1 #1#1 <><#>1#1 #1#1 #1#1 #1#1 #1#1 #1#1 ˜˜126

[1]

P. Agarwal,
J. Strötgen,
L.D. Corro,
J. Hoffart and
G. Weikum,
diaNED: Time-Aware Named Entity Disambiguation for Diachronic
in: Proceedings of the 56th Annual Meeting of the Association for
Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20,
2018, Volume 2: Short Papers,
I. Gurevych and
Y. Miyao, eds,
Association for Computational Linguistics,
pp. 686–693.

[2]

T. Al-Moslmi,
M.G. Ocaña,
A.L. Opdahl and
C. Veres,
Named Entity Extraction for Knowledge Graphs: A Literature
IEEE Access

[3]

M. Albooyeh,
R. Goel and
S.M. Kazemi,
Out-of-Sample Representation Learning for Knowledge Graphs,
in: Findings of the Association for Computational Linguistics: EMNLP
2020, Online Event, 16-20 November 2020,
T. Cohn,
Y. He and
Y. Liu, eds,
Findings of ACL,
Vol. EMNLP 2020,
Association for Computational Linguistics,
pp. 2657–2666.

[4]

P.D. Almeida,
J.G. Rocha,
A. Ballatore and
A. Zipf,
Where the Streets Have Known Names,
in: Computational Science and Its Applications - ICCSA 2016 - 16th
International Conference, Beijing, China, July 4-7, 2016, Proceedings, Part
O. Gervasi,
B. Murgante,
S. Misra,
A.M.A.C. Rocha,
C.M. Torre,
D. Taniar,
B.O. Apduhan,
E.N. Stankova and
S. Wang, eds,
Lecture Notes in Computer Science,
Vol. 9789,
pp. 1–12.

[5]

G. Angeli,
M.J.J. Premkumar and
C.D. Manning,
Leveraging Linguistic Structure For Open Domain Information
in: Proceedings of the 53rd Annual Meeting of the Association for
Computational Linguistics and the 7th International Joint Conference on
Natural Language Processing of the Asian Federation of Natural Language
Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long
The Association for Computer Linguistics,
pp. 344–354.

[6]

J. Baek,
D.B. Lee and
S.J. Hwang,
Learning to Extrapolate Knowledge: Transductive Few-shot Out-of-Graph
Link Prediction,
in: Advances in Neural Information Processing Systems 33: Annual
Conference on Neural Information Processing Systems 2020, NeurIPS 2020,
December 6-12, 2020, virtual,
H. Larochelle,
M. Ranzato,
R. Hadsell,
M. Balcan and
H. Lin, eds,

[7]

I. Balazevic,
C. Allen and
T.M. Hospedales,
Multi-relational Poincaré Graph Embeddings,
in: Advances in Neural Information Processing Systems 32: Annual
Conference on Neural Information Processing Systems 2019, NeurIPS 2019,
December 8-14, 2019, Vancouver, BC, Canada,
H.M. Wallach,
H. Larochelle,
A. Beygelzimer,
F. d'Alché-Buc,
E.B. Fox and
R. Garnett, eds,
pp. 4465–4475.

[8]

D. Banerjee,
D. Chaudhuri,
M. Dubey and
J. Lehmann,
PNEL: Pointer Network Based End-To-End Entity Linking over Knowledge
in: The Semantic Web - ISWC 2020 - 19th International Semantic Web
Conference, Athens, Greece, November 2-6, 2020, Proceedings, Part I,
J.Z. Pan,
V.A.M. Tamma,
C. d'Amato,
K. Janowicz,
B. Fu,
A. Polleres,
O. Seneviratne and
L. Kagal, eds,
Lecture Notes in Computer Science,
Vol. 12506,
pp. 21–38.

[9]

A. Bastos,
A. Nadgeri,
K. Singh,
I.O. Mulang,
S. Shekarpour,
J. Hoffart and
M. Kaul,
RECON: Relation Extraction using Knowledge Graph Context in a Graph
Neural Network,
in: WWW '21: The Web Conference 2021, Virtual Event / Ljubljana,
Slovenia, April 19-23, 2021,
J. Leskovec,
M. Grobelnik,
M. Najork,
J. Tang and
L. Zia, eds,
ACM / IW3C2,
pp. 1673–1685.

[10]

D. Beckett,
T. Berners-Lee,
E. Prud’hommeaux and
G. Carothers,
RDF 1.1 Turtle,
World Wide Web Consortium

[11]

K.D. Bollacker,
C. Evans,
P. Paritosh,
T. Sturge and
J. Taylor,
Freebase: a collaboratively created graph database for structuring
human knowledge,
in: Proceedings of the ACM SIGMOD International Conference on
Management of Data, SIGMOD 2008, Vancouver, BC, Canada, June 10-12, 2008,
J.T. Wang, ed.,
pp. 1247–1250.

[12]

A. Bordes,
N. Usunier,
A. García-Durán,
J. Weston and
O. Yakhnenko,
Translating Embeddings for Modeling Multi-relational Data,
in: Advances in Neural Information Processing Systems 26: 27th Annual
Conference on Neural Information Processing Systems 2013. Proceedings of a
meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States,
C.J.C. Burges,
L. Bottou,
Z. Ghahramani and
K.Q. Weinberger, eds,
pp. 2787–2795.

[13]

A. Bordes,
N. Usunier,
S. Chopra and
J. Weston,
Large-scale Simple Question Answering with Memory Networks,

[14]

E. Boros,
E.L. Pontes,
L.A. Cabrera-Diego,
A. Hamdi,
J.G. Moreno,
N. Sidère and
A. Doucet,
Robust Named Entity Recognition and Linking on Historical Multilingual
in: Working Notes of CLEF 2020 - Conference and Labs of the
Evaluation Forum, Thessaloniki, Greece, September 22-25, 2020,
L. Cappellato,
C. Eickhoff,
N. Ferro and
A. Névéol, eds,
CEUR Workshop Proceedings,
Vol. 2696,

[15]

J.A. Botha,
Z. Shan and
D. Gillick,
Entity Linking in 100 Languages,
in: Proceedings of the 2020 Conference on Empirical Methods in Natural
Language Processing, EMNLP 2020, Online, November 16-20, 2020,
B. Webber,
T. Cohn,
Y. He and
Y. Liu, eds,
Association for Computational Linguistics,
pp. 7833–7845.

[16]

M. Burtsev,
A. Seliverstov,
R. Airapetyan,
M. Arkhipov,
D. Baymurzina,
N. Bushkov,
O. Gureenkova,
T. Khakhulin,
Y. Kuratov,
D. Kuznetsov ,
Deeppavlov: Open-source library for dialogue systems,
in: Proceedings of ACL 2018, System Demonstrations,
pp. 122–127.

[17]

A. Cetoli,
M. Akbari,
S. Bragaglia,
A.D. O'Harney and
M. Sloan,
Named Entity Disambiguation using Deep Learning on Graphs,

[18]

I. Chami,
A. Wolf,
D. Juan,
F. Sala,
S. Ravi and
C. Ré,
Low-Dimensional Hyperbolic Knowledge Graph Embeddings,
in: Proceedings of the 58th Annual Meeting of the Association for
Computational Linguistics, ACL 2020, Online, July 5-10, 2020,
D. Jurafsky,
J. Chai,
N. Schluter and
J.R. Tetreault, eds,
Association for Computational Linguistics,
pp. 6901–6914.

[19]

M. Cornolti,
P. Ferragina and
M. Ciaramita,
A framework for benchmarking entity-annotation systems,
in: 22nd International World Wide Web Conference, WWW '13, Rio de
Janeiro, Brazil, May 13-17, 2013,
D. Schwabe,
V.A.F. Almeida,
H. Glaser,
R. Baeza-Yates and
S.B. Moon, eds,
International World Wide Web Conferences Steering Committee /
pp. 249–260.

[20]

L.D. Corro and
R. Gemulla,
ClausIE: clause-based open information extraction,
in: 22nd International World Wide Web Conference, WWW '13, Rio de
Janeiro, Brazil, May 13-17, 2013,
D. Schwabe,
V.A.F. Almeida,
H. Glaser,
R. Baeza-Yates and
S.B. Moon, eds,
International World Wide Web Conferences Steering Committee /
pp. 355–366.

[21]

A.A. Daruna,
M. Gupta,
M. Sridharan and
S. Chernova,
Continual Learning of Knowledge Graph Embeddings,
IEEE Robotics Autom. Lett.

[22]

F. de Sá Mesquita,
M. Cannaviccio,
J. Schmidek,
P. Mirza and
D. Barbosa,
KnowledgeNet: A Benchmark Dataset for Knowledge Base Population,
in: Proceedings of the 2019 Conference on Empirical Methods in Natural
Language Processing and the 9th International Joint Conference on Natural
Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7,
K. Inui,
J. Jiang,
V. Ng and
X. Wan, eds,
Association for Computational Linguistics,
pp. 749–758.

[23]

A. Delpeuch,
OpenTapioca: Lightweight Entity Linking for Wikidata,
in: Proceedings of the 1st Wikidata Workshop (Wikidata 2020)
co-located with 19th International Semantic Web Conference(OPub 2020),
Virtual Conference, November 2-6, 2020,
L. Kaffee,
O. Tifrea-Marciuska,
E. Simperl and
D. Vrandecic, eds,
CEUR Workshop Proceedings,
Vol. 2773,

[24]

L. Derczynski,
D. Maynard,
G. Rizzo,
M. van Erp,
G. Gorrell,
R. Troncy,
J. Petrak and
K. Bontcheva,
Analysis of named entity recognition and linking for tweets,
Inf. Process. Manag.

[25]

J. Devlin,
M. Chang,
K. Lee and
K. Toutanova,
BERT: Pre-training of Deep Bidirectional Transformers for Language
in: Proceedings of the 2019 Conference of the North American Chapter
of the Association for Computational Linguistics: Human Language
Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume
1 (Long and Short Papers),
J. Burstein,
C. Doran and
T. Solorio, eds,
Association for Computational Linguistics,
pp. 4171–4186.

[26]

M. Dubey,
D. Banerjee,
A. Abdelkawi and
J. Lehmann,
LC-QuAD 2.0: A Large Dataset for Complex Question Answering over
Wikidata and DBpedia,
in: The Semantic Web - ISWC 2019 - 18th International Semantic Web
Conference, Auckland, New Zealand, October 26-30, 2019, Proceedings, Part
C. Ghidini,
O. Hartig,
M. Maleshkova,
V. Svátek,
I.F. Cruz,
A. Hogan,
J. Song,
M. Lefrançois and
F. Gandon, eds,
Lecture Notes in Computer Science,
Vol. 11779,
pp. 69–78.

[27]

L. Ehrlinger and
W. Wöß,
Towards a Definition of Knowledge Graphs,
in: Joint Proceedings of the Posters and Demos Track of the 12th
International Conference on Semantic Systems - SEMANTiCS2016 and the 1st
International Workshop on Semantic Change & Evolving Semantics
(SuCCESS'16) co-located with the 12th International Conference on Semantic
Systems (SEMANTiCS 2016), Leipzig, Germany, September 12-15, 2016,
M. Martin, M. Cuquet and E. Folmer, eds,
CEUR Workshop Proceedings,
Vol. 1695,

[28]

M. Ehrmann,
M. Romanello,
A. Flückiger and
S. Clematide,
Extended overview of CLEF HIPE 2020: named entity processing on
historical newspapers,
in: CLEF,

[29]

M. Ehrmann,
M. Romanello,
A. Flückiger and
S. Clematide,
Extended overview of CLEF HIPE 2020: named entity processing on
historical newspapers,
in: CLEF,

[30]

C.B. El Vaigh,
G. Le Noé-Bienvenu,
G. Gravier and
P. Sébillot,
IRISA System for Entity Detection and Linking at CLEF HIPE 2020,
in: CEUR Workshop Proceedings,

[31]

R. Ellgren,
Exploring Emerging Entities and Named Entity Disambiguation in News Articles,
Master's thesis,
Linköping University,

[32]

H. ElSahar,
P. Vougiouklis,
A. Remaci,
C. Gravier,
J.S. Hare,
F. Laforest and
E. Simperl,
T-REx: A Large Scale Alignment of Natural Language with Knowledge
Base Triples,
in: Proceedings of the Eleventh International Conference on Language
Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018,
N. Calzolari,
K. Choukri,
C. Cieri,
T. Declerck,
S. Goggi,
K. Hasida,
H. Isahara,
B. Maegaard,
J. Mariani,
H. Mazo,
A. Moreno,
J. Odijk,
S. Piperidis and
T. Tokunaga, eds,
European Language Resources Association (ELRA),

[33]

A. Fader,
S. Soderland and
O. Etzioni,
Identifying Relations for Open Information Extraction,
in: Proceedings of the 2011 Conference on Empirical Methods in Natural
Language Processing, EMNLP 2011, 27-31 July 2011, John McIntyre Conference
Centre, Edinburgh, UK, A meeting of SIGDAT, a Special Interest Group of the
pp. 1535–1545.

[34]

M. Färber,
F. Bartscherer,
C. Menne and
A. Rettinger,
Linked data quality of DBpedia, Freebase, OpenCyc, Wikidata, and
Semantic Web

[35]

Freebase Blog,
Introducing the Freebase RDF service,

[36]

M. Galkin,
J. Wu,
E. Denis and
W.L. Hamilton,
NodePiece: Compositional and Parameter-Efficient Representations of Large
Knowledge Graphs.

[37]

M. Galkin,
P. Trivedi,
G. Maheshwari,
R. Usbeck and
J. Lehmann,
Message Passing for Hyper-Relational Knowledge Graphs,
in: Proceedings of the 2020 Conference on Empirical Methods in Natural
Language Processing, EMNLP 2020, Online, November 16-20, 2020,
B. Webber,
T. Cohn,
Y. He and
Y. Liu, eds,
Association for Computational Linguistics,
pp. 7346–7359.

[38]

O. Ganea and
T. Hofmann,
Deep Joint Entity Disambiguation with Local Neural Attention,
in: Proceedings of the 2017 Conference on Empirical Methods in Natural
Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11,
M. Palmer,
R. Hwa and
S. Riedel, eds,
Association for Computational Linguistics,
pp. 2619–2629.

[39]

M. Gardner,
J. Grus,
M. Neumann,
O. Tafjord,
P. Dasigi,
N.F. Liu,
M.E. Peters,
M. Schmitz and
L. Zettlemoyer,
AllenNLP: A Deep Semantic Natural Language Processing Platform,

[40]

K. Gashteovski,
R. Gemulla and
L.D. Corro,
MinIE: Minimizing Facts in Open Information Extraction,
in: Proceedings of the 2017 Conference on Empirical Methods in Natural
Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11,
M. Palmer,
R. Hwa and
S. Riedel, eds,
Association for Computational Linguistics,
pp. 2630–2640.

[41]

D. Gillick,
S. Kulkarni,
L. Lansing,
A. Presta,
J. Baldridge,
E. Ie and
D. García-Olano,
Learning Dense Representations for Entity Retrieval,
in: Proceedings of the 23rd Conference on Computational Natural
Language Learning, CoNLL 2019, Hong Kong, China, November 3-4, 2019,
M. Bansal and
A. Villavicencio, eds,
Association for Computational Linguistics,
pp. 528–537.

[42]

A. Gionis,
P. Indyk and
R. Motwani,
Similarity Search in High Dimensions via Hashing,
in: VLDB'99, Proceedings of 25th International Conference on Very
Large Data Bases, September 7-10, 1999, Edinburgh, Scotland, UK,
M.P. Atkinson,
M.E. Orlowska,
P. Valduriez,
S.B. Zdonik and
M.L. Brodie, eds,
Morgan Kaufmann,
pp. 518–529.

[43]

M.R. Glass and
A. Gliozzo,
A Dataset for Web-Scale Knowledge Base Population,
in: The Semantic Web - 15th International Conference, ESWC 2018,
Heraklion, Crete, Greece, June 3-7, 2018, Proceedings,
A. Gangemi,
R. Navigli,
M. Vidal,
P. Hitzler,
R. Troncy,
L. Hollink,
A. Tordai and
M. Alam, eds,
Lecture Notes in Computer Science,
Vol. 10843,
pp. 256–271.

[44]

T. Hamaguchi,
H. Oiwa,
M. Shimbo and
Y. Matsumoto,
Knowledge Transfer for Out-of-Knowledge-Base Entities : A Graph
Neural Network Approach,
in: Proceedings of the Twenty-Sixth International Joint Conference on
Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25,
C. Sierra, ed.,
pp. 1802–1808.

[45]

B. Haradizadeh and
S. Singh,
Tweeki: Linking Named Entities on Twitter to a Knowledge Graph,
in: Workshop on Noisy User-generated Text (W-NUT),

[46]

S. Heindorf,
M. Potthast,
B. Stein and
G. Engels,
Vandalism Detection in Wikidata,
in: Proceedings of the 25th ACM International Conference on
Information and Knowledge Management, CIKM 2016, Indianapolis, IN, USA,
October 24-28, 2016,
S. Mukhopadhyay,
C. Zhai,
E. Bertino,
F. Crestani,
J. Mostafa,
J. Tang,
L. Si,
X. Zhou,
Y. Chang,
Y. Li and
P. Sondhi, eds,
pp. 327–336.

[47]

N. Heist,
S. Hertling,
D. Ringler and
H. Paulheim,
Knowledge Graphs on the Web - An Overview,
in: Knowledge Graphs for eXplainable Artificial Intelligence:
Foundations, Applications and Challenges,
I. Tiddi,
F. Lécué and
P. Hitzler, eds,
Studies on the Semantic Web,
Vol. 47,
IOS Press,
pp. 3–22.

[48]

S. Hochreiter and
J. Schmidhuber,
Long Short-Term Memory,
Neural Comput.

[49]

J. Hoffart,
Y. Altun and
G. Weikum,
Discovering emerging entities with ambiguous names,
in: 23rd International World Wide Web Conference, WWW '14, Seoul,
Republic of Korea, April 7-11, 2014,
C. Chung,
A.Z. Broder,
K. Shim and
T. Suel, eds,
pp. 385–396.

[50]

J. Hoffart,
M.A. Yosef,
I. Bordino,
H. Fürstenau,
M. Pinkal,
M. Spaniol,
B. Taneva,
S. Thater and
G. Weikum,
Robust Disambiguation of Named Entities in Text,
in: Proceedings of the 2011 Conference on Empirical Methods in Natural
Language Processing, EMNLP 2011, 27-31 July 2011, John McIntyre Conference
Centre, Edinburgh, UK, A meeting of SIGDAT, a Special Interest Group of the
pp. 782–792.

[51]

A. Hogan,
E. Blomqvist,
M. Cochez,
C. d'Amato,
G. de Melo,
C. Gutierrez,
J.E.L. Gayo,
S. Kirrane,
S. Neumaier,
A. Polleres ,
Knowledge graphs,
arXiv preprint arXiv:2003.02320

[52]

B. Huang,
H. Wang,
T. Wang,
Y. Liu and
Y. Liu,
Entity Linking for Short Text Using Structured Knowledge Graph via
Multi-grained Text Matching,
Amazon Science

[53]

M. Jarke,
B. Neumann,
Y. Vassiliou and
W. Wahlster,
KBMS requirements of knowledge-based systems,
in: Foundations of knowledge base management,
pp. 381–394.

[54]

H. Ji,
X. Pan,
B. Zhang,
J. Nothman,
J. Mayfield,
P. McNamee and
C. Costello,
Overview of TAC-KBP2017 13 Languages Entity Discovery and Linking,
in: Proceedings of the 2017 Text Analysis Conference, TAC 2017,
Gaithersburg, Maryland, USA, November 13-14, 2017,

[55]

Kensho R&D group,
Kensho Derived Wikimedia Dataset,

[56]

B. Kitchenham,
Procedures for performing systematic reviews,
Keele, UK, Keele University

[57]

M. Klang and
P. Nugues,
Named entity disambiguation in a question answering system,
in: TheFifth Swedish Language Technology Conference (SLTC2014),

[58]

M. Klang and
P. Nugues,
Hedwig: A Named Entity Linker,
in: Proceedings of The 12th Language Resources and Evaluation
Conference, LREC 2020, Marseille, France, May 11-16, 2020,
N. Calzolari,
F. Béchet,
P. Blache,
K. Choukri,
C. Cieri,
T. Declerck,
S. Goggi,
H. Isahara,
B. Maegaard,
J. Mariani,
H. Mazo,
A. Moreno,
J. Odijk and
S. Piperidis, eds,
European Language Resources Association,
pp. 4501–4508.

[59]

M. Klang,
F. Dib and
P. Nugues,
Overview of the Ugglan Entity Discovery and Linking System,

[60]

J.M. Kleinberg,
Hubs, authorities, and communities,
ACM Comput. Surv.

[61]

J. Klie,
R.E. de Castilho and
I. Gurevych,
From Zero to Hero: Human-In-The-Loop Entity Linking in Low Resource
in: Proceedings of the 58th Annual Meeting of the Association for
Computational Linguistics, ACL 2020, Online, July 5-10, 2020,
D. Jurafsky,
J. Chai,
N. Schluter and
J.R. Tetreault, eds,
Association for Computational Linguistics,
pp. 6982–6993.

[62]

N. Kolitsas,
O. Ganea and
T. Hofmann,
End-to-End Neural Entity Linking,
in: Proceedings of the 22nd Conference on Computational Natural
Language Learning, CoNLL 2018, Brussels, Belgium, October 31 - November 1,
A. Korhonen and
I. Titov, eds,
Association for Computational Linguistics,
pp. 519–529.

[63]

T. Kristanti and
L. Romary,
DeLFT and entity-fishing: Tools for CLEF HIPE 2020 Shared Task,
in: CLEF 2020-Conference and Labs of the Evaluation Forum,
Vol. 2696,

[64]

K. Labusch and
C. Neudecker,
Named Entity Disambiguation and Linking on Historic Newspaper OCR with BERT,
Working Notes of CLEF 2020 - Confer- ence and Labs of the Evaluation

[65]

P. Le and
I. Titov,
Improving Entity Linking by Modeling Latent Relations between
in: Proceedings of the 56th Annual Meeting of the Association for
Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20,
2018, Volume 1: Long Papers,
I. Gurevych and
Y. Miyao, eds,
Association for Computational Linguistics,
pp. 1595–1604.

[66]

J. Lehmann,
R. Isele,
M. Jakob,
A. Jentzsch,
D. Kontokostas,
P.N. Mendes,
S. Hellmann,
M. Morsey,
P. van Kleef,
S. Auer and
C. Bizer,
DBpedia - A large-scale, multilingual knowledge base extracted from
Semantic Web

[67]

X. Lin and
L. Chen,
Canonicalization of Open Knowledge Bases with Side Information from
the Source Text,
in: 35th IEEE International Conference on Data Engineering, ICDE
2019, Macao, China, April 8-11, 2019,
pp. 950–961.

[68]

X. Lin,
H. Li,
H. Xin,
Z. Li and
L. Chen,
KBPearl: A Knowledge Base Population System Supported by Joint
Entity and Relation Linking,
Proc. VLDB Endow.

[69]

Y. Lin,
Z. Liu and
M. Sun,
Neural Relation Extraction with Multi-lingual Attention,
in: Proceedings of the 55th Annual Meeting of the Association for
Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4,
Volume 1: Long Papers,
R. Barzilay and
M. Kan, eds,
Association for Computational Linguistics,
pp. 34–43.

[70]

X. Ling,
S. Singh and
D.S. Weld,
Design Challenges for Entity Linking,
Trans. Assoc. Comput. Linguistics

[71]

Y. Liu,
M. Ott,
N. Goyal,
J. Du,
M. Joshi,
D. Chen,
O. Levy,
M. Lewis,
L. Zettlemoyer and
V. Stoyanov,
RoBERTa: A Robustly Optimized BERT Pretraining Approach,

[72]

L. Logeswaran,
M. Chang,
K. Lee,
K. Toutanova,
J. Devlin and
H. Lee,
Zero-Shot Entity Linking by Reading Entity Descriptions,
in: Proceedings of the 57th Conference of the Association for
Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2,
2019, Volume 1: Long Papers,
A. Korhonen,
D.R. Traum and
L. Màrquez, eds,
Association for Computational Linguistics,
pp. 3449–3460.

[73]

M. Manske,
Wikidata Stats,

[74]

T. Mikolov,
K. Chen,
G. Corrado and
J. Dean,
Efficient Estimation of Word Representations in Vector Space,
in: 1st International Conference on Learning Representations, ICLR
2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings,
Y. Bengio and Y. LeCun, eds.

[75]

D. Moussallem,
R. Usbeck,
M. Röder and
A.N. Ngomo,
MAG: A Multilingual, Knowledge-base Agnostic and Deterministic
Entity Linking Approach,
in: Proceedings of the Knowledge Capture Conference, K-CAP 2017,
Austin, TX, USA, December 4-6, 2017,
Ó. Corcho,
K. Janowicz,
G. Rizzo,
I. Tiddi and
D. Garijo, eds,
pp. 9:1–9:8.

[76]

I.O. Mulang,
K. Singh,
A. Vyas,
S. Shekarpour,
M. Vidal and
S. Auer,
Encoding Knowledge Graph Entity Aliases in Attentive Neural Network
for Wikidata Entity Linking,
in: Web Information Systems Engineering - WISE 2020 - 21st
International Conference, Amsterdam, The Netherlands, October 20-24, 2020,
Proceedings, Part I,
Z. Huang,
W. Beek,
H. Wang,
R. Zhou and
Y. Zhang, eds,
Lecture Notes in Computer Science,
Vol. 12342,
pp. 328–342.

[77]

I.O. Mulang,
K. Singh,
C. Prabhu,
A. Nadgeri,
J. Hoffart and
J. Lehmann,
Evaluating the Impact of Knowledge Graph Context on Entity
Disambiguation Models,
in: CIKM '20: The 29th ACM International Conference on Information
and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020,
M. d'Aquin,
S. Dietze,
C. Hauff,
E. Curry and
P. Cudré-Mauroux, eds,
pp. 2157–2160.

[78]

M. Nayyeri,
S. Vahdati,
C. Aykul and
J. Lehmann,
5* Knowledge Graph Embeddings with Projective Transformations,
in: Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI
2021, Thirty-Third Conference on Innovative Applications of Artificial
Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in
Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021,
AAAI Press,
pp. 9064–9072.

[79]

K. Noullet,
R. Mix and
M. Färber,
KORE 50\({}^{\mbox{DYWC}}\): An Evaluation Data Set for Entity
Linking Based on DBpedia, YAGO, Wikidata, and Crunchbase,
in: Proceedings of The 12th Language Resources and Evaluation
Conference, LREC 2020, Marseille, France, May 11-16, 2020,
N. Calzolari,
F. Béchet,
P. Blache,
K. Choukri,
C. Cieri,
T. Declerck,
S. Goggi,
H. Isahara,
B. Maegaard,
J. Mariani,
H. Mazo,
A. Moreno,
J. Odijk and
S. Piperidis, eds,
European Language Resources Association,
pp. 2389–2395.

[80]

I.L. Oliveira,
R. Fileto,
R. Speck,
L.P. Garcia,
D. Moussallem and
J. Lehmann,
Towards holistic Entity Linking: Survey and directions,
Information Systems

[81]

Oxford Online Dictionary,

[82]

L. Page,
S. Brin,
R. Motwani and
T. Winograd,
The PageRank citation ranking: Bringing order to the web.,
Technical Report,
Stanford InfoLab.

[83]

H. Paulheim,
Knowledge graph refinement: A survey of approaches and evaluation
Semantic Web

[84]

J. Pennington,
R. Socher and
C.D. Manning,
Glove: Global Vectors for Word Representation,
in: Proceedings of the 2014 Conference on Empirical Methods in Natural
Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A
meeting of SIGDAT, a Special Interest Group of the ACL,
A. Moschitti,
B. Pang and
W. Daelemans, eds,
pp. 1532–1543.

[85]

D. Perkins,
Separating the Signal from the Noise: Predicting the Correct Entities in
Named-Entity Linking,
Master's thesis,
Uppsala University,
<http://uu.diva-portal.org/smash/record.jsf?pid=diva2:1437921   dswid=6063>.

[86]

M.E. Peters,
M. Neumann,
M. Iyyer,
M. Gardner,
C. Clark,
K. Lee and
L. Zettlemoyer,
Deep Contextualized Word Representations,
in: Proceedings of the 2018 Conference of the North American Chapter
of the Association for Computational Linguistics: Human Language
Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018,
Volume 1 (Long Papers),
M.A. Walker,
H. Ji and
A. Stent, eds,
Association for Computational Linguistics,
pp. 2227–2237.

[87]

F. Petroni,
A. Piktus,
A. Fan,
P.S.H. Lewis,
M. Yazdani,
N.D. Cao,
J. Thorne,
Y. Jernite,
V. Karpukhin,
J. Maillard,
V. Plachouras,
T. Rocktäschel and
S. Riedel,
KILT: a Benchmark for Knowledge Intensive Language Tasks,
in: Proceedings of the 2021 Conference of the North American Chapter
of the Association for Computational Linguistics: Human Language
Technologies, NAACL-HLT 2021, Online, June 6-11, 2021,
K. Toutanova,
A. Rumshisky,
L. Zettlemoyer,
D. Hakkani-Tür,
I. Beltagy,
S. Bethard,
R. Cotterell,
T. Chakraborty and
Y. Zhou, eds,
Association for Computational Linguistics,
pp. 2523–2544.

[88]

V. Provatorova,
S. Vakulenko,
E. Kanoulas,
K. Dercksen and
J.M. van Hulst,
Named Entity Recognition and Linking on Historical Newspapers: UvA.ILPS & REL
at CLEF HIPE 2020,
Working Notes of CLEF 2020 - Conference and Labs of the Evaluation

[89]

J. Raiman and
O. Raiman,
DeepType: Multilingual Entity Linking by Neural Type System
in: Proceedings of the Thirty-Second AAAI Conference on Artificial
Intelligence, (AAAI-18), the 30th innovative Applications of Artificial
Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances
in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February
2-7, 2018,
S.A. McIlraith and
K.Q. Weinberger, eds,
AAAI Press,
pp. 5406–5413.

[90]

L. Ratinov,
D. Roth,
D. Downey and
M. Anderson,
Local and Global Algorithms for Disambiguation to Wikipedia,
in: The 49th Annual Meeting of the Association for Computational
Linguistics: Human Language Technologies, Proceedings of the Conference,
19-24 June, 2011, Portland, Oregon, USA,
D. Lin,
Y. Matsumoto and
R. Mihalcea, eds,
The Association for Computer Linguistics,
pp. 1375–1384.

[91]

S. Rijhwani,
J. Xie,
G. Neubig and
J.G. Carbonell,
Zero-Shot Neural Transfer for Cross-Lingual Entity Linking,
in: The Thirty-Third AAAI Conference on Artificial Intelligence,
AAAI 2019, The Thirty-First Innovative Applications of Artificial
Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on
Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu,
Hawaii, USA, January 27 - February 1, 2019,
AAAI Press,
pp. 6924–6931.

[92]

D. Ringler and
H. Paulheim,
One Knowledge Graph to Rule Them All? Analyzing the Differences
Between DBpedia, YAGO, Wikidata & co,
in: KI 2017: Advances in Artificial Intelligence - 40th Annual
German Conference on AI, Dortmund, Germany, September 25-29, 2017,
G. Kern-Isberner,
J. Fürnkranz and
M. Thimm, eds,
Lecture Notes in Computer Science,
Vol. 10505,
pp. 366–372.

[93]

M. Röder,
R. Usbeck and
A.N. Ngomo,
GERBIL - Benchmarking Named Entity Recognition and Linking
Semantic Web

[94]

M. Röder,
R. Usbeck,
S. Hellmann,
D. Gerber and
A. Both,
N\({^3}\) - A Collection of Datasets for Named Entity Recognition
and Disambiguation in the NLP Interchange Format,
in: Proceedings of the Ninth International Conference on Language
Resources and Evaluation, LREC 2014, Reykjavik, Iceland, May 26-31, 2014,
N. Calzolari,
K. Choukri,
T. Declerck,
H. Loftsson,
B. Maegaard,
J. Mariani,
A. Moreno,
J. Odijk and
S. Piperidis, eds,
European Language Resources Association (ELRA),
pp. 3529–3533.

[95]

H. Rosales-Méndez,
A. Hogan and
B. Poblete,
VoxEL: A Benchmark Dataset for Multilingual Entity Linking,
in: The Semantic Web - ISWC 2018 - 17th International Semantic Web
Conference, Monterey, CA, USA, October 8-12, 2018, Proceedings, Part II,
D. Vrandecic,
K. Bontcheva,
M.C. Suárez-Figueroa,
V. Presutti,
I. Celino,
M. Sabou,
L. Kaffee and
E. Simperl, eds,
Lecture Notes in Computer Science,
Vol. 11137,
pp. 170–186.

[96]

H. Rosales-Méndez,
A. Hogan and
B. Poblete,
Fine-Grained Entity Linking,
Journal of Web Semantics

[97]

P. Rosso,
D. Yang and
P. Cudré-Mauroux,
Beyond Triplets: Hyper-Relational Knowledge Graph Embedding for Link
in: WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24,
Y. Huang,
I. King,
T. Liu and
M. van Steen, eds,
ACM / IW3C2,
pp. 1885–1896.

[98]

A. Sakor,
I.O. Mulang,
K. Singh,
S. Shekarpour,
M. Vidal,
J. Lehmann and
S. Auer,
Old is Gold: Linguistic Driven Approach for Entity and Relation
Linking of Short Text,
in: Proceedings of the 2019 Conference of the North American Chapter
of the Association for Computational Linguistics: Human Language
Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume
1 (Long and Short Papers),
J. Burstein,
C. Doran and
T. Solorio, eds,
Association for Computational Linguistics,
pp. 2336–2346.

[99]

A. Sakor,
K. Singh,
A. Patel and
M. Vidal,
Falcon 2.0: An Entity and Relation Linking Tool over Wikidata,
in: CIKM '20: The 29th ACM International Conference on Information
and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020,
M. d'Aquin,
S. Dietze,
C. Hauff,
E. Curry and
P. Cudré-Mauroux, eds,
pp. 3141–3148.

[100]

Ö. Sevgili,
A. Shelmanov,
M. Arkhipov,
A. Panchenko and
C. Biemann,
Neural Entity Linking: A Survey of Models based on Deep Learning,

[101]

W. Shen,
J. Wang and
J. Han,
Entity Linking with a Knowledge Base: Issues, Techniques, and
IEEE Trans. Knowl. Data Eng.

[102]

A. Singhal,
Introducing the knowledge graph: things, not strings,
Official Google blog

[103]

D. Sorokin and
I. Gurevych,
Context-Aware Representations for Knowledge Base Relation Extraction,
in: Proceedings of the 2017 Conference on Empirical Methods in Natural
Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11,
M. Palmer,
R. Hwa and
S. Riedel, eds,
Association for Computational Linguistics,
pp. 1784–1789.

[104]

D. Sorokin and
I. Gurevych,
Mixing Context Granularities for Improved Entity Linking on Question
Answering Data across Entity Categories,
in: Proceedings of the Seventh Joint Conference on Lexical and
Computational Semantics, *SEM@NAACL-HLT 2018, New Orleans, Louisiana, USA,
June 5-6, 2018,
M. Nissim,
J. Berant and
A. Lenci, eds,
Association for Computational Linguistics,
pp. 65–75.

[105]

A. Sperduti and
A. Starita,
Supervised neural networks for the classification of structures,
IEEE Trans. Neural Networks

[106]

A. Spitz,
J. Geiß and
M. Gertz,
So far away and yet so close: augmenting toponym disambiguation and
similarity with text-based networks,
in: Proceedings of the Third International ACM SIGMOD Workshop on
Managing and Mining Enriched Geo-Spatial Data, GeoRich@SIGMOD 2016, San
Francisco, California, USA, June 26 - July 1, 2016,
A. Züfle,
B. Adams and
D. Wu, eds,
pp. 2:1–2:6.

[107]

Y. Su,
H. Sun,
B.M. Sadler,
M. Srivatsa,
I. Gur,
Z. Yan and
X. Yan,
On Generating Characteristic-rich Question Sets for QA Evaluation,
in: Proceedings of the 2016 Conference on Empirical Methods in Natural
Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016,
J. Su,
X. Carreras and
K. Duh, eds,
The Association for Computational Linguistics,
pp. 562–572.

[108]

T.P. Tanon,
G. Weikum and
F.M. Suchanek,
YAGO 4: A Reason-able Knowledge Base,
in: The Semantic Web - 17th International Conference, ESWC 2020,
Heraklion, Crete, Greece, May 31-June 4, 2020, Proceedings,
A. Harth,
S. Kirrane,
A.N. Ngomo,
H. Paulheim,
A. Rula,
A.L. Gentile,
P. Haase and
M. Cochez, eds,
Lecture Notes in Computer Science,
Vol. 12123,
pp. 583–596.

[109]

K. Teru,
E. Denis and
W. Hamilton,
Inductive Relation Prediction by Subgraph Reasoning,
in: Proceedings of the 37th International Conference on Machine
Learning, ICML 2020, 13-18 July 2020, Virtual Event,
Proceedings of Machine Learning Research,
Vol. 119,
pp. 9448–9457.

[110]

A. Thawani,
M. Hu,
E. Hu,
H. Zafar,
N.T. Divvala,
A. Singh,
E. Qasemi,
P.A. Szekely and
J. Pujara,
Entity Linking to Knowledge Graphs to Infer Column Types and
in: Proceedings of the Semantic Web Challenge on Tabular Data to
Knowledge Graph Matching co-located with the 18th International Semantic Web
Conference, SemTab@ISWC 2019, Auckland, New Zealand, October 30, 2019,
E. Jiménez-Ruiz,
O. Hassanzadeh,
K. Srinivas,
V. Efthymiou and
J. Chen, eds,
CEUR Workshop Proceedings,
Vol. 2553,
pp. 25–32.

[111]

C. Tsai and
D. Roth,
Cross-lingual Wikification Using Multilingual Embeddings,
in: NAACL HLT 2016, The 2016 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, San Diego California, USA, June 12-17, 2016,
K. Knight,
A. Nenkova and
O. Rambow, eds,
The Association for Computational Linguistics,
pp. 589–598.

[112]

R. Usbeck,
A.N. Ngomo,
M. Röder,
D. Gerber,
S.A. Coelho,
S. Auer and
A. Both,
AGDISTIS - Agnostic Disambiguation of Named Entities Using Linked
Open Data,
in: ECAI 2014 - 21st European Conference on Artificial Intelligence,
18-22 August 2014, Prague, Czech Republic - Including Prestigious
Applications of Intelligent Systems (PAIS 2014),
T. Schaub,
G. Friedrich and
B. O'Sullivan, eds,
Frontiers in Artificial Intelligence and Applications,
Vol. 263,
IOS Press,
pp. 1113–1114.

[113]

R. Usbeck,
A.N. Ngomo,
B. Haarmann,
A. Krithara,
M. Röder and
G. Napolitano,
7th Open Challenge on Question Answering over Linked Data (QALD-7),
in: Semantic Web Challenges - 4th SemWebEval Challenge at ESWC 2017,
Portoroz, Slovenia, May 28 - June 1, 2017, Revised Selected Papers,
M. Dragoni,
M. Solanki and
E. Blomqvist, eds,
Communications in Computer and Information Science,
Vol. 769,
pp. 59–69.

[114]

J.M. van Hulst,
F. Hasibi,
K. Dercksen,
K. Balog and
A.P. de Vries,
REL: An Entity Linker Standing on the Shoulders of Giants,
in: Proceedings of the 43rd International ACM SIGIR conference on
research and development in Information Retrieval, SIGIR 2020, Virtual
Event, China, July 25-30, 2020,
J. Huang, Y. Chang, X. Cheng, J. Kamps, V. Murdock, J. Wen and Y. Liu, eds,
pp. 2197–2200.

[115]

T. van Veen,
J. Lonij and
W.J. Faber,
Linking Named Entities in Dutch Historical Newspapers,
in: Metadata and Semantics Research - 10th International Conference,
MTSR 2016, Göttingen, Germany, November 22-25, 2016, Proceedings,
E. Garoufallou,
I.S. Coll,
A. Stellato and
J. Greenberg, eds,
Communications in Computer and Information Science,
Vol. 672,
pp. 205–210.

[116]

A. Vaswani,
N. Shazeer,
N. Parmar,
J. Uszkoreit,
L. Jones,
A.N. Gomez,
L. Kaiser and
I. Polosukhin,
Attention is All you Need,
in: Advances in Neural Information Processing Systems 30: Annual
Conference on Neural Information Processing Systems 2017, December 4-9, 2017,
Long Beach, CA, USA,
I. Guyon,
U. von Luxburg,
S. Bengio,
H.M. Wallach,
R. Fergus,
S.V.N. Vishwanathan and
R. Garnett, eds,
pp. 5998–6008.

[117]

O. Vinyals,
M. Fortunato and
N. Jaitly,
Pointer Networks,
in: Advances in Neural Information Processing Systems 28: Annual
Conference on Neural Information Processing Systems 2015, December 7-12,
2015, Montreal, Quebec, Canada,
C. Cortes,
N.D. Lawrence,
D.D. Lee,
M. Sugiyama and
R. Garnett, eds,
pp. 2692–2700.

[118]

D. Vrandecic,
Building a multilingual Wikipedia,
Commun. ACM

[119]

D. Vrandečić and
M. Krötzsch,
Wikidata: a free collaborative knowledgebase,
Commun. ACM

[120]

W3C Microposts Community Group,
Making Sense of Microposts (#Microposts2016),

[121]

P. Wang,
J. Han,
C. Li and
R. Pan,
Logic Attention Based Neighborhood Aggregation for Inductive Knowledge
Graph Embedding,
in: The Thirty-Third AAAI Conference on Artificial Intelligence,
AAAI 2019, The Thirty-First Innovative Applications of Artificial
Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on
Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu,
Hawaii, USA, January 27 - February 1, 2019,
AAAI Press,
pp. 7152–7159.

[122]

X. Wang,
T. Gao,
Z. Zhu,
Z. Zhang,
Z. Liu,
J. Li and
J. Tang,
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained
Language Representation,
Trans. Assoc. Comput. Linguistics

[123]

G. Weikum,
X.L. Dong,
S. Razniewski and
F.M. Suchanek,
Machine Knowledge: Creation and Curation of Comprehensive Knowledge
Found. Trends Databases

[124]

Wikimedia Foundation,
Index of /wikidatawiki/entities/,

[125]

Wikimedia Foundation,

[126]

M.D. Wilkinson,
M. Dumontier,
I.J. Aalbersberg,
G. Appleton,
M. Axton,
A. Baak,
N. Blomberg,
J.-W. Boiten,
L.B. da Silva Santos,
P.E. Bourne ,
The FAIR Guiding Principles for scientific data management and
Scientific data

[127]

L. Wu,
F. Petroni,
M. Josifoski,
S. Riedel and
L. Zettlemoyer,
Scalable Zero-shot Entity Linking with Dense Entity Retrieval,
in: Proceedings of the 2020 Conference on Empirical Methods in Natural
Language Processing, EMNLP 2020, Online, November 16-20, 2020,
B. Webber,
T. Cohn,
Y. He and
Y. Liu, eds,
Association for Computational Linguistics,
pp. 6397–6407.

[128]

T. Wu,
A. Khan,
H. Gao and
C. Li,
Efficiently Embedding Dynamic Knowledge Graphs,

[129]

Z. Wu,
S. Pan,
F. Chen,
G. Long,
C. Zhang and
S.Y. Philip,
A comprehensive survey on graph neural networks,
IEEE Transactions on Neural Networks and Learning Systems

[130]

X. Yang,
X. Gu,
S. Lin,
S. Tang,
Y. Zhuang,
F. Wu,
Z. Chen,
G. Hu and
X. Ren,
Learning Dynamic Context Augmentation for Global Entity Linking,
in: Proceedings of the 2019 Conference on Empirical Methods in Natural
Language Processing and the 9th International Joint Conference on Natural
Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7,
K. Inui,
J. Jiang,
V. Ng and
X. Wan, eds,
Association for Computational Linguistics,
pp. 271–281.

[131]

Z. Yang,
Z. Dai,
Y. Yang,
J.G. Carbonell,
R. Salakhutdinov and
Q.V. Le,
XLNet: Generalized Autoregressive Pretraining for Language
in: Advances in Neural Information Processing Systems 32: Annual
Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14
December 2019, Vancouver, BC, Canada,
H.M. Wallach,
H. Larochelle,
A. Beygelzimer,
F. d'Alché-Buc,
E.B. Fox and
R. Garnett, eds,
pp. 5754–5764.

[132]

W. Yih,
M. Richardson,
C. Meek,
M. Chang and
J. Suh,
The Value of Semantic Parse Labeling for Knowledge Base Question
in: Proceedings of the 54th Annual Meeting of the Association for
Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany,
Volume 2: Short Papers,
The Association for Computer Linguistics,

[133]

F. Yu and
V. Koltun,
Multi-Scale Context Aggregation by Dilated Convolutions,
in: 4th International Conference on Learning Representations, ICLR
2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings,
Y. Bengio and
Y. LeCun, eds,

[134]

A. Zaveri,
A. Rula,
A. Maurino,
R. Pietrobon,
J. Lehmann and
S. Auer,
Quality assessment for linked data: A survey,
Semantic Web

[135]

Z. Zhang,
J. Cai,
Y. Zhang and
J. Wang,
Learning Hierarchy-Aware Knowledge Graph Embeddings for Link
in: The Thirty-Fourth AAAI Conference on Artificial Intelligence,
AAAI 2020, The Thirty-Second Innovative Applications of Artificial
Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on
Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY,
USA, February 7-12, 2020,
AAAI Press,
pp. 3065–3072.

[136]

S. Zhou,
S. Rijhwani,
J. Wieting,
J.G. Carbonell and
G. Neubig,
Improving Candidate Generation for Low-resource Cross-lingual Entity
Trans. Assoc. Comput. Linguistics

[137]

S. Zwicklbauer,
C. Seifert and
M. Granitzer,
DoSeR - A Knowledge-Base-Agnostic Framework for Entity
Disambiguation Using Semantic Embeddings,
in: The Semantic Web. Latest Advances and New Domains - 13th
International Conference, ESWC 2016, Heraklion, Crete, Greece, May 29 -
June 2, 2016, Proceedings,
H. Sack,
E. Blomqvist,
M. d'Aquin,
C. Ghidini,
S.P. Ponzetto and
C. Lange, eds,
Lecture Notes in Computer Science,
Vol. 9678,
pp. 182–198.


# Addressee and Response Selection for Multilingual Conversation

Motoki Sato1 Hiroki Ouchi2,3 Yuta Tsuboi1∗
1Preferred Networks, Inc.,
2RIKEN Center for Advanced Intelligence Project
3Tohoku University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>This work was
conducted when the first author worked at Nara Institute of Science and
Technology, and portions of this research were done while the third author was
at IBM Research - Tokyo.

###### Abstract

Developing conversational systems that can converse in many languages is an
interesting challenge for natural language processing. In this paper, we
introduce multilingual addressee and response selection. In this task, a
conversational system predicts an appropriate addressee and response for an
input message in multiple languages. A key to developing such multilingual
responding systems is how to utilize high-resource language data to compensate
for low-resource language data. We present several knowledge transfer methods
for conversational systems. To evaluate our methods, we create a new
multilingual conversation dataset. Experiments on the dataset demonstrate the
effectiveness of our methods.

##  1 Introduction

00footnotetext:  This work is licensed under a Creative Commons Attribution
4.0 International License. License details:
http://creativecommons.org/licenses/by/4.0/.

Open-domain conversational systems, such as chatbots, are attracting a vast
amount of interest and play their functional and entertainment roles in real-
world applications. Recent conversational models are often built in an end-to-
end fashion using neural networks, which require a large amount of training
data [Vinyals and Le (2015, Serban et al. (2016]. However, it is challenging
to collect enough data to build such models for many languages. Consequently,
most work has targeted high-resource languages, such as English and Chinese
[Shang et al. (2015, Serban et al. (2016].

In this work, we aim to develop multilingual conversational systems that can
return appropriate responses in many languages. Specifically, we assume the
two types of systems: (i) language-specific systems and (ii) language-
invariant systems. A language-specific system consists of multiple
conversational models, each of which returns responses in a corresponding
language. By contrast, a language-invariant system consists of a single
unified model, which returns responses in all target languages. A key to
building these multilingual models is how to utilize high-resource language
data to compensate for low-resource language data. We present several
knowledge-transfer methods. To the best of our knowledge, this is the first
work focusing on low-resource language enablement of conversational systems.

One challenge when developing conversational systems is how to evaluate the
system performance. For generation-based conversational systems, which
generate each word for a response one by one, many studies adopt human
judgments. However, it is costly and impractical to adopt this evaluation
method for multilingual systems, especially for minor-language systems. Thus,
as a first step, we develop retrieval-based conversational systems and
evaluate the ability to select appropriate responses from a set of candidates.

Fig. 1 shows the overview of our multilingual responding systems.

Figure 1: Example of multilingual response selection.

This paper provides: (i) formal task definitions, (ii) several knowledge-
transfer methods and (iii) a multilingual conversation dataset. First, we
introduce and formalize the two task settings: single-language adaptation for
language-specific systems and multi-language adaptation for language-invariant
systems (Sec. 4).

Second, we present several methods leveraging high-resource language data to
compensate for low-resource language in the two settings (Sec. 5). Our basic
method uses multilingual word embeddings and transfers source-language
knowledge to target languages (Sec. 5.1 (a)). We also design three extended
methods. Among them, the fine-tuning method builds a model specific to a
single target language (Sec. 5.2 (b)). The joint loss training and the multi-
language adversarial training methods build a unified model invariant for
multiple target languages (Sec. 5.2 (c) and (d)).

Third, we create a multilingual conversation corpus and dataset111Our code and
dataset are publicly available at https://github.com/aonotas/multilingual_ASR
(Sec. 6).

From the Ubuntu IRC Logs222http://irclogs.ubuntu.com/, we collect the logs in
$12$ languages.

To show benchmark results, we perform experiments on the created dataset (Sec.
7). The results demonstrate that our methods allow models to effectively adapt
to low-resource target languages.

In particular, our method using Wasserstein GAN [Arjovsky et al. (2017]
achieves high-performance for simultaneously dealing with multiple languages
with a single unified model.

##  2 Related Work

### Short Text Conversation

In short text conversation, a system predicts an appropriate response for an
input message in single-turn, two-party conversation [Ritter et al. (2011].
One major approach to it is the generation-based approach, which generates a
response using a sequence-to-sequence model [Shang et al. (2015, Vinyals and
Le (2015, Serban et al. (2016, Li et al. (2016, Mei et al. (2017]. Another
popular approach is the retrieval-based approach, which retrieves candidate
responses from a repository and returns the highest scoring one using a
ranking model [Wang et al. (2013, Lu and Li (2013, Ji et al. (2014, Wang et
al. (2015]. ?) proposed next utterance classification (NUC), in which a model
has to select an appropriate response from a fixed set of candidates.

### Evaluation for Conversational Systems

Evaluation methods for conversational systems are an open question [Lowe et
al. (2015, Liu et al. (2016, Lowe et al. (2017].

While many of previous studies on conversational systems used human
judgements, automatic evaluation methods are attractive because it is much
easier and less costly to use. However, according to ?), for generation-based
systems, automatic evaluation metrics, such as BLEU [Papineni et al. (2002],
correlate very weakly with human judgements.

For retrieval-based systems, some studies used ranking-based metrics, such as
mean average precision and accuracy [Ji et al. (2014, Wang et al. (2015]. ?)
confirmed the feasibility of NUC as a surrogate task for building
conversational systems. Although there are controversial issues for these
evaluation methods [Lowe et al. (2015], as a practical choice, we adopt the
accuracy-based metric for evaluating multilingual conversational systems.

### Addressee and Response Selection

NUC focuses on two-party, multi-turn conversation. As an extension of it, ?)
proposed addressee and response selection (ARS) for multi-party conversation.
ARS integrates the addressee detection problem, which has been regarded as a
problematic issue in multi-party conversation [Traum (2003, Jovanović and
Akker (2004, Bohus and Horvitz (2011, Uthus and Aha (2013]. Mainly, this
problem has been tackled in spoken/multimodal dialog systems [Jovanović et al.
(2006, Akker and Traum (2009, Nakano et al. (2013, Ravuri and Stolcke (2014].
While these systems largely rely on acoustic signal or gaze information, ARS
focuses on text-based conversations. Extending these studies, we tackle
multilingual, multi-turn, and multi-party text conversation settings.

### Cross-Lingual Conversation

The motivation of our task is similar with that of ?). They tackled cross-
lingual dialog state tracking in English and Chinese. While they transfer
knowledge from English to Chinese, we transfer knowledge between a high-
resource and several low-resource languages.

##  3 Addressee and Response Selection

Addressee and response selection (ARS)333Due to the space limitation, we give
a brief overview of ARS. For the complete task definition, please refer to
?)., proposed by ?), assumes the situation where a responding agent returns a
response to an addressee following a conversational context.

Formally, given an input conversational situation $\mbox{\boldmath$x$}\in X$,
a system predicts $\mbox{\boldmath$y$}\in Y$, which consists of an addressee
$a$ and a response $r$:

$\textsc{Given}:\mbox{\boldmath$x$}=(a_{\rm
res},\>\mathcal{C},\>\mathcal{R}),\hskip
14.22636pt\textsc{Predict}:\mbox{\boldmath$y$}=(a,\mbox{\boldmath$r$})$

where $a_{\rm res}$ is a responding agent, $\mathcal{C}$ is a context (a
sequence of previous utterances) and $\mathcal{R}$ is a set of candidate
responses. To predict an addressee $a$, we select an agent from a set of the
agents appearing in a context $\mathcal{A}(\mathcal{C})$. To predict a
response $r$, we select a response from a set of candidate responses
$\mathcal{R}=\\{\mbox{\boldmath$r$}_{1},\cdots,\mbox{\boldmath$r$}_{|\mathcal{R}|}\\}$.

This task evaluates accuracy on the three aspects: addressee-response pair
selection (ADR-RES), addressee selection (ADR), and response selection (RES).
In ADR-RES, we regard the answer as correct if both the addressee and response
are correctly selected. In ADR/RES, we regard the answer as correct if the
addressee/response is correctly selected.

##  4 Multilingual Addressee and Response Selection

As an extension of monolingual ARS, we propose multilingual addressee and
response selection (M-ARS). In ARS, a system is given as input a set of
candidate responses and a conversational context in a single language. By
contrast, in M-ARS, a system receives the inputs in one of multiple languages.
In the following, we first explain our motivation for tackling M-ARS and then
describe the formal task definitions.

###  4.1 Motivative Situations

We assume the two multilingual conversational situations:

  * •

You want to build language-specific systems, each of which responds in a
single language.

  * •

You want to build one language-invariant system, which responds in multiple
languages.

The first situation is that we build $K$ models, each of which is specialized
for one of $K$ target languages. The second one is that we build one unified
model that can deal with all $K$ target languages. Taking these situations
into account, we present the corresponding two tasks: (i) single-language
adaptation and (ii) multi-language adaptation.

###  4.2 Task Overview

The goal of single-language adaptation is to develop and evaluate a language-
specific ARS model for a single target language. For example, using English,
German and Italian training data, we build a model specialized for German
conversation. The goal of multi-language adaptation is to develop and evaluate
a language-invariant ARS model for multiple target languages. For example,
using English, German and Italian training data, we build a model that can
respond to not only German but also Italian and English conversation. In the
following subsections, we formalize each of these tasks.

###  4.3 Formal Task Definition

We assume that we have conversation data in each of a set of languages
$\mathcal{K}$.

Training
In the training phase, a training dataset is given for each language
$k\in{\mathcal{K}}$:

$\displaystyle\begin{aligned} &\mathcal{D}^{(k)}_{\rm
train}=\\{\>\mbox{\boldmath$x$}^{(k)}_{i},\mbox{\boldmath$y$}^{(k)}_{i}\>\\}_{i=1}^{N^{(k)}},\>\>k\in\mathcal{K}\\\
&\mathcal{D}_{\rm train}=\bigcup_{k}\>\mathcal{D}^{(k)}_{\rm
train}\end{aligned}$

where $\mbox{\boldmath$x$}^{(k)}$ and $\mbox{\boldmath$y$}^{(k)}$ are a
conversational situation and the target output in language $k$, respectively.
We train a model $\mathcal{F}:\mathcal{X}\rightarrow\mathcal{Y}$ on these
training samples.

Evaluation
In single-language adaptation, we evaluate a trained model for a single target
language $t\in\mathcal{K}$. The trained model receives an input of the target
language, $\mbox{\boldmath$x$}^{(t)}\sim\mathcal{D}^{(t)}_{\rm eval}$, and
predicts $\hat{\mbox{\boldmath$y$}}^{(t)}$. As evaluation metrics, we use the
three accuracies (ADR-RES, ADR and RES) used in ARS (Sec. 3).

In multi-language adaptation, given evaluation datasets for all the languages
$\mathcal{K}$, i.e., $\bigcup_{k}\mathcal{D}^{(k)}_{\rm eval}$, the trained
model receives an input of each language
$\mbox{\boldmath$x$}^{(k)}\sim\mathcal{D}^{(k)}_{\rm eval}$ and predicts
$\hat{\mbox{\boldmath$y$}}^{(k)}$. As evaluation metrics, we use macro average
over all the languages: $\text{\tt ADR-RES}=\frac{\sum_{k}\>\text{\tt ADR-
RES}^{(k)}}{|\mathcal{K}|}$. ADR and RES are also computed in the same way.

##  5 Methods

In this section, we firstly describe a model used for addressee and response
selection, and then explain our proposed methods to train parameters of the
model.

Our model $\mathcal{F}$ consists of a feature extractor $f^{E}$, addressee
scoring function $f^{A}$ and response scoring function $f^{R}$. $f^{A}$ and
$f^{R}$ return relevance scores (probabilities) for an addressee and response:

$\displaystyle f^{A}(\mbox{\boldmath$x$},a_{i})$ $\displaystyle=$
$\displaystyle\sigma([{\bf a}_{\rm res},{\bf h}_{c}]^{\rm T}\>{\bf
W}_{a}\>{\bf a}_{i})$ (1) $\displaystyle
f^{R}(\mbox{\boldmath$x$},\mbox{\boldmath$r$}_{j})$ $\displaystyle=$
$\displaystyle\sigma([{\bf a}_{\rm res},{\bf h}_{c}]^{\rm T}\>{\bf
W}_{r}\hskip 4.26773pt{\bf r}_{j})$ (2)

where ${\bf a}_{\rm res}$ is a responding agent vector, ${\bf h}_{c}$ is a
conversational context vector, ${\bf a}_{i}$ is an agent vector, and ${\bf
r}_{j}$ is a candidate response vector. All these vectors are encoded by the
feature extractor $f^{E}$. We use the dynamic model [Ouchi and Tsuboi (2016]
as $f^{E}$. Fig. 3 shows the overview of the dynamic model. This model
represents each agent as a hidden state vector that dynamically changes along
with time steps in GRU [Cho et al. (2014]. 444In this example, a responding
agent vector ${\bf a}_{\rm res}$ is ${\bf a}_{3}$. Note that the states of the
agents that are not speaking at the time are updated by zero vectors.

A model $\mathcal{F}$ is parameterized by $\theta=\\{\theta_{E}\cup\\{{\bf
W}_{a},{\bf W}_{r}\\}\\}$, where $\theta_{E}$ is parameters of $f^{E}$. To
train these parameters, we present four methods. These methods assume that we
have training sets for a set of languages $\mathcal{K}$: some of them are
high-resource languages $\mathcal{S}\subseteq\mathcal{K}$ and others are
relatively low-resource languages $\mathcal{T}=\bar{\mathcal{S}}$.

###  5.1 A Basic Method

Figure 2: Overview of Dynamic Model. Figure 3: Overview of our W-GAN training
method for multiple target languages.
---

### (a) Multilingual Embedding Replacement

This method trains a model $\mathcal{F}$ on high-resource language data
$\mathcal{D}^{(s)}_{\rm train}$, where $s\in\mathcal{S}$, and uses the trained
model for responding conversations in other languages $\bar{\mathcal{S}}$. To
realize this transfer, we use multilingual embeddings.

Consider the case where the high-resource language is English (En) and low-
resource language is German (De). In the training phase, we use English word
embeddings to train model parameters.555The embeddings are fixed, not fine-
tuned, during training. In the testing phase, instead of the English
embeddings, we use German embeddings:

Train: ${\bf w}={\bf W}^{({\tt En})}_{\rm emb}\>w$ Test: ${\bf w}={\bf
W}^{({\tt De})}_{\rm emb}\>w$
where $w$ is a one-hot vector. We just replace the English word embeddings
${\bf W}^{({\tt En})}_{\rm emb}$ with the German ones ${\bf W}^{({\tt
De})}_{\rm emb}$. After looking up each word embedding ${\bf w}$, the neural
model computes the hidden states. One advantage of this method is to require
no target language data. As multilingual embeddings, we use MultiCCA666The
pre-trained MultiCCA embeddings are provided at
http://128.2.220.95/multilingual/data/ proposed by ?). In these embeddings,
semantically similar words in the same language or translationally equivalent
words in different languages are projected onto nearby.

Besides multilingual embeddings, another option to build a conversational
model without no conversation data in a target language is to translate high-
resource language data to low-resource one and train a conversational model on
the translations. One limitation of this approach is that it is costly to
prepare parallel corpora for building the translation model. We discuss this
approach in Sec. 7.3.

###  5.2 Extended Methods

We present the two types of methods which use target language data for
building (i) language-specific models and (ii) language-invariant models.

####  5.2.1 Methods for Language-Specific Models

### (b) Fine-Tuning with Target Language Data

To compensate for the lack of the low-resource language data, this method
firstly trains a model $\mathcal{F}_{\theta}$ on high-resource language data
(pre-training phase). Then, using the pre-trained parameters $\theta$ as the
initial values, this method re-trains them on low-resource language data
(fine-tuning phase). We can expect that by gaining the better initial
parameters, the tuning effectively adapts the model to the target language.

####  5.2.2 Methods for Language-Invariant Models

In order to build language-invariant models, it is critical to consider the
two perspectives: (i) avoiding catastrophic forgetting and (ii) learning
language-invariant features. Catastrophic forgetting [Kirkpatrick et al.
(2017] is the phenomenon that a model forgets knowledge of previously trained
tasks (languages) by incorporating knowledge of the current task (language).
Language-invariant features are the features that are common and unchanged in
different languages. Taking these two perspectives into account, we present
the following two methods.

### (c) Joint Loss Training

This method aims to avoid catastrophic forgetting by jointly training model
parameters on all the language data at a time. Assuming that we have a set of
languages $\mathcal{K}$, this method uses the joint loss function:
$\mathcal{J}_{\text{joint}}(\theta)=\sum_{k}\mathcal{J}(\mathcal{D}^{(k)},\theta)$
where the loss function $\mathcal{J}$ is the cross-entropy loss used in ?).

### (d) Multi-Language Adversarial Training

To learn language-invariant features, we use a framework of Wasserstein-GAN
(W-GAN) [Arjovsky et al. (2017], a recently proposed technique to improve
stability for generative adversarial nets (GANs) [Goodfellow et al. (2014].
The aim of this method is to match the distributions of feature
representations in two languages.

Fig. 3 illustrates an example. English is the high-resource language
$s\in\mathcal{S}$, and German and Croatian are low-resource languages
$t\in\mathcal{T}$. For each language, the feature extractor $f^{E}$ receives
an input conversation $x$ and computes the hidden features ${\bf
h}=f^{E}(\mbox{\boldmath$x$})$777Hidden feature representation $\bf h$ is the
concatenation of the responding speaker vector and context vector in Eqs. 1
and 2, i.e. ${\bf h}=[{\bf a}_{\rm res},{\bf h}_{c}]$.. Thus, by using
$f^{E}$, we obtain the hidden feature ${\bf h}^{(s)}$ and ${\bf h}^{(t)}$ for
English and the others, respectively.

A pair of the high- and low-resource language features ${\bf h}^{(s)}$ and
${\bf h}^{(t)}$ is given to a critic $g_{\pi}$ to minimize the Wasserstein
distance between the distributions $p({\bf h}^{(s)})$ and $p({\bf h}^{(t)})$:

$\displaystyle\mathcal{W}(\>p({\bf h}^{(s)}),\>p({\bf
h}^{(t)})\>)=\max_{\pi}\>\mathbb{E}_{{\bf h}^{(s)}\sim p({\bf
h}^{(s)})}[g_{\pi}({\bf h}^{(s)})]-\mathbb{E}_{{\bf h}^{(t)}\sim p({\bf
h}^{(t)})}[g_{\pi}({\bf h}^{(t)})]$ (3)

where the maximum is taken over the set of all 1-Lipschitz functions
$g_{\pi}$.888A function $g$ is 1-Lipschitz when $|g(x)-g(y)|\leq|x-y|$ for all
$x$ and $y$. To constrain the critic $g$ to a 1-Lipschitz function, the
parameters of $g$ are clipped to a fixed range. By maximizing this equation,
the distributions of the feature representations, $p({\bf h}^{(s)})$ and
$p({\bf h}^{(t)})$, are made as close as possible. In this paper, as the
critic $g_{\pi}$, we use multi-layer perceptron.

Eq. 3 is designed for the two distributions. Thus, we generalize this W-GAN
equation to deal with $|\mathcal{S}|$ high-resource languages and
$|\mathcal{T}|$ low-resource languages:

$\mathcal{J}_{\text{wgan}}(\theta)=\sum_{s\in\mathcal{S}}\sum_{t\in\mathcal{T}}\>\mathcal{W}(\>p({\bf
h}^{(k)}),\>p({\bf h}^{(\ell)})\>)$

This loss function $\mathcal{J}_{\text{wgan}}$ is integrated with the joint
loss:
$\mathcal{J}_{\text{adv}}(\theta)=\mathcal{J}_{\text{joint}}(\theta)+\lambda\>\mathcal{J}_{\text{wgan}}(\theta)$
where $\lambda$ is a hyper-parameter that balances these loss functions and we
used $\lambda=0.5$.

Corpus
---
Language | Docs | Utters | Words
English (en) | 7355 | 2.4 M | 27.0 M
Italian (it) | 357 | 165 k | 1.1 M
Croatian (hr) | 254 | 80 k | 630 k
German (de) | 248 | 38 k | 335 k
Portuguese (pt) | 211 | 52 k | 285 k
Slovenian (sl) | 179 | 59 k | 357 k
Polish(pl) | 67 | 8.8 k | 51 k
Dutch (nl) | 57 | 7.2 k | 75 k
Spanish (es) | 36 | 7.1 k | 49 k
Swedish (sv) | 26 | 1.7 k | 6.8 k
Russian (ru) | 5 | 0.3 k | 1.5 k
French (fr) | 3 | 0.5 k | 3.0 k
Table 1: Statistics of M-ARS corpus.

Dataset
---
Language | Train | Dev | Test
English (en) | 665.6 k | 45.1 k | 51.9 k
Italian (it) | 38,511 | 2,561 | 3,873
Croatian (hr) | 11,387 | 512 | 1,145
German (de) | 5,500 | 354 | 569
Portuguese (pt) | 5,951 | 285 | 975
Table 2:  Statistics of the M-ARS dataset.

##  6 Corpus and Dataset

One of our goals is to provide a multilingual conversation corpus/dataset that
can be used over a wide range of conversation research. We follow the corpus
and data creation method of ?). First, we crawl the Ubuntu IRC Logs999We use a
collection of the logs during one year (2015). We plan to expand it by
collecting the logs over all the years., and preprocess the logs in many
languages. Each language is identified by using a language detection library
[Nakatani (2010]. The resulting corpus consists of multilingual conversations
in $12$ languages, shown in Tab. 2.

Then, we create an M-ARS dataset. For each language, we set the ground-
truth/false addressees and responses following ?). Note that the addressed
usernames in utterances have been removed for addressee selection. Thus, we
have to predict the addressees without seeing the addressed usernames. The
number of candidate responses ($|\mathcal{R}|$) is set to $2$ or $10$. The
dataset is then randomly partitioned into a training set (90%), a development
set (5%) and a test set (5%). Tab. 2 shows the statistics of the top $5$
largest language sections of this resulting dataset.

|  | $|\mathcal{R}|=2$ | $|\mathcal{R}|=10$
---|---|---|---
Setting | Method | ADR-RES | ADR | RES | ADR-RES | ADR | RES
Single Language Adaptation | Chance |  3.97 |  7.94 | 50.00 |  0.80 |  7.94 | 10.00
TF-IDF | 39.51 | 64.97 | 60.61 | 12.54 | 64.97 | 18.50
TrgOnly | 47.35 | 69.27 | 67.35 | 19.42 | 69.73 | 26.13
EnOnly | 38.07 | 65.72 | 57.65 |  8.50 | 62.38 | 13.75
FineTune | 49.58 | 69.59 | 69.84 | 21.15 | 70.33 | 28.15
Joint | 51.55 | 70.30 | 71.88 | 22.32 | 70.36 | 29.38
Wgan | 53.17 | 70.99 | 73.25 | 23.34 | 70.20 | 30.39
Two Language Adaptation | Chance |  2.30 |  4.59 | 50.00 |  0.46 |  4.59 | 10.00
TF-IDF | 38.32 | 60.29 | 64.25 | 13.99 | 60.29 | 23.84
EnOnly | 46.77 | 67.62 | 67.90 | 19.88 | 65.86 | 27.83
FineTune | 50.98 | 68.79 | 72.60 | 24.30 | 68.89 | 32.96
Joint | 53.37 | 69.75 | 74.94 | 26.60 | 69.75 | 35.59
Wgan | 54.14 | 70.07 | 75.63 | 27.23 | 69.76 | 36.11
Five Language Adaptation | TF-IDF | 39.04 | 63.10 | 62.06 | 13.12 | 63.10 | 20.64
EnOnly | 41.55 | 66.48 | 61.75 | 13.05 | 63.77 | 19.38
Joint | 50.69 | 69.00 | 72.18 | 22.80 | 69.18 | 31.11
Wgan | 52.11 | 69.74 | 73.34 | 23.39 | 69.35 | 31.88
Table 3:  Results for Single-/Two-/Five- language adaptation. Each number
represents accuracy on addressee-response selection (ADR-RES), addressee
selection (ADR) or response selection (RES).

##  7 Experiments

###  7.1 Experimental Setup

####  7.1.1 Task Settings

We use the following languages: (i) English (En) as the high-resource
language, and (ii) Italian (It), Croatian (Hr), German (De), Portuguese (Pt)
as the low-resource languages. In the following, we describe the languages
used in each task.

### (a) Single-Language Adaptation

Train: English + 1 Low-Res. Language, Dev & Test: 1 Low-Res. Language

For example, in the case that the target is Italian (It), we use the En and It
training sets to train a model, and evaluate the trained model on the It test
set. As evaluation metrics, we use the three types of accuracies, ADR-RES, ARD
and RES (described in Sec. 3). We report the macro average accuracies of all
source-target language pairs (En-It, En-Hr, En-De, and En-Pt).

### (b) Multi-Language Adaptation

Train: English + $|\mathcal{T}|$ Low-Res. Languages, Dev & Test: English +
$|\mathcal{T}|$ Low-Res. Languages

We use the En, It, Hr, Pt and De training sets to train a unified model, and
evaluate it on the test sets for all the languages (En, It, Hr, Pt, De). As
evaluation metrics, we use the macro averages of ADR-RES, ARD and RES for all
the languages. For example, for two language adaptation ($|\mathcal{T}|=1$),
we report the macro averages over all the language pairs (En-It, En-Hr, En-De,
and En-Pt). For five language adaptation ($|\mathcal{T}|=4$), we report the
macro averages over all the five languages. Note that while we evaluate the
performance on only the test set of the target low-resource language in
single-language adaptation, we evaluate it on the test sets of English and the
low-resource languages in multi-language adaptation.

####  7.1.2 Comparative Methods

We compare several methods. Our proposed methods (Sec. 5) are orthogonal, so
that we can combine a method with others. In the following, we list the
methods used in the comparison.

  * •

TrgOnly: A dynamic model proposed by ?) trained on only the low-resource
target language data.

  * •

EnOnly: A model built by (a) multilingual embedding replacement in Sec. 5.1:
training a model on the English data and replacing the English word embeddings
with the embeddings of the low-resource language.

  * •

FineTune: A model built by (b) fine-tuning in Sec. 5.2: training a model on
the high-resource language (English), and retraining it on the low-resource
language.

  * •

Joint: A model built by (b) fine-tuning and (c) joint loss training: building
a model by FineTune as an initial model, and retraining it with the joint loss
functions.

  * •

Wgan: A model built by (b) fine-tuning and (d) multi-language adversarial
training: building a model by FineTune as an initial model, and retraining it
with W-GAN.

Besides the neural models, we also use the TF-IDF model used in ?). This model
firstly creates TF-IDF vectors for the context and each candidate response.
Then, it computes the cosine similarity for each pair of the context vector
and a response vector. Finally, it selects the candidate response with the
highest similarity.

####  7.1.3 Optimization

We use stochastic gradient descent (SGD) with a mini-batch method. To update
parameters, we use Adam [Kingma and Ba (2014]. We describe the details of
hyper-parameter settings in Supplementary Material.

###  7.2 Results

Tab. 3 shows the results of single-language and multi-language adaptation.
Note that Tab. 6, Tab. 7, and Tab. 8 shows the detailed results for each
language.

#### Single-Language Adaptation

Wgan achieved the best scores for most of the metrics. This suggests that the
W-GAN method successfully transfers knowledge of high-resource language to a
low-resource language. Also, FineTune outperformed TrgOnly. This means that
pre-training parameters on the high-resource language data improves a model
for a target low-resource language. Interestingly, EnOnly achieved higher
scores than chance-level without any target language data. One possible
explanation is that the multilingual embeddings have good alignments to some
extent between similar meaning words in different languages.

#### Multi-Language Adaptation

In both two- and five-language adaptation, Wgan achieved the best scores.
Specifically, In five language adaptation, regardless of using a single,
unified model, Joint and Wgan achieved high-performance.101010Since FineTune
builds a model for each target language, it cannot analyze multiple languages
with a single model. That is why there is no result of FineTune in five-
language adaptation.

Also, Wgan outperformed Joint in all the metrics. This suggests that Wgan
learns language-invariant features more effectively.

In NLP tasks, ?) applied W-GAN to cross-lingual sentiment classification and
successfully transferred the source-side knowledge to the target one. In this
paper, we have extended the adaptation of single source-target pair to the
adaptation of multiple pairs. Our experimental results show that our method
works well for multi-language adaptation in conversation domain.

###  7.3 Data Augmentation with NMT

Figure 4: Effects of data augmentation with NMT. Figure 5: Effects of the
number of agents in the context. Left axis: ADR-RES accuracy on test sets
(drawn as lines). Right axis: the average number of test samples (drawn as
bars).
---

As we mentioned in Sec. 5.1 (a), as another approach to compensating for low-
resource language, we use data augmentation. To increase the amount of the
training set of a low-resource language, we translate high-resource language
(English) samples to low-resource ones by using Neural Machine Translation
(NMT). Although some translations are noisy, we can obtain much more training
samples for a low-resource language. One limitation of this method is that it
is costly to prepare parallel corpora, which is often unavailable for low-
resource languages. For reproducibility, we use publicly available NMT models
already trained on a parallel corpus. Since
OpenNMT111111http://opennmt.net/Models/ provides an English-German model, we
conduct experiments for the English-German pair.

We investigate the effects of translated German data for pre-training a model.
Translating English (En) training utterances to German (De) ones by using the
trained NMT model, we can obtain translated German training data (De’). We
compare the two settings: (i) pre-training a model on English data and (ii)
pre-training a model on translated German data. After pre-training, we apply
the methods used in Sec. 7. We evaluate the performance with ADR-RES accuracy
on the German test set.

Fig. 5 shows the results ($|\mathcal{R}|=10$). The red dotted line is the
performance of the modes trained on only original German training data (De).
In each method, the blue bar at left hand is a model pre-trained on English
(En), and the green bar at right hand is a model pre-trained on (De’).

PreTrainOnly, the left-most method in Fig.5, uses the pre-trained models. The
results were almost the same between models pre-trained on English or
translated German data, and worse than the model trained on only the original
German training data (red dotted line). This suggests that only the
translations by NMT are not sufficient for building good multilingual ARS
models.

Furthermore, we re-train the pre-trained models by using the three methods,
FineTune, Joint and Wgan. In other words, each method uses a pre-trained model
as the initial model and re-trains the parameters. In all methods, the models
re-trained from the De’ pre-trained model (green bars) were better than the
ones from the En pre-trained ones (blue bars). This suggests that by combining
the NMT-based data augmentation method with the knowledge-transfer methods,
the performance is boosted. Another point is that Wgan consistently
outperformed the other methods, which supports the utility of Wgan.

###  7.4 Analysis of Number of Agents

We investigate how accuracy fluctuates according to the number of agents in
the context of length $15$, as shown in Fig. 5. Overall, as the number of
agents increases, the accuracies of all the methods tend to decline. Among
them, Wgan achieved the best results in most of the cases. This suggests that
Wgan can stably predicts addressees and responses in conversations with many
participants.

##  8 Conclusion and Future Work

We have introduced multilingual addressee and response selection by providing
(i) formal task definitions, (ii) several knowledge-transfer methods and (iii)
a multilingual conversation corpus and dataset.

Experimental results have demonstrated that our methods allow models to adapt
multiple target languages. In particular, methods for language-invariant
models can simultaneously deal with multiple languages with a single model.

Since our methods and dataset can apply to response generation, tackling the
multilingual response generation tasks is an interesting line of our future
work. In addition, our language-invariant systems can receive conversation in
a language (e.g., English ) and reply to it in another language (e.g.,
German). It can lead to interesting findings that our system is evaluated on
code-mixing situations, where two or more languages are used in the same
context.

## References

  * [Akker and Traum (2009]  Rieks Akker and David Traum.  2009\.  A comparison of addressee detection methods for multiparty conversations.  In Workshop on the Semantics and Pragmatics of Dialogue.
  * [Ammar et al. (2016]  Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A. Smith.  2016\.  Massively multilingual word embeddings.  CoRR.
  * [Arjovsky et al. (2017]  Martin Arjovsky, Soumith Chintala, and Léon Bottou.  2017\.  Wasserstein gan.  arXiv preprint arXiv:1701.07875.
  * [Bohus and Horvitz (2011]  Dan Bohus and Eric Horvitz.  2011\.  Multiparty turn taking in situated dialog: Study, lessons, and directions.  In Proceedings of the SIGDIAL 2011 Conference, SIGDIAL ’11, pages 98–109, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * [Chen et al. (2016]  Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger.  2016\.  Adversarial deep averaging networks for cross-lingual sentiment classification.
  * [Cho et al. (2014]  Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  2014\.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  In EMNLP.
  * [Goodfellow et al. (2014]  Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  2014\.  Generative adversarial nets.  In Proceedings of NIPS, pages 2672–2680.
  * [Ji et al. (2014]  Zongcheng Ji, Zhengdong Lu, and Hang Li.  2014\.  An information retrieval approach to short text conversation.  arXiv preprint arXiv: 1408.6988.
  * [Jovanović and Akker (2004]  Natasa Jovanović and op den Rieks Akker.  2004\.  Towards automatic addressee identification in multi-party dialogues.  In Proceedings of SIGDIAL.
  * [Jovanović et al. (2006]  Natasa Jovanović, op den Rieks Akker, and Anton Nijholt.  2006\.  Addressee identification in face-to-face meetings.  In Proceedings of EACL.
  * [Kim et al. (2016]  Seokhwan Kim, Luis Fernando D’Haro, Rafael E Banchs, Jason D Williams, Matthew Henderson, and Koichiro Yoshino.  2016\.  The fifth dialog state tracking challenge.  In Proceeding of Spoken Language Technology Workshop (SLT), pages 511–517.
  * [Kingma and Ba (2014]  Diederik P. Kingma and Jimmy Lei Ba.  2014\.  Adam: A method for stochastic optimization.  arXiv preprint arXiv: 1412.6980.
  * [Kirkpatrick et al. (2017]  James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  2017\.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the National Academy of Sciences of the United States of America, pages 3521–3526.
  * [Li et al. (2016]  Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan.  2016\.  A persona-based neural conversation model.  In Proceedings of ACL.
  * [Liu et al. (2016]  Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau.  2016\.  How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation.  In Proceedings of EMNLP, pages 2122–2132.
  * [Lowe et al. (2015]  Ryan Lowe, Nissan Pow, Iulian V. Serban, and Joelle Pineau.  2015\.  The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems.  In Proceedings of SIGDIAL, pages 285–294.
  * [Lowe et al. (2016]  Ryan Lowe, Iulian Vlad Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau.  2016\.  On the evaluation of dialogue systems with next utterance classification.  In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 264–269, Los Angeles, September. Association for Computational Linguistics.
  * [Lowe et al. (2017]  Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau.  2017\.  Towards an automatic turing test: Learning to evaluate dialogue responses.  In Proceedings of ACL, pages 1116–1126.
  * [Lu and Li (2013]  Zhengdong Lu and Hang Li.  2013\.  A deep architecture for matching short texts.  In Proceedings of NIPS, pages 1367–1375.
  * [Mei et al. (2017]  Hongyuan Mei, Mohit Bansal, and Matthew R Walter.  2017\.  Coherent dialogue with attention-based language models.  In Proceedings of AAAI.
  * [Nakano et al. (2013]  Yukiko I. Nakano, Naoya Baba, Hung-Hsuan Huang, and Yuki Hayashi.  2013\.  Implementation and evaluation of a multimodal addressee identification mechanism for multiparty conversation systems.  In Proceedings of the 15th ACM on International Conference on Multimodal Interaction, ICMI ’13, pages 35–42, New York, NY, USA. ACM.
  * [Nakatani (2010]  Shuyo Nakatani.  2010\.  Language detection library for java.
  * [Ouchi and Tsuboi (2016]  Hiroki Ouchi and Yuta Tsuboi.  2016\.  Addressee and response selection for multi-party conversation.  In Proceedings of EMNLP, pages 2133–2143.
  * [Papineni et al. (2002]  Kishore Papineni, Salim E. Roucos, Todd Ward, and Wei-Jing Zhu.  2002\.  Bleu: a method for automatic evaluation of machine translation.  In ACL.
  * [Ravuri and Stolcke (2014]  Suman V Ravuri and Andreas Stolcke.  2014\.  Neural network models for lexical addressee detection.  In Proceedings of INTERSPEECH, pages 298–302.
  * [Ritter et al. (2011]  Alan Ritter, Colin Cherry, and William B. Dolan.  2011\.  Data-driven response generation in social media.  In Proceedings of EMNL, pages 583–593.
  * [Serban et al. (2016]  Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau.  2016\.  Building end-to-end dialogue systems using generative hierarchical neural network models.  In Proceedings of AAAI, pages 3776–3783.
  * [Shang et al. (2015]  Lifeng Shang, Zhengdong Lu, and Hang Li.  2015\.  Neural responding machine for short-text conversation.  In Proceedings of ACL/IJCNLP, pages 1577–1586.
  * [Traum (2003]  David Traum.  2003\.  Issues in multiparty dialogues.  Advances in Agent communication, pages 201–211.
  * [Uthus and Aha (2013]  David C Uthus and David W Aha.  2013\.  Multiparticipant chat analysis: A survey.  Artificial Intelligence, pages 106–121.
  * [Vinyals and Le (2015]  Oriol Vinyals and V. Quoc Le.  2015\.  A neural conversational model.  arXiv preprint arXiv: 1506.05869.
  * [Wang et al. (2013]  Hao Wang, Zhengdong Lu, Hang Li, and Enhong Chen.  2013\.  A dataset for research on short-text conversations.  In Proceedings of EMNLP, pages 935–945.
  * [Wang et al. (2015]  Mingxuan Wang, Zhengdong Lu, Hang Li, and Qun Liu.  2015\.  Syntax-based deep matching of short texts.  In Proceedings of IJCAI, pages 1354–1361.

##  Appendix A Hyper-Parameters

Hyper-parameter | Values
---|---
Embedding size | 512
GRU state size | 256
WGAN $\lambda$ | 0.50
WGAN iterations | 5
Critic hidden size | 512
Critic activation function | ReLU
Batch size | 32
Max epoch | 30
Adam alpha | {0.001, 0.0005, 0.0001}
L2 weight decay | {0.001, 0.0005, 0.0001}
Table 4:  Hyper-parameters for our experiments.

##  Appendix B Statics of Dataset

Tab. 5 shows the details of our dataset. “Docs” is documents, “Utters” is
utterances, “W. / U.” is the number of words per utterance, “A. / D.” is the
number of agents per document.

| Dataset (Train / Dev / Test)
---|---
| English (En) | Italian (It) | Croatian (Hr) | German (De) | Portuguese (Pt)
No. of Docs | 7355 | 357 | 254 | 248 | 211
(6,606/ 367/ 382) | (306/ 17/ 34) | (216/ 12/ 26) | (216/ 12/ 20) | (180/ 10/ 21)
No. of Utters | 2.4 M | 165 k | 80 k | 38 k | 52 k
(2.1 M / 13.2 k / 15.1 k) | (144 k / 7 k / 14 k) | ( 71 k / 3.4 k / 6.9 k) | ( 33 k / 1.9 / 2.9 k) | ( 44 k / 2.1 / 6.1 k )
No. of Words | 27.0 M | 1.1 M | 630 k | 335 k | 285 k
(23.8 M/ 1.5 M/ 1.7 M) | ( 1.0 M/ 54 k/ 100 k) | ( 553 k/ 25 k/ 52 k) | ( 294 k/ 16 k/ 24 k) | ( 243 k/ 11 k/ 30 k)
No. of Samples | - | - | - | - | -
665.6 k/ 45.1 k/ 51.9 k | 38511 / 2561 / 3873 | 11387 / 512 / 1145 | 5500 / 354 / 569 | 5951 / 285 / 975
Avg. W. / U. | 11.1 | 7.2 | 7.5 | 8.5 | 5.2
(11.1/ 11.2/ 11.3) | ( 6.9/ 7.7/ 7.1) | ( 7.7/ 7.3/ 7.5) | ( 8.9/ 8.4/ 8.2) | ( 5.5/ 5.2/ 4.9)
Avg. A. / D. | 26.8 | 25.6 | 12.9 | 16.4 | 19.0
(26.3/ 30.68/ 32.1) | (24.9/ 26.2/ 25.6) | (12.7/ 13.5/ 12.7) | (17.4/ 15.9/ 15.8) | (19.7/ 18.6/ 18.8)
Table 5:  Statistics of the multilingual dataset.

##  Appendix C Results for each language

Tab. 6 shows the detailed results of single-language adaptation. Tab. 7 shows
the detailed results of two-language adaptation. Tab. 8 shows the detailed
results of five-language adaptation.

|  |  | $|\mathcal{R}|=2$ | $|\mathcal{R}|=10$
---|---|---|---|---
Target | #Train | Method | ADR-RES | ADR | RES | ADR-RES | ADR | RES
It | 38,511 | Chance | 2.99 | 5.97 | 50.00 | 0.60 | 5.97 | 10.00
TF-IDF | 43.89 | 67.49 | 64.58 | 16.63 | 67.49 | 23.42
TrgOnly | 63.28 | 79.86 | 78.36 | 32.87 | 80.92 | 38.73
EnOnly | 44.54 | 72.37 | 60.11 | 9.91 | 66.74 | 16.29
FineTune | 64.81 | 80.79 | 79.14 | 34.57 | 81.44 | 41.26
Joint | 63.44 | 79.81 | 78.36 | 34.37 | 80.30 | 40.82
Wgan | 65.17 | 80.56 | 79.94 | 35.71 | 80.76 | 42.14
Hr | 11,387 | Chance | 5.39 | 10.78 | 50.00 | 1.08 | 10.78 | 10.00
TF-IDF | 35.63 | 58.78 | 61.05 | 10.22 | 58.78 | 17.29
TrgOnly | 40.52 | 63.06 | 64.10 | 14.32 | 63.23 | 22.97
EnOnly | 34.06 | 60.87 | 54.24 | 7.95 | 60.52 | 12.31
FineTune | 40.00 | 62.62 | 63.23 | 14.67 | 64.72 | 22.79
Joint | 44.37 | 62.97 | 69.26 | 15.98 | 62.97 | 24.54
Wgan | 45.07 | 62.62 | 70.48 | 16.59 | 63.76 | 25.68
De | 5,500 | Chance | 4.09 | 8.17 | 50.00 | 0.82 | 8.17 | 10.00
TF-IDF | 36.38 | 64.67 | 55.36 | 10.19 | 64.67 | 14.41
TrgOnly | 43.94 | 67.49 | 64.15 | 16.52 | 66.96 | 22.50
EnOnly | 36.56 | 63.27 | 59.75 | 5.98 | 57.12 | 12.13
FineTune | 50.44 | 68.89 | 72.06 | 20.39 | 68.19 | 26.71
Joint | 50.79 | 70.30 | 70.47 | 21.79 | 69.24 | 27.94
Wgan | 52.90 | 71.53 | 72.23 | 22.50 | 68.89 | 28.30
Pt | 5,951 | Chance | 3.42 | 6.84 | 50.00 | 0.68 | 6.84 | 10.00
TF-IDF | 42.15 | 68.92 | 61.44 | 13.13 | 68.92 | 18.87
TrgOnly | 41.64 | 66.67 | 62.77 | 13.95 | 67.79 | 20.31
EnOnly | 37.13 | 66.36 | 56.51 | 10.15 | 65.13 | 14.26
FineTune | 43.08 | 66.05 | 64.92 | 14.97 | 66.97 | 21.85
Joint | 47.59 | 68.10 | 69.44 | 17.13 | 68.92 | 24.21
Wgan | 49.54 | 69.23 | 70.36 | 18.56 | 67.38 | 25.44
Avg. | - | Chance |  3.97 |  7.94 | 50.00 |  0.80 |  7.94 | 10.00
TF-IDF | 39.51 | 64.97 | 60.61 | 12.54 | 64.97 | 18.50
TrgOnly | 47.35 | 69.27 | 67.35 | 19.42 | 69.73 | 26.13
EnOnly | 38.07 | 65.72 | 57.65 |  8.50 | 62.38 | 13.75
FineTune | 49.58 | 69.59 | 69.84 | 21.15 | 70.33 | 28.15
Joint | 51.55 | 70.30 | 71.88 | 22.32 | 70.36 | 29.38
Wgan | 53.17 | 70.99 | 73.25 | 23.34 | 70.20 | 30.39
Table 6:  Results for single-language adaptation. Each number represents accuracy on addressee-response selection (ADR-RES), addressee selection (ADR) or response selection (RES). #Train is the number of training data. |  | $|\mathcal{R}|=2$ | $|\mathcal{R}|=10$
---|---|---|---
Target | Method | ADR-RES | ADR | RES | ADR-RES | ADR | RES
En, It | Chance |  1.80 ( 0.62, 2.95) |  3.61 | 50.00 |  0.36 ( 0.12, 0.60) | 3.61 | 10.00
TF-IDF | 40.51 (37.13, 43.89) | 61.56 | 66.24 | 16.04 (15.44, 16.63) | 61.56 | 26.31
EnOnly | 50.01 (55.47, 44.54) | 70.95 | 69.13 | 20.59 (31.27, 9.91) | 68.05 | 29.11
FineTune | 59.13 (53.44, 64.81) | 74.71 | 77.78 | 31.07 (27.56, 34.57) | 74.62 | 39.34
Joint | 59.56 (55.67, 63.44) | 74.63 | 78.50 | 33.04 (31.71, 34.37) | 74.77 | 41.72
Wgan | 60.20 (55.23, 65.17) | 74.94 | 79.10 | 33.38 (31.04, 35.71) | 75.09 | 41.87
En, Hr | Chance |  2.35 ( 0.62, 5.39) | 4.71 | 50.00 |  0.47 ( 0.12, 1.08) |  4.71 | 10.00
TF-IDF | 36.76 (37.13, 35.63) | 60.15 | 61.63 | 12.82 (15.44, 10.22) | 60.15 | 21.80
EnOnly | 44.77 (55.47, 34.06) | 65.20 | 66.20 | 19.61 (31.27, 7.95) | 64.94 | 27.12
FineTune | 46.03 (52.06, 40.00) | 65.29 | 69.15 | 20.96 (27.24, 14.67) | 66.01 | 30.28
Joint | 49.49 (55.66, 43.32) | 65.98 | 73.14 | 22.95 (31.74, 14.15) | 66.15 | 32.23
Wgan | 49.80 (54.53, 45.07) | 65.70 | 73.96 | 23.61 (30.62, 16.59) | 66.38 | 33.52
En, De | Chance |  3.01 ( 0.62, 4.08) |  6.01 | 50.00 |  0.60 ( 0.12, 0.82) |  6.01 | 10.00
TF-IDF | 36.38 (37.13, 36.38) | 57.20 | 64.47 | 12.83 (15.44, 10.19) | 57.20 | 23.24
EnOnly | 46.02 (55.47, 36.56) | 66.40 | 68.95 | 18.63 (31.27, 5.98) | 63.24 | 27.03
FineTune | 52.22 (53.99, 50.44) | 68.79 | 74.52 | 24.66 (28.93, 20.39) | 68.20 | 33.09
Joint | 53.04 (55.28, 50.79) | 69.73 | 74.30 | 26.09 (31.97, 20.21) | 68.98 | 35.14
Wgan | 54.05 (55.20, 52.90) | 70.35 | 75.19 | 27.05 (31.42, 22.67) | 69.43 | 35.31
En, Pt | Chance |  2.02 ( 0.62, 3.42) |  4.04 | 50.00 |  0.40 ( 0.12, 0.68) |  4.04 | 10.00
TF-IDF | 39.64 (37.13, 42.15) | 62.27 | 64.67 | 14.29 (15.44, 13.13) | 62.27 | 24.03
EnOnly | 46.30 (55.47, 37.13) | 67.94 | 67.33 | 20.71 (31.27, 10.15) | 67.24 | 28.09
FineTune | 46.53 (49.98, 43.08) | 66.39 | 68.97 | 20.52 (26.06, 14.97) | 66.74 | 29.13
Joint | 51.39 (55.18, 47.59) | 68.66 | 73.81 | 24.32 (31.51, 17.13) | 69.11 | 33.28
Wgan | 52.49 (55.44, 49.54) | 69.31 | 74.29 | 24.88 (31.19, 18.56) | 68.16 | 33.75
Avg. | Chance |  2.30 |  4.59 | 50.00 |  0.46 |  4.59 | 10.00
TF-IDF | 38.32 | 60.29 | 64.25 | 13.99 | 60.29 | 23.84
EnOnly | 46.77 | 67.62 | 67.90 | 19.88 | 65.86 | 27.83
FineTune | 50.98 | 68.79 | 72.60 | 24.30 | 68.89 | 32.96
Joint | 53.37 | 69.75 | 74.94 | 26.60 | 69.75 | 35.59
Wgan | 54.14 | 70.07 | 75.63 | 27.23 | 69.76 | 36.11
Table 7:  Results for two-language adaptation. Each number is macro average of the accuracies over all the languages. Each of the parenthesized numbers in the ADR-RES column is ADR-RES accuracy for each language. |  | $|\mathcal{R}|=2$ | $|\mathcal{R}|=10$
---|---|---|---
Target | Method | ADR-RES | ADR | RES | ADR-RES | ADR | RES
En, It | TF-IDF | 39.04 | 63.10 | 62.06 | 13.12 | 63.10 | 20.64
(37.13, 43.89, 35.63, 36.38, 42.15) |  |  | (15.44, 16.63, 10.22, 10.19, 13.13) |  |
EnOnly | 41.55 | 66.48 | 61.75 | 13.05 | 63.77 | 19.38
(55.47, 44.54, 34.06, 36.56, 37.13) |  |  | (31.27, 9.91, 7.95, 5.98, 10.15) |  |
Hr, De, Pt | Joint | 50.69 | 69.00 | 72.18 | 22.80 | 69.18 | 31.11
(54.49, 61.71, 43.41, 47.80, 46.05) |  |  | (31.26, 30.29, 14.59, 20.21, 17.64) |  |
| Wgan | 52.11 | 69.74 | 73.34 | 23.39 | 69.35 | 31.88
| (53.88, 63.18, 44.19, 52.02, 47.28) |  |  | (30.6, 31.29, 14.67, 22.32, 18.05) |  |
Table 8:  Results for five-language adaptation. Each number is macro average
of the accuracies over all the languages. Each of the parenthesized numbers in
the ADR-RES column is ADR-RES accuracy for each language.