* Nguyen et al. [2015] Anh Nguyen, Jason Yosinski, and Jeff Clune.  Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 427–436. IEEE, 2015.
  * Song et al. [2018] Qingquan Song, Haifeng Jin, Xiao Huang, and Xia Hu.  Multi-label adversarial perturbations.  In _2018 IEEE International Conference on Data Mining (ICDM)_ , pages 1242–1247. IEEE, 2018.
  * Zhou et al. [2020] Nan Zhou, Wenjian Luo, Xin Lin, Peilan Xu, and Zhenya Zhang.  Generating multi-label adversarial examples by linear programming.  In _2020 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–8. IEEE, 2020.
  * Lu et al. [2021] Shaohao Lu, Yuqiao Xian, Ke Yan, Yi Hu, Xing Sun, Xiaowei Guo, Feiyue Huang, and Wei-Shi Zheng.  Discriminator-free generative adversarial attack.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , pages 1544–1552. ACM, 2021.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25, 2012.
  * Boutell et al. [2004] Matthew R Boutell, Jiebo Luo, Xipeng Shen, and Christopher M Brown.  Learning multi-label scene classification.  _Pattern recognition_ , 37(9):1757–1771, 2004\.
  * Zhang and Zhou [2007] Min-Ling Zhang and Zhi-Hua Zhou.  Ml-knn: A lazy learning approach to multi-label learning.  _Pattern recognition_ , 40(7):2038–2048, 2007\.
  * Read et al. [2011] Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank.  Classifier chains for multi-label classification.  _Machine learning_ , page 333, 2011.
  * Chen et al. [2018a] Shang-Fu Chen, Yi-Chen Chen, Chih-Kuan Yeh, and Yu-Chiang Wang.  Order-free rnn with visual attention for multi-label classification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_. AAAI, 2018a.
  * Allwein et al. [2000] Erin L Allwein, Robert E Schapire, and Yoram Singer.  Reducing multiclass to binary: A unifying approach for margin classifiers.  _Journal of machine learning research_ , 1(Dec):113–141, 2000.
  * Chen et al. [2019a] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo.  Multi-Label Image Recognition with Graph Convolutional Networks.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2019a.
  * Cai et al. [2022] Zikui Cai, Shantanu Rane, Alejandro E Brito, Chengyu Song, Srikanth V Krishnamurthy, Amit K Roy-Chowdhury, and M Salman Asif.  Zero-query transfer attacks on context-aware object detectors.  _arXiv preprint arXiv:2203.15230_ , 2022.
  * Cai et al. [2021] Zikui Cai, Xinxin Xie, Shasha Li, Mingjun Yin, Chengyu Song, Srikanth V Krishnamurthy, Amit K Roy-Chowdhury, and M Salman Asif.  Context-aware transfer attacks for object detection.  _arXiv preprint arXiv:2112.03223_ , 2021.
  * Sitawarin et al. [2018] Chawin Sitawarin, Arjun Nitin Bhagoji, Arsalan Mosenia, Mung Chiang, and Prateek Mittal.  Darts: Deceiving autonomous cars with toxic signs.  _arXiv preprint arXiv:1802.06430_ , 2018.
  * Chen et al. [2018b] Shang-Tse Chen, Cory Cornelius, Jason Martin, and Duen Horng Polo Chau.  Shapeshifter: Robust physical adversarial attack on faster r-cnn object detector.  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , pages 52–68. Springer, 2018b.
  * Yin et al. [2021] Mingjun Yin, Shasha Li, Zikui Cai, Chengyu Song, M Salman Asif, Amit K Roy-Chowdhury, and Srikanth V Krishnamurthy.  Exploiting multi-object relationships for detecting adversarial attacks in complex scenes.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7858–7867, 2021.
  * Xiao et al. [2018a] Chaowei Xiao, Bo Li, Jun-Yan Zhu, Warren He, Mingyan Liu, and Dawn Song.  Generating Adversarial Examples with Adversarial Networks.  _arXiv preprint arXiv:1801.02610_ , 2018a.
  * Wah et al. [2011] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * PyTorch [2022] PyTorch.  Imagenet models and pre-trained weights.  PyTorch, 2022.  URL https://pytorch.org/vision/stable/models.html.
  * Badue et al. [2021] Claudine Badue, Rânik Guidolini, Raphael Vivacqua Carneiro, Pedro Azevedo, Vinicius B Cardoso, Avelino Forechi, Luan Jesus, Rodrigo Berriel, Thiago M Paixao, Filipe Mutz, et al.  Self-driving cars: A survey.  _Expert Systems with Applications_ , 165:113816, 2021.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International Conference on Machine Learning_ , pages 4904–4916. PMLR, 2021.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pages 8748–8763. PMLR, 2021.
  * Mu et al. [2021] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie.  Slip: Self-supervision meets language-image pre-training.  _arXiv preprint arXiv:2112.12750_ , 2021.
  * Li et al. [2021b] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan.  Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.  _arXiv preprint arXiv:2110.05208_ , 2021b.
  * Yao et al. [2021] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  Filip: Fine-grained interactive language-image pre-training.  _arXiv preprint arXiv:2111.07783_ , 2021.
  * Luo et al. [2021] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li.  Clip4clip: An empirical study of clip for end to end video clip retrieval.  _arXiv preprint arXiv:2104.08860_ , 2021.
  * Patashnik et al. [2021] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2085–2094, 2021.
  * Khandelwal et al. [2021] Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi.  Simple but effective: Clip embeddings for embodied ai.  _arXiv preprint arXiv:2111.09888_ , 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Conde and Turgutlu [2021] Marcos V Conde and Kerem Turgutlu.  Clip-art: contrastive pre-training for fine-grained art classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3956–3960, 2021.
  * Gal et al. [2021] Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or.  Stylegan-nada: Clip-guided domain adaptation of image generators.  _arXiv preprint arXiv:2108.00946_ , 2021.
  * Sanghi et al. [2021] Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Yi Cheng, and Marco Fumero.  Clip-forge: Towards zero-shot text-to-shape generation.  _arXiv preprint arXiv:2110.02624_ , 2021.
  * Kim and Ye [2021] Gwanghyun Kim and Jong Chul Ye.  Diffusionclip: Text-guided image manipulation using diffusion models.  _arXiv preprint arXiv:2110.02711_ , 2021.
  * Kwon and Ye [2021] Gihyun Kwon and Jong Chul Ye.  Clipstyler: Image style transfer with a single text condition.  _arXiv preprint arXiv:2112.00374_ , 2021.
  * Rao et al. [2022] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu.  Denseclip: Language-guided dense prediction with context-aware prompting.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Li et al. [2022] Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, and Shih-Fu Chang.  Clip-event: Connecting text and images with event structures.  _arXiv preprint arXiv:2201.05078_ , 2022.
  * Fawzi et al. [2018] Alhussein Fawzi, Omar Fawzi, and Pascal Frossard.  Analysis of Classifiers’ Robustness to Adversarial Perturbations.  _Machine Learning_ , pages 481–508, 2018.
  * Liu et al. [2019] Aishan Liu, Xianglong Liu, Jiaxin Fan, Yuqing Ma, Anlan Zhang, Huiyuan Xie, and Dacheng Tao.  Perceptual-Sensitive GAN for Generating Adversarial Patches.  In _Proceedings of the AAAI conference on artificial intelligence_ , pages 1028–1035. AAAI, 2019.
  * Han et al. [2019] Jiangfan Han, Xiaoyi Dong, Ruimao Zhang, Dongdong Chen, Weiming Zhang, Nenghai Yu, Ping Luo, and Xiaogang Wang.  Once a MAN: Towards Multi-Target Attack via Learning Multi-Target Adversarial Network Once.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5158–5167. IEEE, 2019.
  * Xiao et al. [2018b] Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song.  Spatially transformed adversarial examples.  _arXiv preprint arXiv:1801.02612_ , 2018b.
  * Nakka and Salzmann [2020] Krishna Kanth Nakka and Mathieu Salzmann.  Indirect local attacks for context-aware semantic segmentation networks.  In _European Conference on Computer Vision_ , pages 611–628. Springer, 2020.
  * Hu et al. [2021] Shu Hu, Lipeng Ke, Xin Wang, and Siwei Lyu.  Tkml-ap: Adversarial attacks to top-k multi-label learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7649–7657. IEEE, 2021.
  * Dong et al. [2018] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li.  Boosting Adversarial Attacks with Momentum.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 9185–9193. IEEE, 2018.
  * Fan et al. [2020] Yanbo Fan, Baoyuan Wu, Tuanhui Li, Yong Zhang, Mingyang Li, Zhifeng Li, and Yujiu Yang.  Sparse adversarial attack via perturbation factorization.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16_ , pages 35–50. Springer, 2020.
  * Bhattad et al. [2019] Anand Bhattad, Min Jin Chong, Kaizhao Liang, Bo Li, and David A Forsyth.  Unrestricted adversarial examples via semantic manipulation.  _arXiv preprint arXiv:1904.06347_ , 2019.
  * Qiu et al. [2020] Haonan Qiu, Chaowei Xiao, Lei Yang, Xinchen Yan, Honglak Lee, and Bo Li.  Semanticadv: Generating adversarial examples via attribute-conditioned image editing.  In _European Conference on Computer Vision_ , pages 19–37. Springer, 2020.
  * Jolicoeur-Martineau [2018] Alexia Jolicoeur-Martineau.  The relativistic discriminator: a key element missing from standard gan.  In _International Conference on Learning Representations_ , 2018.
  * Fang et al. [2021] Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen.  Clip2video: Mastering video-text retrieval via image clip.  _arXiv preprint arXiv:2106.11097_ , 2021.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very Deep Convolutional Networks for Large-Scale Image Recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Johnson et al. [2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution.  In _European conference on computer vision_ , pages 694–711. Springer, 2016.
  * Mechrez et al. [2018] Roey Mechrez, Itamar Talmi, and Lihi Zelnik-Manor.  The contextual loss for image transformation with non-aligned data.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 768–783, 2018.
  * Hessel et al. [2021] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  CLIPScore: a reference-free evaluation metric for image captioning.  In _EMNLP_ , 2021.
  * Zhou et al. [2022] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In _CVPR_ , 2022.
  * Chen et al. [2019b] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo.  Multi-label image recognition with graph convolutional networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5177–5186, 2019b.
  * Islam et al. [2021] Md Amirul Islam, Matthew Kowal, Konstantinos G Derpanis, and Neil DB Bruce.  Segmix: Co-occurrence driven mixup for semantic segmentation and adversarial robustness.  _arXiv preprint arXiv:2108.09929_ , 2021.
  * Zhang et al. [2019] Hang Zhang, Han Zhang, Chenguang Wang, and Junyuan Xie.  Co-occurrent features in semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 548–557, 2019.
  * Islam et al. [2020] Md Amirul Islam, Matthew Kowal, Konstantinos G Derpanis, and Neil DB Bruce.  Feature binding with category-dependant mixup for semantic segmentation and adversarial robustness.  _arXiv preprint arXiv:2008.05667_ , 2020.
  * Hadsell et al. [2006] Raia Hadsell, Sumit Chopra, and Yann LeCun.  Dimensionality reduction by learning an invariant mapping.  In _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , volume 2, pages 1735–1742. IEEE, 2006.
  * Chuang et al. [2020] Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka.  Debiased contrastive learning.  _Advances in neural information processing systems_ , 33:8765–8775, 2020.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Naseer et al. [2020] Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli.  A self-supervised approach for adversarial robustness.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 262–271, 2020.
  * Madry et al. [2017] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  _arXiv preprint arXiv:1706.06083_ , 2017.
  * Everingham et al. [2010] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The Pascal Visual Object Classes (VOC) Challenge.  _International journal of computer vision_ , 88(2):303–338, 2010.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft COCO: Common Objects in Context.  In _European conference on computer vision_ , pages 740–755. Springer, 2014.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778. IEEE, 2016.
  * Huang et al. [2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely Connected Convolutional Networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4700–4708. IEEE, 2017.
  * Godbole and Sarawagi [2004] Shantanu Godbole and Sunita Sarawagi.  Discriminative methods for multi-labeled classification.  In _Pacific-Asia conference on knowledge discovery and data mining_ , pages 22–30. Springer, 2004.
  * Sorower [2010] Mohammad S Sorower.  A literature survey on algorithms for multi-label learning.  _Oregon State University, Corvallis_ , 18:1–25, 2010.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Krause et al. [2013] Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei.  Collecting a large-scale dataset of fine-grained cars.  2013\.
  * Maji et al. [2013] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  _arXiv preprint arXiv:1306.5151_ , 2013.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. IEEE, 2009.
  * Chen [2022] Aaron Chen.  Coarse-grain models and pre-trained weights.  GitHub link, 2022.  URL https://github.com/aaron-xichen/pytorch-playground.
  * Hu et al. [2018] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7132–7141. IEEE, 2018.
  * Alibaba-AAIG [2022] Alibaba-AAIG.  Fine-grain models and pre-trained weights.  GitHub link, 2022.  URL https://github.com/Alibaba-AAIG/Beyond-ImageNet-Attack/releases/download/Pretrained_DCL_model/model.zip.
  * Girshick [2015] Ross Girshick.  Fast r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , 2015.
  * Lin et al. [2017] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In _Proceedings of the IEEE international conference on computer vision_ , 2017.
  * Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _ECCV_ , 2020.
  * Zhu et al. [2021] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=gZ9hCDWe6ke.
  * Chen et al. [2019c] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin.  MMDetection: Open mmlab detection toolbox and benchmark.  _arXiv preprint arXiv:1906.07155_ , 2019c.
  * Li et al. [2020] Shasha Li, Shitong Zhu, Sudipta Paul, Amit Roy-Chowdhury, Chengyu Song, Srikanth Krishnamurthy, Ananthram Swami, and Kevin S Chan.  Connecting the Dots: Detecting Adversarial Perturbations Using Context Inconsistency.  In _European Conference on Computer Vision_ , pages 396–413. Springer, 2020.
  * Aich et al. [2021] Abhishek Aich, Meng Zheng, Srikrishna Karanam, Terrence Chen, Amit K. Roy-Chowdhury, and Ziyan Wu.  Spatio-temporal representation factorization for video-based person re-identification.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 152–162, October 2021.
  * Aich et al. [2020] Abhishek Aich, Akash Gupta, Rameswar Panda, Rakib Hyder, M. Salman Asif, and Amit K. Roy-Chowdhury.  Non-adversarial video synthesis with learned priors.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * Gupta et al. [2020] Akash Gupta, Abhishek Aich, and Amit K Roy-Chowdhury.  Alanet: Adaptive latent attention network for joint video deblurring and interpolation.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pages 256–264, 2020.
  * Abid et al. [2018] Abubakar Abid, Martin J Zhang, Vivek K Bagaria, and James Zou.  Exploring patterns enriched in a dataset with contrastive principal component analysis.  _Nature communications_ , 9(1):1–7, 2018.
  * Fujiwara et al. [2019] Takanori Fujiwara, Oh-Hyun Kwon, and Kwan-Liu Ma.  Supporting analysis of dimensionality reduction results with contrastive learning.  _IEEE transactions on visualization and computer graphics_ , 26(1):45–55, 2019.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Nair and Hinton [2010] Vinod Nair and Geoffrey E Hinton.  Rectified linear units improve restricted boltzmann machines.  In _ICML_. ICML, 2010.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4401–4410. IEEE, 2019.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Advances in Neural Information Processing Systems_ , pages 8026–8037. NeurIPS, 2019.


# Controllable Generation from Pre-trained Language Models via Inverse
Prompting

Xu Zou12, Da Yin12, Qingyang Zhong12, Ming Ding12, Hongxia Yang4, Zhilin
Yang∗123,Jie Tang∗12 1 Department of Computer Science and Technology, Tsinghua
University  2 Beijing Academy of Artificial Intelligence  3 Recurrent AI, Ltd.
4 DAMO Academy, Alibaba Inc<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
(2021; 2021)

###### Abstract.

Large-scale pre-trained language models have demonstrated strong capabilities
of generating realistic text. However, it remains challenging to control the
generation results. Previous approaches such as prompting are far from
sufficient, which limits the usage of language models. To tackle this
challenge, we propose an innovative method, inverse prompting, to better
control text generation. The core idea of inverse prompting is to use
generated text to inversely predict the prompt during beam search, which
enhances the relevance between the prompt and the generated text and provides
better controllability. Empirically, we pre-train a large-scale Chinese
language model to perform a systematic study using human evaluation on the
tasks of open-domain poem generation and open-domain long-form question
answering. Our results show that our proposed method substantially outperforms
the baselines and that our generation quality is close to human performance on
some of the tasks.††∗Corresponding authors: Zhilin Yang and Jie Tang.

Narrators can try our poem generation demo at
https://pretrain.aminer.cn/apps/poetry.html, while our QA demo can be found at
https://pretrain.aminer.cn/app/qa. For researchers, the code is provided in
https://github.com/THUDM/InversePrompting.

††journalyear: 2021††copyright: acmlicensed††conference: Proceedings of the
27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August
14–18, 2021; Virtual Event, Singapore††booktitle: Proceedings of the 27th ACM
SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’21), August
14–18, 2021, Virtual Event, Singapore††price: 15.00††copyright:
acmcopyright††journalyear: 2021††copyright: none

##  1\. Introduction

Figure 1. The generation process of open-domain traditional Chinese poems
under inverse prompting. Using title New York as an example.

The field of text generation has made tremendous progress recently. Large-
scale autoregressive Transformer models (Vaswani et al., 2017) optimized with
maximum likelihood estimation have shown the ability of generating realistic
text (Dai et al., 2018; Radford et al., 2018; Brown et al., 2020). For real-
world applications of text generation such as essay writing and story
generation, it is essential for the users to be able to control the generation
results. One of the most common approaches is to use prompting; i.e., a user
shall manually write a few sentences to serve as the prompt and the language
model generates the subsequent tokens given the prompt. For example, a user
might input “this is a sad story about a disease named COVID-19” as a prompt
to expect the generation of a COVID-19 story.

However, prompting is far from sufficient for controllable text generation. It
is not uncommon for a language model to deviate the generation process from
the original prompt and start generating text of unrelated topics. Figure 2
shows an example of how language models fail to maintain the coherence between
the prompt and the generated text. In the example, the language model is asked
to answer the question “which moment did you want to live in forever”. The
baseline using conventional prompting generates a story that deviates a lot
from the prompt; i.e., most of the generated content is irrelevant to the
question. There were also unnatural expressions that do not make much sense in
the context.

To tackle this challenge, we propose a novel method, inverse prompting, to
refine the process of text generation from pre-trained language models.
Inverse prompting can be decoupled into three steps. First, given a piece of
generated text, an inverse prompt is constructed using the generated text.
Second, the conditional likelihood of the original prompt given the inverse
prompt is computed based on the pre-trained language model. Third, the
conditional likelihood is used as a score in beam search for selecting the
best generation candidates. As a result, inverse prompting ensures that the
pre-trained language model predicts the prompt given the generated text with
high likelihood, which encourages the relevance of the generated text to the
prompt. For example, in the case of question answering, a language model
generates an answer given the question; during beam search, we use the same
language model to compute the likelihood of the question given the generated
answer for candidate selection. In the example of Figure 2, text generated by
inverse prompting describes a beautiful moment of high school times which is
closely related to the question. As a result, inverse prompting achieves an
average score of 8.60 out of 10 under human evaluation on this example,
compared to the baseline’s 5.40.

Figure 2. An example showing how the prompting baseline model may fail to
maintain relevance in generated text, and how inverse prompting alleiates this
issue. The relevance and overall scores were obtained from human evaluation.

To systematically evaluate the performance of inverse prompting, we conduct an
experiment using human evaluation on the tasks of open-domain long-form
question answering and open-domain traditional Chinese poem generation. We
pre-train a Chinese language model to serve as the base model in our
experiments. The task of long-form question answering is similar to answering
questions on Quora or Zhihu. On this task, we show that inverse prompting
achieves much higher scores in all aspects than the prompting baseline and the
previous state-of-the-art Chinese language model CPM (Zhang et al., 2020). The
task of traditional Chinese poem generation targets generating poems of an
ancient form but with contemporary topics including rocket science,
relativity, or artificial intelligence, which tests the generalization ability
of different approaches. Figure 1 illsutrates an example of traditional
Chinese poem generation under the title New York. It combines contemporary
notions of New York like Manhattan and the financial center with a traditional
form and traditional poetic imagery of cloud and rain. On this task, human
expert evaluation demonstrates that inverse prompting performs significantly
better than the prompting baseline and is comparable to Jiuge (Zhipeng et al.,
2019), a well-known state-of-the-art system for traditional Chinese poem
generation. When we combine inverse prompting with self training, i.e.,
finetuning the model with self-generated poems, our system outperforms Jiuge
under human evaluation by a large margin. Our results of human evaluation
demonstrate that inverse prompting improves the controllability and quality of
text generation significantly and achieves close-to-human results.

##  2\. Related Work

###  2.1. Pre-training and Language Models

Language modeling has been widely used as an objective for pretraining and
demonstrates strong generalization abilities. Originating from word embedding
methods such as word2vec (Mikolov et al., 2013) and GloVe (Pennington et al.,
2014), pretraining methods have displayed an increased level of importance in
the field of natural language processing (Devlin et al., 2018; Liu et al.,
2019; Dai et al., 2018). These models are more general and require less
domain-specific data to achieve strong performance.

Specifically, a main type of pretrained models are autoregressive language
models. Generative pretraining (GPT) (Radford et al., 2018, 2019; Brown et
al., 2020) and Transformer-XL (Dai et al., 2018) achieve substantial
improvement in terms of perplexity and also improves generation quality. The
approach has also been adapted to different languages (Zhang et al., 2020; de
Vries and Nissim, 2020).

Although realistic text can now be generated automatically by large-scale
pretrained language models, it is challenging but essential for users to be
able to control the generation results. Prompting (Dai et al., 2018; Radford
et al., 2018) has been widely used but is rather limited in controlling the
generation results. CTRL (Keskar et al., 2019) proposes to use control codes
to provide conditions for a language model. Different from their method, our
method does not rely on modification of pretraining paradigms or human-
designed attributes. PPLM (Dathathri et al., 2019) performs backpropagation
during test time to adjust generation to maximize the scores given by
attribute models. Compared to PPLM, inverse prompting does not require any
gradient update to the original model and is free of any additional attribute
models.

The idea of using dual process to strengthen the quality of AI generation by
the dual property that the outputs and inputs are inputs and outputs under an
inverse prespective has long been researched. (Xia et al., 2016) introduces
dual learning for the task of machine translation. The method uses multiple
different models to form a translation loop and hopes the contexts will remain
unchanged after passing through the loop. CycleGAN (Chu et al., 2017) and VAE
(An and Cho, 2015) also shares the similar idea of reconstruction in their
applications. Different from these works that uses different forward and
inverse models, in this paper, we exploit the existence of inverse format in
natural languages and use the same language model for prompting and inverse
prompting.

###  2.2. Open-Domain Long-Form Question-Answering

Question answering is a well-studied problem in artificial intelligence
(Simmons, 1970). There are various paradigms of question answering. Short-form
question answering focuses on using a short phrase or sentence to answer the
question (Rajpurkar et al., 2016; Yang et al., 2018). On the other hand, long-
form question answering targets generating multiple sentences or paragraphs to
answer a question in a more comprehensive way. Online question answering
platforms such as Quora and Zhihu can be viewed as good examples of long-form
question answering. While short-form question answering is easier to evaluate
and more more widely studied, we are interested in investigate the ability of
open-domain long-form question answering using text generation models in this
work. Because it is challenging to evaluate the qualities of long-form
question answering, we employ human evaluation in our experiments.

###  2.3. Traditional Chinese Poem Generation

Traditional Chinese poetry is an important genre of Chinese literature with a
history of tens of centuries (Liu, 1966). A few years ago, researchers
experimented with generating traditional Chinese poems using statistical
machine learning methods (Jiang and Zhou, 2008). Later, Jiuge (Zhipeng et al.,
2019; Yi et al., 2020) advanced traditional Chinese poem generation to a new
level. As the well-recognized state of the art for open-domain Chinese poem
generation, Jiuge is able to generate multiple forms of poems under any given
titles, keywords or even images. Despite its ability to handle arbitrary open-
domain inputs, Jiuge performs well on domain-specific contexts such as giant
deserts or iron horses but does not generalize well to contemporary notions
such as Donald Trump, quantum computation, and Europe. Different from Jiuge,
we employ a large-scale language model pretrained on a general-purpose corpus
and leverage inverse prompting to enhance its generation qualities.

##  3\. Methodology

In this section, we discuss the proposed inverse prompting method. The problem
of text generation is modeled as generating $c_{g}$ given the prompt $c_{p}$,
where both $c_{p}$ and $c_{g}$ are sequences of tokens.

###  3.1. Baseline: Prompting and Beam Search

Given a language model with probability distribution $p$, a simple and widely-
used approach is to generate text by maximizing the conditional probability
$p(c_{g}|c_{p})$. This is usually achieved with beam search (Medress et al.,
1977). With a beam size of $n$, beam search keeps the top-$n$ sequences during
decoding time according to a beam scoring function $f(\cdot)$. An illustration
is shown in Algorithm 1. The baseline method uses the log likelihood to define
the scoring function, i.e., $f(c_{g}|c_{p})=\log p(c_{g}|c_{p})$

Figure 3. Language model generation and language model inverse prompting
scoring for generating a poem sentence.

Result: Generated Context $c_{g}$

1 Given a language model $p$, a prompt $c_{p}$, the number of beams $n$, the
number of steps $s$, exploration steps for each beam $m$. Initialize current
step $k=0$. For each beam $j$, initialize the generated context for this beam
$c_{j}=^{\prime\prime}$.

2while  _k¡s_ do

3 For each $c_{j}$, generate $m$ next token $t_{j1}...t_{jm}$ sampled from
$p(\cdot|c_{p}+c_{j})$, update $c_{jl}=c_{j}+t_{jl}$

4 Choose the best $n$ texts $c_{b1},c_{b2}...c_{bn}$ with highest
$f(c_{jl}|c_{p})$.

5 For all $j$, update $c_{j}=c_{bj}$.

6 Update k=k+1.

7 end while

Output the best beam $c_{g}=c_{1}$.

Algorithm 1 Beam search. Inverse prompting follows the beam search framework
with a novel scoring function $f$ being the inverse log likelihood of the
prompt given the generated text.

###  3.2. Inverse Prompting

In this paper, we introduce a new method based on a new scorer $f$ in beam
search, called inverse prompting. Unlike previous controllable methods such as
CTRL or PPLM which needs additional attribute model training or manually-
defined codes, inverse prompting directly uses the original language model
itself to improve its generation.

One main issue that reduces the quality of the generated text is the run-away
phenomena shown in Table 2. The text gradually becomes irrelevant to the given
prompt as the sentences being generated. As the distance between the given
prompt and the generated sentence becomes larger, it hinders the generator to
keep a close connection with the prompt.

To alleviate this issue, our main idea is to design a novel beam search
scoring function that evaluates the log likelihood in an inverse direction;
e.g., if the prompt can be generated back from the text, they ought to be very
related with each other:

(1) $f(c_{g}|c_{p})=\log p(c_{p}|c_{g}).$

Texts are not always fluent if we read them from an inverse way. In question-
answering, the prompt may be ”Question:${Question} Answer:”. It is natural to
follow the answer after that, yielding ”Question:${Question}
Answer:${Answer}”. However, it is very unlikely that in natural language the
order will present in the inverse way ”${Answer} Question:${Question}
Answer:”. Simply using equation 1 only results in failure.

However, thanks to the nature of natural language, there do exist ways to
rearrange contexts to make them appear in an inverse order properly. Let’s
continue with the above instance: For ”Question:${Question} Answer:${Answer}”,
there do exist a way in natural language to inverse it: ”${Answer} answers the
question:${Question}”.

To achieve the core idea of Eqn. 1, we simply need to alter the format of the
prompts and texts:

(2) $f(c_{g}|c_{p})=\log p(c^{\prime}_{p}|c^{\prime}_{g}),$

where $c^{\prime}_{g}$ is inverse prompt under a new format, and
$c^{\prime}_{p}$ being the inverse text. Figure 4 displays some examples of
this transformation format. For $c_{p}=$”Question:${Question} Answer:” and
$c_{g}=$${Answer}, we only need to set $c^{\prime}_{p}=$”${Question}” and
$c^{\prime}_{g}=$”${Answer} answers the question:”, equation 2 shall work.
Inverse prompting ranks different beams by their likelihood to generate back
the original prompt in an inverse way, promoting the most relevant generated
texts. Inverse prompting can be used as long as the language supports an
inverse structure to rearrange the prompt and context in a proper way.
Detailed illustration for language model generation and language model inverse
prompting is presented in Figure 3.

The top rows in figure 4 are the formats of inverse prompting in the two
experiments of our paper—long-form question answering and poem generation,
while the bottom rows are additional examples of how inverse prompting can be
used in other text generation tasks.

Figure 4.  Inverse prompting transformation Table. The first rows represents
the inverse prompts used in experiments.(in Chinese and English) Some
additional examples of inverse prompting format are also displayed.

Inverse prompting is a simple method and easy to implement. The method
requires no additional models or data processing, as the inverse prompting
score can be simply computed by the same language model used for generation.
However, inverse prompting offers large improvements to the qualities of the
generated texts, which we will show in Sections 4 and 5.

##  4\. Implementation

We mainly use two long-term text generation tasks, Open-Domain Long-Term
Chinese Question-Answering, and Open-Domain Chinese Poem Generation, which
require the AI to generate long, in-depth contexts according to relatively
short prompts, to demonstrate the fantastic performance of inverse prompting.

We believe that as the relevance between generated texts and the given prompt
(questions/titles) improves, the generation quality will increase too. So we
conduct inverse prompting on questions/titles in our experiments, as shown in
the first four rows in Figure 4.

###  4.1. Base Language Model

We train our base Chinese language model using Megatron-LM(Shoeybi et al.,
2019) with Transformer-XL(Dai et al., 2018). The model has 2.86 billion
parameters. The training set contains 302GB of raw Chinese data abstracted
from multiple Chinese websites including Baidu, Zhihu and Sougou. We train the
base model using 64 GPUs for 160,000 steps. Details of training settings and
datasets are displayed in Appendix 7.1.

###  4.2. Open-Domain Long-Form Question-Answering

Long-Form Question-Answering, like the QAs on Quora, Zhihu or Sougou, is a
form of question-answering that questions and descriptions are short and
described in one or two sentences, while the answers have to be long,
informative and in-depth. The questioner expects a thorough answer answering
the question he asks in detail.

We apply inverse prompting in this way to generate Long-Form Answers given
Question prompts. We generate sub-sentences randomly according to language
model $\mathcal{LM}$, and do beam-search with inverse prompting in sub-
sentence level. To ensure the answer follows the question, we apply inverse
prompting (Equation 2 for each sub-sentence and sum up their scores. To keep
the generated context fluent, we combine the scores with normalized forward
perplexity,

(3) $f(c_{g}|c_{p})=\frac{1}{n}\sum_{s\in c_{g}}\lambda_{1}\log
p(c^{\prime}_{p}|s^{\prime})+\lambda_{2}\frac{\log
p(c_{g}|c_{p})}{n(c_{g})^{\lambda}}.$

###  4.3. Open-Domain Poem Generation

Traditional Chinese Poem generation is the pearl of domain-specific long-form
Chinese text generation tasks. Traditional Chinese poems have their specific
complex format and word usages different from modern Chinese language. Most of
the poems are written by ancient poets to express their feelings, describe
things they are doing, or ideas on different items. Generation of meaningful
text under the poem format given open-domain information is very hard for both
state-of-the-art AI models and humans.

In this paper, besides Open-Domain Long-Form QA, we challenge our inverse
prompting for a seemingly impossible task– To use the language model trained
on modern texts to generate Open-Domain Traditional Chinese Poems.

We basically keep the inverse prompting format of equation 3 while adding a
poem-format term to the beam-search (Equation 4), which penalizes contexts by
the degree they disobey with the poem format in rhythms or tones.

(4) $f(c_{g}|c_{p})=\frac{1}{n}\sum_{s\in c_{g}}\lambda_{1}\log
p(c^{\prime}_{p}|s^{\prime})+\lambda_{2}\frac{\log
p(c_{g}|c_{p})}{n(c_{g})^{\lambda}}-\lambda_{3}l_{format}(c_{g})$

###  4.4. Self Training for Poem Generation

Given that the model is trained on modern Chinese texts including very few
poem-format texts, it can hardly generate texts fully obeying the poem format
while maintaining a strong relevance to the given title.

Therefore, to improve its performance, we try the generate-and-fine-tune self
training protocol in AlphaGo-Zero(Silver et al., 2017) for this task.

We randomly select 1500 titles and let the model to produce poems based on
them. Then we fine-tune the model on these generated poems for 2000 steps.
This cycle can be repeated multiple times and in our experiments we repeat
this cycle for 2 times. We expect the fine-tuned model to be more likely to
generate sentences with better poem formats and other poem-specific properties
like aesthetics without losing their relevance to the given title.

##  5\. Experiments

In this section, we display the human-evaluation results of inverse prompting
on two long-form text generation tasks, open-domain long-form QA and open-
domain poem generation.

###  5.1. Human Evaluation Protocol

We first introduce how our human evaluation on the two tasks is conducted.
Table 1 illustrates the statistics for human evaluation experiments. For open-
domain long-form QA, we recruit 45 people, mostly university students, to
evaluate the quality of the generated answers. As for the evaluation of poem
generation, we invite 11 experts on traditional Chinese poems. Some of them
previously participated in the evaluation of Jiuge, the previous state-of-the-
art Chinese poem generator. The others are either similarly known to be
knowledgeable on poems or guaranteed to be qualified for the evaluation by
some poem contests.

Table 1. Human Evaluation Statistics. We filter out the answers from participants who does not finish all questions and pass the consistency check. Task | Participants | Prompts | Methods | Scores
---|---|---|---|---
|  | Compared | Collected
long-form QA | 30 | 100 | 4 | 12,000
Poem generation | 10 | 100 | 4 | 4,000

Each task contains 100 prompts and for each prompt, we provide 4 different
contexts for evaluators to evaluate. An evaluator needs to score each context
from multiple aspects and give an overall rating for each context on our
online evaluation platform within one week time.

To ensure participants making evaluations seriously, for each prompt we ask
the participants to select the best context. Then we will check if this answer
is consistent with the overall ratings for those 4 contexts additionally. If
the proportion of inconsistent answers reaches over 20%, we will treat this
participant as invalid. Finally, we only collect the answers submitted by
valid participants.

As listed in Table 1, 32 evaluators in long-form QA evaluation and 10 experts
for traditional Chinese poems finished his/her evaluation. 30 of the finished
evaluators in long-form QA experiment are valid, while all the 10 finished
experts in our traditional Chinese poem experiment are valid.

###  5.2. Open-domain long-form Chinese QA

For open-domain long-form Chinese QA evaluation, we randomly select 100
questions from various domains in Zhihu, a Quora-like Chinese open-domain
long-form QA platform. In Zhihu, users can ask questions with some background
descriptions, or provide informative long-form answers to the raised
questions. Besides, users can ”Upvote” or ”Downvote” answers based on their
opinions.

In this experiment, we only select questions that are excluded in the training
set of our base model. For each question, we display one highly-upvoted human
answer and three AI-generated answers produced by CPM (Zhang et al., 2020),
prompting baseline, and inverse prompting respectively.

We shuffle the order of all answers and ask human evaluators to score the
answers through four aspects including:

  1. (1)

Fluency Whether the answer is well-formed and logical to read. Rated from 1 to
5.

  2. (2)

Informativeness Whether the answer contains useful information for the given
question. Rated from 1 to 5.

  3. (3)

Relevance Whether the answer is relevant to the given question. Rated from 1
to 5.

  4. (4)

Overall The overall quality of the answer. Rated from 1 to 10.

Table 2. Performance for open-domain long-form Chinese QA under Human
Evaluation.

Method | Fluency |  Inform.11footnotemark: 1 | Relevance | Overall
---|---|---|---|---
(1-5) | (1-5) | (1-5) | (1-10)
CPM (Zhang et al., 2020) | 2.66 | 2.47 | 2.36 | 4.32
Prompting Baseline | 3.44 | 3.25 | 3.21 | 5.97
Inverse Prompting | 3.61 | 3.43 | 3.59 | 6.51
Human Answers | 3.80 | 3.61 | 3.67 | 6.85

  * 1

Informativeness

Table 2 shows that inverse prompting outperforms both the prompting baseline
and the previous SOTA Chinese language model CPM by a large margin in all
individual aspects, as well as the overall quality of the generated answers.

Despite inverse prompting only forces the answer to be more related to the
question in this experiment, an interesting finding is that by producing more
relevant answers, inverse prompting also makes improvements on the fluency and
informativeness of the generated answers, raising the overall quality as well.
This supports our hypothesis in section 4.

###  5.3. Open-domain Poem Generation

The second experiment is to evaluate the task of open-domain poem generation.
This task is similar to the long-form QA experiment described above. We
randomly design 100 poem titles including various open domains for evaluation.
These poem titles never appear in any real poems in the training set or being
used as randomized titles in our reinforcement learning process.

For each title, we apply four different methods to generate pomes, including
Jiuge (the SOTA model for open-domain Chinese poem generation), the beam
search baseline with poem format loss $l_{format}$, inverse prompting with
poem format loss (Equation 4) and inverse prompting with the self-training
mentioned in section 4.4. These four poems are shuffled for evaluation. For
each generated poem, we request evaluators for 5 ratings:

  1. (1)

Format Whether the generated poem follows the rule of rhythm in traditional
Chinese poetry. Rated from 1 to 5.

  2. (2)

Innovation Whether the sentences are copied from existing poems or created
with innovative expressions. Rated from 1 to 5.

  3. (3)

Relevance Whether the content of the poem is related to the given title. Rated
from 1 to 5.

  4. (4)

Aesthetics Whether the poem has obscure meanings apart from its obvious
meanings, making it aesthetically better? Rated from 1 to 5.

  5. (5)

Overall The overall quality of the poem. Rated from 1 to 10.

Table 3. Performance for open-domain Traditional Chinese Poem Generation under
human expert evaluation.

Method | Format |  Innov.11footnotemark: 1 | Relevance |  Aes.22footnotemark: 2 | Overall
---|---|---|---|---|---
(1-5) | (1-5) | (1-5) | (1-5) | (1-10)
Jiuge (Zhipeng et al., 2019) | 3.60 | 2.47 | 1.99 | 3.12 | 3.57
Search Baseline | 2.79 | 1.10 | 1.16 | 2.44 | 1.35
Inverse Prompting | 2.56 | 2.71 | 2.92 | 2.33 | 4.00
Inverse Prompting +ST | 2.42 | 2.92 | 3.65 | 2.18 | 4.40

  * 1

Innovation

  * 2

Aesthetics

Table 3 illustrates the experimental results. The average scores for all
methods are low as all of the experts are extremely critical. They only give
high scores to very few perfect poems. One of the experts says she’ll give
less than 5 ”overall” score to an average TC-Poem written by ancient
celebrities, while scoring results indicate that other experts are even more
critical than her.

The prompting baseline can hardly generate appropriate poems. Even with the
poem format loss, it only outputs unrelated sentences copied from existing
famous poems that appear in modern Chinese languages.

However, with the help of inverse prompting, the overall quality of generated
poems surpasses Jiuge. Moreover, the self-training can further improve the
performance on top of inverse prompting.

Generally, Jiuge is good at generating sentences with beautiful words and
gorgeous rhythm, since it is designed to focus strictly on poem formats.
Nevertheless, according to human evaluation results, despite it sometimes does
generate relevant poems, most of its generation results are not quite related
to the given title and comparably weak at innovation.

inverse prompting offers innovative and relevant expressions in the form of
traditional Chinese poems. However, as the model is trained on modern Chinese
texts, it is inevitably not so good in following traditional poem formats like
rhythms or tones. It also doesn’t handle aesthetics well, as this is common in
ancient Chinese poems, but rarely appears in modern Chinese.

Despite these disadvantages, the experts still agree to give poems generated
by inverse prompting a much higher average overall score than Jiuge due to
their high relevance to titles and innovative expressions.

In section 4.4, we expect the self-training can bring better format and
aesthetics to the generated poems. However, to our surprise, the self-training
further enhances the innovation and relevance by a large margin at the cost of
a minor decrease in format and aesthetics, suggesting that what the model
really learns under our reinforcement learning scheme is to be more relevant.
By generating more relevant poems to the title with more innovative
expressions, its average overall score becomes much higher additionally.
Eventually, inverse prompting with self-training gets 4.40 average overall
score, compared with Jiuge’s 3.57.

One possible explanation for this phenomenon is that in order to be more
relevant to open-domain titles which may never appear in the domain of
Traditional Chinese Poems, the model has to be more innovative in language
arrangement and less focused on formats or aesthetics.

In Appendix 7.3, we discuss our deviation analysis and calculate p-values for
different methods on the above two tasks.

###  5.4. Poem Turing Test

Apart from human evaluation for open-domain titles, we also test the
performance of it on domain-specific titles.

Back to the result of long-form QA in Table 2, answers generated by inverse
prompting are only slightly inferior to human answers. Their average score is
6.51 compared with human answers’ 6.85. This enlightens our mind that the
generated poems may be comparable in quality to human-made poems.

Inspired by turing test (Turing, 2009), we similarly designed a traditional
Chinese poem turing test to further evaluate the generated poems quality of
inverse prompting.

In the turing test, also known as the imitation game, a human interrogator is
requested to distinguish between generated poems and human poems. We implement
an online game platformwhere any player can participate without limitation. In
the game, each player is given several pairs of poems with each pair contains
one poem written by a human poet and the other one generated by AI under the
same title. The human poems are randomly selected from Quan Tang Shi, the most
famous collection of traditional Chinese poems. The collection was published
in 1705 and consists of high-quality traditional Chinese poems mostly written
in Tang Dynasty (618-907). In our designed game, the player needs to figure
out which poem is written by the human poet. We generate 1,500 pairs of poems
and randomly displays 5 pairs for each game.

As displayed in Table 4, 4,592 game records are gathered from 755 different
users. Each game record involves a binary selection between a human poem and
an AI poem. 46.4% of the user records choose AI poems while the rest 53.6%
chooses human poems, suggesting that the quality of poems generated by inverse
prompting +self-training on domain-specific titles may be close to human level
for average online users.

Table 4. Poem turing test on 755 identical online users. Human poems are randomly selected from Quan Tang Shi. AI Poems are generated given the same title. Method | Total | Selected | Selection Rate
---|---|---|---
Inverse Prompting +ST | 4,592 | 2,132 | 46.4%
Ancient Human Poems | 4,592 | 2,460 | 53.6%

###  5.5. Case Study

In this section, we discuss explicitly the pros and cons of inverse prompting
by performing series of case studies on the two tasks – Open-domain long-form
QA, and Open-domain Poem Generation.

Figure 5. A Perfect Example of inverse prompting generating better answer than
human in open-domain long-form QA.

Figure 5 exhibits a comparison between two answers for a question on how to
deal with stress at work during pregnancy. We list the answer generated by
inverse prompting and the human answer. In this case, the evaluators even give
higher scores to the inverse prompting generated answer than the human answer.
Both answers provide comprehensive and practical aids related to the question.
The human answer is more emotional and gives advice based on the answerer’s
own experience. The generated answer, on the other hand, raises the point that
pregnant mothers should insist on working and overcome the difficulties
straightforwardly. Then it uses several sentences to enhance this point, which
turns out to be more informative, reasonable and persuasive.

Figure 6. A bad case for inverse prompting generated texts. It can’t overcome
the barrier of maths.

While the proposed method seems to be able to understand questions, integrate
learned knowledge and generate logical answers, we found that numbers in the
task are comparatively difficult, which often lead to some chaotic outputs. In
Figure 6 we show a bad case generated by inverse prompting that only receives
a 4.10 score in overall quality. While the question is asking about Chapter
882 in the One Piece manga, the model is clearly unable to understand the
differences between Chapter 885 and the asked 882. Besides, the answer itself
is self-contradictory. It is worth noticing that such a chaotic problem in
maths universally exists for language models. Previous study (Saxton et al.,
2019) also shows that it is extremely hard for neural models to do
mathematical reasoning.

Figure 7. A representative case in open-domain long-form QA that the quality
of answers in this problem reflects the overall performance of different
methods.

In Figure 7, we display all 4 answers for the question “How to wash purple
sweet potato on clothes” with the average overall scores. The best answer is
written by a human, which comprehensively introduces the solution to the
problem step by step. The answer generated by inverse prompting offers a
similar solution but with fewer details. The prompting baseline does not give
a precise answer to the original question. Instead, it tries to answer another
question “How to wash out the liquids on clothes such as milk or beer?”. This
tells us why we need to use inverse prompting to force the generated answer to
be closely related to the original question. Finally, CPM can neither produce
fluent answers nor provide useful information. This example illustrates how
the difference in overall ratings for different methods in Table 2 come from
in a representative way.

Figure 8. A representative case in poem generation that reflects the overall qualities of different methods.  Figure 9. A bad case that inverse prompting is not successful in understanding the given title Iroquois even under after self training.  Table 5. A GPT-2 345M paragraph generation case.  Title |  Arctic Cat
---|---
GPT-2 |  VIRGO (Aug. 23-Sept. 22): A business partner invites a group of friends to a casino but no money is exchanged, causing a misunderstanding. A lawyer accidentally chases away a fish when someone else asks for the name of a housemate. LIBRA (Sept. 23-Oct. 22): A business partner mentions missing money on a Web site, triggering a conversation about receiving the money in time. Expect to be caught by someone who knows the truth. SCORPIO (Oct. 23-Nov. 21): When others make big bets, you have to say no. If you don’t like it, they can make it worse. But don’t do it. SAGITTARIUS (Nov. 22-Dec. 21): Take a stand and speak your mind. Don’t let others tell you you can’t do it. You can. CAPRICORN (Dec. 22-Jan. 19): Speaking too much may seem harsh, but it’s important. For example, if someone tries to take over your career, you must not cave in. AQUARIUS (Jan. 20-Feb. 18): You are correct to think that everyone agrees with you when you talk about this topic. However, this person may not be a professional so don’t break bread with them. PISCES (Feb. 19-March 20): When someone tells you they’re really sorry for throwing away something, it is OK to say yes. Keep it short and simple. IF APRIL GOES AWAY Aries (April 20-May 20): You are able to talk easily with someone who is far away from you because of a move of public events. Taurus (May 21-June 20): If you trust someone, you might get involved with them by going out with them. GEMINI (June 21-July 22): Don’t act your age when you are with someone and they get angry. You are mature enough for such a thing. CANCER (July 23-Aug. 22): You can take on anything in life that you like, but others don’t like it. That is OK. Leo (Aug. 23-Sept. 22): They see you as someone they can manipulate and can be loud about. Don’t like it.
Inverse Prompting |  Around the globe, it’s still hard to find a cold-weather car. However’s will now serve as the closest thing to snow on the road yet. The new Arctic Cat Aruba features two levels of navigation that allow drivers to get a glimpse of location and what’s happening on the road. Additional intercom-compatible pieces will be available to start the journey when it makes its return to the United States. Further out Arctic Cat Aruba For the US.
Table 6. Another GPT-2 345M paragraph generation case.  Title |  Boris Johnson in Delhi
---|---
GPT-2 |  And far from showing relief that the switch-back from the hard-nosed - and potentially racist - EU-UK relations to a softer, conciliatory approach, Tony Blair yesterday morning raised expectations of another barrage of retaliatory tariffs against the UK’s soaring exports to America, and said he was in no mood to settle for the kind of bilateral exchange once favoured by George W Bush and Tony Blair. Downing Street has been irritated that the free-trade deal with America was unexpectedly allowed to lapse just when the G7 summit was over - something it had tried hard to avoid. But the prime minister, in a direct intervention before the G7 summit in Northern Ireland, suggested the long-awaited deal would take time to complete. He told Sir Michael Dawe: ”We want to get to the point where we are able to settle trade in terms of good terms with the United States on a relationship that is not just good for the United States, but is good for the UK as well. ”It is an old parochial argument that trading relations are too important to be put at the mercy of political relations. That’s a fundamentally flawed argument.” Mr Blair has faced particular criticism from the Tory backbenchers over the continued access of the US and British airlines to market dominance through the ”free movement” of goods, which has now been abandoned and which Mr Blair publicly admitted as a feature of the global economy. That prompted the shadow trade secretary, Chris Grayling, to accuse the prime minister of trying to impose unilateral change on the economic process. In a letter to Mr Grayling, he said: ”It seems that for the moment, Mr Blair is pushing for an extra ten days in the EU deal for our exports to America without consulting us. The UK has been up against that earlier history in a number of ways - with the currency exchange rate as well as the fact that exports and imports have been flowing at cross purposes. ”But there are some substantive things that Mr Blair might be looking at that we need to have on our side, and that would be an effort to get these and other issues sorted out before the G7 summit. It would probably be a necessity before the G7 summit, before this financial rescue package.” The Labour leader, Harriet Harman, added: ”The proposed U.S.-UK free trade agreement that the prime minister has opposed is going to fail if we don’t deal properly with it.
Inverse Prompting |  May 24, 2009 - Boris Johnson arrives at the National Portrait Gallery to accept a prize for his portrait of Italian Chancellor Romano Prodi July 28, 2011 - Boris Johnson makes his first visit to south Asia, in the Indian capital, Delhi June 4, 2012 - Boris Johnson attends a US delegation in New Delhi June 20, 2013 - Boris Johnson visits the Taj Mahal monument in Agra, the Raj, Bihar holy city of Lalbagh November 21, 2014 - Boris Johnson in his office, New Delhi January 23, 2014 - The foreign with which Boris Johnson has most bonded during the ten-month tour of India and China was Prince Charles. He met the billionaire at Buckingham Palace before heading off to Delhi.

Figure 8 shows poems generated by different methods under title Zebra . Zebra
is an open-domain concept that never appears in any traditional Chinese poems
(as there’s no zebra in ancient China). However, there exist lots of
traditional Chinese poems for different types of animals. We would like to see
how different methods generalize the traditional Chinese poem format for
zebras.

Note that the direct meaning for “zebra” in Chinese is ”spotted horses”, so
models are likely to misuse the concept of ”horse”. The prompting baseline
copies sentences from famous existing poems for horses and gets only 2.20 for
being an awkward copycat. Jiuge treats zebras as horses and applies a lot of
analogy to glorify the ”horses”, with good representation and perfect format
it gets a 5.10 overall score. Inverse prompting offers a description between
horses and zebras and gets a 4.80 overall score, while inverse prompting with
self-training states the hoof, the fur and the behaviors of zebras in a subtle
way, differing zebras from horses, this poem is scored the highest (5.70) by
expert evaluators.

This is a representative instance for poem generation. The other 99 poems are
also evaluated in such a way that the title is in a category that exists a lot
in traditional poems. However, the precise concept of the title is new. For
example, climbing different mountains or towers is popular in traditional
Chinese poems, and we design open-domain titles like “Climbing Eiffel Tower”
and “Climbing Mt. Everest” which never appear in them. The prompting baseline
often copies from existing poems. Jiuge usually gives an poem with good format
between related and unrelated. inverse prompting seems to understand the title
while inverse prompting with self-training understands the title better.

However, inverse prompting does not guarantee understanding of the given
title. Figure 9 illustrates a bad case under title Sigh for the Iroquois .
Inverse prompting +self-training fails to understand the meanings of the
native American tribe and mistreat it as some forms of monsters living in
deserts and killing people. This may due to the low occurrence of the Iroquois
in modern Chinese texts that the base language model itself cannot fully
understand its meanings, and the self-training process is unable to help the
model understand more about such concepts as well.

###  5.6. Inverse Prompting for English(GPT-2)

We also practice inverse prompting under an open-sourced toy English language
model: GPT-2 345M111https://github.com/NVIDIA/Megatron-LM(Shoeybi et al.,
2019).

As can be concluded from Table 5,6, inverse prompting greatly improves the
relativeness of GPT-2 345M for generated English contexts. Under title ”Arctic
Cat”, inverse prompting refers to a cold-weather car brand, while direct
generation generates totally unrelated contexts. Under title ”Boris Johnson in
Delhi”,GPT-2 is successful in referring the title to UK, while completely
ignoring ”Delhi”. However, using inverse prompting, the generated context
suddenly becomes very relative.

Although the quality of the base model limits the performance, in these cases
inverse prompting still achieves obvious improvements.

##  6\. Conclusion

In this paper, we present a new method, inverse prompting for text generation.
inverse prompting offers a new option for controllable generation using
language models by exploiting the inverse form of natural languages.

We experiment the text generation using inverse prompting, which is to assign
inverse prompts that forces the generated texts to be related to the prompt.
Extensive experiments of human evaluations demonstrate that as the relevance
increases, the overall quality of long-form Chinese texts also improves
significantly. On long-form open-domain QA, inverse prompting improves AI-
generated texts one step closer towards human level. More promising results
occur in open-domain Traditional Chinese Poem Generation, as with inverse
prompting, language models trained on modern Chinese context can generate
poems that surpass the previous SOTA despite the disadvantages in format and
aesthetics. Furthermore, with self-training, the inverse prompting poem
generator can do even better in human evaluations.

Future works may include improving the poem generation format, applying
inverse prompting to other languages like English, or enhancing the self
training-process.

Acknowledgements

We thank Zhengxiao Du and Jifan Yu for helpful discussions.

Ming Ding contributed for the speed acceleration after submission to KDD. So
we decide to list him in the author list of the arxiv version.

##  7\. Appendix

###  7.1. Implementation Details

Training Details for Base Model.  For training of base model, we use a
training set of 302GB, the distribution of these data is shown in Table 7. The
evaluation set contains 400MB Open-Domain QA contexts that is not used during
training. We select the 100 questions in human evaluation from this evaluation
set.

Table 7. Dataset Distribution. Source | Format | Size
---|---|---
Baidu & Sougou Baike | Online Encyclopedia | 133GB
Zhihu | Open-domain QA | 131GB
Baidu QA | Open-domain QA | 33GB
Generated TC-Poems | Traditional Chinese Poetry | 1.6MB
Evaluation | Open-domain QA | 440MB

As mentioned in section 4, we use GPT framework with its transformer model
substituted to Transformer-XL. For optimization, we use the AdamW optimizer
with $\beta_{1}=0.9,\beta_{2}=0.95,\epsilon=1e-6$ and a $0.1$ L2-weight decay.
The learning rate is warmed up linearly over the first 3,000 steps to a peak
value of $1e-4$, then is tuned with cosine decay to $10\%$ of its peak value.
The total training steps is 160,000. The training process uses 8 servers with
8 Nvidia V100 GPUs on each server. Each server has 96 Intel CPU cores and
376GB Memory. Serves are connected by 100G RoCEv2 network.

For reinforcement learning, on each cycle we first generate a few poems for
each of the 1500 prompts, resulting in around 800KB of generated poem data.
Our fine-tuning inherits the previous conditions of the optimizer from the
previous model and train on generated poem data for 2,000 steps. We repeat
this process twice, so the final size of train poems generated is 1.6MB. The
fine-tuning uses one server with 8 Nvidia V100 GPUs.

Table 8. Training Dataset Distribution. Task | beam | generations | max short
---|---|---|---
size | per beam | sentences
Long-form QA | 5 | 5 | 30
Poem (train/turing) | 10 | 7 | 8
Poem (eval) | 10 | 12 | 8

Parameters for Beam Search.  Table 8 displays the beam search parameters we
use. For long-form QA, we use a beam size of 5, and for each beam we generate
5 samples for the next short sentence, and we limit the length of the answer
to 30 short sentences. For Poem Generation, we use a beam size of 10, for each
beam we generate 7 samples for the next short sentence in reinforcement
learning and the Turing Test, and 12 samples for open-domain title human
evaluation. We limit the length of the generated poems to 8 short sentences.

For $\lambda,\lambda_{1},\lambda_{2},\lambda_{3},l_{format}$ mentioned in
Section 4, we take $\lambda=\lambda_{1}=\lambda_{3}=1$, $\lambda_{2}=0.75$ for
poem generation and use $\lambda_{2}=1.5$ for open-domain QA.

For Poem generation, we design $l_{format}$ using various format information
like number of characters in each short sentence, repetitive, rhythm, and
tones. Despite it doesn’t work when applied to prompting baseline directly, it
cooperates well with inverse prompting.

###  7.2. Human Evaluation Details

Table 9. Performance and Deviation for open-domain long-form Chinese QA under Human Evaluation.  Method | Fluency | Informativeness | Relevance | Overall
---|---|---|---|---
(1-5) | (1-5) | (1-5) | (1-10)
CPM | 2.66$\pm$0.19 | 2.47$\pm$0.19 | 2.36$\pm$0.20 | 4.32$\pm$0.37
Prompting Baseline | 3.44$\pm$0.19 | 3.25$\pm$0.20 | 3.21$\pm$0.22 | 5.97$\pm$0.42
Inverse Prompting | 3.61$\pm$0.17 | 3.43$\pm$0.19 | 3.59$\pm$0.20 | 6.51$\pm$0.38
Human Answers | 3.80$\pm$0.18 | 3.61$\pm$0.19 | 3.67$\pm$0.21 | 6.85$\pm$0.39
Table 10. Performance and Deviation for open-domain Traditional Chinese Poem Generation under human expert evaluation.  Method | Format | Innovation | Relevance | Aesthetics | Overall
---|---|---|---|---|---
(1-5) | (1-5) | (1-5) | (1-5) | (1-10)
Jiuge | 3.60$\pm$0.25 | 2.47$\pm$0.28 | 1.99$\pm$0.31 | 3.12$\pm$0.31 | 3.57$\pm$0.54
Search Baseline | 2.79$\pm$0.37 | 1.10$\pm$0.13 | 1.16$\pm$0.16 | 2.44$\pm$0.38 | 1.35$\pm$0.27
Inverse Prompting | 2.56$\pm$0.28 | 2.71$\pm$0.28 | 2.92$\pm$0.37 | 2.33$\pm$0.28 | 4.00$\pm$0.52
Inverse Prompting +ST | 2.42$\pm$0.29 | 2.92$\pm$0.28 | 3.65$\pm$0.33 | 2.18$\pm$0.28 | 4.40$\pm$0.47

Our human evaluation is conducted on a platform. Figure 10 illustrates how the
evaluation platform looks like. The whole task of evaluating 100 prompts is
divided into 10 sub-tasks, and in each sub-task, the evaluator is required to
score 4 contexts for 10 prompts in multiple aspects, like an online
questionnaire.

Figure 10. An illustration of our human evaluation platform. The whole task of
evaluating 100 prompts is divided into 10 sub-tasks, and in each sub-task, the
evaluator is required to score 4 contexts for 10 prompts in multiple aspects.

The evaluation does not necessarily need to be finished at once. People can
login and logout, change their answers for already completed problems, or
continue evaluation from their current positions freely in one week’s time.
They only need to ensure that all evaluation questions have been answered
before the deadline, with the ratings being consistent.

Valid evaluators for open-domain QA are paid 150 RMB yuan each (about $\$25$),
while each TCP evaluator receives 300 RMB yuan (about $\$50$), as evaluation
for traditional poems requires more expert reviewers. The payment is not high
but due to the flexible time arrangement for online and interesting content,
the task still attracted a lot of participants.

We recruit 11 experts for TCP evaluation, 10 of them finished and all of those
finished provide valid evaluations, we recruit 45 people for open-domain QA,
32 of them finish their experiments and 30 of them provide consistent
evaluations.

For generating baseline texts, for QA, we generate the prompting baseline
using the base text generation code under the prompt format of
”Question:$\$\textbf{Question}$ Description:$\$\textbf{Description}$ Answer:”,
for CPM we apply the same prompt format and use its recommended text
generation code.

For poem generation using Jiuge, we write code that can automatically make
online queries to its public generation website http://jiuge.thunlp.org/ and
get generated poems. Jiuge has a lot of format modes and we choose four most
general modes without additional restrictions
”5-Jueju”,”7-Jueju”,”5-Lvshi”,”7-Lvshi”. For each title Jiuge generates one
best poem for each mode. However, it offers no hint about which poem it
considers the best so we randomly choose one from the 4 generated for human
evaluation.

###  7.3. Deviation for Human Evaluators and p-values

Table 9,10 displays the deviation of the scorings for human evaluators.

The deviation is calculated in a per-capita basis that we first average the
scorings for each method on for every evaluator, then we compute the deviation
based on the average scores of each human evaluators.

As can be seen, evaluators agree more on the quality for Chinese QA, while
less agree on the qualities for poems.

With those standard deviations, assuming evaluators are independent with each
other, we can calculate p-score. For poems we have $N=10$, the p-value for
Jiuge $\geq$ Inverse Prompting is 0.0544 while the p-value for Jiuge $\geq$
Inverse Prompting+self-training is 0.0009, suggesting that under $p<.05$ we
cannot fully reject the hypothesis that Jiuge is not worse to Inverse
Prompting. However, Inverse Prompting with self-training is statistically
better than Jiuge.

For QA, with $N=30$ the p-value for Prompting Baseline $\geq$ Inverse
Prompting is $<.00001$, while the p-value for Inverse Prompting $\geq$ Human
is 0.0006. So inverse prompting is statistically better than the prompting
baseline but is still worse than human.

###  7.4. Online Demo Platforms

We further developed the poem generation and add some other formats of poems,
including heading, which pre-defines the first word of each short sentence
before poem generation, and SongCi, which is another form of traditional
Chinese context that involves much higher format standard. All of these
downstream tasks are based on the inverse prompting+self training protocol ,
with tiny format adjustments for each downstream task.

We display these applications on our demo Wudao Poetry
222https://pretrain.aminer.cn/apps/poetry.html. Users can also submit their
customized titles and generate poems of their own. There is also a QA demo
named Wudao QA 333https://pretrain.aminer.cn/os/qa where users can submit
their own questions and descriptions to get an AI answer.

Figure 11. Selected examples of inverse prompting poetry, poems with heading,
and SongCi.

Figure 11 displays some of the generated poems for these downstream tasks on
the platform. More cases can be found on the platform, or generated according
to users’ submissions.

## References

  * (1)
  * An and Cho (2015) Jinwon An and Sungzoon Cho. 2015.  Variational autoencoder based anomaly detection using reconstruction probability.  _Special Lecture on IE_ 2, 1 (2015), 1–18.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ (2020).
  * Chu et al. (2017) Casey Chu, Andrey Zhmoginov, and Mark Sandler. 2017\.  Cyclegan, a master of steganography.  _arXiv preprint arXiv:1712.02950_ (2017).
  * Dai et al. (2018) Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2018.  Transformer-xl: Language modeling with longer-term dependency.  (2018).
  * Dathathri et al. (2019) Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019\.  Plug and play language models: A simple approach to controlled text generation.  _arXiv preprint arXiv:1912.02164_ (2019).
  * de Vries and Nissim (2020) Wietse de Vries and Malvina Nissim. 2020.  As good as new. How to successfully recycle English GPT-2 to make models for other languages.  _arXiv preprint arXiv:2012.05628_ (2020).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Jiang and Zhou (2008) Long Jiang and Ming Zhou. 2008.  Generating Chinese couplets using a statistical MT approach. In _Coling’08_. 377–384.
  * Keskar et al. (2019) Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019\.  Ctrl: A conditional transformer language model for controllable generation.  _arXiv preprint arXiv:1909.05858_ (2019).
  * Liu (1966) James JY Liu. 1966\.  _The art of Chinese poetry_.  University of Chicago Press.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Medress et al. (1977) Mark F. Medress, Franklin S Cooper, Jim W. Forgie, CC Green, Dennis H. Klatt, Michael H. O’Malley, Edward P Neuburg, Allen Newell, DR Reddy, B Ritea, et al. 1977\.  Speech understanding systems: Report of a steering committee.  _Artificial Intelligence_ 9, 3 (1977), 307–316.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Distributed representations of words and phrases and their compositionality.  _arXiv preprint arXiv:1310.4546_ (2013).
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP’14)_. 1532–1543.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.  (2018).
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ 1, 8 (2019), 9.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  _arXiv preprint arXiv:1606.05250_ (2016).
  * Saxton et al. (2019) D. Saxton, Edward Grefenstette, Felix Hill, and P. Kohli. 2019.  Analysing Mathematical Reasoning Abilities of Neural Models.  _ArXiv_ abs/1904.01557 (2019).
  * Shoeybi et al. (2019) Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019\.  Megatron-lm: Training multi-billion parameter language models using model parallelism.  _arXiv preprint arXiv:1909.08053_ (2019).
  * Silver et al. (2017) David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. 2017\.  Mastering the game of go without human knowledge.  _nature_ 550, 7676 (2017), 354–359.
  * Simmons (1970) Robert F Simmons. 1970\.  Natural language question-answering systems: 1969.  _Commun. ACM_ 13, 1 (1970), 15–30.
  * Turing (2009) Alan M Turing. 2009\.  Computing machinery and intelligence.  In _Parsing the turing test_. Springer, 23–65.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is all you need.  _arXiv preprint arXiv:1706.03762_ (2017).
  * Xia et al. (2016) Yingce Xia, Di He, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016.  Dual learning for machine translation.  _arXiv preprint arXiv:1611.00179_ (2016).
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018\.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  _arXiv preprint arXiv:1809.09600_ (2018).
  * Yi et al. (2020) Xiaoyuan Yi, Ruoyu Li, Cheng Yang, Wenhao Li, and Maosong Sun. 2020.  MixPoet: Diverse poetry generation via learning controllable mixed latent space. In _AAAI’20_ , Vol. 34. 9450–9457.
  * Zhang et al. (2020) Zhengyan Zhang, Xu Han, Hao Zhou, Pei Ke, Yuxian Gu, Deming Ye, Yujia Qin, Yusheng Su, Haozhe Ji, Jian Guan, et al. 2020\.  CPM: A Large-scale Generative Chinese Pre-trained Language Model.  _arXiv preprint arXiv:2012.00413_ (2020).
  * Zhipeng et al. (2019) Guo Zhipeng, Xiaoyuan Yi, Maosong Sun, Wenhao Li, Cheng Yang, Jiannan Liang, Huimin Chen, Yuhui Zhang, and Ruoyu Li. 2019.  Jiuge: A human-machine collaborative chinese classical poetry generation system. In _ACL’19_. 25–30.