酷应用

如何快速跟进NLP领域最新技术？(文献阅读清单)

百家作者：新智元 2019-05-24 10:29:49

新智元报道

来源：medium

作者：huggingface 编辑：肖琴

【新智元导读】NLP领域发展迅速，初入坑者阅读哪些论文才能快速跟上现代NLP的最新趋势？HuggingFace团队近日发布这份论文列表和资源清单，紧跟研究最前沿，必备收藏。

在过去的两年中，NLP在各种不同任务和应用上的进展十分迅速。这些进展是由于构建NLP系统的经典范式发生了转变带来的：很长一段时间以来，研究人员都使用预训练的词嵌入(如word2vec或GloVe)来初始化神经网络，然后使用一个特定于任务的架构，该架构使用单个数据集以监督方法训练。

最近，一些研究证明，我们可以利用非监督(或自监督)信号，如语言建模，在web规模的数据集上学习分层上下文表示，并将这种预训练转移到下游任务(迁移学习)。令人兴奋的是，这种转变带来了下游应用领域的重大进展，从问题回答到通过句法分析进行自然语言推理……

“我可以读哪些论文来跟上现代NLP的最新趋势?”

几周前，我的一位朋友决定入坑NLP。他已经有机器学习和深度学习的背景，所以他真诚地问我：“我可以阅读哪些论文来跟上现代NLP的最新趋势?”

这是一个非常好的问题，尤其是考虑到NLP会议(以及普遍的ML会议)收到的论文投稿呈指数级增长时：NAACL 2019收到投稿比2018增加了80%， ACL 2019收到的投稿比2018年增加了90%……

因此，我为他整理了这份论文列表和资源清单，并与大家分享。

免责声明：本列表并非详尽无遗，也无法涵盖NLP中的所有主题(例如，没有涵盖语义解析、对抗性学习、NLP强化学习等)。所选择的论文主要是过去几年/几个月最具影响力的工作。

一般而言，开始进入一个新领域的好方法是阅读介绍性或总结性的博客（比如这篇），可以让你在花时间阅读论文之前快速了解背景。

一种新的范式：迁移学习

以下的参考文献涵盖了NLP迁移学习的基本思想：

Deep contextualized word representations (NAACL 2018)

Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer

Universal Language Model Fine-tuning for Text Classification (ACL 2018)

Jeremy Howard, Sebastian Ruder

Improving Language Understanding by Generative Pre-Training

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever

Language Models are Unsupervised Multitask Learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (NAACL 2019)
Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova

Cloze-driven Pretraining of Self-attention Networks (arXiv 2019)
Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, Michael Auli

Unified Language Model Pre-training for Natural Language Understanding and Generation (arXiv 2019)
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon

MASS: Masked Sequence to Sequence Pre-training for Language Generation (ICML 2019)
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, Tie-Yan Liu

Transformer结构已经成为序列建模任务流行结构。Source: Attention is all you need

表示学习(Representation Learning)

What you can cram into a single vector: Probing sentence embeddings for linguistic properties (ACL 2018)
Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, Marco Baroni

No Training Required: Exploring Random Encoders for Sentence Classification(ICLR 2019)
John Wieting, Douwe Kiela

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (ICLR 2019)
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel R. Bowman

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (arXiv 2019)
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel R. Bowman

Linguistic Knowledge and Transferability of Contextual Representations (NAACL 2019)
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, Noah A. Smith

To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks(arXiv 2019)
Matthew Peters, Sebastian Ruder, Noah A. Smith

神经对话(Neural Dialogue)

A Neural Conversational Model (ICML Deep Learning Workshop 2015)
Oriol Vinyals, Quoc Le

A Persona-Based Neural Conversation Model (ACL 2016)
Jiwei Li, Michel Galley, Chris Brockett, Georgios P. Spithourakis, Jianfeng Gao, Bill Dolan

A Simple, Fast Diverse Decoding Algorithm for Neural Generation (arXiv 2017)
Jiwei Li, Will Monroe, Dan Jurafsky

Neural Approaches to Conversational AI (arXiv 2018)
Jianfeng Gao, Michel Galley, Lihong Li

TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents (NeurIPS 2018 CAI Workshop)
Thomas Wolf, Victor Sanh, Julien Chaumond, Clement Delangue

Wizard of Wikipedia: Knowledge-Powered Conversational agents (ICLR 2019)
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, Jason Weston

Learning to Speak and Act in a Fantasy Text Adventure Game (arXiv 2019)
Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, Jason Weston

其他

Pointer Networks (NIPS 2015)
Oriol Vinyals, Meire Fortunato, Navdeep Jaitly

End-To-End Memory Networks (NIPS 2015)
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, Rob Fergus

Get To The Point: Summarization with Pointer-Generator Networks (ACL 2017)
Abigail See, Peter J. Liu, Christopher D. Manning

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (EMNLP 2017)
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, Antoine Bordes

End-to-end Neural Coreference Resolution (EMNLP 2017)
Kenton Lee, Luheng He, Mike Lewis, Luke Zettlemoyer

StarSpace: Embed All The Things! (AAAI 2018)
Ledell Wu, Adam Fisch, Sumit Chopra, Keith Adams, Antoine Bordes, Jason Weston

The Natural Language Decathlon: Multitask Learning as Question Answering(arXiv 2018)
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, Richard Socher

Character-Level Language Modeling with Deeper Self-Attention (arXiv 2018)
Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, Llion Jones

Linguistically-Informed Self-Attention for Semantic Role Labeling (EMNLP 2018)
Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, Andrew McCallum

Phrase-Based & Neural Unsupervised Machine Translation (EMNLP 2018)
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, Marc’Aurelio Ranzato

Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (ICLR 2018)
Sandeep Subramanian, Adam Trischler, Yoshua Bengio, Christopher J Pal

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context (arXiv 2019)
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov

Universal Transformers (ICLR 2019)
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Łukasz Kaiser

An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models (NAACL 2019)
Alexandra Chronopoulou, Christos Baziotis, Alexandros Potamianos

其他年代较远的论文，在选择阅读内容时，通常可以用引用数量作为选择的指标。

我的经验是，你应该阅读那些你觉得有趣、并能让你快乐的文章！

其他资源

有很多很赞的资源可以使用，不一定是论文。以下是一些:

书籍：

Speech and Language Processing (3rd ed. draft)
Dan Jurafsky and James H. Martin

Neural Network Methods for Natural Language Processing
Yoav Goldberg

课程资料：

Natural Language Understanding and Computational Semantics with Katharina Kann and Sam Bowman at NYU

CS224n: Natural Language Processing with Deep Learning with Chris Manning and Abigail See at Standford

Contextual Word Representations: A Contextual Introduction from Noah A. Smith’s teaching material at UW

博客/播客：