Machine Learning for Natural Language Processing (ML-for-NLP)
This reading group focuses on Machine Learning techniques that may be applied to the field of Natural Language Processing. Participants are encouraged to suggest topics, papers, or tutorials (which need not involve any current application in NLP) by adding them to the lists below. Suggesting a paper does not constitute any sort of commitment to presenting that paper.
Meetings are approximately every week on Mondays. Meetings will be in 3.02 at 3pm unless otherwise stated. Announcements for this group will be made by email, and it is possible to sign up to the mailing list here.
Schedule for Spring 2020
February 3
- Neural Text Generation With Unlikelihood Training (ICLR 2020)
Slides: https://drive.google.com/file/d/10SdZpjh50pWF-bInZdhnvjBKbCOfZyLZ/view
Abstract: Neural text generation is a key tool in natural language applications, but it is well known there are major problems at its core. In particular, standard likelihood training and decoding leads to dull and repetitive outputs. While some post-hoc fixes have been proposed, in particular top-k and nucleus sampling, they do not address the fact that the token-level probabilities predicted by the model are poor. In this paper we show that the likelihood objective itself is at fault, resulting in a model that assigns too much probability to sequences containing repeats and frequent words, unlike those from the human training distribution. We propose a new objective, unlikelihood training, which forces unlikely generations to be assigned lower probability by the model. We show that both token and sequence level unlikelihood training give less repetitive, less dull text while maintaining perplexity, giving superior generations using standard greedy or beam search. According to human evaluations, our approach with standard beam search also outperforms the currently popular decoding methods of nucleus sampling or beam blocking, thus providing a strong alternative to existing techniques.
Schedule for Spring 2019
April 15
- Tutorial on spectral learning for NLP (Naomi)
April 8
- Tutorial on Normalizing Flows and Latent Normalizing Flows for Discrete Sequences (Nicola and Tom)
April 1
- Syntax-Directed Variational Autoencoder for Structured Data (Nicola)
March 25
- Tutorial on Variational Inference (Tom)
March 18
March 11
- Tutorial on generative adversaria networks for text generation part II (Christos)
March 4
February 27
February 20
- Tutorial on generative adversaria networks for text generation part I (Christos)
February 13
- Tutorial on generative adversaria networks, adversarial training and adversarial examples (Nicola)
February 4
- Discussion/planning for future meetings
January 28
- Adversarial Sampling and Training for Semi-Supervised Information Retrieval [pdf] (Nelly)
January 21
- A Structured Variational Autoencoder for Contextual Morphological Inflection [pdf] (Caio)
January 14
- Simple and Effective Semi-Supervised Question Answering [pdf] (Nicola)
Schedule for Fall 2018
November - December 2018
- break for NAACL and winter
November 19
- Neural Language Modeling by Jointly Learning Syntax and Lexicon [pdf] (Hao)
November 12
November 5
October 29
- Learning to Compose Task-Specific Tree Structures [pdf] (Rei)
October 22
- Understanding deep learning requires rethinking generalization [pdf] (Naomi)
October 15
- Semantically Equivalent Adversarial Rules for Debugging NLP Models [pdf] (Jon)
October 8
- On the Importance of Single Directions for Generalization [pdf] (Naomi)
October 1
Schedule for Spring 2018
May 7
- Imagine This! Scripts to Compositions to Videos [pdf] (David)
April 30
- SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient [pdf] (Chunchuan)
April 23
- A Deep Reinforced Model for Abstractive Summarization [pdf] (Ratish)
April 16
- Counterfactual fairness [pdf] (Naomi)
April 9
- Poincaré Embeddings for Learning Hierarchical Representations [pdf] (Javad)
April 2
- Semi-supervised learning with deep generative models [pdf] (Caio)
March 26
- Tensor Fusion Network for Multimodal Sentiment Analysis [pdf] (Nikos)
February 26 - March 19
February 19
- Morphological Inflection Generation with Hard Monotonic Attention [pdf] (Joana)
February 12
- A hybrid approach to dialogue management based on probabilistic rules [pdf] (Craig)
February 5
- Semantic Composition via Probabilistic Model Theory [pdf] (Matt)
January 29
January 22
- Emergent translation in multi-agent communication [pdf] (Serhii)
January 15
Schedule for Fall 2017
December 4
- Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model [pdf] (Ratish)
November 27
- Non-Autoregressive Neural Machine Translation [pdf] (Nikos)
November 14
November 6
- Word Representation Models for Morphologically Rich Languages in Neural Machine Translation [pdf] (Sander)
October 30
- Cross-lingual, Character-Level Neural Morphological Tagging [pdf] (Clara)
October 23
- Probabilistic Models for Segmenting and Labeling Sequence Data [pdf] (Joana)
October 16
- A joint many-task model: Growing a neural network for multiple NLP tasks [pdf] (Marco)
October 9
- Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling [pdf] (Diego)
October 2
- Human Centered NLP with User-Factor Adaptation [pdf] (Pippa)
September 25
Schedule for Fall 2016
December 12
December 5
November 28
- A Decomposable Attention Model for Natural Language Inference (Imigo)
November 21
- Jointly Combining Implicit Constraints Improves Temporal Ordering (Nate)
November 14
- Policy distillation (Mihai)
November 7
- Distilling the Knowledge in a Neural Network (Naomi)
October 31
October 24
- Learning hidden unit contributions for unsupervised acoustic model adaptation (Joachim)
October 17
October 10
- A fully bayesian approach to unsupervised POS tagging (Craig)
October 3
- Beam sampling with Infinite HMM (Joana)
September 26
- Introduction to Bayesian statistics (Shay)
September 19
Schedule for Spring 2016
May 30
- Crash-course on Clustering (Akash)
May 23
- Probabilistic program induction (Pablo)
May 2
- Composing Neural Networks (Shashi)
- Learning to compose neural networks for question answering, Jacob Andreas, Marcus Rohrbach, Trevor Darrell and Dan Klein. NAACL 2016. http://arxiv.org/pdf/1601.01705v1.pdf
Apr 25
- Bayesian optimization (Nicolas)
Apr 18
- Planning meeting (Herman and Shashi)
Apr 11
- A quick introduction to Torch (Stef)
- Differences to Theano and Tensorflow
Apr 4
- A quick introduction to TensorFlow (Rafael and Clara)
Mar 28
- Optimization Stuffs (Michael)
Mar 21
- Generative Bayesian-ish Neural Network Stuffs (Lea)
- "Neurons as Monte Carlo Samplers: Bayesian Inference and Learning in Spiking Networks" from Huang and Rao :link
Mar 7
- Generative Bayesian-ish Neural Network Stuffs (Jianpeng = JP)
Feb 29
- Generative Bayesian-ish Neural Network Stuffs (Harri)
Feb 22
- Generative Bayesian-ish Neural Network Stuffs (Herman)
Feb 15
- Parsing and Generation (Joana)
Feb 8
- Parsing and Generation (John)
Feb 1
- Parsing and Generation (Marco)
Jan 25
- Parsing and Generation (Sorcha)
Jan 18
- Parsing and Generation (Rafael)
Jan 11
- Planning meeting (Herman & Shashi)
Schedule for Fall 2015
Dec 7
- Neural Networks: Attention models (Liang & Xingxing)
Nov 30
- Neural Networks: RNNs (Daniel)
Nov 23
- Neural Networks: RNNs (Herman)
- Deep Learning, An MIT Press book in preparation. Ian Goodfellow, Aaron Courville, and Yoshua Bengio. http://goodfeli.github.io/dlbook/ Section 10.2.2 in the RNN chapter and Section 8.2.5 in the optimization chapter.
Nov 16
- Word embeddings: Application (Rafael)
Nov 9
- Word embeddings: Application (Michael Hahn)
Nov 2
Oct 26
- Reinforcement learning and Neural Networks (Xingxing)
Oct 19
- Reinforcement Learning(Mihai)
Oct 12
- Reinforcement learning (Ben)
Oct 5
- Planning meeting (Herman & Shashi)
- Major topics
- Tool/Resource sessions
Previous Meetings
Visit Previous meetings for a list of previous meetings.
Useful Information
Visit Useful Information for information about useful tools and paper recommendations from previous meetings.
Other Reading Groups
Cambridge http://www.wiki.cl.cam.ac.uk/rowiki/NaturalLanguage/ReadingGroup
Heriot-Watt https://sites.google.com/site/hwmlreadinggroup/
Toronto http://learning.cs.toronto.edu/mlreading.html
- Planning Meeting 2017: