Probabilistic Inference Group (PIGS)

The Probabilistic Inference Group (PIGS) is a paper discussion group with meetings held fortnightly. The group focuses on probabilistic and information theoretic approaches to machine learning problems. Meetings are generally held fortnightly on Mondays at 10:30am in room 1.16 of the Informatics Forum. Announcements are made through the PIGS mailing list.

Instructions for presenters:

1) Choose a mainstream ML paper (or two).
2) Provide paper(s) at least one week in advance of the meeting.
3) Lead a discussion of the paper(s) in the meeting.

Mainstream means a paper that does not depend heavily on domain specific background to be comprehensible. A possible test would be to consider if the techniques could fairly readily be transferred to another application area. Papers from conferences like NIPS, ICML, UAI, AISTATS, and journals like JMLR and ML papers from IEEE PAMI are likely to be in scope; but note that papers from other sources could well fit too.

Students should discuss their paper selections with their supervisor to make sure they are reasonable choices. It is acceptable to relate the selected papers to the presenter's research, but not at the expense of discussion of the selected paper.

If people want to make thematic groupings of readings it should be possible to arrange swaps in the rota in order to make this happen.

Upcoming discussions:

Jun 4: Sohan 1.16

Differential Privacy and Machine Learning: a Survey and Review; Zhanglong Ji, Zachary C. Lipton, Charles Elkan, 2014 https://arxiv.org/abs/1412.7584

Jun 18: 1.16

Past discussions:

May 7: ICLR 1.16

Saphra: On the importance of single directions for generalization. Ari S. Morcos, David G.T. Barrett, Neil C. Rabinowitz, Matthew Botvinick. https://arxiv.org/abs/1803.06959

Sohan: Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments. Maruan Al-Shedivat, Trapit Bansal, Yura Burda, Ilya Sutskever, Igor Mordatch, Pieter Abbeel. https://openreview.net/forum?id=Sk2u1g-0-

Apr 23: AISTATS 1.16

Sohan: A generic approach for excaping saddle points, Reddi et al. https://arxiv.org/abs/1709.01434

Amos: On the challenges of learning with inference networks on sparse, high-dimensional data http://proceedings.mlr.press/v84/krishnan18a/krishnan18a.pdf

Amos: Gradient Layer: Enhancing the Convergence of Adversarial Training for Generative Models http://proceedings.mlr.press/v84/nitanda18a/nitanda18a.pdf

Other good papers people might want to do:

Tracking the gradients using the Hessian: A new look at variance reducing stochastic methods. Robert Gower, Nicolas Le Roux, Francis Bach ; PMLR 84:707-715

Can clustering scale sublinearly with its clusters? A variational EM acceleration of GMMs and k-means. Dennis Forster, Jörg Lücke

Delayed Sampling and Automatic Rao-Blackwellization of Probabilistic Programs. Lawrence Murray, Daniel Lundën, Jan Kudlicka, David Broman, Thomas Schön

Apr 9: Iain 1.16 (confirmed)

Vprop: Variational Inference using RMSprop, Mohammad Emtiyaz Khan, Zuozhu Liu, Voot Tangkaratt, Yarin Gal. https://arxiv.org/abs/1712.01038

Appeared at this workshop: http://bayesiandeeplearning.org/

Noisy Natural Gradient as Variational Inference, Guodong Zhang, Shengyang Sun, David Duvenaud, Roger Grosse. https://arxiv.org/abs/1712.02390

Mar 26: Chris 1.16

Generative Models of Visually Grounded Imagination, Vedantam et al . ICLR 2018. https://arxiv.org/pdf/1705.10762.pdf

Feb 26: Simao 1.16 (confirmed)

Wasserstein Auto-Encoders, Tolstikhin et al. ICLR 2018 https://arxiv.org/pdf/1711.01558.pdf

Feb 12: George 1.16 (Confirmed)

Mastering the game of Go without human knowledge, Silver et al. Nature. https://www.nature.com/articles/nature24270

Additional reading

https://arxiv.org/abs/1712.01815

https://www.nature.com/articles/nature16961

Jan 29: Charlie 1.16 (Confirmed)

Variational Lossy Autoencoder, Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, Pieter Abbeel, ICLR 2017 https://arxiv.org/abs/1611.02731

Jan 15: 1.16

We will watch the talk “Information Theory of Deep Learning” by Naftali Tishby, https://www.youtube.com/watch?v=bLqJHjXihK8&t=912s. The related paper can be found here, https://arxiv.org/pdf/1503.02406.pdf

Dec 18: NIPS 1.16

Sohan: Convolutional Gaussian Processes

Sohan: Non-stationary Spectral Kernels

Serhii: REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models

Naomi: Poincaré Embeddings for Learning Hierarchical Representations

Chris: Unsupervised Image-to Image Translation Networks

George: A Linear-Time Kernel Goodness-of-Fit Test

Dec 4: 1.16 (overlaps with NIPS)

Nov 20: James 1.16 (confirmed)

1. WaveNet: A Generative Model for Raw Audio by Aaron van den Oord et al. (blog)

1a. Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders by Jesse Engel et al (blog)

2. A note on the evaluation of generative models by Lucas Theis et al.

Nov 6: Naomi 1.16 (confirmed)

1. Learning the Curriculum with Bayesian Optimization for Task-Specific Word Representation Learning by Tsvetkov et al.

2. Self-paced Curriculum Learning by Jiang et al.

Additional reading,

3. Curriculum Dropout by Morerio et al.

4. Repeat before Forgetting: Spaced Repetition for Efficient and Effective Training of Neural Networks by Amiri et al.

Oct 23: Cancelled 1.16 (confirmed)

Oct 9: No PIGS

Sep 25: UAI 1.16

Sohan: Learning Approximately Objective Priors by Nalisnick and Smyth

Per: Bayesian Inference of Log Determinants by Fitzsimons et al.

Chris: Green Generative Modeling: Recyclling Dirty Data using Recurrent Variational Autoencoders

Simao: Learning to Draw Samples with Amortized Stein Variational Gradient Descent by Feng et al.

Sep 11: ICML - 1.16

Sohan: Post-Inference Prior Swapping by Neiswanger and Xing

Iain: Efficient softmax, maybe Neural Optimizer Search with Reinforcement Learning

Per: Asynchronous Distributed Variational Gaussian Process for Regression by Peng et al.

Chris: iSurvive: An Interpretable, Event-time Prediction Model for mHealth by Dempsey et al.

June 19: ICLR - 2.33

Sohan: Revisiting classifier two-sample tests; Lopez-Paz and Oquab: https://openreview.net/pdf?id=SJkXfE5xx

Chris: PixelVAE : A latent variable model for natural images; Gulrajani et al.: https://openreview.net/pdf?id=BJKYvt5lg

What does it take to generate natural textures? Ustyuzhaninov et al.: https://openreview.net/pdf?id=BJhZeLsxx

Iain: Highway and Residual Networks learn Unrolled Iterative Estimation, Greff et al.: https://openreview.net/forum?id=Skn9Shcxe

June 5: Gavin (confirmed) 2.33

1. Stochastic Gradient Descent with Restarts: https://arxiv.org/abs/1608.03983.

2. Probabilistic Line Searches for Stochastic Optimisation: https://arxiv.org/abs/1502.02846

May 22: James (cancelled due to NIPS deadline) 2.33

May 8: AISTATS - 4.02 (4.31-4.33 cancelled)

Chris: Prediction Performance After Learning in Gaussian Process Regression, Wagberg et al.

Matt: Annular augmentation sampling; Fagan, Bhandari and Cunningham.

Sohan: Detecting Dependencies in Sparse, Multivariate Databases Using Probabilistic Programming and Non-parametric Bayes, Saad and Mansinghka

April 24: Harri (confirmed - cancelled due to ICLR ) 2.33

April 10: Antonio (confirmed) 5.42

"Wasserstein GAN" by Arjovsky et al. link

March 27: Cancelled due to job candidate presentation

March 13: Iain (confirmed) 5.42

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer: Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean; To appear, ICLR 2017 link

March 6: Tim (confirmed) 4.31/4.33

Tim will discuss "modular neural networks” as illustrated by two recent papers:

Neural Module Network, Andreas et al. link

Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfer, Devin et al. link

February 27: Vaishak (confirmed) 2.33

Vaishak will give a brief overview of the following papers,

Probabilistic Inference in Hybrid Domains by Weighted Model Integration. Belle, V.; Passerini, A.; and Van den Broeck, G. In IJCAI, 2015. link

Hashing-based Approximate Probabilistic Inference in Hybrid Domains. Belle, V.; Van den Broeck, G.; and Passerini, A. In UAI, 2015. link

Component Caching in Hybrid Domains with Piecewise Polynomial Densities. Belle, V.; Van den Broeck, G.; and Passerini, A. In AAAI, 2016. link

February 13: Elliot (confirmed) 4.31-4.33

Elliot will lead the discussion on "Adversarial Feature Learning" (arXiv), and perhaps also touch upon "Energy-based Generative Adversarial Networks" (arXiv).

January 30: Matt (confirmed) 5.42

Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space.
Anh Nguyen, Jason Yosinski, Yoshua Bengio, Alexey Dosovitskiy, Jeff Clune. arXivwebsite

January 16: Chris (confirmed) (5.42)

The session will be about Generative Adversial Networks (GANs) and Noise Contrastive Estimation (NCE).

We will read

On distinguishability criteria for estimating generative models
Ian J. Goodfellow
https://arxiv.org/abs/1412.6515

and

Statistical Inference of Intractable Generative Models via Classification
Michael U. Gutmann, Ritabrata Dutta, Samuel Kaski, and Jukka Corander
https://arxiv.org/pdf/1407.4981v2.pdf

December 19: NIPS 2016 review

George: Learning to Learn by Gradient Descent by Gradient Descent, https://arxiv.org/abs/1606.04474

Sohan: Bayesian optimization for automated model selection, Malkomes et al., PDF

Chris: Improving Variational Autoenciders with Inverse Autoregressive Flow, Kingma et al, PDF

Michael: Bayesian Optimization with Robust Bayesian Neural Networks, Springenberg et al, pdf

Gavin: A Probabilistic Framework for Deep Learning, Patel et al, https://arxiv.org/abs/1612.01936

Naomi: Residual networks behave like ensembles of relatively shallow networks, Veit et al, https://arxiv.org/abs/1605.06431

Matt: Measuring the reliability of MCMC inference with bidirectional Monte Carlo. Grosse, Anche and Roy PDF

November 21: George (confirmed)

"Rényi Divergence Variational Inference" by Li and Turner (pdf)

November 7: Sohan (confirmed)

"Variational Autoencoder for Deep Learning of Images, Labels and Captions" by Pu et al. (pdf)

“Attribute2Image: Conditional Image Generation from Visual Attributes” by Yan et al. (pdf)

August 8: Jaroslav

Presentation of his KDD 2016 paper on: A Subsequence Interleaving Model for Sequential Pattern Mining

Abstract:

Recent sequential pattern mining methods have used the minimum description length (MDL) principle to define an encoding scheme which describes an algorithm for mining the most compressing patterns in a database. We present a novel subsequence interleaving model based on a probabilistic model of the sequence database, which allows us to search for the most compressing set of patterns without designing a specific encoding scheme. Our proposed algorithm is able to efficiently mine the most relevant sequential patterns and rank them using an associated measure of interestingness. The efficient inference in our model is a direct result of our use of a structural expectation-maximization framework, in which the expectation-step takes the form of a submodular optimization problem subject to a coverage constraint. We show on both synthetic and real world datasets that our model mines a set of sequential patterns with low spuriousness and redundancy, high interpretability and usefulness in real-world applications. Furthermore, we demonstrate that the quality of the patterns from our approach is comparable to, if not better than, existing state of the art sequential pattern mining algorithms.

July 11: ICML 2016 review

Matt: Slice Sampling on Hamiltonian Trajectories, Benjamin Bloem-Reddy and John P. Cunningham

Harri: http://arxiv.org/abs/1602.03032 Associative Long Short-Term Memory | Ivo Danihelka et al.

Pol: Autoencoding beyond pixels using a learned similarity metric Larsen et. al

Gavin: Noisy Activation Functions Caglar Gulcehre et al

June 20: Lukasz

Attend, Infer, Repeat: Fast Scene Understanding with Generative Models

S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, Koray Kavukcuoglu, Geoffrey E. Hinton

Learning to decompose for object detection and instance segmentation

Eunbyung Park, Alexander C. Berg

June 6: Charlie

Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation Jonathan Thompson, Arjun Jain, Yann LeCun and Christoph Bregler

Conditional Random Fields as Recurrent Neural Networks Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip H. S. Torr

May 23: Weng-Keen

Presentation of his ICML 2016 paper on: Efficient Multi-Instance Learning for Activity Recognition from Time Series Data Using an Auto-Regressive Hidden Markov Model

Abstract:

Activity recognition from sensor data has spurred a great deal of interest due to its impact on health care. Prior work on activity recognition from multivariate time series data has mainly applied supervised learning techniques which require a high degree of annotation effort to produce training data with the start and end times of each activity. In order to reduce the annotation effort, we present a weakly supervised approach based on multi-instance learning. We introduce a generative graphical model for multi-instance learning on time series data based on an auto-regressive hidden Markov model. Our approach models both the structure within an instance as well as the structure between instances in a bag. Our model has a number of advantages, including the ability to produce both bag and instance-level predictions as well as an efficient exact inference algorithm based on dynamic programming.

April 25: Amos

Knowledge Matters: Importance of Prior Information for Optimization C. Gulcehre and Yoshua Bengio

April 11: Chris

A survey of techniques for incremental learning of HMM parameters Wael Khreich Eric Granger, Ali Miri, Robert Sabourin

March 14: Theo

NICE: NON-LINEAR INDEPENDENT COMPONENTS ESTIMATION Laurent Dinh, David Krueger and Yoshua Bengio

Feb 29: George

Automatic Variational Inference in Stan Alp Kucukelbir, Rajesh Ranganath, Andrew Gelman and David Blei

Feb 15: Matt

A note on the evaluation of generative models Lucas Theis, Aäron van den Oord and Matthias Bethge

Feb 1: Sohan

Robust Spectral Inference for Joint Stochastic Matrix Factorization Moontae Lee, David Bindel and David Mimno

Jan 18: NIPS 2015 review

George: Bayesian Dark Knowledge Anoop Korattikara, Vivek Rathod, Kevin Murphy, Max Welling

Mingjun: Optimization Monte Carlo: Efficient and Embarrassingly Parallel Likelihood-Free Inference Edward Meeds, Max Welling

Harri: Semi-supervised learning with ladder networks Antti Rasmus, Harri Valpola et al

Theo: Generative Image Modeling Using Spatial LSTMs Lucas Theis, Matthias Bethge

Chris: Unsupervisd Learning by Program Synthesis Ellis, Solar-Lezama, Tenenbaum

Krzysztof: : Training Very Deep Networks Rupesh Kumar Srivastava, Klaus Greff, Jurgen Schmidhuber

Oct 26: Harri

Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks Emily Denton, Soumith Chintala, Arthur Szlam, Rob Fergus

Oct 12: Gavin

Variational Dropout and the Local Reparameterization Trick Diederik P. Kingma, Tim Salimans, Max Welling

Meetings in 2015

• Visit PIGS meetings 2015 for a list of meetings held in 2014 (Sep 2014 - Jul 2015).

Meetings in 2014

• Visit PIGS meetings 2014 for a list of meetings held in 2014 (Sep 2013 - Jul 2014).

Meetings in 2013

• Visit PIGS meetings 2013 for a list of meetings held in 2013 (until May).

Meetings in 2012

• Visit PIGS meetings 2012 for a list of meetings held in 2012.

Meetings in 2011

• Visit PIGS meetings 2011 for a list of meetings held in 2011.

Meetings in 2010

• Visit PIGS meetings 2010 for a list of meetings held in 2010.

Meetings in 2009

• Visit PIGS meetings 2009 for a list of meetings held in 2009.

Meetings in 2008

• Visit PIGS meetings 2008 for a list of meetings held in 2008.

Meetings in 2007

• Visit PIGS meetings 2007 for a list of meetings held in 2007.

Earlier meetings (2002-2006) on old website

• Visit the PIGSTwoThousandAndSixAndEarlier for earlier meetings