PIGlets

The purpose of the session is for PhD students interested in machine learning to get to grips with advanced material sitting in difficulty above the taught courses here and below the discussions of cutting-edge material in the usual PIGS meetings. Anyone else who's interested in joining us for the same is welcome, but the point is to get a solid handle on the techniques rather than debate their finer points. Meetings are organised by Gavin Gray and generally held every second Monday at 10:30am in room 2.33 in the Informatics Forum. Announcements are made through the PIGS mailing list.

Remit from 2016

To solve problems of persistence in teaching material and to encourage group discussion, we're going to try some changes to the format. Previously, one person lead the discussion with the expectation that everyone in the room had read the paper and wanted to discuss it. Unfortunately, this means that one person is stuck with a lot of work to learn the material in order to teach it to others. Also, just reading a paper has limited scope for discussion and is not necessarily the best way to learn about every topic. Alternatives to this format may be useful, such as a tutorial format with exercises released the week before, or a practical session introducing the problem that motivated the topic of interest. Volunteers in charge of a topic are free to choose whichever format they think is best, or even outsource leading the discussion (if they can).

Another problem we're hoping to solve is not having enough teaching material, because it's a lot of work to prepare it. Making the sessions more flexible, and urging people who have volunteered for a topic to prepare as much material as they can, whenever they are studying the topic, should let us schedule greedily based on what's available. For example, if someone presents on variational inference, preparing a talk around a fundamental paper, but at the same time comes across some good exercises, these exercises could be greedily scheduled at a later date for a tutorial session. We're hoping to aggregate this material in a Git Book. This has the added advantage of making the material persistent; future students in ANC will have access to an index of materials for advanced topics.

Proposed topics for 2017

Volunteers to own a given topic are next to each topic:

• Conditional Random Fields
• Autoencoders: Gavin Gray
• Approximate Bayesian Computation: Harri Edwards
• Dimensionality Reduction
• Probabilistic Programming: Svetlin Penkov
• Sparse Regression
• Time-series Analysis: Alina Selega
• Gaussian Processes
• Learning Theory
• Reinforcement Learning: Harri Edwards
• Bayesian Neural Networks Part 2: George Papamakarios
• Experiment Logging: Gavin Gray

Topics Covered in 2016

So far, as of 22nd of February 2016:

• Bayesian Neural Networks: George Papamakarios
• Advanced MCMC: HMC, SGLD etc.: Matt Graham
• Variational Inference: Gavin Gray
• Bayesian Optimisation: James Wilson

Proposed topics for 2014/2015

• Graphical Models Review/ Factor graphs/ EP
• Dimensionality reduction
• Probabilistic programming
• Variational inference
• Sparse regression
• Sampling methods
• Time-series analysis
• Gaussian processes

Future meetings

• Gavin (25 January 2016): Black Box Variational Inference: paper, related code
• Matt (8 February 2016): Gradient based MCMC methods (part I) more details and links to resources slides

Previous meetings

Gaussian Processes

• Iain (16 March 2015): Introduction to GPs [ slides] [ book]

Variational bayesian inference

• Gavin (02 Feb 2015) : Variational Bayes EM (Mixture of Gaussians). Section 21.6 from Murphy or 10.2 from Bishop.
• Agamemnon (19th Jan 2015) : Variational inference introduction (Section 21.1 - 21.4 from Murphy's textbook).
• George (02 March 2015): Variational message passing

Graphical models review / Expectation propagation

• Agamemnon (17th Nov 2014): Understanding Belief Propagation and its Generalizations (Yedidia, Freeman and Weiss) - Sections 1 - 2. http://www.cs.ubc.ca/~schmidtm/Software/UGM.html">Toolbox for inference in Undirected graphical models:
• Tom (1st Dec 2014): Understanding Belief Propagation and its Generalizations (Yedidia, Freeman and Weiss) - Sections 3 - 5. Expectation Propagation for Approximate Bayesian Inference: . Thomas Minka's PhD thesis might also be useful:
• Matt (15th Dec 2014): Expectation propagation (2nd part): Expectation Propagation for Approximate Bayesian Inference . http://research.microsoft.com/en-us/um/people/minka/papers/ep/minka-thesis.pdf">Thomas Minka's PhD thesis might also be useful. For the second half of the meeting my plan is to run through how to apply EP to an example problem. I will look at using the Infer.NET framework from Microsoft Research (with Minka being one of the developers of this) to apply expectation propagation to the Bayes Point Machine classification model discussed as an example at the end of Minka's UAI paper (*). Use of probabilistic programming frameworks like Infer.NET was mentioned as a possible topic that people were interested in covering so I thought this might be a useful chance to combine this with getting some practical experience of using EP. If anyone wants to have a go at playing around with this themselves there is a tutorial for how to implement this in C# with Infer.NEThereand I've put my (Iron)Python code based on the same tutorial and instructions for setting up Infer.NET on Ubuntuhere in case it's of any help to non-Windows users.

Time Series / Sequential Data

• Miltos (11th Jun): The Analysis of Time Series: An Introduction - Chris Chatfield (2009) - Chapter 4
• Kostas (25th Jun): Machine Learning: A Probabilistic Perspective - Kevin Murphy (2012) - Chapter 18

Conditional Random Fields

• Krzysztof (2nd Apr): An Introduction to Conditional Random Fields - Charles Sutton and Andrew McCallum (2012) - Chapter 1, 2 and 3
• James (16th Apr): An Introduction to Conditional Random Fields - Charles Sutton and Andrew McCallum (2012) - Chapter 4, 5 and 6

Bayesian Non-parametrics

• Daniel Renshaw (22nd Jan): A Tutorial on Bayesian Nonparametric Models - Samuel J. Gershman, David M. Blei (2011)
• Simon (5th Feb): Markov Chain Sampling Methods for Dirichlet Process Mixture Models - Radford M. Neal (2000)
• Daniel Trejo (5th Mar): Variational Inference for Dirichlet Process Mixtures - David M. Blei and Michael I. Jordan (2006)

Variational Inference and EP

• Peter (23rd Oct): Bayesian Reasoning and Machine Learning - David Barber (2012) - Section 28.1 - 28.7
• Ioan (6th Nov): Understanding Belief Propagation and its Generalizations - Jonathan S. Yedidia, William T. Freeman, and Yair Weiss (2001)
• Ali (20th Nov): A Family of Algorithms for Approximate Bayesian Inference - Thomas P Minka (2001) - Chapter 2, Section 3.1 & 3.2
• Ali (20th Nov): TrueSkill(TM): A Bayesian Skill Rating System - Ralf Herbrich, Tom Minka, and Thore Graepel (2007)

Large Data/Online and Streaming

• Peter (14th Aug): Large-Scale Machine Learning at Twitter - Jimmy Lin and Alek Kolcz (2012)

MCMC

• Jono (8th May): An Introduction to MCMC for Machine Learning - Andrieu, C., De Freitas, N., Doucet, A., Jordan, M.I. (2003) - Pages 1-27
• Miha (22nd May): Slice Sampling - Radford Neal (2000) - Sections 1, 2, 3 and 4
• Miha (22nd May): MCMC using Hamiltonian dynamics - Radford Neal (2000) - Sections 1, 2, and 3

Boltzmann machines, MRFs etc.

• Simon (27th Mar): Information Theory, Inference, and Learning Algorithms, Ch 43 - Boltzmann Machines - D. MacKay .
• Simon (27th Mar): Section six of Learning Deep Architectures for AI - Yoshua Bengio
• Beni (10th Apr): Chapter 5 (up to 5.4.2) of Learning Deep Architectures for AI - Yoshua Bengio
• Beni (10th Apr): section 2 of Modelling Human Motion Using Binary Hidden Variables - Taylor, G.W. Hinton, G.E., Rowels, S. (2006)
• James (24th Apr): On Contrastive Divergence Learning - Miguel A. Carreira-Perpinan and Geoffrey E. Hinton (2005)
• James (24th Apr): Training Restricted Boltzmann Machines using Approximations to the Likelihood Gradient - Tijmen Tieleman (2008)

Theory of kernel methods

• Dominik (14th Feb): A Tutorial Review of RKHS Methods in Machine Learning, Hofmann, Schölkopf, Smola. Pages 1-19
• Athina (28th Feb): Kernel Methods for Pattern Analysis (available from the library website) - Shawe-Taylor and Christianini. Chapter 6.
• Ali (13th Mar): Learning a kernel matrix for nonlinear dimensionality reduction, Kilian Q. Weinberger, Fei Sha, Lawrence K. Saul (2004)

Dynamical Models

• Simon (Jan 17th): Continuous Time Bayesian Networks, U. Nodelman, C.R. Shelton, and D. Koller (2002)
• Simon (Jan 17th): Learning continuous time Bayesian networks, Nodelman, U., Shelton, C. R., & Koller, D. (2003)
• Benigno (Dec 6th): Towards Better Understanding of the Model Implied by the use of Dynamic Features in HMMs, Bridle, J. S. (2004)
  • For those who are interested in the subject, the seminal paper is Speech Parameter Generation Algorithms for HMM-Based Speech Synthesis, however, the notation in that paper is horrible. A clearer notation can be found on the first 4 pages of Reformulating the HMM as a trajectory model by imposing explicit relationships between static and dynamic feature vector sequences (the rest of the paper is not related).
  • For the mathematically inclined, see: How to pretend that correlated variables are independent by using difference observations. Beni isn't going to talk about this, but this paper may also be of interest as it was written by Chris Williams.

Conditional Random Fields

• Dominik (Nov 22nd): Shallow Parsing with Conditional Random Fields, Sha, F. and Pereira, F. (2003)
• Dominik (Nov 22nd): Hidden-state Conditional Random Fields, Quattoni, A., Wang, S., Morency, L.P., Collins, M. and Darrell, T. (2007)
• Ioan (Nov 8th): An Introduction to Conditional Random Fields, Sutton, C. and McCallum , A. (2010)
• Ioan (Nov 8th): Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data, Lafferty, J., McCallum , A. and Pereira, F. (2001)

Model Combination

• Krzysztof (Oct 25th): Supervised aggregation of classifiers using artificial prediction markets, Lay, N. and Barbu, A. (2010)
• Krzysztof (Oct 25th): An Introduction to Artificial Prediction Markets for Classification, Barbu, A. and Lay, N. (2011)
• Krzysztof (Oct 25th): Machine Learning Markets, Storkey, A. (2011)
• Yichuan (Oct 11th): Adaptive Mixtures of Local Experts, Jacobs, R.A., Jordan, M.I., Nowlan, S.J. and Hinton, G.E. (1991)
• Yichuan (Oct 11th): Training products of experts by minimising contrastive divergence, Hinton, G.E. (2002)
• Peter (Sept 13th): Pattern Recognition and Machine Learning - Ch 14: Combining Models, C Bishop (2006)
• Konrad (Sept 27th): Integration of Stochastic Models by Minimizing α-Divergence, Amari, S. (2007)
• Konrad (Sept 27th): Parameter Estimation for α-GMM Based on Maximum Likelihood Criterion, Wu, D. (2009)

Optimisation

• Frank (Aug 30th): Convex Optimisation, Boyd and Vandenberghe (2004)

Sampling

• James (Apr 26th): Adaptive Direction Sampling, Gilks, W., Roberts, G., George, E. (1994)
• James (Apr 26th): Adaptive Independence Sampler, Keith, J., Kroese, D., Sofronov, G. (2008)
• Grigoris (May 24th): MCMC Using Ensembles of States for Problems with Fast and Slow Variables such as Gaussian Process Regression, Radford, N (2010)
• Simon (Jul 18th): The Gaussian Process Density Sampler, Adams, R., Murray, I., MacKay , D. (2009)

Topic Models

• Athina (Feb 15th): Bayesian Reasoning and Machine Learning, Barber, B. (2010)
• Athina (Feb 15th): The Author-Topic Model for Authors and Documents, Rosen-Zvi M., Griffiths T., Steyvers M., Smyth P. (2004)
• Ioan (Mar 1st): Finding Scientific Topics., Thomas L. Griffiths, Mark Steyvers. (2004)
• Ioan (Mar 1st): Topic modeling: beyond bag-of-words., Hanna M. Wallach. (2006)
• Jyri (Mar 15th): Probabilistic Topic Models, David Blei, Lawrence Carin, and David Dunson (2010)
• Jyri (Mar 15th): Hierarchical Bayesian nonparametric models with applications (Sections 1-5: pages 1-25), Yee W. Teh, and Michael I. Jordan (2010)
• Ondrej (Mar 29th): Discrete Component Analysis, W. Buntine and A. Jakulin (2006)

Discriminative and generative methods

• Principled Hybrids of Generative and Discriminative models, Lasserre, J. Bishop, C. M. and Minka, T. (2006)
• Hannes (February 1st): On Discriminative vs Generative Classifiers: A comparison of logistic regression and naive Bayes, Ng, A. and Jordan, M. (2008)
• Hannes (February 1st): An Asymptotic Analysis of Generative, Discriminative, and Pseudolikelihood Estimators, Liang. and Jordan, M. (2006)

Sparse ICA

1. Ali (September 14th): Independent Components Analysis: Algorithms and Applications, Aapo Hyvärinen and Erkki Oja (2000)
2. Konrad (September 28th): Sparse Coding with an Overcomplete Basis Set: A Strategy Employed by V1?,, Olshausen BA, Field DJ (1997)
3. Peter (October 12th): An Introduction to Compressive Sampling, Emmanuel J. Candes and Michael B. Wakin (2008)

Markov Chain Monte Carlo methods

Probabilistic inference using Markov chain Monte Carlo methods, Radford M. Neal (1993)

1. Jakub (July 6th): Pages 1-59
2. Ali (July 20th): Pages 59-87
3. Ali (August 3rd): Pages 59-87
4. Nicolas (August 17th): Pages 87-121

Slice Sampling , Radford M. Neal (2003)

1. Iain (August 31st): Pages 710-720

Archives

Visit PIGlets 2009 for a list of meetings held in 2009.