NAVER LABS Europe seminars are open to the public. This seminar is virtual and requires registration.
Date: 26th January 2021, 10:00 am (GMT +01.00)
Learning with sparse latent structure
Speaker: Vlad Niculae is an assistant professor in the Language Technology Lab, part of the Informatics Institute at the University of Amsterdam. Vlad’s research lies at the intersection of machine learning and natural language processing, building upon techniques from optimization, geometry, and probability, in order to develop and analyze better models of language structures and phenomena. He obtained his PhD in Computer Science from Cornell University in 2018, and has worked until 2020 as post-doctoral researcher in the DeepSPIN project (Deep Structured Prediction for Natural Language Processing) at the Instituto de Telecomunicações, Lisbon, Portugal.
Abstract: Structured representations are a powerful tool in machine learning, in particular for natural language: The discrete, compositional nature of words and sentences leads to natural combinatorial representations such as trees, sequences, segments, or alignments, among others. Such representations are at odds with deep neural networks, which conventionally perform smooth, soft computations, learning dense, inscrutable hidden representations.
We present SparseMAP, a strategy for inferring differentiable combinatorial latent structures, alleviating the tension between discrete and continuous representations through sparsity. SparseMAP computes a globally-optimal combination of a very small number of structures, and can be extended to arbitrary factor graphs (LP-SparseMAP), only requiring access to local maximization oracles. Our strategy is fully deterministic and compatible with familiar gradient-based methods for training neural networks. We demonstrate sparse and structured neural hidden layers, with successful empirical results and visualisation properties.