Energy Based Models – Podcast

Host: Can you describe what Energy Based Models are?

Marc Dymetman: EBMs are a specific machine learning approach where the focus is on assigning a quality score, a so-called “energy”, to an object, where the object can be an image, a sentence, or a grammatical structure.  A ‘quality score’ is a number which doesn’t have to be between 0 and 1, like in standard probabilistic machine learning models. It can be any number that lets you compare 2 objects. This difference from standard models is quite important because you can design much more expressive models, such as models controlling global properties of the object, like the general sentiment of a sentence, or models that allow you to combine different aspects of the same object.

For example, in a speech/dialogue generation model you may want to control the ‘naturalness’ of the voice to make it sound like a real dialogue rather than a predefined script.  Or, you might want to specify the gender, age, subject matter or even the attitude (i.e. positive or negative) of an utterance. These are all ‘aspects’. So EBMs allow you to model these and combine them to get an overall score. You can then easily compare the scores of two utterances to see which one is better.

In a way, EBMs give you architectural freedom by allowing  you to decouple the modeling aspect from the implementation aspect. To help explain this, think of when you build a house  –  well there’s the modelling team – the architect(s)- and the implementation team – the contractors. It would be very limiting to force the architect to have to think in terms of implementation details right from the start, right? It’s much better for them to think in terms of the general plan and to have some freedom at that level.

Host: Well that makes sense! But surely it can’t be that simple – isn’t there a downside to this freedom of decoupling the model from the implementation?

Marc Dymetman: Yes there is. EBMs have more expressive power, but less direct implementation than standard models. In the example of the utterance – it’s a completely different matter to be able to measure with scores whether an utterance satisfies a desired property than to actually produce utterances that satisfy the criteria. That’s even difficult for humans. An extreme case of this is artistic work, where you know what you like but that doesn’t mean you can produce such a thing. And this implementation aspect is where the main technical challenges lie, with different approaches to implementations (for the same EBMs), with their own pros and cons.

Host: Are there other examples of the flexibility of EBMs?

Hady Elsahar: Let me start by saying that the key point, as Marc said, is that modeling things using scores is much more flexible than modeling things using normalized probabilities. So let me give a good example where this flexibility could be useful – such as the task of compositionality where you want to combine 2 scores or 2 probability distributions.  Imagine you want to model something that has 2 attributes like ‘dogs’ with ‘spots’. However, you don’t have training data for these 2 attributes – you only have training data for dogs and training data for spotted things, like shirts, for example. Using EBMs, you would build a first model (Model 1) that gives you a good score (low energy) when it sees a data point that’s a dog. Then you build a second model (Model 2) that gives you good scores when it sees data points that are spotted. To identify or to model ‘spotted dogs’ is easy with EBMs because you can just combine these two scores.With normalized probabilities, even when Model 1 and Model 2 give you normalized probabilities, combining these probabilities will not be normalized. You would need to renormalize again and that would be quite limiting.

Host: Why is ‘renormalizing’ so limiting?

Hady Elsahar: Right, so people always say that normalizing is intractable so, let me explain what that means. Normalization basically means you have a score, but you don’t know what that score means in the absolute i.e. how it compares to all scores. So, what you do is you list all the possible other classes, or things, or possible categories that could appear and calculate the scores, and then you divide the score that you got for the object you want by all the other scores. And in some cases that might be easy i.e. if you have only 10 numbers and you want to check which number is present in your object. However, in high dimensional data, like images, that would be impossible because we don’t have a way of listing all the possible images and normalizing over them. So, in the example of spotted dogs, if you want to normalize the scores of spotted dogs, you would have to divide them by the sum of scores given of all possible images of dogs and all possible images of spotted things. That would be hard – and expensive! So, this gives you a bit of intuition on why using scores or modeling things using scores is much more flexible.

Host: Can you say a bit more about this advantage of decoupling which seems to be the crux of Energy Based Models – decoupling the method from the implementation in some use cases or applications?

Hady Elsahar: Let me illustrate that by continuing with the example I gave of images, where there could be a limitless number of categories. In standard machine learning we have something very useful that calculates probabilities, something we all love called the ‘softmax’.  Softmax basically lists all the possible categories and calculates the scores and gives you normalized probabilities. But it’s very prohibitive having to list all the possible categories for high dimensional data. Now, in natural language processing we solve this by, instead of treating sentences as atomic units, we break them down into tokens or words.

Host: So you’re cutting up the sentences into smaller parts to limit the number of possibilities?

Hady Elsahar: Yes. So, instead of outputting one sentence at a time, your generation process would output the next word one step at a time until you generate a sentence – we call it ‘left to right’. That’s basically how many (but not all) big language models work. And, although by limiting the size of the vocabulary and only calculating next word probable output (local normalization of probabilities), you make the problem tractable, you can’t plan or control or calculate for things that are based on the global aspects of a sentence like ‘politeness’ or ‘sentiment’.

Host: Are there other use cases that show the breadth of applications of EBMs?

Hady Elsahar: In a different field,  you can use EBMs to add prior knowledge to a pre-trained model or even inside your training objective. Without going into too much detail, imagine you’d like to combine the rules of physics inside the (high-dimensional) image generation process. This could be useful for video generation where, given previous frames, the task is to generate the next frame. Now, you might find that your model is very accurate in producing high quality frames but that they defy the rules of gravity (objects are flying around) because you didn’t capture those rules. Now, if I know the rules of gravity and want to incorporate them in the training loop or even impose them on a pre-trained model that’s something that EBMs would do well and in an easier way than normalized probabilities.

Similarly, and something we’ve worked on recently, is generate programming languages using a model which helps you write code (1, 2).  Well again, the fact that language models can’t plan ahead and that they can make mistakes mean you might generate code that can’t be compiled. Using an EBM you could impose this compilability constraint to make the compiler and the language model collaborate together to produce good code. Again, this is easy to model with an EBM and hard with a probabilistic model.

Host: Are EBMs generally applicable in applying prior knowledge to machine learning models?

Hady Elsahar: I’m giving these examples of prior knowledge because these are the types of problems we’re currently interested in and working on but the applications of EBMs are limitless with different applications appearing every day and in many fields. For example, one task that seemed to be useful to model using EBMs is ‘Continual Learning’ (3). Continual learning is a concept where you learn a model on a sequence of tasks but at ‘Task 1’ you don’t see what’s coming after. So, if you ‘classify cats and dogs’ and the next task is ‘classify cows and horses’ and so on well, if you remember what I said earlier about listing all the categories and normalizing over them, well when you’re classifying cats and dogs you don’t see cows and horses and when you’re classifying cows and horses you don’t see cats and dogs. This leads to problems like forgetting what you learned before so normalizing on all these categories might not be straightforward. By freeing yourself from these normalized probabilities, EBMs give you more flexibility in thinking of ways to solve the problem.

Host: Being able to explore more ideas and their applicability to many fields explains why EBMs are popular but can you tell me why they’re popular now when they’ve actually been around for nearly 20 years?

Marc Dymetman: EBMs attracted a lot of attention around 2000, particularly in the vision community but back then neural networks were not as developed as today and computing power was insufficient for the large simulations often required by EBMs. In the following 15 years or so, neural networks were developed very rapidly yet this progress was mostly orthogonal to EBMs. In fact the attention of most researchers was focused on techniques such as CNN architectures and developing them in the field of computer vision, or autoregressive models in natural language processing (NLP) which were very effective.

The recent comeback of EBMs is partly due to the increase in feasibility of certain approaches. For instance, in continuous domains such as computer vision, advanced techniques which capitalize on the differentiability of the energy  (aka ‘score’) relative to the image have become more practical due to progress both in terms of algorithms and in terms of computing infrastructure.

Host: Is it for the same reason as in computer vision that EBMs are now popular in the field of natural language processing (NLP)? 

Marc Dymetman: In the case of NLP the interest came later. Our interest for EBMs here at NAVER LABS  Europe started a few years ago with the realization that they could help resolve certain limitations of standard neural training techniques relative to global properties of texts. As Hady explained earlier, standard techniques in text focus on predicting the next word in a way that imitates existing text at a very local level. It’s a bit like someone who is walking being entirely focussed on where to place their foot for the next step as opposed to concentrating on the entire trajectory of where they’re going. By observing properties of the whole trajectory you can better generalize from the training data; we were working on pretty synthetic toy data at the time – but some of the important ideas were there (4).

Host: you recently organized a workshop on EBMs at ICLR-2021 in May this year with colleagues at Facebook AI Research and Google Brain. What were some of the highlights of the workshop?

Hady Elsahar: At the EBM workshop (5) we had 4 invited talks, a poster session, contributed talks and a debate. The invited talks were personally very informative for me. We had for example Yann LeCun from Facebook AI Research who talked about converting all self-supervised learning tasks into the Energy Based Model framework. Yingzhen Li from Imperial College London talked about Inference from Energy Based Models (one of the challenges of EBS) in a very interesting way framed as a personal story including all the failures, challenges and successes she encountered. Stefano Ermon from Stanford University talked about score-based generative models. These work with gradients with respect to the input dimensions rather than gradients with respect to model parameters.  These have recently been successful, especially with deep Energy Based Models.

The last speaker was Benjamin Scellier from Mila in Montreal and he talked about EBMs and physics.
EBMs are inspired by physics, hence the term ‘energy’ but this is just an analogy because energy in EBMs mean ‘a quality score’. Benjamin gave an interesting talk about his work with Yoshua Bengio about how to use the laws of physics to minimize energy – ‘energy’ here being more related to physical energy. And, as far as I understood, that would be very useful because you could rely on physics to optimize your machine learning model and it could even yield better hardware.

The accepted papers yielded a very interactive poster session and the 4 contributed talks covered a range of topics from better techniques for training EBMs to more EBM applications such as the biomedical domain, Molecular Graph Generation and, as I said earlier, Continual Learning.

Host: I understand there was a panel during the workshop which created some debate on probabilistic vs non-probabilistic EBMs. Can you explain why?

Marc Dymetman: That was one of the most exciting parts of the workshop for me-  this lively debate about probabilistic versus non-probabilistic views of EBMs. Everybody agrees that EBMs do not need to be described in terms of probability distributions, as we explained earlier, but many researchers still use probabilities to implement EBMs. A strong dissenter with this view during the debate was Yann LeCun from Facebook and NYU, who advocated a radical non-probabilistic approach even at the implementation level. He argued that doing so avoided many difficult problems, such as computing intractable normalization constants (as Hady explained earlier), and that a non-probabilistic view opened the way to effective new architectures, even if the mathematical foundations were not yet as well established as with probabilities.

Host: So he favoured the expressivity that EBMs allow for. What were the views that differed from that which fired the debate?

Marc Dymetman: Will Grathwohl, from the University of Toronto and Yingzhen Li, from Imperial College, defended the probabilistic approach to implementation, by insisting on its formal consistency, its very nice mathematical properties, and also, on the practical side, the recent advances in inference and learning techniques addressing tractability concerns. Hady has mentioned some of these advances in terms of score matching and other things.

Overall, the exchange greatly helped in clarifying the strong and weak points of each position, and this debate should be useful for the EBM community as a whole going forward.

Host: You are both researchers in Natural Language Processing. How did you personally come to be interested in the topic of Energy Based Models?

Marc Dymetman: Maybe I can start. My interest for some approaches that are strongly related to EBMs  – like Maximum Entropy Models, Graphical Probability Models, Weighted Automata – all these things are very much related and you could see EBMs as a convergence of these techniques with neural models in a way. And this interest goes back a very long time. However, my direct involvement with EBMs proper started less than three years ago. As I mentioned earlier, some colleagues and I noticed how EBMs could be used to exploit global properties in standard neural models for language modelling and also how some ideas from reinforcement learning could be transposed to such models. Hady became interested in this work and saw potential connections with his own interests in controlled text generation. Together with another colleague, we investigated the use of EBMS for controlling certain properties of text generation models, in particular the avoidance of social bias, an important and timely problem for many companies, including NAVER.

Host: Bias is indeed a very hot topic in AI.  Please explain how EBMs are applicable to bias in text generation.

Hady Elsahar: Bias in text generation is a trendy topic because, basically these large pre-trained models are trained on large uncurated data from the web. You could think of un-biasing or de-biasing these models as a form of controlled text generation which is something I work on. After some research, I found that, because of what we said earlier about these language models not looking ahead a lot and being narrowly focused, that it wasn’t possible to impose global constraints like sequence level constraints and it was in general very limiting working with probabilities. Along with many other researchers I was having to compromise on the types of constraints or optimizing a non-optimal objective. The potential we saw in EBMs is they could really free you so that you could model and focus on the right questions.

Host: Have you had some success with EBMs in text generation?

Hady Elsahar: Yes great success in fact! I’ll give you an example about de-biasing language models – something it wasn’t possible to even think of before. De-biasing is a bit different from toxicity. In the latter you want to eliminate – you don’t want to see toxic outputs of your model. That’s easy as you focus on each output and you just eliminate it (with point-wise constraints). De-biasing is rather challenging because it works at what we call the ‘distributional level’. An example is that we trained GPT-2 on wikipedias and biographies from wikipedias and, because of the gender imbalance, especially in some categories like scientists, when you sample from GPT2 you get only 7% output containing the pronouns ‘she’ or ‘her’ (meaning that it’s a female biography). Now, if we would like to impose what we call ‘distributional constraint’ on GPT-2 to balance this gender distribution to produce 30% female, 30% male and 30% ‘other’  – thinking of how to even model this problem is very limiting with a probabilistic model. With EBMs you’re freed from that so, for example, we created this general framework called ‘GDC’ – Generation with Distributional Control (6) – and it’s very flexible with types of constraints so you can not only do point-wise constraints and filter toxicity, but you can also de-bias with respect to anything and can even write, for example, the exact percentage of the features that you want to see.
We also worked on other applications like compilable text generation and we’re exploring lots of other ideas at the moment.

Host: What you just described in controlling text generation and reducing bias is recent work. Is this something you’re going to be continuing or will you be turning your attention and the use of EBMs to other areas in the future?

Marc Dymetman: You can tell Hady and I are very much convinced about the potential of this whole line of research and have been successful in attracting more colleagues to collaborate with us. We have been investigating in more depth the fine-tuning of EBMs for text generation and connections to reinforcement learning, and are also moving towards “conditional models” such as EBMs for Neural Machine Translation. This can address well-known issues such as gender clichés: for example, a tendency of translation models trained on bilingual data  to systematically translate an English word such as “nurse” into the French feminine “infirmière” and not the masculine ‘infirmier’ – showing strong bias in these models.

Another line of research focusses on situations where responses to dynamically changing conditions need to be generated in real time without the luxury of retraining the EBM models.

Host: Can you give an example of dynamically changing conditions?

Marc Dymetman: There are many applications. Let me give you two examples. The first is searching for facts, in real time, in the implicit “knowledge base” of large pre-trained language models such as GPT-3. Another example is providing authoring tips to writers, to help them better reformulate something they’ve written using these large pre-trained models. These conditions can’t be predicted in advance and depend on the needs of the user at a specific point in time. This is ‘dynamicity’.

There are many more opportunities for EBMs going forward, especially in NLP, which has been less explored than vision up to now and has its own specific challenges, such as the discrete nature of text data which is quite different from the continuous nature of vision data. For discrete models, and in NLP, this involves questions of non-differentiability which make certain advanced techniques developed for vision difficult to use at this time.

On the other hand, the sequential nature of text based models has deep relations to the sequential nature of many decision processes, which makes for intriguing connections to reinforcement learning techniques in robotics for instance, a domain of great interest to NAVER LABS. Overall, this is a great domain to be working on right now and we’re very excited about the whole thing!

References

1: Energy-based models for code generation under compilability constraints.  Tomasz Korbak, Hady Elsahar, Marc Dymetman, German Kruszewski, 1^st Workshop on Natural Language Processing for Programming, ACL 2021. arXiv:2106.04985.

2:  https://copilot.github.com/

3: Continual learning of visual representations without catastrophic forgetting. Riccardo Volpi, Diane Larlus, Gregory Rogez, 2021 (blog).

4: Global Autoregressive Models for Data-Efficient Sequence Learning. Tetiana Parshakova, Jean-Marc Andreoli, Marc Dymetman, arXiv:1909.07063, 2019.

5: Workshop on Energy Based Models: Current Perspectives, Challenges and Opportunities, ICRL, May 7^thv2021 [Slideslive recording].

6: A distributional approach to controlled text generation. Hady Elsahar, Marc Dymetman, Muhammad Khalifa, 2021 (blog).