NAVER LABS AI research in Europe – a podcast with lab manager Matthias Gallé

Edited transcript

Matthias Gallé: There are a number of ways of answering that question. All our projects have three key ingredients: they have a team, they have a challenging problem for NAVER and they have a research challenge that has as yet not been solved. So, to answer your question let me first answer it by telling you what kind of problems we work on which have relevance for NAVER.

We have two main missions as a lab. One is to reinforce NAVER’s current leading position in search and online services and the second one is to establish NAVER LABS Europe as a worldwide leader in AI for Robotics.

For examples of this first mission, we recently deployed new models that provide you with a more diverse set of pictures when you do image search. So now, when you search for “Korean barbecue”, instead of having dozens and dozens of images of the same round grill with meat, you now get much more varied images from your search. In a similar way, we’ve contributed to translation models that integrate recent advances in learning from monolingual data instead of only from bilingual data.

Concerning the second mission and as you mention, NAVER, like many other digital companies are making several bets for the future. One of those bets is in robotics and we are collaborating with our sister lab in Korea to embed robots with advanced AI capabilities. We’re leveraging our network to collaborate with top academic places world-wide and, something that’s a bit special that we’ve done there, is to release a library called kapture (with a ‘k’) that provides a unified format to describe Structure-from-Motion data and which allows fast experimentation for localization and reconstruction pipelines so you can take still images and reconstruct 3D scenes of where these images were taken. kapture is being used not only by us and by our partners, but in other institutions across the world and is behind the recent localization challenge at CVPR.

Matthias Gallé: So, that’s actually one of those instances when the work from home situation actually helped because all of a sudden we’re all doing videoconferencing and it doesn’t really make much difference where you’re located. Now, there are challenges because of the time difference as you said, 8 hours – and cultural differences whereby things are done a bit different.

Matthias Gallé: I believe we have the possibility of combining several great characteristics from both places. Korea is a fast-moving culture so, as a consequence, NAVER is a fast-moving company. You might have heard of this expression ‘pali-pali’ which means to ‘move quickly’ or ‘faster’. One quote that gets cited quite often around the place here is that ‘all internet companies are one click away from bankruptcy’ meaning that we have to reinvent ourselves all the time to remain relevant.  So, this calls for lots of innovation and we have to make sure we’re at the forefront of what’s going on in the research community.  This means to say that we can’t assume our current position as a digital leader is a given. A bet on robotics, like the one you mentioned, is very much in line with the idea of reinventing the company all the time and this is of course exciting for a research lab like ours.

At the same time, being in France, we can benefit from a long-standing tradition of doing solid and long-term research where there are universities with extremely good mathematical training which helps a lot in our field. Korea brings a lot of technological progress, and it’s one of the countries that has most deployed 5G which will revolutionize the way that brainless robots will work – robots that are not being controlled on the robot but controlled remotely. And then again, from the European side, what we can bring to the table is extensive multi-culturalism which is the norm here. English is our lingua franca but we’re very proud that in the lab we can count 25, no it’s actually 26 different nationalities (out of just over 100 people) not only from Europe but also from the Americas, from Africa and even from Asia.

Matthias Gallé: Yes. Of course having the different languages helps in this specific field, but having different viewpoints on how to solve a problem is key to providing new insights.

Matthias Gallé: This is something that is very close to my heart because, as an industrial lab, we are guided by the applications and the impact they can have for NAVER. But everything we do is grounded in research so every team has their own research agenda, where they spell out what they believe are the important trends in the coming years they would like to work on, and where the solutions can have an impact on the company. Reviewing these, we discovered there are a few underlying trends which pop out in several of the research agendas.

One which is well studied but still a problem is something called “frugal learning”. Current machine learning techniques are extremely hungry in that they need to consume lots of annotated data in order to be able to solve a specific task you want them to do. This also means that they need to process all this data so, in turn, they need lots of computational power which entails a huge consumption of energy. So, the problem gets worse the more complex the task you want to solve because you need more data and so on. For instance, in natural language processing which is my own background, if you want to solve a problem called ‘named entity recognition’ which is the problem of extracting names and organizations from text, a training dataset has maybe 5000 sentences with which you can get good performance. If you want to do ‘question answering’, which is a system that can answer natural language questions, you need a couple of hundreds of thousands of examples. And if you want to do machine translation, so if you want to translate from Korean to English, you will need at least a few million parallel sentences. So, the obvious question is how can we do more, with less data or with less computational power?

Let me give you a few examples of the things we’re working on to tackle this specific problem. One is a direction which is called ‘self-supervised learning’. It’s a problem defined as trying to extract a training signal out of data which has no annotation. This data is called unsupervised because there has not been any supervision by humans, but you want to extract some kind of supervised signal so that you can feed the unsupervised data and the supervised signal into your standard supervised machine learning. Now, in computer vision, one of things that we’re concerned about is learning good mathematical representations of images and, if you have a good vector representation of an image, you can feed this into an algorithm to perform some kind of final task like classification or retrieval. So, we’ve done some work in supervised learning where we provide a learning signal – something that we call a contrastive loss – where we take one image and modify it by either shifting the angle or changing the colours. We want the two representations of these two different images to be very close to each other and, at the same time, be very far apart from the representation of some random image. So, to achieve this what we do is we create artificial synthetic, negative data and we’ve shown that this ‘hard negative data’ can help us improve the accuracy of out-of-domain data by a significant margin over state of the art baselines.

Matthias Gallé: The problem of mining hard examples is something that’s been studied by a number of people. Our contribution to this is the problem of creating synthetic hard examples by interpolating between some which already exist.

This idea of self-supervision also comes into play in a totally different field which is the one of text summarization. So, we have a problem where we want to summarize reviews of a place or of a product and, instead of reading hundreds and hundreds of reviews, you’d like a short summary. So, we assume that we don’t have any supervised signals, so we take dozens of reviews of one product and we just assume that one of them is a summary of the others. So, we create ‘artificial supervised data points’ like this which we then can feed into a traditional supervised summarization model.

A very different way of learning with less annotated data is by leveraging information that is not there in the dataset in itself but which comes for free. Another project related to this learning representations of images is to leverage the caption that comes together with images that we find on the internet. So, what we do is leverage the text modality to improve the representation of the image and we can use this new representation to perform much better on downstream tasks like ranking, retrieval or classification.

Matthias Gallé: It’s closely related because one of the key problems of a robot is that it needs to know exactly where in the world it is at any given time. One way of solving this problem is with what we call visual localization where you take a picture of a place then you go back to your database where you have taken lots of pictures of the same place and use these (with an algorithm) to know where exactly you’re currently located. The library I mentioned earlier called ‘kapture’ is useful here because it lets you test many different algorithms in a very quick way.

Matthias Gallé: Yes, and there are a number of ways of solving that. One way is this idea of brainless robots where the brain of the robot is not on top of the robot so it doesn’t have to go around with this huge computational server. The computational server is far away and you just communicate with it. This is where the promise of 5G will revolutionize the number of things we can do with our robots because now the amount of information that can fly between the robot and the server is much quicker.

Another way of doing this is trying to have the same model running on an edge device i.e. one which doesn’t sit in your data center. Some of the work we’re doing in this field is pruning neural networks because one of the problems of neural networks is that they have millions of parameters and you need lots of memory to store them.  So, there are methods to remove some of the nodes in these networks to reduce the memory consumption. We propose a model called ‘Force’ which allows us to prune up to 99% of the nodes. We retrain the remaining nodes and, as the network is now much smaller than the original one whilst maintaining the same level of performance, it can fit in a device which doesn’t have the capacity of a server.

Matthias Gallé: We do indeed have a number of robots. We have commercial robots we test and we have prototypes which are developed in Korea that our colleagues have sent over.

Matthias Gallé: There are a number of challenges where we try to improve their intelligence. One is localization, which I already mentioned, and the other is visualization and how to navigate around a place. This is where we have an example of a second trend appearing in the research we’re doing  which is how to control a robot which has a learned model that is extremely powerful. Let me give you an example. When you have small kids, you want them to absorb and learn lots of different things but once they become older, you realise you need to control them a bit so that they don’t make too many stupid mistakes.

Matthias Gallé: Well you need to leave them with a certain amount of liberty! The problem we’re facing these days is that there are extremely powerful agents behind existing digital services or robots that we can’t just freely deploy without having some kind of control over them.

I mentioned the example of the robot and the way that those navigational policies and rules which control it are learned, is by trying to optimize the average reward of actually reaching the target so, let’s assume you have a 200kg robot and you want it reach your living-room. You run the experiment a number of times giving your robot a reward and in the end you just take the average of how many times it reached its destination and then you improve your policy.

Matthias Gallé: It’s just a score, it’s a carrot, it’s something which tells you that you did a very good job. Now, if for instance on the way to its destination the robot knocks over a stroller with a (fake) baby in it you give it a negative reward. However, if you just take the average and use it to improve the policy it might still knock over the stroller with the baby because, on average, it still reached the target a lot. Well, that’s obviously not what you want. You need to make sure that under absolutely no circumstances whatsoever it can do any harm to anyone on the way and by just optimizing with an average reward you can’t ensure that. What you need is some kind a worst-case analysis to make sure your policy is learned in a risk–sensitive way which requires and entirely different set of techniques.

Matthias Gallé: Right, and in traditional computer science the way that algorithms were analysed is the worst-case scenario, so you want that in the worst-case it runs in a certain time. Machine learning algorithms are optimised in an ‘average’ way meaning that, on average, you want them to perform very well. The problem is that you don’t penalise enough those worst-case scenarios and we realise that they do happen and they happen a lot more than we would like. They can result in physical harm as in the case of the 200kg robot but there are different kinds of harm with digital services where there is not physical contact. An example is in natural language where there has been huge progress made in the last few years with models that can generate text that is extremely similar to text written by humans.

Matthias Gallé: Yes, right, which are being used for instance to create fictional stories which is fun and for chat bots, which is a bit less fun because you don’t really know what they are saying.  So, assume you want your language model to generate fictional stories and you want to make sure that your story doesn’t include only male protagonists because you care very deeply about that. You want to have 50% male and 50% female protagonists. With current models, you can’t do these kinds of things. You can have some kind of pointwise control so, given one text, you can say ‘I like it’ or ‘I don’t like it, please improve’ – but you can’t tell it ‘I want you to start generating different texts and that in that distribution, I want there to be 50% male and 50% female protagonists’.

So, we’re proposing a new model to do these kinds of distributional control where you can take a language model and you can combine this with sampling techniques to make sure that every time you sample from this model you approach a distribution which is the one that you want.

Matthias Gallé: Well if you start producing only stories with a 100% of males there will be different consequences. That may not the ‘worst-case’ but we have also seen many examples of racism and bias encoded in existing language models.

Matthias Gallé: This is something that really excites us because it allows us to work on not only the robots that are in our robotics space and in our very reduced simulation environment. We have the opportunity of deploying our algorithms on top of robots which are going to be used, and as you said, it’s the first time that such a tall building be constructed from the beginning having in mind that there will be robots there. We’re working very tightly with the people developing robots specifically for this building on the visualization and navigation aspects that I mentioned earlier. We also have teams on Korea and France that are closely connected on studying human robot interaction and how robots should interact in a way that doesn’t scare people and where the robots actually help them.

Matthias Gallé: Yes, we have a dedicated team who analyse the interaction between humans and AI agents at large – these can be embodied as in the case of robots or not. I am convinced that a lab of our size in the times that we’re living needs a dedicated team of people studying these types of interactions to make sure not only that obtain better performance but also that the AI agents perform in a way that is not harmful to users.

Matthias Gallé: The pandemic was beyond doubt challenging for us as it was for everyone not only because of the sanitary conditions but also because of having to suddenly start working remotely. Now I believe having to work remotely has actually created a number of very interesting opportunities which I hope will continue into the future. One of these is that we were able to continue with our internship programme even with the work from home situation and that’s not something which many companies were able to do. We have a great support team who put everything in place. We also have a very lively series of seminars where almost every week, even more sometimes, we have an external speaker and this situation led us to opening up the event to everyone so people beyond the lab can dial in from wherever they are and I hope we can continue to leverage those ways of reaching people beyond the area. We have a lot of learning to do in that I recently co-organised a winter school which was supposed to take place in the middle of the mountains and which was virtual and we had some nice ideas and some things worked well but there a lot of challenges in how to organise these events in a purely virtual way.