AI and democracy – AI could make everyone’s voice heard

Many of us have freedom of speech and can easily publish our opinions online, but how many of us feel that our voice is heard? Unless you’re an influencer, the majority of blog posts are only read by a few friends, and voting in elections or giving star ratings to restaurants are not exactly rich in self-expression. This means there’s a fundamental asymmetry in a society where the opinions of only a few are heard by everyone, but where no one can possibly listen to everyone’s opinions.

But computers can. What’s to stop them from telling us what everyone out there is saying? What would it mean? Well, if a computer could automatically summarise everyone’s opinions and broadcast that summary to all, then each of us would be able to see that our opinion had been heard. This is something on the not so distant horizon that citizens will come to expect, as well as customers, employees and members of any kind of organisation.

At XRCE, we’re working on Artificial Intelligence software that can understand and summarise opinions. It includes specific technology for understanding sentiments and aspects of opinions, and generic deep learning architectures for understanding text. But the area where the most progress is needed is how to reduce the vast diversity of individual opinions into a comprehensible summary.

The core technology missing to be able to do this is ‘abstraction’. We need to be able to abstract away from the diverse details of individual opinions and find the consensus amongst large groups. For example, if one person says “Universal health care systems have lower long term health costs as they encourage patients to seek preventative care” and someone else says “Public health insurance is less costly than private insurance to the overall economy” we want to know they agree that “Public healthcare is less expensive”.

In natural language processing, abstraction is known as ‘textual entailment’. A statement ‘y’ entails a statement ‘x’ if ‘x’ is an abstraction of ‘y’, meaning all the information in ‘x’ is also in ‘y’. Any information in ‘y’ which is not in ‘x’ is the information being abstracted away. For example, “Public healthcare is less expensive” abstracts away from the claim that public healthcare encourages patients to seek preventative care.

Our recent work on entailment in text has focussed on abstraction in the meaning of words. The meaning of words is a core issue in natural language understanding because they are the fundamental building blocks of the meaning of text and because there are so many words and their meanings are so hard to define.

One very successful approach to the meaning of words has been distributional semantics. This hypothesises that you can infer the meaning of a word by looking at the distribution of other words which appear near it in a very large corpus of text. This distribution is then compressed into a vector of real numbers, such that words with similar distributions, and therefore similar meanings, also have similar vectors. These vectors have played a key role in the success of deep learning in language understanding.

In the European labs, we’ve developed a version of distributional semantics to model abstraction in the meaning of words[i].  We first developed a new deep learning architecture based on entailment between vectors, as opposed to similarity between vectors. We then used these entailment vectors to define a new model of distributional semantics, and trained new entailment vectors for words. The resulting model of abstraction in words gives the best published results on predicting hyponym-hypernym word pairs, such as “cat” (hyponym) entails “animal” (hypernym).

Standard deep learning architectures are based on the dot product between vectors, which measures the symmetric similarity of the two vectors. We propose vector operators which measure the asymmetric inclusion of one vector within another. This is derived from an interpretation of vectors in terms of things we know; if the things we know given vector ‘x’ are included in the things we know given vector ‘y’, then ‘y’ entails ‘x’, and the operator will give a high score. This framework also allows one vector to be calculated from other vectors which it entails or is entailed by. By specifying networks of vectors which entail each other and performing these vector calculations, we get a new form of deep learning architecture where the vectors and the structure of the model have a clear interpretation in terms of entailment.

The intuition behind our distributional semantic model is that words which occur together in a text should (on average) be consistent and redundant with each other. For example, in “furry cat”, there is nothing inconsistent between something being both furry and a cat, and both these concepts share the property of having fur. We model this with a new vector ‘y’ which is the unification of the vectors for the two words, meaning that, in this example, ‘y’ must entail both “furry” and “cat”. If these entailments have good scores and ‘y’ has a high probability, then the words are consistent and redundant. By training the word vectors so that this score reflects the distributions of word co-occurrences in a very large corpus of text, we get vectors which reflect what we know given a word.

To evaluate these vectors, we used data on abstraction relationships between words, called hyponymy. A hyponym-hypernym pair, like the “cat-animal” example given above, is one where the hypernym is entailed by the hyponym. In this example, knowing something is a cat includes knowing it is an animal. Just with the original vectors learned on text, we reach 70% accuracy on classifying word pairs for hyponymy, and if we train a mapping to select the features of the vectors which are relevant to hyponymy, then we reach 86% accuracy on new words. This beats any other published results so far.

Right now, we’re extending this model of entailment between words to a deep-learning model of entailment between sentences, which we expect to have within the year. That’s the critical piece of Artificial Intelligence technology we need to then model abstraction in opinions, and to compress very large numbers of opinions into short comprehensible summaries. Check the XRCE blog for updates on our progress. Once we have this summarisation technology, we’re optimistic that everyone’s right to be heard will quickly follow.

Author of this article: James Henderson who was part of NAVER LABS Europe until September 2017. 

——————————–

[i] Henderson, J. and Popa, D., 2016 “A Vector Space for Distributional Semantics for Entailment’, Proceedings of the 54^th Annual Meeting of the Association of Computational Linguistics, pp 2052-2062, Berlin, Germany, August 7-12, 2016. [PDF]

Listen to the author on how social mass communication will change society: