Natural Language Processing (‘NLP’) has become a buzzword. Not only to those in academia but to the wider world too. In particular, smartphone users are no strangers to autocorrection and text prediction nowadays. These are just two familiar applications of NLP which make the daily lives of billions of people easier. We take them for granted but one may wonder: how do they actually work?
The answer lies within more buzzwords: Machine Learning (‘ML’) and Neural Network (‘NN’). Autocorrection and prediction occur while we type, and the suggestions change as we type more and more. But what happens exactly?
Obviously, the machine does not understand what each word means, unlike humans. But machines are good at executing things per specific instructions, like checking on patterns of usage. For example, it can spot which words are often used together and memorize such patterns.
But now the texting world has not only text to deal with but also this invention called emojis. Which makes the whole text prediction technology even more challenging, because the usage pattern is less trivial: a smiley face can mean one is truly glad or, sarcastically, ‘thanks but no thanks!’.
How do we at Typewise build our prediction AI to predict your next emoji?
Keyword Lookup
Remember when you type ‘Fire’ and the emoji ‘🔥’ shows up on the suggestion bar? This is the most basic yet most effective approach towards emoji prediction. Each emoji is in fact explained by several annotations. You can view these on the emoji database from the emoji creator, Unicode. Emoji ‘🔥’ has the annotation ‘fire | flame | tool’, so the keyword ‘fire’ is looked up and found there.
Neural Network Approach
But when we type ‘Lit’, the emoji ‘🔥’ also shows up (and correctly!) although ‘Lit’ is not a keyword associated with ‘🔥’ in the database. Instead, the power of the ‘Neural Network’ comes into play.
When we learn something, the brain propagates the just-learned knowledge through its neural network, keeping the knowledge in memory, and reusing it later when triggered. This is exactly how neural networks in computers work. The only difference lies in that the ‘knowledge’ is no longer a faceless abstract concept; in a computer, it is real numbers. Like 0, 1, 0.375.
The process called the pre-processing step first turns the input text (‘knowledge’) into individual words and emojis (known as ‘tokens’) and assigns them numerical labels to be understood by the neural network. A ‘probability’ is assigned to each and every emoji, and the one with the highest chance is picked.
But how are these probabilities calculated? This is the neural network’s work. Basically, it does a lot of mathematical computations, from the first randomly-assigned probabilities, then where the neural is connected or joined, it does some addition, multiplication, or scaling to ‘update’ the probabilities for each emoji again. It is like when you know that an egg is cheaper than a chocolate bar and someone tells you that a chocolate bar is cheaper than a chicken, then jointly you know an egg is cheaper than a chicken. The difference is that here it uses mathematics.
Afterward, you let the neural network ‘propagate’ such a ‘learning’ process, and after some time, you get the updated probabilities of emojis and pick the most likely one in any given scenario.
This is how our text prediction engine is able to predict the emoji users want to use. You can see a live example of the API code in action here.
Creating A Personalized Solution
If you are the sort who is always nice and kind in your texts (😊), you do not want the smiley face emoji to get suggested in any sarcastic ways. This is where the personalized solution must kick in.
The personalized solution is even more important for emoji prediction than text prediction. With the writing skills we learn at school, we use most words in the way we are taught, thereby following a known and accepted structure based on meaning and context. But for emoji usage, there are no such rules so consequently, people tend to be more creative with the way they use them; recall the smiley face example. So, a personalized solution helps with emoji prediction, as shown in Google’s work.
So, what do we do with the neural network we have already trained? It is like updating your knowledge: last year your mum said she likes chocolate on top of ice cream and this year she is being more specific, saying she likes only vanilla ice cream. You still know from last year that she likes chocolate on top (and this is unchanged) but now only with vanilla ice cream, not just any ice cream. In short, you remember the new information and add it to the parts of the old information that remain unchanged.
With a neural network, it works exactly like this. One first has the neural network trained aggregately on texts from thousands or millions of people. As users type more and more, the model re-learns with new texts and updates the numbers, then gets the updated probabilities for the emojis.
Hidden Caveats and Last Words
Emoji prediction sounds very simple, so you may wonder why even big tech companies like Google and Apple need to make so much effort in order to just make it work. Well, it’s actually a very complex situation.
In order to train the neural network, just like when we learn something, one needs a lot of data. In this case a lot of texts with emojis that people use. And not just use, but really use when they talk to friends and family. Not to their boss. Not in a book. But in everyday real text communication. Because we would not want to get old-fashioned Shakespearean words predicted when talking to friends, especially with emojis 😊. (Unless you are a Shakespearean type! Then the neural network will eventually learn that from you and start suggesting it!).
