Natural Language Processing: The Secret Sauce

We’ve all seen the numbers in one form or another: 90% of the world’s data was created in the last two years; 30 billion pieces of content are shared on Facebook daily; an estimated 8 zettabytes of data will be generated in 2015. Great! With so much information available, times have never been better for making informed, data-driven decisions, right? Not if you don’t have the right tools to make sense of it all.

Enter natural language processing. It has been around since the 1950’s, but as the age of Big Data has gained momentum, so has activity and interest in the field. Today, there are a host of tools available that collect huge sets of data from social media outlets, blogs, news sites, and other online sources of conversational information. These tools can perform deep analysis on the text, identifying themes and gauging sentiment, all with a high degree of accuracy and without much help from human users. Now brands can rapidly distill an enormous mess of unstructured data into actionable information about their product or service.

Just like IBM’s chess grandmaster-humbling Deep Blue and the same company’s Jeopardy champion Watson computing platform, it can seem that NLP applications have been magically granted a mind of their own. However, IBM is no Hogwarts. There is hard NLP and machine learning science that social listening and analysis tools stand on, and much of it is easier to understand at a high level than you might think. In this post, we’ll take a brief look into some of the techniques that these tools leverage to spark insight.

iPhones may or may not be awesome

One of the common functions of social listening platforms nowadays is sentiment analysis. The tools can observe a collection of textual data about a topic and display to the user how the authors generally feel about the topic, often expressed as a percentage value of positive or negative sentiment. But how does it work? In the early days of NLP, many analytical methods were based almost entirely on word frequency, what we would call a lexical approach. It is a very simplistic but not altogether incorrect solution: It certainly stands to reason that words that occur most often in a particular body of text more often than they do in general could be considered the most important for that body. However, what if a major concept in the text is better described by a phrase as opposed to a single word? How can we know what attitude the authors of a collection of tweets generally have toward our topics of interest? “iPhone” and “awesome” might end up being two of the most frequent in those tweets, but that doesn’t necessarily mean that those saying “awesome” were talking about the iPhone at all! The point is this: effective text analysis needs to consider context.

Of course, a computer program doesn’t have all of the context clues that humans do in determining emotion: They don’t get body language or facial expressions and they pick up on sarcasm about as often as Cleveland celebrates a professional sports championship. However, from a purely textual standpoint, modern social listening tools function in much the same way as humans. They use syntax, or sentence structure, to give them some context.

Take, for example, the sentence, “I don’t think oranges are very good, but apples are amazing!” A way to go about determining the sentiments expressed in the sentence might be to make a couple lists of words that suggest a positive or negative sentiment and tell the computer to determine the sentiment of the sentence by comparing the words within to those on our list. In this case, the computer would likely see “good” and “amazing” and tell us that this is an indisputably positive sentence. If we were doing an analysis on how people feel about apples, our program would be in luck since the author does indeed speak favorably of apples. However, if we are concerned about oranges, then our program will get it completely wrong here.

Modern tools take a more sophisticated approach to this problem. They observe not just what words occur, but also where the words occur in the sentence and how they are connected with the other words. I’ll spare the details of how programs specifically implement this idea, but the general process goes like this:

  1. The analysis tool reads a body of text, sentence by sentence.
  2. The structure of each sentence is determined by identifying parts of speech, figuring out which adjectives modify which nouns, considering negations, etc. (This part leverages machine learning techniques. The algorithms that do this work are trained on a corpus of text that helps them to identify patterns in sentence structure. The algorithms can then apply these patterns to new pieces of text in order to make probabilistic best guesses at the structure and meaning of a sentence.)
  3. The program keeps a running tally of what it sees on a per-sentence basis.
  4. Additional analysis is conducted on the aggregate data to provide users with meaningful metrics about the text.

You’ll notice that I mentioned in step two how analysis algorithms make a “probabilistic best guess” at how the sentence is structured and what sentiments it expresses. By definition, this means that the algorithms are not perfect. However, the algorithms in social listening and analysis tools have been fine-tuned to be accurate 85% of the time or more! Pretty good batting average if you ask me.

If we tell our program to apply this syntactical analysis to our sentence comparing apples and oranges, it will be more accurately identify what’s going on: It now has the ability to tell us that the sentence expresses a positive sentiment toward apples, but a negative one toward oranges. Among other advantages, this powerful idea enables brands to get an on-demand, accurate view of how consumers feel about their product.

It’s a means, not an end

NLP applications are surely a valuable tool. It would be utterly impossible for people to read through every bit of content that is created on the web each day. However, people are still essential for making sense of the distilled information that these tools put at our fingertips. It is beyond the reach of a computer to be given a problem and to know what kind of content of look for or how to connect the dots between what the data says and what actions that it should prompt. The experience and intuition of social media experts are key to turning the information into actionable insight about about a product or brand.