NLP or NLU: How to Choose the Best Option for Your Project

In a previous post we talked about how organizations can benefit from machine learning (especially natural language processing) without making a big investment. Now we'll delve deeper into natural language processing (NLP), explain the differences between NLP and natural language understanding (NLU), and offer some tips for choosing the best solution for your company.

NLP vs NLU

As we explained in a recent post, NLP is a subfield of artificial intelligence that focuses on natural (human) language processing. However, we have yet to discuss what exactly "natural language processing" means. Let’s talk a little about it now.

As the name suggests, NLP focuses on processing natural language. What is not clear directly from the definition is the level of processing that you can apply. Analyzing a natural language sentence from a linguistic perspective can be done in different ways – syntactically, grammatically, semantically, among others. In the same way, NLP can execute processes at different levels. Some NLP tasks that reflect the range of levels in NLP are:

• Name Entity Recognition (NER)

• Parts of Speech Tagging (POS Tagging)

• Syntactic Parsing

• Semantic Parsing

• Question Answering

• Semantic Analysis

• Automatic-Summarization

From this list, the first three tasks (NER, POS tagging and syntactic parsing) are syntactic processes, while the last four tasks are semantic.The complexity and mechanisms needed to resolve each one of those problems vary. The syntactic problems are easier to solve, and there are a lot of mechanisms and algorithms (some of them pretty old) to deal with them. Most of them are easy to define and you can implement an acceptable solution without applying machine learning.

On the other hand, semantic problems are more complex to solve. The main reason for this is that defining semantic concepts is not trivial, and there are usually discrepancies in how different humans define them. Moreover, resolving a semantic problem involves understanding what a sentence means. This task looks simple to humans; however, it's very complex for a computer. Trying to solve a semantic problem without using machine learning algorithms usually gives poor results in terms of precision or recall.

Now that we have defined the scope of NLP and the diversity of problems that are included in it, we can define and explain the concept of natural language understanding (NLU). NLU is a subfield within NLP, and it includes all the algorithms that try to understand natural language. In this context, the word "understand" refers to the semantic meaning of language.

So, understanding natural language is different than processing natural language. Returning to the example problems listed before, most of the syntactic problems can be resolved using NLP algorithms without applying any NLU techniques. For example, there are several examples of NLP algorithms that resolve the POS tagging problem with an acceptable performance without taking into account the semantic meaning of the language (for example, the NLTK package). A solution that includes a semantic approach possibly achieves better results, but you can solve the problem without it.

Semantic problems are better suited to NLU because the concepts of "understanding" and “semantic” are similar. Sometimes the similarity of these terms causes people to assume that all NLP algorithms that solve a semantic problem are applying NLU. This is incorrect because understanding a language involves more than the ability to solve a semantic problem. Applying NLU involves a solution that understands the semantics of the language and has the ability to generalize. That means that an NLU solution should be able to understand a never-before-seen situation and give the expected results.

Let’s use an example to better explain the difference. Imagine an algorithm designed to solve a sentiment analysis problem. The idea is that when given a sentence, the algorithm returns Positive or Negative taking into account the sentiment of the sentence. This is clearly a semantic task. Imagine that you want to apply that solution to a well-defined scope – for example, movie reviews. So, a possible solution could be to download a corpus of movie reviews and train a neural network to detect the sentiment of the sentence.

That solution is an NLP algorithm that resolves a semantic problem. However, what happens when the input of the sentence is "Today is Monday"? This sentence doesn’t have any sentiment in it, and it was probably never seen by the algorithm before (remember that we trained the algorithm with movies reviews). The output of our algorithm probably will answer with Positive or Negative, when the expected result should be, “That sentence doesn’t have a sentiment,” or something like, “I am not trained to process that kind of sentence."

If we take an NLU approach, instead of training our model with only movie reviews, we use more data (such as a Wikipedia corpus). Then the model will probably be able to detect that the sentence doesn’t have any sentiment information and give the expected result.

So, taking into account that the NLU approach generalizes better than a traditional NLP approach in some semantic tasks, why don’t we always use NLU for semantic tasks?. Well, there are two main reasons. First of all, training an algorithm that efficiency processes NLU is complex and requires a lot of data. Languages are very complex and are in continuous development (new words appear, new expressions are used, etc). Keeping an NLU algorithm updated requires retraining it periodically. The second reason is that better generalization usually involves worse precision. In the example mentioned before, the first approach(training with only the movie review corpus) will probably yield better results with a new review than the NLU approach.

How to Choose the Best Option

Now that we have defined the different NLP problems that we can process and have given a brief definition of NLU, our next question is, how do you choose the best option for your company? Here are some tips that can help you to answer that question.

Check whether NLU is a valid approach for your business.

As we mentioned earlier, NLP involves many subfields. NLU is very useful for semantic problems. Before thinking about how to apply NLU, it's important to understand what kind of problem you are dealing with. Is it a semantic or syntactic problem?

Analyze which metric is more important for your business: generalization or precision/recall.

As we discussed before, NLU solutions usually generalize very well, but sacrifice precision/recall. Before choosing which strategy to use for your problem, it's crucial to define what is more important.

Analyze the characteristics of your input data.

If your input data comes from a well-known source and is always written in a certain style, generalization might not be necessary, so you won't need NLU. On the other hand, if the input data is diverse, NLU is possibly the best approach.

Determine whether an NLU approach is doable.

This is a very important point that you'll need to define before starting to implement anything. Sometimes an NLU approach is impossible because you don’t have enough data to train your model. For example, if you want to deal with a language that is not widely spoken or you are working in a particular field (medicine, transport, etc) that uses very specific terminology, an NLU approach may not be able to process it.

NLP and NLU are fascinating fields that provide a lot of great opportunities for businesses to create innovative, competitive solutions. We hope this post has helped you understand the key differences between NLP and NLU and identify the important questions you'll need to answer before you implement NLP or NLU in your product.