Natural Language Processing

Overview

Language allows you to analyze text to collect data from content such as ideas, entities, key phrases, subgroups, feelings, emotions, relationships, textual norms, using the understanding of the natural language, and to identify specific entities and relationships in unstructured text.

Beginner's Guide

Computers are great at working with structured data like spreadsheets and database tables. But we humans usually communicate in words, not in tables. That’s unfortunate for computers. A lot of information in the world is unstructured — raw text in English or another human language. How can we get a computer to understand the unstructured text and extract data from it?

NLP is short for natural language processing while NLU is the shorthand for natural language understanding. Similarly named, the concepts both deal with the relationship between natural language (as in, what we as humans speak, not what computers understand) and artificial intelligence.

They share a common goal of making sense of concepts represented in unstructured data, like language, as opposed to structured data like statistics, actions, etc. To that end, NLP and NLU are opposites of a lot of other data mining techniques. But that’s where the comparisons stop: NLU and NLP aren’t the same things.

Quickstart

The following 5 services/APIs are provided by Language as part of Release 1.0.0.

POS Tagger : Part Of Speech tagging for a piece of text.
Sentiment Analysis (English) : Analyses the sentiment of the text and returns if it falls under the positive, neutral, or negative sentiment.
NER Tagger : Named Entity Recognition Tagging for a piece of text.
Dependency Parser : Dependency Parser represents the Grammatical Structure of the Sentence.
Tags : Tags Extractor for extracting keywords from a paragraph.

APIs

All URIs below are relative to https://prod-kong.dltk.ai

POS Tagger	`POST`/core/nlp/pos/
Sentiment Analysis	`POST`/core/nlp/sentiment/
NER Tagger	`POST`/core/nlp/ner/
Dependency Parser	`POST`/core/nlp/dependency-parser/
Tags	`POST`/core/nlp/tags/

POS Tagger

Description

This API would enable you to do a part of speech tagging of the text.

URI
POST /core/nlp/pos

Headers

api-key

Your App’s API Key

Attributes

text	Text Sentence (String)

Request Example:

{
  "text": "He did a terrific job at the construction site"
}

Response:

{
  "text": "He did a terrific job at the construction site.",
  "result": {
    "the": "DT",
    "a": "DT",
    "site": "NN",
    "at": "IN",
    "terrific": "JJ",
    "construction": "NN",
    "job": "NN",
    "He": "PRP",
    "did": "VBD"
  }
}

Sentiment Analysis

Description

This API would enable you to do a sentiment analysis and classify if the text is positive, neutral or negative.

URI
POST /core/nlp/sentiment

Headers

api-key

Your App’s API Key

Attributes

text	Text Sentence (String)

Request Example:

{
  "text": "He has been always cheerful ever since he moved to Hyderabad."
}

Response:

{
    "scores": {
        "neu": 0.741,
        "pos": 0.259,
        "compound": 0.5423,
        "neg": 0
    },
    "text": "He has been always cheerful ever since he moved to Hyderabady",
    "emotion": "POSITIVE",
    "polarity": 3
}

Named Entity Recognition(NER) Tagger

Description

This API would enable you to do a named entity recognition tagging of the text. This API also finds nearest the match for the person detected and gives information such as occupation, nationality, etc.

URI
POST /core/nlp/ner

Headers

api-key

Your App’s API Key

Attributes

text	Text Sentence (String)

Request Example:

{
  "text": "Madan Puri bought 300 shares of Acme Corp. in 2006."
}

Response:

{
    "result": {
        "300": "CARDINAL",
        "2006": "DATE",
        "Madan Puri": "PERSON",
        "Acme Corp.": "ORG"
    },
    "persons": [
        {
            "Link": "https://en.wikipedia.org/wiki/Madan_Puri",
            "Name": "Madan Puri",
            "Nationality": "Indian",
            "Occupation": "Actor"
        }
    ],
    "text": "Madan Puri bought 300 shares of Acme Corp. in 2006."
}

Dependency Parser

Description

This API would enable you to do dependency-parser.

URI
POST /core/nlp/dependency-parser/

Headers

api-key

Your App’s API Key

Attributes

text	Text Sentence (String)

Request Example:

{
  "text": "He is a cheerful guy."
}

Response:

{
    "cheerful": {
        "children": [],
        "headPOS": "NOUN",
        "dep": "amod",
        "headText": "leader"
    },
    "is": {
        "children": [
            "He",
            "leader",
            "."
        ],
        "headPOS": "VERB",
        "dep": "ROOT",
        "headText": "is"
    },
    "He": {
        "children": [],
        "headPOS": "VERB",
        "dep": "nsubj",
        "headText": "is"
    },
    ".": {
        "children": [],
        "headPOS": "VERB",
        "dep": "punct",
        "headText": "is"
    },
    "a": {
        "children": [],
        "headPOS": "NOUN",
        "dep": "det",
        "headText": "leader"
    },
    "leader": {
        "children": [
            "a",
            "cheerful"
        ],
        "headPOS": "VERB",
        "dep": "attr",
        "headText": "is"
    }
}

Tags Extractor

Description

This API would enable you to get tags from a paragraph.

URI
POST /core/nlp/tags/

Headers

api-key

Your App’s API Key

Attributes

text	Text Sentence (String)

Request Example:

{
"text": "TweetDeck is a social media dashboard application for management of Twitter accounts. Originally an independent app, TweetDeck was subsequently acquired by Twitter Inc. and integrated into
                Twitter's interface."
}

Response:

{
    "tags": [
        "tweetdeck",
        "twitter"
    ],
    "text": "TweetDeck is a social media dashboard application for management of Twitter accounts. Originally an independent app, TweetDeck was subsequently acquired by Twitter Inc. and integrated
                into Twitter's interface."
}

Annotations

NER Supports below entities

Type	Description
PERSON	People, including fictional.
NORP	Nationalities or religious or political groups.
LANGUAGE	Any named language.
QUANTITY	Measurements, as of weight or distance.
ORDINAL	“First”, “second”, etc.
CARDINAL	Numerals that do not fall under another type.
MONEY	Monetary values, including unit.
PERCENT	Percentage, including “%”.
TIME	Times smaller than a day.
DATE	Absolute or relative dates or periods.
EVENT	Named hurricanes, battles, wars, sports events, etc.
WORK_OF_ART	Titles of books, songs, etc.
LAW	Named documents made into laws.
LOC	Non-GPE locations, mountain ranges, bodies of water.
PRODUCT	Objects, vehicles, foods, etc. (not services).
FAC	Buildings, airports, highways, bridges, etc.
ORG	Companies, agencies, institutions, etc.
GPE	Countries, cities, states, etc.

Annotations

POS TAGGER supports below entities:

POS	Description
PUNCT	Punctuations
CD	Cardinal Number
SYM	Symbol
DT	Determiner
NN, NNS	Noun
NNP, NNPS	Proper Noun
PRP	Pronoun
UH	Interjection
JJ, JJR, JJS, WDT	Adjective, Comparative and Superlative
ADP	AdPosition like in, during
VB, VBG, VBN, VBP, VBZ	Verb
RB, RBR, RBS	Adverb
AUX	Auxiliary
CC	Conjunction
CCONJ	Coordinating Conjunction
IN	Subordinating Conjunction

SDK

Installation

DLTK requires Python version greater than 3.5. One can install DLTK SDK using following command:

pip install dltk_ai

Creating Client

Create DLTK client to perform a different task.

client = dltk_ai.DltkAiClient('Your API Key')

To use these services, one needs to register to dltk website and create a project. Copy your API key to use different APIs.

Sentiment Analysis

Use to get the sentiment of the given text.

sentiment_analysis_response = client.sentiment_analysis("The stay was great.")
sentiment_analysis_response

To use these services, one needs to register to dltk website and create a project. Copy your API key to use different APIs.

POS Tagger

Use to get the Part Of Speech of the given text.

pos_tagging_response = client.pos_tagger("The stay was great.")
pos_tagging_response

Dependency Parser

Use to get the dependency parser of the given text.

dependency_parser_response = client.dependency_parser('He is a cheerful guy.')
dependency_parser_response

NER Tagger

Use to get the ner tagger of the given text.

ner_tagger_response = client.ner_tagger('Madan Puri bought 300 shares of Acme Corp. in 2006.')
ner_tagger_response

Tag Extraction

Use to extract tags from given paragraph.

tagger_response = client.tags('TweetDeck is a social media dashboard application for management of
                                Twitter accounts. Originally an independent app, TweetDeck was
                                 subsequently acquired by Twitter Inc. and integrated intoTwitter's
                                 interface..')
tagger_response

Sarcasm Analysis

Check if the given sentence is sarcastic in nature.

sarcasm_analysis_response = client.sarcasm_analysis("I am not insulting you, I am describing you")
sarcasm_analysis_response

Abuse Analysis

Check if the given sentence is abusive in nature.

abuse_analysis_response = client.abuse_analysis("You need to shut the fuck up")
abuse_analysis_response

Release Notes

Following are the release notes as part of Release 1.0.0

POS Tagger tags supported are DT, NN, IN, JJ, PRP, VBD etc.
NER Tagger tags supported are PER, ORG, LOC, DATE, GPE, PRODUCT, LANGUAGE, TIME, MONEY, QUANTITY etc.
Sentiment Analysis is supported only in English.
Tags Extractor works on paragraphs and minimum frequency of any word to be detected as keyword must be at least two.
The Info of the person detected in NER is the nearest match.