Forem: Pallavi Ratra

How to tokenize a string?

Pallavi Ratra — Tue, 09 Feb 2021 15:59:21 +0000

To tokenize a string using winkNLP, read the text using readDoc. Then use the tokens method to extract a collection of tokens from the string. Follow this with the out method to get this collection as a JavaScript array. This is how you can tokenize a string:

// Load wink-nlp package  & helpers.
const winkNLP = require( 'wink-nlp' );
// Load "its" helper to extract item properties.
const its = require( 'wink-nlp/src/its.js' );
// Load english language model — light version.
const model = require( 'wink-eng-lite-model' );
// Instantiate winkNLP.
const nlp = winkNLP( model );

// Input string
const text = '#Breaking:D Can’t get over this #Oscars selfie from 
@TheEllenShow🤩https://pic.twitter.com/C9U5NOtGap';
// Read text
const doc = nlp.readDoc( text );
// Tokenize the string
const tokens = doc.tokens();
console.log( tokens.out() );

This returns an array of tokens:

[
  '#Breaking', ':D', 'Ca', 'n’t', 'get', 'over', 'this', '#Oscars', 
'selfie','from', '@TheEllenShow', '🤩', 
'https://pic.twitter.com/C9U5NOtGap'
]

winkNLP has a lossless tokenizer which preserves and reproduces the original text. The tokenizer intelligently handles hyphenation, contractions and abbreviations. It also detects token types like ‘word’, ‘number’, ‘punctuation’, ‘symbol’, etc.

How to run NLP in the browser?

Pallavi Ratra — Tue, 02 Feb 2021 18:13:57 +0000

While winkNLP was initially made to run in NodeJS, it can just as easily be used in the browser. To do this, we need to use the web version of the English lite model — wink-eng-lite-web-model. We'll also need a tool that can bundle all the required modules, something like Webpack or Browserify. For the purpose of this tutorial we'll use Browserify. First, lets install the required packages:

We'll be installing Broswerify globally, refer to its documentation for more details.

npm install wink-nlp --save
npm install wink-eng-lite-web-model --save
npm install -g browserify

Next, we'll write our JavaScript as we normally would. Here, we'll make a new file called token-counter.js and require winkNLP, some helpers, and the web model:

const winkNLP = require( 'wink-nlp' );
const its = require( 'wink-nlp/src/its.js' );
const as = require( 'wink-nlp/src/as.js' );
const model = require( 'wink-eng-lite-web-model' );
const nlp = winkNLP( model )

const text = `Its quarterly profits jumped 76% to $1.13 billion 
for the three months to December, from $639 million of previous 
year.`;
const doc = nlp.readDoc( text );

doc.entities().each((e) => e.markup());
document.getElementById("result").innerHTML = doc.out(its.markedUpText);

Now, we'll use Broswerify to bundle all the required modules into a single file:

browserify token-counter.js -o bundle.js

This will create a new file called bundle.js which you can include in your HTML as you would any other:

<div id="result"></div>
<script src="bundle.js" charset="utf-8"></script>

This will create the following output:

Its quarterly profits jumped 76% to $1.13 billion for the three months to December, from $639 million of previous year.

Performance considerations

It is important to note that this is a fully-featured English language model. Make sure to use gzip when you serve it on the web. This will reduce its size to under 1MB (from the uncompressed 3.5MB). Also, setting an appropriate cache header will ensure that the client doesn't have to download it multiple times.

How to split text into sentences in NLP?

Pallavi Ratra — Tue, 26 Jan 2021 10:53:25 +0000

To split any text into sentences using winkNLP, read the text using readDoc. Then use the sentences method to get a collection of sentences from the text. Follow this with the out method to get this collection as a JavaScript array. This is how you can split a text into sentences:

// Load wink-nlp package  & helpers.
const winkNLP = require( 'wink-nlp' );
// Load "its" helper to extract item properties.
const its = require( 'wink-nlp/src/its.js' );
// Load english language model — light version.
const model = require( 'wink-eng-lite-model' );
// Instantiate winkNLP.
const nlp = winkNLP( model );

// Input text
const text = 'AI Inc. is focussing on AI. It is based in 
              the U.S.A. It was started on 06.12.2007.';
// Read text
const doc = nlp.readDoc( text );
// Extract sentences from the data
const sentences = doc.sentences().out();
console.log( sentences );

This returns an array of sentences:

[
  'AI Inc. is focussing on AI.',
  'It is based in the U.S.A.',
  'It was started on 06.12.2007.'
]

If no sentence-break is found in the input text, the output is the complete text as an array with a single member.

A sentence is usually split at a full-stop, question mark or exclamation mark. Even in the presence of abbreviations, honorifics, etc., winkNLP attempts to intelligently identify the sentence boundary.

How to do sentiment analysis?

Pallavi Ratra — Mon, 18 Jan 2021 15:13:15 +0000

Sentiment analysis is used to check if a phrase has positive or negative sentiment. To get the sentiment score, start by reading the phrase using readDoc. Then using the its.sentiment property of the out() method, get the sentiment score of the phrase. This is how you can check if a phrase is positive or negative:

// Load wink-nlp package & helpers.
const winkNLP = require( 'wink-nlp' );
const its = require( 'wink-nlp/src/its.js' );
const model = require( 'wink-eng-lite-model' );
const nlp = winkNLP( model );

const text = 'Very excited to be part of the @winkjs_org team:D!';
const doc = nlp.readDoc( text );

// Calculate sentiment score
const sentiment = doc.out( its.sentiment );
console.log( 'Sentiment Score:', sentiment );

if( sentiment > 0 ) {
  console.log( 'Sentiment: Positive' );
} else {
  console.log( 'Sentiment: Negative' );
}

// -> Sentiment Score: 0.6
// -> Sentiment: Positive

The sentiment score will be between -1 and 1, with 0 representing an overall neutral sentiment. You can get the sentiment of the entire document, or a single sentence in a text.

Sentiment analysis can be used to judge the overall response to an event, product etc. In chatbots, it can be used to change the response based on the sentiment of the message it has received. See our Twitter Hashtag showcase to see this in action.

winkNLP's English language lite model uses ML-SentiCon as a base with further training. For emojis it uses the Emoji Sentiment Ranking. Together, they deliver an f-score of about 84.5%.