Hewlett Packard has developed a set of JSON-based REST API’s which enable “Big Data”-type processing capabilities allowing developers to process information embedded in unstructured text and images in previously inaccessible formats. This platform is called IDOL OnDemand, the APIs are published here https://www.idolondemand.com/developer/apis
In this tutorial, you will learn how to create a Node.js web application that pulls in tweets from a twitter feed on a specific term (such as '#f1'). The HP IDOL OnDemand APIs will be applied on each tweet that is pulled in. Each tweet will be processed using the following APIs
The language of each tweet will be detected. Sentiment analysis will be performed on each tweet, and the aggregate sentiment (positive, negative, or neutral) will be determined, along with the aggregate score, and also the individual positive and negative sentiments that make up this overall score. Finally, the Highlight Text API will be used to highlight the previous individual positive/negative sentiment terms amongst the overall text of the tweet.
The accompanying tutorial video for also demonstrates the same concepts.
The accompanying tutorial video for also demonstrates the same concepts.
Technologies/Frameworks Used
API Keys
To use the HP IDOL OnDemand API's, you must have a valid apiKey. Go to https://www.idolondemand.com/signup.html to sign up for a developer account. After you verify your account, you will be able to access the API key that you can use in your calls.
To use the Twitter library for the Twitter REST API, you need to register your application on twitter. Sign up on twitter here - http://twitter.com/signup, or use an existing account if you have one. After signing in, go to https://dev.twitter.com/ and sign in. Create a new Twitter app, fill in the necessary details and generate the API keys. You will get access to the API key, API key secret, access token, and access token secret.
APP SETUP
Create a new NodeJS application that has the following dependencies -
Create a new NodeJS application that has the following dependencies -
Create a code.js file. Setup a basic express app, that uses ejs as a view engine, and has two routes -
app.get('/', function(req, res) {});
app.get('/search/:searchTerm', function(req, res) {});
The first route will serve a template that will input the search term from the user. So, we have the option of either hard-coding the string to search for in twitter, or to ask the user each time. This route will then call the second route, that contains the main application logic.
We will also specify the number of tweets to pull in. Later we will see how to take it from the user, or store in a config file. We will also hard-code the apikey for HP IDOL OnDemand, and the API keys for twit library. We will store them in a config file later.
var twitter_searchTerm = '#f1';
var number_ofTweets = 20;
var apikey = 'xxx';
var host_url = 'api.idolondemand.com';
Initialize the twit library with your 4 twitter API keys -
var T = new Twit({
consumer_key: 'xxx',
consumer_secret: 'xxx',
access_token: 'xxx',
access_token_secret: 'xxx'
});
Set the request headers, which we will use in all the API calls -
var headers = {
'User-Agent': 'Super Agent/0.0.1',
'Content-Type': 'application/x-www-form-urlencoded'
};
Set the options for the Language Identification API call. Note that the text variable is first passed to the encodeURIComponent() function, and all space characters(%20) are replaced by '+' characters.
var get_identifyLanguage_opt = function(text) {
return {
host: host_url ,
port: 443,
path: '/1/api/sync/identifylanguage/v1?text=' +
encodeURIComponent(text).replace(/%20/g,'+') +
'&apikey=' + apikey,
headers: headers
};
};
Set the options for the Sentiment Analysis API call -
var get_analyzeSentiment_opt = function(text) {
return {
host: host_url ,
port: 443,
path: '/1/api/sync/analyzesentiment/v1?text=' +
encodeURIComponent(text).replace(/%20/g,'+') +
'&apikey=' + apikey,
headers: headers
};
};
Set the options for the Highlight Text API call -
var get_hightlightText_opt = function(text, hlight_expr) {
return {
host: host_url ,
port: 443,
path: '/1/api/sync/highlighttext/v1?text=' +
encodeURIComponent(text).replace(/%20/g,'+') +
'&highlight_expression=' + encodeURIComponent(hlight_expr).replace(/%20/g,'+') +
'&apikey=' + apikey,
headers: headers
};
};
CALLING THE API
twitter_searchTerm = req.params.searchTerm;
And call the twit library -
T.get('search/tweets', { q: twitter_searchTerm, count: number_ofTweets }, function(err, data, response) {});
The response returned in 'data' is contains an array named 'statuses'. We will loop through data.statuses to process each tweet. Now, to handle the asynchronous nature of the API call, we will use async.each and async.waterfall. First, declare an array all_tweets (that will contain the processed information about all tweets, that we will use to render the page at the end of the async.each function). Then use async.each -
async.each(data.statuses, function(status, callbackMain) {
}, function(err, result) {});
This will process each item of the data.statuses in parallel. This is efficient for our needs since each tweet can be processed in isolation, and the info stored in all_tweets array. Now, inside this function, we call async.waterfall. This will execute each API serially, so that the result of one API call can be used in the next. First declare some variables, so that they can be used through the scope of the async.waterfall.
var single_tweet = [],
global_language,
global_sentiment,
global_highlight_positives = [],
global_highlight_negatives = [];
We will store the processed information about each tweet in them as we get it, push them all into single_tweet in the final function of async.waterfall, and push single_tweet itself into the all_tweets array.
async.waterfall([
function(callbackInner) {
//call language identification API
callbackInner();
},
function(callbackInner) {
//call sentiment analysis API
callbackInner();
},
function(callbackInner) {
//call highlight text API(positive sentiments)
callbackInner();
},
function(callbackInner) {
//call highlight text API(negative sentiments)
callbackInner();
}],
function (err, result) {
//push processed information about tweet in the global all_tweets array
callbackMain();
});
In the language identification API call, we pass the options specified earlier into https.get and parse the response returned. Remember, status is the individual tweet, status.text contains the actual text content, and the response is a stream that must be captured in a string. The response is returned according to this specification.
So we parse the JSON and save the 'language' string in the variable global_language. Notice we call callbackInner because we want to go to the next function in async.waterfall. If we called callbackMain(), then we would go straight to the end of async.each.
var lang = https.get(get_identifyLanguage_opt(status.text), function(response) {
var str = '';
response.on('data', function(chunk) {
str += chunk;
});
response.on('end', function() {
var json = JSON.parse(str);
global_language = json.language;
callbackInner();
});
});
In the next Sentiment Analysis API call, we use roughly the same format. We send status.text to https.get and receive a response that we parse and store in global_sentiment.
var sent = https.get(get_analyzeSentiment_opt(status.text), function(response) {
var str = '';
response.on('data', function(chunk) {
str += chunk;
});
response.on('end', function() {
var json = JSON.parse(str);
global_sentiment = json;
callbackInner();
});
});
So in the third waterfall function, we will use a for loop. As we are executing an asynchronous API call inside the for loop (which is executed instantly), we need to prevent the loop index j from incrementing instantaneously. So we use a closure to keep the index variable.
We will push each returned highlighted expression into a global_highlight_positives array, to pick up all the returned strings for each iteration of the loop.
for(var j in global_sentiment.positive)
{
(function(j) {
//send the global_sentiment.positive[j].original_text as the text, and the global_sentiment.positive[j].sentiment string as the highlight_expression)
var sent = https.get(get_hightlightText_opt(global_sentiment.positive[j].original_text, global_sentiment.positive[j].sentiment), function(res3) {
var str = '';
res3.on('data', function(chunk) {
str += chunk;
});
res3.on('end', function() {
var json = JSON.parse(str);
global_highlight_positives.push({ text: json.text, score: global_sentiment.positive[j].score });
//if all the sentiments have been processed then only proceed to the next callback
if (j == (global_sentiment.positive.length - 1))
{
callbackInner();
}
});
});
})(j);
}
The fourth waterfall function is similar to the third, with negative replacing positive everywhere.
One addition to these third and fourth Highlight Text functions is to first check if the global_sentiment.positive and global_sentiment.negative arrays are empty or not. If the sentiment is neutral, then there may be no sentiments and these arrays will be empty, so looping through them will throw an error. So, before calling https.get, use a simple if-construct. We must call callbackInner(to go to the next function) and also call return (to not go any further in this one).
if (typeof global_sentiment == 'undefined' || global_sentiment == null) {
callbackInner(); return;
}
if (global_sentiment.positive.length == 0) {
callbackInner(); return;
}
The final function in async.waterfall will push the 5 variables - the tweet text, global_language, global_sentiment, global_highlight_positives and global_highlight_negatives - into the single_tweet array. We then push the single_tweet variable into the all_tweets array, and call callbackMain() to end the async.waterfall.
The final function in async.each is called when all the tweets have been processed. It will simply render the ejs template, passing the all_tweets array in as a variable so we can access it in the front-end.
DISPLAYING THE TWEETS
Now we look at the front-end part, that is, handling and displaying the all_tweets array in the ejs template. First, the user will reach 'localhost/', that means 'homePage.ejs' will be rendered. This page will input a search term from the user, and use some simple javascript to call the second route.
<script>
function searchTwitter() {
var URL = '/search/' + encodeURIComponent($('#searchTerm').val());
window.location.href = URL;
}
</script>
<div class="container">
<div class="row">
<div class="panel panel-primary">
<div class="panel-heading">
<h3>Enter Twitter Search Term</h3>
</div>
<div class="panel-body">
<input type="text" id="searchTerm">
<button class="btn btn-primary" onclick="searchTwitter()">Search</button>
</div>
</div>
</div>
</div>
Now, in the indexPage.ejs, create a table element with table headers as
<script>
function searchTwitter() {
var URL = '/search/' + encodeURIComponent($('#searchTerm').val());
window.location.href = URL;
}
</script>
<div class="container">
<div class="row">
<div class="panel panel-primary">
<div class="panel-heading">
<h3>Enter Twitter Search Term</h3>
</div>
<div class="panel-body">
<input type="text" id="searchTerm">
<button class="btn btn-primary" onclick="searchTwitter()">Search</button>
</div>
</div>
</div>
</div>
Now, in the indexPage.ejs, create a table element with table headers as
- Tweet
- Language
- Sentiment Score
- Positives
- Negatives
<tbody>
<% tweets.forEach(function(twit) { %>
<tr class="<%= twit[0]['analysis']['aggregate']['sentiment'] %>">
<td><p><%= twit[0]['text'] %></p></td>
<td><p><%= twit[0]['language'] %></p></td>
<td><p><%= twit[0]['analysis']['aggregate']['score'] %></p></td>
<td>
<% if (twit[0]['positive_terms'].length != 0) { %>
<table class="table table-bordered inner">
<thead>
<tr>
<th>Sentiment</th>
<th>Score</th>
</tr>
</thead>
<tbody>
<% twit[0]['positive_terms'].forEach(function(positive_term) { %>
<tr>
<td><%- positive_term['text'] %></td>
<td><%- positive_term['score'] %></td>
</tr>
<% }) %>
</tbody>
</table>
<% } %>
</td>
<% }) %>
</tbody>
This code will display the tweet text, language, and the aggregate score in three different columns. In the last 2 columns, it will loop through the array populated by the Highlight Text API. It creates a table for each tweet's negative and positive sentiments array (there may be multiple sentiments with highlighted text) and loops through each array.
We are appending the sentiment(positive, negative, or neutral) to the class of the table row. So we can define styles in CSS by picking up tr.positive, tr.negative etc
tr.neutral {
background-color: white;
}
tr.positive {
background-color: rgb(154, 238, 154);
}
tr.negative {
background-color: rgb(255, 98, 98);
}
So we render this indexPage.ejs by calling response.render in the final function of the async.each construct. Remember not to mix up the 'response' variables, as they are nested inside. Place the template in a /views sub-directory of the main folder, and specify the views in express -
app.set('views', __dirname + '/views');
Place the template i
FINISHING TOUCHES
We can create a config.js file to store the properties used in the code.js file.
module.exports = {
//HP keys
apikey: 'xxx',
host_url: 'api.idolondemand.com',
//twitter keys
consumer_key: 'xxx',
consumer_secret: 'xxx',
access_token: 'xxx',
access_token_secret: 'xxx',
//tweet analyzer terms
twitter_searchTerm: '#f1',
number_ofTweets: 20
}
var settings = require('./config.js');
Then use the variables as settings.apikey, settings.twitter_searchTerm etc
To specify a date for tweets and other options, modify the T.get call according to its library options - https://github.com/ttezel/twit. If you use console.log in the functions for debugging, you will see all the API calls being executed in parallel, so everything will come in a torrent. So for debugging, change async.each to async.eachSeries. This simple change will make sure that one tweet is processed only after the previous one has finished processing.
