Quick Start Guide

To learn how integrations work we will step through creating a simple integration that makes a GET request to the https://httpbin.org website. The request will always return back the same information so isn't particularly useful but it will allow us to cover the basics of writing an integration including making a REST API call.

At a minimum each integration must include a config.js file, a package.json file, and a main module file (typically named integration.js). We also recommend including a README.md file where you can provide documentation for your integration using Markdown syntax.

To get started setup the following directory/file structure:

generic-rest/
├── config/
│   └── config.js
├── integration.js
├── package.json
└── README.md

We will start by editing the package.json file:

package.json

Open the empty package.json file and paste the following content into it:

package.json

{
  "name": "generic-rest",
  "version": "0.1.0",
  "main": "./integration.js",
  "private": true,
  "dependencies": {
    "request": "2.76.0",
    "async": "2.1.2"
  }
}

The name field is used to name our integration and is required (it should not include uppercase characters or spaces). The version field specifies the version and should follow the semver 2.0.0 versioning scheme. The main field specifies the main module for our integration (i.e., the javascript file that contains our integration lookup logic). The main file is the entry point into our integration. The private field set to true indicates that the integration (an NPM module) should not be published to the public NPM registry. Finally, we list out the dependencies and their versions that we are going to use when developing our integration.

For more information on the package.json file click here

The package.json file is JSON which means all strings must be double quoted. This is more strict than in Javascript where the key does not have to be quoted and you can use either single or double quotes around string values.

config/config.js

Now that our package.json file is setup we can move on to configuring the integration using the Polarity config file. The config file must called config.js and must exist in a config directory.

The config file should export a javascript object containing various configuration properties. In the example below we include the minimum required fields and one optional field (the description).

Click here for a full list of configuration values

Copy the contents below into your own config/config.js file.

config.js

// Generic REST Config File
module.exports = {
    name: 'Generic REST',
    acronym: 'REST',
    description: 'A sample integration that makes a generic REST call',
    entityTypes: ['IPv4']
};

Our Generic REST integration config file contains a name field which is displayed to the end user, an acronym for the integration which is used as part of the notifications and a description which is displayed in the Polarity integration interface. In addition, we add IPv4 as one of the entity types that this integration will receive. If we don't specify any entity types then our integration won't receive any data.

README.md

While optional, we highly recommend including a README.md file for your integration which at a minimum should describe what your integration does. The content of the README can be formatted using Markdown. See the README.md guide for more information.

Copy the contents below into your own README.md file.

readme.md

# Generic REST Integration

This integration sends a GET request to https://httpbin.org for each IPv4 entity that is processes.

integration.js

This is where we will begin to implement the logic of the Generic REST integration. This is the main "driver" for our integration and is developed as the main Node.js module.

Export doLookup

At a minimum, the integration must implement and export a doLookup method. Add the module.exports statement to the bottom of your integration.js file.

integration.js

 module.exports = {
     doLookup: doLookup
 };

Require Dependencies

We also want to import our dependencies so that we can use them when writing our integration. At the top of the integration.js file you can import the required dependencies by using the require statement. Include the request and async libraries at the top of your integration.js file like this:

integration.js

 const request = require('request');
 const async = require('async');

We're able to require the request and async libraries because we included them in our package.json file under dependencies.

request

Request is designed to be the simplest way possible to make HTTP calls. It supports HTTPS and follows redirects by default.

There are a wide range of node modules for accessing REST APIs including the built-in Node libraries which can be used as a replacement for request. We chose the request module for this example as it is actively maintained, simple to use, and full featured. We have also standardized on using the request library in all of our official open source integrations.

async

Async is a utility module which provides straight-forward, powerful functions for working with asynchronous JavaScript.

doLookup Implementation

We now need to implement the doLookup method. To start, we we should define a new function in our integration.js file (typically this will be placed above your module.exports statement but below your require statements). Add the following function to your integration.js file:

integration.js

/*
 * Given an array of entity objects, performs lookup requests and returns an array of result objects.
 * 
 * @method doLookup
 * @param {Array} entities
 * @param {Object} options
 * @param {Function} callback
 */
function doLookup(entities, options, cb){

 }

doLookup Parameters

entities

The doLookup method receives as its first parameter an array of entity objects which we have named entities. An entity object describes an entity that was seen on a user's screen. The entity object contains a value property that contains the value of the entity as extracted from a user's screen as well as additional boolean flags that can be used to quickly determine the type of the entity.

 // entities
 [{
     type: 'IPv4',
     types: ['IP', 'IPv4']
     isIP: true,
     isIPv4: true,
     isIPv6: false,
     isPrivateIP: true,
     IPType: 'IPv4',
     isDomain: false,
     isHash: false,
     isMD5: false,
     isSHA1: false,
     isSHA256: false,
     hashType: '',
     isEmail: false,
     isURL: false,
     value: '192.168.0.1',
     IPLong: 939655937,
     channels: []
 }]

options

The second parameter is an options object which contains integration options specified by the user. For example, if we had an option called apiKey and an option called username then the options object might look like this.

//options
{
    apiKey:'lkajsdlkajsq131232kjahosdaisd389dkl',
    username: 'ed'
}

For our simple Generic REST integration we are not using the options parameter.

cb

The third parameter is our callback which is typically abbreviated cb. The callback should return any errors (null if there are none) as well as the lookupResults array.

// callback definition
cb(errors, lookupResults);

Note that the lookupResults array contains resultObjects that must follow a well defined structure. The structure of the resultObject is covered later in this guide.

Iterate over Entities

We can use the isIPv4 property of the entity object to determine whether the entity is an IPv4 address. We use the async module to iterate over the entities array and execute a callback for each entity. We will perform our lookup inside the callback so that our REST requests are executed in parallel and do not block the main integration thread.

If the entity is an IPv4 address then we pass the value of the entity to a new method called _lookupIPv4 which we will implement next.

For more information on the async.each call please see the official async module documentation.

integration.js

function doLookup(entities, options, cb){
    // store results in this array
    let lookupResults = [];
    async.each(entities, function(entity, next){
        if(entity.isIPv4){
            // this entity is an IPv4 address so lets process it
            _lookupIPv4(entity, function(err, result){
                if(!err){
                    // add to our results if there was no error
                    lookupResults.push(result);
                }
                // processing complete
                next(err);
            });
        }else{
            // not an IPv4 entity so ignore this entity
            next(null);
        }
    }, function(err){
        cb(err, lookupResults);
    });
}

Lookup IPv4 Entity

The next step in developing the integration is to implement the _lookupIPv4() method. The method is called from doLookup() and is passed an entity object as well as a done callback which is a function that should be executed when the lookup is complete.

integration.js

 function _lookupIPv4(entity, done){
    request({url:'http://httpbin.org/get', json:true}, function(err, response, body){
       // Our GET request has completed and returned
       // We need to handle any errors and then process the result
    });
 }

We will make use of the request module to execute a GET statement to the URL http://httpbin.org/get which is a test endpoint you can use to test HTTP GET requests. The request() call will return via callback, any errors, the HTTP response, and the body of the response.

We can make a GET request by passing a configuration object to the request() method as the first parameter. The configuration object contains a url property which points to where we want to make the request, and a json boolean property set to true which indicates the response will be in JSON. The second parameter to the request method is a callback that will be executed when the HTTP request completes. The callback includes the following three parameters:

err

The first parameter (as should be the case will all async callbacks in Node.js) is an error object. You should check this object in the event that there was an HTTP error when attempting to make the REST request.

response

The second parameter is the response object which is the raw Node.js http.IncomingMessage object. Generally, you will use this object to access response headers and the HTTP status code. More information on what is contained in this object can be found in the official Node documentation.

body

The final parameter is the body parameter which contains the deserialized contents of the response body. This is where the content fetched from the GET request will be found and it is automatically deserialized from JSON into a javascript object literal.

Handling Errors

It's important that we handle any errors that might have occurred when making the GET request. In addition to checking the value of the err object we also consider any HTTP StatusCode that is not 200 OK to be an error. For example, if the server responded with the code 500 then we would consider this an error.

As is convention in Node.js we pass back the error as the first parameter in our callback (if there is no error we would pass back null for this value).

The error that we pass back here will be displayed to the user in their notification window and will also be visible to Polarity admins through the Polarity integration interface.

As you become more familiar with developing integrations it is usually a good practice to return errors messages that are as specific as possible. For example, if a specific HTTP code has a special meaning for your integration, you should check for that HTTP code and then return a human readable message.

integration.js

 function _lookupIPv4(entity, done){
    request({url:'http://httpbin.org/get', json: true}, function(err, response, body){
        // handle any errors that might have occurred
        if(err || response.statusCode !== 200){
            // return either the error object if it exists,
            // or the statusMessage if the statusCode is not 200 
            done(err || response.statusMessage);
            return;
        }
    });
 }

The Result Object

Once we've handled any errors we move on to creating our result object.

The following is a sample result object:

{
    entity: entity,
    data: {
        summary: ["tag1", "tag2"],
        details: {
           key1: "value1",
           key2: "value2"
       }
   }
}

The result object contains a top level entity property and data property.

entity

required | object

The entity object for this particular lookup.

data

required | object

The data that we want to return to be displayed. Note that the value for the data property can be null which means the entity in question had no data. Returning a null value for data indicates to Polarity that this lookup should be cached as a miss (thereby preventing future lookups for data we know does not exist). For more details see the main module guide.

data.summary

required | array of strings

An array of strings which will be converted into tags and displayed in the summary block of the notification window. Note that the strings support HTML which means you can add icons or modify the CSS style inline of the tag.

data.details

required | object

The details object is passed through the Integration template file. The default template will render the values in the details object in tabular form.

integration.js

function _lookupIPv4(entity, done){
    request({url:'http://httpbin.org/get', json: true}, function(err, response, body){
        if(err || response.statusCode !== 200){
            // return either the error object, or the body as an error
            done(err || body);
            return;
        }

        // there was no error in making the GET request so process the body here
        done(null, {
            entity: entity,
            data:{
                summary: [entity.value],
                details: body
            }
        });
    });
}

Reviewing our doLookup method we can see that the result object passed back from the _lookupIPv4 method will be added to a results array (lookupResults) as part of the async.each call. We then return the full set of result objects using the callback passed into the doLookup method by the Polarity Integration platform. If at any point an error is returned by our _lookupIPv4 method, that error is immediately sent back via the callback cb.

integration.js

function doLookup(entities, options, cb){
    // store results in this array
    let lookupResults = [];
    async.each(entities, function(entity, next){
        if(entity.isIPv4){
            // this entity is an IPv4 address so lets process it
            _lookupIPv4(entity, function(err, result){
                if(!err){
                    // add to our results if there was no error
                    lookupResults.push(result);
                }
                // processing complete
                next(err);
            });
        }else{
            // not an IPv4 entity
            next(null);
        }
    }, function(err){
        cb(err, lookupResults);
    });
}

Caching Misses

The Polarity integration framework includes a built-in caching mechanism to help improve performance of integrations. In general, as an integration developer you do not need to worry about the cache as the caching layer is implemented independent of your integration code. The one exception to this rule is if you want to cache misses. In other words, if you want to cache the fact that a specific entity had no data for it. If your integration caches misses then the Polarity sever can tell users no data exists for a particular entity without forcing your integration to do a lookup.

We can modify our _lookupIPv4 method to cache misses. In our case the REST API we're working with will return a 404 HTTP status code in the event that no data exists for the entity we looked up. We can check for this 404 response and if we receive it we add a null data result object. The Polarity Server will then know that no data exists for the specified entity.

A null data result object still specifies the entity property but sets the data property to null.

integration.js

function _lookupIPv4(entity, done){
    request({url:'http://httpbin.org/get', json: true}, function(err, response, body){
        if(err){
            // return either the error object, or the body as an error
            done(err || body);
            return;
        }

        if(response.statusCode === 404){
            done(null, {
                entity: entity,
                data: null // this entity will be cached as a miss
            });
            return;
        }

        if(response.statusCode !== 200){
            done(err || body);
            return;
        }

        // there was no error in making the GET request so process the body here
        done(null, {
            entity: entity,
            data:{
                summary: [entity.value],
                details: body
            }
        });
    });
}

Installing the Integration

You should copy the generic-rest directory and its files to your Polarity Server and place them inside the integrations directory of the polarity-server which by default will be /app/polarity-server/integrations. Your completed directory structure should look like this:

/app/polarity-server/integrations/generic-rest/
├── config/
│   └── config.js
├── integration.js
├── package.json
└── README.md

Once the files are copied you will need to install the Node.js dependencies (async, and request) that we included in our package.json file. To do this change directory into the generic-rest directory and run the command npm install. Finally, you should ensure that the entire generic-rest directory is owned by the polarityd user.

cd /app/polarity-server/integrations/generic-rest
npm install
chown -R polarityd:polarityd /app/polarity-server/integrations/generic-rest

The integration is now fully installed. You will need to restart the Polarity server for the integration to show up on the integrations page:

service polarityd restart

Login to Polarity, navigate to the Integrations page and subscribe to the integration. You should now start seeing results for any valid IPv4 addresses on your screen.

Full integration.js Code

The following is the complete integration.js file as covered in this example.

const request = require('request');
const async = require('async');

function doLookup(entities, options, cb){
    // store results in this array
    let lookupResults = [];
    async.each(entities, function(entity, next){
        if(entity.isIPv4){
            // this entity is an IPv4 address so lets process it
            _lookupIPv4(entity, function(err, result){
                if(!err){
                    // add to our results if there was no error
                    lookupResults.push(result);
                }
                // processing complete
                next(err);
            });
        }else{
            // not an IPv4 entity
            next(null);
        }
    }, function(err){
        cb(err, lookupResults);
    });
}

function _lookupIPv4(entity, done){
    request({url:'http://httpbin.org/get', json: true}, function(err, response, body){
        if(err || response.statusCode !== 200){
            // return either the error object, or the body as an error
            done(err || body);
            return;
        }

        // there was no error in making the GET request so process the body here
        done(null, {
            entity: entity,
            data:{
                summary: [entity.value],
                details: body
            }
        });
    });
}

module.exports = {
    doLookup: doLookup
};