# Quick Start Guide

To learn how integrations work we will step through creating a simple integration that makes a GET request to the <https://httpbin.org> website. The request will always return back the same information so isn't particularly useful but it will allow us to cover the basics of writing an integration including making a REST API call.

At a minimum each integration must include a `config.js` file, a `package.json` file, and a main module file (typically named `integration.js`). We also recommend including a `README.md` file where you can provide documentation for your integration using Markdown syntax.

To get started setup the following directory/file structure:

```
generic-rest/
├── config/
│   └── config.js
├── integration.js
├── package.json
└── README.md
```

We will start by editing the `package.json` file:

## package.json

Open the empty package.json file and paste the following content into it:

{% code title="package.json" %}

```javascript
{
  "name": "generic-rest",
  "version": "0.1.0",
  "main": "./integration.js",
  "private": true,
  "dependencies": {
    "request": "2.76.0",
    "async": "2.1.2"
  }
}
```

{% endcode %}

The `name` field is used to name our integration and is required (it should not include uppercase characters or spaces). The `version` field specifies the version and should follow the [semver 2.0.0 versioning scheme](http://semver.org/). The `main` field specifies the main module for our integration (i.e., the javascript file that contains our integration lookup logic). The `main` file is the entry point into our integration. The `private` field set to `true` indicates that the integration (an NPM module) should not be published to the public NPM registry. Finally, we list out the dependencies and their versions that we are going to use when developing our integration.

[For more information on the `package.json` file click here](https://docs.polarity.io/integrations/build-an-integration/package_json)

{% hint style="info" %}
The `package.json` file is JSON which means all strings must be double quoted.  This is more strict than in Javascript where the `key` does not have to be quoted and you can use either single or double quotes around string values.
{% endhint %}

## config/config.js

Now that our package.json file is setup we can move on to configuring the integration using the Polarity config file. The config file must called `config.js` and must exist in a `config` directory.

The config file should export a javascript object containing various configuration properties. In the example below we include the minimum required fields and one optional field (the `description`).

[Click here for a full list of configuration values](https://docs.polarity.io/integrations/build-an-integration/configuration-file)

Copy the contents below into your own `config/config.js` file.

{% code title="config.js" %}

```javascript
// Generic REST Config File
module.exports = {
    name: 'Generic REST',
    acronym: 'REST',
    description: 'A sample integration that makes a generic REST call',
    entityTypes: ['IPv4']
};
```

{% endcode %}

Our Generic REST integration config file contains a `name` field which is displayed to the end user, an `acronym` for the integration which is used as part of the notifications and a `description` which is displayed in the Polarity integration interface. In addition, we add `IPv4` as one of the entity types that this integration will receive. If we don't specify any entity types then our integration won't receive any data.

## README.md

While optional, we highly recommend including a `README.md` file for your integration which at a minimum should describe what your integration does. The content of the README can be formatted using [Markdown](https://guides.github.com/features/mastering-markdown/). See the [README.md guide for more information](https://docs.polarity.io/integrations/build-an-integration/readme-guide).

Copy the contents below into your own `README.md` file.

{% code title="readme.md" %}

```
# Generic REST Integration

This integration sends a GET request to https://httpbin.org for each IPv4 entity that is processes.
```

{% endcode %}

## integration.js

This is where we will begin to implement the logic of the Generic REST integration. This is the main "driver" for our integration and is developed as the main Node.js module.

### Export doLookup

At a minimum, the integration must implement and export a `doLookup` method. Add the `module.exports` statement to the bottom of your `integration.js` file.

{% code title="integration.js" %}

```javascript
 module.exports = {
     doLookup: doLookup
 };
```

{% endcode %}

### Require Dependencies

We also want to import our dependencies so that we can use them when writing our integration. At the top of the `integration.js` file you can import the required dependencies by using the `require` statement. Include the `request` and `async` libraries at the top of your `integration.js` file like this:

{% code title="integration.js" %}

```javascript
 const request = require('request');
 const async = require('async');
```

{% endcode %}

{% hint style="info" %}
We're able to require the `request` and `async` libraries because we included them in our `package.json` file under `dependencies`.
{% endhint %}

#### request

[Request](https://github.com/request/request) is designed to be the simplest way possible to make HTTP calls. It supports HTTPS and follows redirects by default.

{% hint style="info" %}
There are a wide range of node modules for accessing REST APIs including the built-in Node libraries which can be used as a replacement for `request`. We chose the `request` module for this example as it is actively maintained, simple to use, and full featured.  We have also standardized on using the `request` library in all of our official open source integrations.
{% endhint %}

#### async

[Async](https://caolan.github.io/async/index.html) is a utility module which provides straight-forward, powerful functions for working with asynchronous JavaScript.

### `doLookup` Implementation

We now need to implement the `doLookup` method. To start, we we should define a new function in our `integration.js` file (typically this will be placed above your `module.exports` statement but below your `require` statements).  Add the following function to your `integration.js` file:

{% code title="integration.js" %}

```javascript
/*
 * Given an array of entity objects, performs lookup requests and returns an array of result objects.
 * 
 * @method doLookup
 * @param {Array} entities
 * @param {Object} options
 * @param {Function} callback
 */
function doLookup(entities, options, cb){

 }
```

{% endcode %}

### `doLookup` Parameters

#### ***entities***

The `doLookup` method receives as its first parameter an array of entity objects which we have named `entities`. An entity object describes an entity that was seen on a user's screen. The entity object contains a `value` property that contains the value of the entity as extracted from a user's screen as well as additional boolean flags that can be used to quickly determine the type of the entity.

```javascript
 // entities
 [{
     type: 'IPv4',
     types: ['IP', 'IPv4']
     isIP: true,
     isIPv4: true,
     isIPv6: false,
     isPrivateIP: true,
     IPType: 'IPv4',
     isDomain: false,
     isHash: false,
     isMD5: false,
     isSHA1: false,
     isSHA256: false,
     hashType: '',
     isEmail: false,
     isURL: false,
     value: '192.168.0.1',
     IPLong: 939655937,
     channels: []
 }]
```

#### ***options***

The second parameter is an `options` object which contains integration options specified by the user. For example, if we had an option called `apiKey` and an option called `username` then the `options` object might look like this.

```javascript
//options
{
    apiKey:'lkajsdlkajsq131232kjahosdaisd389dkl',
    username: 'ed'
}
```

For our simple Generic REST integration we are not using the `options` parameter.

#### ***cb***

The third parameter is our `callback` which is typically abbreviated `cb`. The callback should return any errors (null if there are none) as well as the `lookupResults` array.

```javascript
// callback definition
cb(errors, lookupResults);
```

{% hint style="info" %}
Note that the `lookupResults` array contains [resultObjects ](https://docs.polarity.io/integrations/build-an-integration/main/dolookup/result-objects)that must follow a well defined structure. The structure of the `resultObject` is covered later in this guide.
{% endhint %}

### Iterate over Entities

We can use the `isIPv4` property of the entity object to determine whether the entity is an IPv4 address. We use the [`async`](http://caolan.github.io/async/) module to iterate over the `entities` array and execute a callback for each entity. We will perform our lookup inside the callback so that our REST requests are executed in parallel and do not block the main integration thread.

If the entity is an IPv4 address then we pass the value of the entity to a new method called `_lookupIPv4` which we will implement next.

For more information on the `async.each` call please see the [official async module documentation](https://caolan.github.io/async/docs.html#each).

{% code title="integration.js" %}

```javascript
function doLookup(entities, options, cb){
    // store results in this array
    let lookupResults = [];
    async.each(entities, function(entity, next){
        if(entity.isIPv4){
            // this entity is an IPv4 address so lets process it
            _lookupIPv4(entity, function(err, result){
                if(!err){
                    // add to our results if there was no error
                    lookupResults.push(result);
                }
                // processing complete
                next(err);
            });
        }else{
            // not an IPv4 entity so ignore this entity
            next(null);
        }
    }, function(err){
        cb(err, lookupResults);
    });
}
```

{% endcode %}

### Lookup IPv4 Entity

The next step in developing the integration is to implement the `_lookupIPv4()` method. The method is called from `doLookup()` and is passed an `entity` object as well as a `done` callback which is a function that should be executed when the lookup is complete.

{% code title="integration.js" %}

```javascript
 function _lookupIPv4(entity, done){
    request({url:'http://httpbin.org/get', json:true}, function(err, response, body){
       // Our GET request has completed and returned
       // We need to handle any errors and then process the result
    });
 }
```

{% endcode %}

We will make use of the [`request`](https://github.com/request/request#requestoptions-callback) module to execute a GET statement to the URL `http://httpbin.org/get` which is a test endpoint you can use to test HTTP GET requests. The `request()` call will return via callback, any errors, the HTTP response, and the body of the response.

We can make a GET request by passing a configuration object to the `request()` method as the first parameter. The configuration object contains a `url` property which points to where we want to make the request, and a `json` boolean property set to `true` which indicates the response will be in JSON. The second parameter to the request method is a callback that will be executed when the HTTP request completes. The callback includes the following three parameters:

#### err

The first parameter (as should be the case will all async callbacks in Node.js) is an error object. You should check this object in the event that there was an HTTP error when attempting to make the REST request.

#### response

The second parameter is the `response` object which is the raw Node.js `http.IncomingMessage` object. Generally, you will use this object to access response headers and the HTTP status code. More information on what is contained in this object can be found in the official [Node documentation](https://nodejs.org/api/http.html#http_class_http_incomingmessage).

#### body

The final parameter is the `body` parameter which contains the deserialized contents of the response body. This is where the content fetched from the GET request will be found and it is automatically deserialized from JSON into a javascript object literal.

### Handling Errors

It's important that we handle any errors that might have occurred when making the GET request. In addition to checking the value of the `err` object we also consider any HTTP StatusCode that is not `200 OK` to be an error. For example, if the server responded with the code `500` then we would consider this an error.

As is convention in Node.js we pass back the error as the first parameter in our callback (if there is no error we would pass back `null` for this value).

The error that we pass back here will be displayed to the user in their notification window and will also be visible to Polarity admins through the Polarity integration interface.

As you become more familiar with developing integrations it is usually a good practice to return errors messages that are as specific as possible. For example, if a specific HTTP code has a special meaning for your integration, you should check for that HTTP code and then return a human readable message.

{% code title="integration.js" %}

```javascript
 function _lookupIPv4(entity, done){
    request({url:'http://httpbin.org/get', json: true}, function(err, response, body){
        // handle any errors that might have occurred
        if(err || response.statusCode !== 200){
            // return either the error object if it exists,
            // or the statusMessage if the statusCode is not 200 
            done(err || response.statusMessage);
            return;
        }
    });
 }
```

{% endcode %}

### The Result Object

Once we've handled any errors we move on to creating our result object.&#x20;

The following is a sample result object:

```
{
    entity: entity,
    data: {
        summary: ["tag1", "tag2"],
        details: {
           key1: "value1",
           key2: "value2"
       }
   }
}
```

The result object contains a top level `entity` property and `data` property.&#x20;

#### entity

required | *object*

The entity object for this particular lookup.

#### data

required | *object*

The data that we want to return to be displayed. Note that the value for the `data` property can be `null` which means the `entity` in question had no data. Returning a `null` value for `data` indicates to Polarity that this lookup should be cached as a miss (thereby preventing future lookups for data we know does not exist). For more details see the [main module guide](https://docs.polarity.io/integrations/build-an-integration/main).

**data.summary**

required | *array of strings*

An array of strings which will be converted into tags and displayed in the summary block of the notification window. Note that the strings support HTML which means you can add icons or modify the CSS style inline of the tag.

**data.details**

required | *object*

The details object is passed through the Integration template file. The default template will render the values in the details object in tabular form.

{% code title="integration.js" %}

```javascript
function _lookupIPv4(entity, done){
    request({url:'http://httpbin.org/get', json: true}, function(err, response, body){
        if(err || response.statusCode !== 200){
            // return either the error object, or the body as an error
            done(err || body);
            return;
        }

        // there was no error in making the GET request so process the body here
        done(null, {
            entity: entity,
            data:{
                summary: [entity.value],
                details: body
            }
        });
    });
}
```

{% endcode %}

Reviewing our `doLookup` method we can see that the result object passed back from the `_lookupIPv4` method will be added to a results array (`lookupResults`) as part of the `async.each` call. We then return the full set of result objects using the `callback` passed into the `doLookup` method by the Polarity Integration platform. If at any point an error is returned by our `_lookupIPv4` method, that error is immediately sent back via the callback `cb`.

{% code title="integration.js" %}

```javascript
function doLookup(entities, options, cb){
    // store results in this array
    let lookupResults = [];
    async.each(entities, function(entity, next){
        if(entity.isIPv4){
            // this entity is an IPv4 address so lets process it
            _lookupIPv4(entity, function(err, result){
                if(!err){
                    // add to our results if there was no error
                    lookupResults.push(result);
                }
                // processing complete
                next(err);
            });
        }else{
            // not an IPv4 entity
            next(null);
        }
    }, function(err){
        cb(err, lookupResults);
    });
}
```

{% endcode %}

### Caching Misses

The Polarity integration framework includes a built-in caching mechanism to help improve performance of integrations. In general, as an integration developer you do not need to worry about the cache as the caching layer is implemented independent of your integration code. The one exception to this rule is if you want to cache misses. In other words, if you want to cache the fact that a specific entity had no data for it. If your integration caches misses then the Polarity sever can tell users no data exists for a particular entity without forcing your integration to do a lookup.

We can modify our `_lookupIPv4` method to cache misses. In our case the REST API we're working with will return a `404` HTTP status code in the event that no data exists for the entity we looked up. We can check for this `404` response and if we receive it we add a `null` data result object. The Polarity Server will then know that no data exists for the specified entity.

A null data result object still specifies the `entity` property but sets the `data` property to `null`.

{% code title="integration.js" %}

```javascript
function _lookupIPv4(entity, done){
    request({url:'http://httpbin.org/get', json: true}, function(err, response, body){
        if(err){
            // return either the error object, or the body as an error
            done(err || body);
            return;
        }

        if(response.statusCode === 404){
            done(null, {
                entity: entity,
                data: null // this entity will be cached as a miss
            });
            return;
        }

        if(response.statusCode !== 200){
            done(err || body);
            return;
        }

        // there was no error in making the GET request so process the body here
        done(null, {
            entity: entity,
            data:{
                summary: [entity.value],
                details: body
            }
        });
    });
}
```

{% endcode %}

## Installing the Integration

You should copy the `generic-rest` directory and its files to your Polarity Server and place them inside the `integrations` directory of the polarity-server which by default will be `/app/polarity-server/integrations`. Your completed directory structure should look like this:

```
/app/polarity-server/integrations/generic-rest/
├── config/
│   └── config.js
├── integration.js
├── package.json
└── README.md
```

Once the files are copied you will need to install the Node.js dependencies (async, and request) that we included in our `package.json` file. To do this change directory into the `generic-rest` directory and run the command `npm install`. Finally, you should ensure that the entire `generic-rest` directory is owned by the `polarityd` user.

```bash
cd /app/polarity-server/integrations/generic-rest
npm install
chown -R polarityd:polarityd /app/polarity-server/integrations/generic-rest
```

The integration is now fully installed. You will need to restart the Polarity server for the integration to show up on the integrations page:

```bash
service polarityd restart
```

Login to Polarity, navigate to the Integrations page and subscribe to the integration. You should now start seeing results for any valid IPv4 addresses on your screen.

## Full `integration.js` Code

The following is the complete `integration.js` file as covered in this example.

```javascript
const request = require('request');
const async = require('async');

function doLookup(entities, options, cb){
    // store results in this array
    let lookupResults = [];
    async.each(entities, function(entity, next){
        if(entity.isIPv4){
            // this entity is an IPv4 address so lets process it
            _lookupIPv4(entity, function(err, result){
                if(!err){
                    // add to our results if there was no error
                    lookupResults.push(result);
                }
                // processing complete
                next(err);
            });
        }else{
            // not an IPv4 entity
            next(null);
        }
    }, function(err){
        cb(err, lookupResults);
    });
}

function _lookupIPv4(entity, done){
    request({url:'http://httpbin.org/get', json: true}, function(err, response, body){
        if(err || response.statusCode !== 200){
            // return either the error object, or the body as an error
            done(err || body);
            return;
        }

        // there was no error in making the GET request so process the body here
        done(null, {
            entity: entity,
            data:{
                summary: [entity.value],
                details: body
            }
        });
    });
}

module.exports = {
    doLookup: doLookup
};
```