# Throttling Lookups

A common concern when developing integrations is throttling the number of lookups an integration makes in a given time interval. We can add this functionality directly to an integration. The following guide walks through how to add basic throttling to your integration. To start you will need to define two constants to your `integration.js` file (or your main integration module if you are not using the default naming conventions).

At the top of your integration file add the following global constants:

{% code title="integration.js" %}

```javascript
 // throttle interval in milliseconds
const THROTTLE_INTERVAL = 5000;
// number of lookups allowed within the THROTTLE_INTERVAL
const THROTTLE_MAX_REQUESTS = 2;
```

{% endcode %}

The `THROTTLE_MAX_REQUESTS` constant is the maximum number of times you want your lookup or function to excecute within the given `THROTTLE_INTERVAL`. Keep in mind that the `THROTTLE_INTERVAL` is specified in milliseconds. As an example, if you want your lookup routine to be called no more than once a second you could set the following values:

{% code title="integration.js" %}

```javascript
const THROTTLE_INTERVAL = 1000;
const THROTTLE_MAX_REQUESTS = 1;
```

{% endcode %}

In addition to the two global constants we will need two more global variables that will track our throttling state. You can add the following right below the throttle constants we defined above:

{% code title="integration.js" %}

```javascript
// Tracks the number of lookups we have made within the throttle window
let numLookupsInThrottleWindow = 0;
// Tracks the last time we started a new throttle window
let lastThrottleWindowStartTime = Date.now();
```

{% endcode %}

The variables `numLookupsInThrottleWindow` and `lastThrottleWindowStartTime` are used to track the current state of our throttling. These are not values you should configure or modify.

Next we will add our `throttle` function into our `integration.js` file.

{% code title="integration.js" %}

```javascript
/**
 * Executes the `execFunc` if we have not reached our throttling limit, 
 * otherwise executes the `throttledCb` callback
 * 
 * @param execFunc
 * @param throttledCb
 */
function throttle(execFunc, throttledCb) {
    // If we are past our throttle interval then we can reset the throttle window start time
    // as well as the number of lookups we have made during the window.
    if(Date.now() - lastThrottleWindowStartTime > THROTTLE_INTERVAL){
        numLookupsInThrottleWindow = 0;
        lastThrottleWindowStartTime = Date.now();
    }

    // As long as the number of lookups we have made during the window is less
    // than the maximum allowed execute the `execFunc`.
    if(numLookupsInThrottleWindow < THROTTLE_MAX_REQUESTS){
        numLookupsInThrottleWindow++;
        execFunc();
    }else{
        // Reached our throttle limit so just call the throttled callback
        throttledCb(null);
    }
}
```

{% endcode %}

The above function takes two parameters. The first parameter is the function we want to throttle. The second parameter is a callback that is executed in the event that we have reached our throttle maximum. As an example, if we wanted to throttle a function called `_lookupIP` we could set it up like this:

{% code title="integration.js" %}

```javascript
throttle(function(){
    _lookupIP(entity);
}, function(){
    // this callback is executed if the lookup was throttled
})
```

{% endcode %}

The more typical case is that our lookup function is executed inside an `async.each` loop. In that case your code might look like this:

{% code title="integration.js" %}

```javascript
function doLookup(entities, options, cb) {
    let lookupResults = [];

    async.each(entities, function (entityObj, next) {
        if (entityObj.isIPv4) {
            throttle(function(){
                // execute _lookupIP in a throttled manner
                _lookupIP(entityObj, options, function (err, result) {
                    if (err) {
                        next(err);
                    } else {
                        lookupResults.push(result);
                        next(null);
                    }
                });
            }, function(){
                // called if the _lookupIP method gets throttled
                // calling next(null) effectively skips this entity
                next(null);
            })
        } else {
            // This is not an IPv4 so we skip it
            next(null);
        }
    }, function (err) {
        // Return our results 
        cb(err, lookupResults);
    });
}
```

{% endcode %}

In the above example we throttle the `_lookupIP` method so that it only executes at most `THROTTLE_MAX_REQUESTS` within the time inteval `THROTTLE_INTERVAL`. If we reach the throttle limit then we simply ignore the entity to be looked up by making the call to `next(null)` thereby skipping over it within our `async.each` loop. You could also return an error message if you wanted to notify the user that their lookups are now being throttled.

{% code title="integration.js" %}

```javascript
throttle(function(){
// execute _lookupIP in a throttled manner
    _lookupIP(entityObj, options, function (err, result) {
        // handle results
    }, function(){
        // called if the _lookupIP method gets throttled
        // In this example we are passing an error message back to the notification window
        next(`Your lookup for [${entityObj.value}] was throttled.`);
    })
});
```

{% endcode %}