Source Analytics Integration with Elasticsearch

Setup the Polarity Source Analytics Integration with Elasticsearch

Download and Install

The Source Analytics integration can be downloaded and installed just like any other Polarity integration. First, change into the integration directory, dowload the integration and untar it:

cd /app/polarity-server/integrations
wget https://dl.polarity.io/C7BFBB1CF5A58C7DBA57CD0941F635BF1895366064713E88DAA7F05A5295B8B1/INT/analyst-telemetry-elasticsearch/analyst-telemetry-elasticsearch-3.9.1-dev-full.tgz
tar -xvzf analyst-telemetry-elasticsearch-3.9.1-dev-full.tgz

Next, ensure permissions are set correctly on the integration:

cd /app/polarity-server/integrations/analyst-telemetry-elasticsearch
chown -R polarityd: .

Finally, restart your Polarity Server process so the new integration can be installed and configured via the Integration Settings page.

systemctl restart polarityd

Setup Elasticsearch API Key

The Source Analytics integration requires a properly configured API key to be able to search the telemetry logs. Login to your Kibana instance and navigate to "Stack Management" -> "Security" -> "API Keys".

Click on the Create API Key button.

Under Name fill in descriptive name for the API Key usage such as Polarity Source Analytics Integration.

Check the "Restrict privileges" check box and then paste the following configuration into the text box area.

{
  "source_analytics": {
    "cluster": [],
    "indices": [
      {
        "names": [
          "<name-of-data-stream>"
        ],
        "privileges": [
          "read"
        ],
        "allow_restricted_indices": false
      }
    ],
    "run_as": [
      "*"
    ]
  }
}

Be sure to replace <name-of-data-stream> with the name of the data stream that that Source Analytic logs are being sent to.

If you used the configuration from the Configuring Source Analytics on Elasticsearch guide then the data stream name would be logs-psa-default

Optionally set the API Key to expire after a certain amount of time.

Next click on the Create API Key button.

Copy and Save the provided API key in Base64 format.

Configure the Integration

Set the following required values for the integration which can be configured from the Integration Settings page under the Options tab.

Integration Options

Elasticsearch URL

This should be the Elasticsearch REST API URL. The URL should include the scheme (https://) and a port if applicable.

Example: https://elastic.prod:9200

Username and Password

While the integration supports authenticating via Username and Password we do not recommend this method of authentication. Leave these two options blank and instead provide the API Key you configured previously.

API Key

Fill in the API Key you generated in the previous Setup Elasticsearch API Key step.

Index

The index where your Source Analytics data is being sent. This can be set to the name of the data stream you generated which in a default configuration would be logs-psa-default

Ignored User Ids

This option allows you to ignore searches from specific users. Typically this field is left blank but in some deployments you may want to ignore lookups performed by the admin user. In this case you can set the value of this field to 1 which is the user ID of the default admin user.

Fields

This is a comma delimited list of the required source analytic fields. For a default configuration you should set this value to:

timestamp:@timestamp, entityValue:polarity.entityValue, userId:polarity.userId, integrationId:polarity.integrationId, isFromCache:polarity.isFromCache, hasResult:polarity.hasResult, windowTitle:polarity.windowTitle, process:polarity.process

Ignore Integrations

Comma delimited list of integration IDs to ignore. Integration IDs match the directory name of the integration but with dashes converted to underscores. For example, if the integration directory is "generic-integration", the ID for the integration would be "generic_integration".

The Source Analytics integration will never include itself in results so this option can typically be left blank.

Display Window Title

If checked, Window Title information will displayed as part of the returned data. In some cases you may not want other users to be able to view Window Title information of previous search requests. In this case, uncheck this option.

Testing

Once the integration options have been set you can subscribe to the integration and try searching on an indicator that has been searched before. You will see the Source Analytics integration return results in the Overlay Window.