# Configuring Source Analytics on Elasticsearch

### Enable Source Analytics

Prior to setting up collection of your source analytic (PSA) logs, please enable the logging on your server.

{% content-ref url=".." %}
[..](https://docs.polarity.io/polarity-admin-guide/GGQxi8U7L785oCyu5JjE/guides/enabling-source-analytics)
{% endcontent-ref %}

### Create a Polarity Agent Policy

Once, source analytic are being collected on your Polarity Server you can configure Elasticsearch to receive those logs.&#x20;

Login to your Elasticsearch Kibana instance.

Navigate to the "**Management**" -> "**Fleet**" page

Click on "**Agent Policies**"

Click on "**Create agent policy**"

<figure><img src="https://1063702622-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fliell3s8GjOV2GWs9poX%2Fuploads%2FajhgQwPr8y6h78zybeKV%2Fimage.png?alt=media&#x26;token=6e503400-f24a-4eda-9152-48558de48bf7" alt=""><figcaption></figcaption></figure>

Name the policy.  For example "polarity-source-analytics"

{% code title="Name" %}

```
polarity-source-analytics
```

{% endcode %}

Decide if you would also like to collect system logs and metrics (note that this is not required for Source Analytics collection)

The default "**Advanced options**" will work but you may want to make changes depending on your organization.

For example, you might want to add an optional description or modify the "**Default namespace**".&#x20;

Click on "**Create agent policy**"

Your new policy will be created but still needs to be configured.

### Configure the Agent Policy

Your new policy will show up in the Fleet list under "Agent Policies". Click on it to view the details:<br>

<figure><img src="https://1063702622-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fliell3s8GjOV2GWs9poX%2Fuploads%2FXhXlsB6RzD3HYV4hWlwZ%2Fimage.png?alt=media&#x26;token=f3f75b8c-f293-4a0c-991b-e9165ede205e" alt=""><figcaption></figcaption></figure>

###

Click on "**Add integration**"

Search for the "**Custom Logs**" integration and click on it:<br>

<figure><img src="https://1063702622-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fliell3s8GjOV2GWs9poX%2Fuploads%2FvrV3uMqiPQ54K90j2ZQE%2Fimage.png?alt=media&#x26;token=f00e60fb-f439-4bd5-a559-f1b062b1de24" alt=""><figcaption></figcaption></figure>

Click on the "**Add Custom Logs**" button.

Under the Custom Logs configuration, set the Integration name:

{% code title="Integration name" %}

```
psa-collection
```

{% endcode %}

Set the description.

{% code title="Description" %}

```
Collects polarity source analytic (PSA) logs
```

{% endcode %}

Ensure "**Custom log file**" is checked and then expand the "**Change defaults**" section:

Set the "**Log file path**"

{% code title="Log file path" %}

```
/app/polarity-server/logs/polarity-server.log
```

{% endcode %}

Click on "Advanced Options"

<figure><img src="https://1063702622-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fliell3s8GjOV2GWs9poX%2Fuploads%2FSFVUPR9cmZMQhlWLNUme%2Fimage.png?alt=media&#x26;token=21625df9-70c7-4300-9071-ba314f78a851" alt=""><figcaption></figcaption></figure>

Set the "**Dataset name**"

{% code title="Dataset name" %}

```
psa
```

{% endcode %}

In the "Processors" section paste the following configuration:

{% code title="Processors" %}

```
- decode_json_fields:
    fields: ["message"]
    target: polarity
    process_array: true      
- drop_event:
    when:
      not:       
        equals:
          polarity.msgType: "integration-lookup"         
- drop_fields:
    fields: ["polarity.req.session.passport.user.integrations", "polarity.req.session.passport.user.settings", "polarity.meta", "polarity.errors", "polarity.req.query", "polarity.error", "polarity.debug"]
    ignore_missing: true
- timestamp:
    field: polarity.timestamp
    layouts:
      - '2006-01-02T15:04:05Z'
      - '2006-01-02T15:04:05.999Z'
      - '2006-01-02T15:04:05.999-07:00'
    test:
      - '2019-06-22T16:33:51Z'
      - '2019-11-18T04:59:51.123Z'
      - '2020-08-03T07:10:20.123456+02:00'
```

{% endcode %}

Click on "**Save and continue**"

When prompted click on "**Add Elastic Agent to your hosts**"

<figure><img src="https://1063702622-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fliell3s8GjOV2GWs9poX%2Fuploads%2F7kJGclDRmjungIJohHaT%2Fimage.png?alt=media&#x26;token=13229d44-3ebb-44de-ba9e-8acdac3bfc1f" alt=""><figcaption></figcaption></figure>

Leave the default settings.  Copy the "**Linux Tar**" command and run it on your Polarity server to install the fleet agent.

After the fleet agent is installed it should automatically connect to your fleet agent console and appear as healthy.

<figure><img src="https://1063702622-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fliell3s8GjOV2GWs9poX%2Fuploads%2FXDigdIx8QGGyIwNmT1Pg%2Fimage.png?alt=media&#x26;token=270f4888-1181-4a05-ad7b-f95aaf84012e" alt=""><figcaption><p>Look for your agent policy on the Fleet "Agents" page</p></figcaption></figure>

If you used the default namespace and dataset name your logs will be collected under the datastream `logs-psa-default`.  To find this data stream navigate to "**Stack Management**" -> "**Index Management**" -> "**Data Streams**":

<figure><img src="https://1063702622-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fliell3s8GjOV2GWs9poX%2Fuploads%2FqM1PjD9Zi8jTWSnT0BrO%2Fimage.png?alt=media&#x26;token=43bfc515-04d2-4cb1-8cf8-4c1985b4ffdf" alt=""><figcaption></figcaption></figure>

### Configure a Data View

If you do not see the data stream and your Agent is reporting as "Healthy", ensure you have PSA enabled on the server and that a search has been run since you enabled it.

To make your data stream searchable you have to create a "Data View".  Navigate to "**Kibana**" -> "**Data Views**" and click on "**Create data view**".

Give the data view a name:

{% code title="Name" %}

```
Polarity Source Analytics
```

{% endcode %}

and then set the "**Index Pattern**":

{% code title="Index pattern" %}

```
logs-psa-default
```

{% endcode %}

You can leave the Timestamp field with the default setting of "**@timestamp**".

Click on "**Save data view to Kibana**"

### View your Data

You can view the raw source analytics by navigating to "**Analytics**" -> "**Discover**"

In the top left, filter to only show data from your newly created "**Polarity Source Analytics**" data view.

<figure><img src="https://1063702622-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fliell3s8GjOV2GWs9poX%2Fuploads%2F8H8vxqA8zukSri1rR9zP%2Fimage.png?alt=media&#x26;token=ed11c083-0236-48db-b832-fb5afcd4378e" alt=""><figcaption></figcaption></figure>

You should now see your Source Analytics Data available in Kibana.  To view the Source Analytics specific data you can click on a log file and then filter fields by the term "Polarity"

<figure><img src="https://1063702622-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fliell3s8GjOV2GWs9poX%2Fuploads%2Ffghq3lcLruevLoTwxfmK%2Fimage.png?alt=media&#x26;token=7904a808-3b8a-4017-8c56-3183e3ad23fd" alt=""><figcaption></figcaption></figure>

From here you can design Dashboards or install the Polarity Source Analytics for Elasticsearch integration.

{% hint style="info" %}
If you don't see any data, try adjusting your search time window and click on "Refresh"
{% endhint %}
