Configuring Source Analytics on Elasticsearch
The following guide walks through how to collect source analytic logs from your server using Elasticsearch.
Last updated
The following guide walks through how to collect source analytic logs from your server using Elasticsearch.
Last updated
Prior to setting up collection of your source analytic (PSA) logs, please enable the logging on your server.
Once, source analytic are being collected on your Polarity Server you can configure Elasticsearch to receive those logs.
Login to your Elasticsearch Kibana instance.
Navigate to the "Management" -> "Fleet" page
Click on "Agent Policies"
Click on "Create agent policy"
Name the policy. For example "polarity-source-analytics"
Decide if you would also like to collect system logs and metrics (note that this is not required for Source Analytics collection)
The default "Advanced options" will work but you may want to make changes depending on your organization.
For example, you might want to add an optional description or modify the "Default namespace".
Click on "Create agent policy"
Your new policy will be created but still needs to be configured.
Your new policy will show up in the Fleet list under "Agent Policies". Click on it to view the details:
Click on "Add integration"
Search for the "Custom Logs" integration and click on it:
Click on the "Add Custom Logs" button.
Under the Custom Logs configuration, set the Integration name:
Set the description.
Ensure "Custom log file" is checked and then expand the "Change defaults" section:
Set the "Log file path"
Click on "Advanced Options"
Set the "Dataset name"
In the "Processors" section paste the following configuration:
Click on "Save and continue"
When prompted click on "Add Elastic Agent to your hosts"
Leave the default settings. Copy the "Linux Tar" command and run it on your Polarity server to install the fleet agent.
After the fleet agent is installed it should automatically connect to your fleet agent console and appear as healthy.
If you used the default namespace and dataset name your logs will be collected under the datastream logs-psa-default. To find this data stream navigate to "Stack Management" -> "Index Management" -> "Data Streams":
If you do not see the data stream and your Agent is reporting as "Healthy", ensure you have PSA enabled on the server and that a search has been run since you enabled it.
To make your data stream searchable you have to create a "Data View". Navigate to "Kibana" -> "Data Views" and click on "Create data view".
Give the data view a name:
and then set the "Index Pattern":
You can leave the Timestamp field with the default setting of "@timestamp".
Click on "Save data view to Kibana"
You can view the raw source analytics by navigating to "Analytics" -> "Discover"
In the top left, filter to only show data from your newly created "Polarity Source Analytics" data view.
You should now see your Source Analytics Data available in Kibana. To view the Source Analytics specific data you can click on a log file and then filter fields by the term "Polarity"
From here you can design Dashboards or install the Polarity Source Analytics for Elasticsearch integration.
