Deploying Polarity Server on AWS with RDS and Elasticache

Pre-requisites

• AWS Security Groups

  • EC2 Instance (allows 80/tcp & 443/tcp inbound from appropriate networks)

  • Postgres Database (allows 5432/tcp inbound from EC2 instance security group)

  • Cache Security Group (allows 6379/tcp inbound from EC2 instance security group)

  • ElastiCache Subnet Group (as required based on the VPC selected for ElastiCache instances)

  • RDS DB Subnet Group (as required based on the VPC selected during RDS setup)

• 4x AWS ElastiCache Redis 6.x Parameter Groups

  • Session Cache

  • Integration Cache

  • Metrics Cache

  • PG Cache

• 4x AWS ElastiCache Redis 6.x clusters (configuration details below)

• 1x AWS RDS PostgreSQL v13.4-r1 database (configuration details below)

• 1x AWS EC2 instance (configuration details below)

Overview

While the entire Polarity platform is typically deployed onto a single EC2 instance you can also leverage Elasticache for Redis and RDS when deploying into an AWS environment. With this setup the EC2 instance runs the NGINX web proxy, REST API, and integration framework while caching and database storage is provided by Elasticache and RDS.

Security Groups

3 security groups are recommended:

Security Group Name	Protocol Allowed	Port Allowed	Source
ec2	TCP (HTTP)	80	Subnets for Authorized Polarity Clients
	TCP (HTTPS)	443	Subnets for Authorized Polarity Clients
rds-polarity-database	TCP (Custom TCP)	5432	ec2 security group
ec-polarity-caches	TCP (Custom TCP)	6379	ec2 security group

ElastiCache

Parameter Groups

The 4 redis instances have specific configuration values that need to be applied. This requires creating four different ElastiCache Parameter Groups (session-cache, integration-cache, metrics-cache, and pg-cache) for redis6.x.

The Postgres database also requires specific configuration values and 1 RDS Parameter Group (postgresql-rds) will need to be created and applied to the RDS instance.

Parameter Group	Parameter Name	Parameter Value
polarity-session-cache	maxmemory-policy	noeviction
polarity-integration-cache	maxmemory-policy	volatile-ttl
	notify-keyspace-events	Ex
polarity-metrics-cache	maxmemory-policy	allkeys-lru
polarity-pg-cache	maxmemory-policy	allkeys-lru
polarity-rds (these settings assume a database with 32GiB of memory)	work_mem (in kb)	32768
	max_connections	200

Elasticache for Redis does not support RDB backups via the save directive in the same way regular Redis does. In addition, the appendonly parameter is not supported for modern versions of Redis. Instead, Elasticache for Redis supports daily backups via their backup and restore parameter. Further availability and reliability should be done via multi-AZ read replicas: Mitigating Failures - Amazon ElastiCache for Redis

Session Cache

Create a Redis 6 ElastiCache cluster for the session cache with the following settings:

• Cluster Engine: Redis

• Cluster Mode Enabled is unchecked

• Engine Version: 6.x

• Port: 6379

• Parameter Group: Choose the polarity-session-cache parameter group you previously created.

• Node Type: cache.t3.small

• Replicas: 2

• Multi-AZ: Enable if you have an appropriate subnet setup, otherwise do not check

• Subnet Group: Select as appropriate

• Security Group: Remove default and select the ec-polarity-caches security group you created earlier

• Encryption in-transit: can optionally be enabled

  • Redis AUTH: must be checked if Encryption in-transit is enabled

  • Access Control Option: Redis AUTH Default user

  • Redis Auth Token: <Set a token of your choice>

  • POLARITY_SESSION_CACHE_PASSWORD environment variable must be set on your polarity-server .env file if Redis AUTH is enabled.

Integration Cache

• Cluster Engine: Redis

• Cluster Mode Enabled is unchecked

• Engine Version: 6.x

• Parameter Group: Choose the polarity-integration-cache parameter group you previously created.

• Node Type: cache.m6g.large

• Replicas: 2

• Multi-AZ: Enable if you have an appropriate subnet setup, otherwise do not check

• Subnet Group: Select as appropriate

• Security Group: Remove default and select the ec-polarity-caches security group you created earlier

• Encryption in-transit: can optionally be enabled

  • Redis AUTH: must be checked if Encryption in-transit is enabled

  • Access Control Option: Redis AUTH Default user

  • Redis Auth Token: <Set a token of your choice>

  • POLARITY_INTEGRATION_CACHE_PASSWORD environment variable must be set on your polarity-server .env file if Redis AUTH is enabled.

Metrics Cache

• Cluster Engine: Redis

• Cluster Mode Enabled is unchecked

• Engine Version: 6.x

• Parameter Group: Choose the polarity-metrics-cache parameter group you previously created.

• Node Type: cache.m6g.large

• Replicas: 2

• Multi-AZ: Enable if you have an appropriate subnet setup, otherwise do not check

• Subnet Group: Select as appropriate

• Security Group: Remove default and select the ec-polarity-caches security group you created earlier

• Encryption in-transit: can optionally be enabled

  • Redis AUTH: must be checked if Encryption in-transit is enabled

  • Access Control Option: Redis AUTH Default user

  • Redis Auth Token: <Set a token of your choice>

  • POLARITY_METRICS_CACHE_PASSWORD environment variable must be set on your polarity-server .env file if Redis AUTH is enabled.

PG Cache

• Cluster Engine: Redis

• Cluster Mode Enabled is unchecked

• Engine Version: 6.x

• Parameter Group: Choose the polarity-pg-cache parameter group you previously created.

• Node Type: cache.t3.small

• Replicas: 2

• Multi-AZ: Enable if you have an appropriate subnet setup, otherwise do not check

• Subnet Group: Select as appropriate

• Security Group: Remove default and select the ec-polarity-caches security group you created earlier

• Encryption in-transit: can optionally be enabled

  • Redis AUTH: must be checked if Encryption in-transit is enabled

  • Access Control Option: Redis AUTH Default user

  • Redis Auth Token: <Set a token of your choice>

  • POLARITY_PG_CACHE_PASSWORD environment variable must be set on your polarity-server .env file if Redis AUTH is enabled.

AWS RDS PostgreSQL

Create an RDS PostgreSQL database using the Standard Creation method with the following settings:

Engine Options

• Engine Type: PostgreSQL

• Version: 13.3-R1

Templates

• Template: Production

Settings

• DB instance identifier: customer specific

• Master username: breach

• Master password: specify or let RDS auto generate

DB Instance Size

• DB instance class

  • Standard class: db.m5.2xlarge

Storage

• Storage Type: Provisioned IOPS (SSD)

• Allocated storage: 100GiB

• Provisioned IOPS: 3000

• Enable storage autoscaling: checked

• Maximum storage threshold: 1000

Availability & durability

• Multi-AZ deployment: create a standby instance

Connectivity

• Virtual private cloud: customer specific

• Subnet group: customer specific (must be created ahead of time)

• Public Access: No

• VPC Security Group: Choose existing

  • Remove default

  • Add rds-polarity-database

• Database port: 5432

Database Authentication

• Password authentication

Additional configuration

Database Options

• Initial database name: breach

• DB parameter group: polarity-rds

• Backup/Encryption/Monitoring: Maintenance: customer specific

When “View Credential Details” opens, please ensure you save your database credentials in a safe location as you cannot retrieve them once you close the window.

Polarity Server

EC2 Instance

Create an EC2 instance for the Polarity Server with the following settings:

• Instance AMI: “CentOS 7 (x86_64) - with Updates HVM”

• Instance Type: m5.xlarge

• EBS Storage: at least 40GB General Purpose SSD

• Existing Security Group: ec2

Install Polarity Server

Pre-Installation Steps

Download Installation TGZ

Untar the installation file and change into the installation directory:

tar -xzvf polarity-server-4.1.0-web-4.1.0-el7.tgz
cd polarity-server-4.1.0-web-4.1.0

After untaring the upgrade package you will find the following files:

• polarity-web-4.1.0-1.el7.noarch.rpm

• polarity-server-4.1.0-1.el7.x86_64.rpm

• polarity-metrics-cache-1.0.16+6.2.4-1.el7.x86_64.rpm

• polarity-integration-cache-1.0.16+6.2.4-1.el7.x86_64.rpm

• polarity-session-cache-1.0.16+6.2.4-1.el7.x86_64.rpm

• polarity-pg-cache-1.0.16+6.2.4-1.el7.x86_64.rpm

• polarity-nodejs-1.0.1+12.22.5-1.el7.x86_64.rpm

• polarity-nodejs-devel-1.0.1+12.22.5-1.el7.x86_64.rpm

• polarity-nodejs-npm-1.0.1+12.22.5-1.el7.x86_64.rpm

• upgrade-to-polarity-server-4.1.0.sh

• SHA256SUMS

We will not be using the upgrade-to-polarity-server-4.1.0.sh script for this install

Installation Steps

Optional: Update System

yum clean all
yum update -y

Install Base Dependencies

yum install -y epel-release
yum install https://download.postgresql.org/pub/repos/yum/\
reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm -y
yum install wget ansible libselinux-python nginx gcc-c++ \
postgresql13-server postgresql13-contrib postgresql13-libs \
git ip4r13 python-psycopg2 policycoreutils-python \
libicu -y

Install Node RPMs

yum install polarity-nodejs-*.rpm -y

Install the Polarity Server. The Polarity Server install process can take up to 15 minutes. Please be patient while the RPM installs.

yum install polarity-server-4.1.0-1.el7.x86_64.rpm -y

After the RPM completes installation you’ll notice an error at the end indicating that the database migrations failed to run. This migration error is expected as we will not be using the local postgresql instance for our deployment.

/// EXPECTED ERROR
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! polarity-server@3.12.8 migrations:run: \
`./migration/scripts/init-db.sh && ./node_modules/.bin/ts-node \
--transpile-only ./node_modules/.bin/typeorm migration:run -t=false`

Stop the local copy of PostgreSQL as we will be using our RDS instance.

systemctl stop postgresql-13
systemctl disable postgresql-13

The local copy of PostgreSQL is required to as it is a requirement for the RPM. We will not be connecting to it however and can safely stop the service.

Install Polarity Web

yum install polarity-web-4.1.0-1.el7.noarch.rpm -y

Modify Cache and PostgreSQL Configuration

Open the environment file to edit connection settings for the database and caches.

vi /app/polarity-server/.env

Set the following environment variables:

POLARITY_INITIALIZE_REDIS_CONFIGS_ON_START=false

POLARITY_SESSION_CACHE_HOST=<your-amazon-polarity-session-cache-host>
POLARITY_SESSION_CACHE_PORT=6379
POLARITY_SESSION_CACHE_DB=0
POLARITY_SESSION_CACHE_TTL=2592000

POLARITY_INTEGRATION_CACHE_HOST=<your-amazon-polarity-integration-cache-host>
POLARITY_INTEGRATION_CACHE_PORT=6379
POLARITY_INTEGRATION_CACHE_DB=0

POLARITY_PG_CACHE_HOST=<your-amazon-polarity-pg-cache-host>
POLARITY_PG_CACHE_PORT=6379
POLARITY_PG_CACHE_DB=0

POLARITY_METRICS_REDIS_HOST=<your metrics cache host on aws>
POLARITY_METRICS_REDIS_PORT=6379
POLARITY_METRICS_REDIS_DB=0

POLARITY_DB_SUPERUSER_USER=breach
POLARITY_DB_SUPERUSER_PASSWORD=<RDS "breach" user password here>
POLARITY_DB_HOST=<RDS Endpoint Hostname Here>
POLARITY_DB_PORT=5432
POLARITY_DB_DATABASE=breach
POLARITY_DB_USER=polarity
POLARITY_DB_SCHEMA=polarity
POLARITY_DB_PASSWORD=<RDS "polarity" user password here>

# The following cache passwords are only required if you enabled 
# Redis AUTH when standing up your Elasticache for Redis instances
POLARITY_INTEGRATION_CACHE_PASSWORD=<integration cache auth password>
POLARITY_METRICS_CACHE_PASSWORD=<metrics cache auth password>
POLARITY_PG_CACHE_PASSWORD=<pg cache auth password>
POLARITY_SESSION_CACHE_PASSWORD=<session cache password>

# Required only if the rds.force_ssl parameter was set in the polarity-rds 
# parameter group
PGSSLMODE=require

Note that you must set POLARITY_DB_SUPERUSER_PASSWORD to password set when standing up the RDS instance in AWS.

Note that you can set POLARITY_DB_PASSWORD to any sufficiently random string

Next open up the Polarity Server config.js file which will require modifications

vi /app/polarity-server/config/config.js

Set the following configuration values for the Caches

pufferfish.redisStateStore.host: "<your session cache host on aws>"
pufferfish.redisStateStore.port: 6379
pufferfish.redisStateStore.db: 2

postgres.database: "breach"
postgres.port: 5432
postgres.host: '<your RDS host>'
postgres.user: 'breach'
postgres.password: '<your RDS breach user password>'
postgres.schema: 'public'

Note that the we are using JSON dot notation to represent the nested JSON structure of the config file

Initialize RDS Database

Save the following script to a file called setup_rds_db.sh

#!/usr/bin/env bash
# setup_rds_db.sh
set -Eeo pipefail
source /app/polarity-server/.env
export PGPASSWORD="$POLARITY_DB_SUPERUSER_PASSWORD"
echo "start IS_INITIALIZED"
IS_INITIALIZED=$(psql -U "$POLARITY_DB_SUPERUSER_USER" \
 -d "$POLARITY_DB_DATABASE" -h "$POLARITY_DB_HOST" -p \
  "$POLARITY_DB_PORT" -t -c "SELECT to_regclass('polarity.users')")
if [[ "${IS_INITIALIZED}" =~ "users" ]]; then
  echo "Polarity Server database has already been initialized"
  exit 0
fi
echo "Initializing Polarity Server database"
echo "start create user"
psql -v ON_ERROR_STOP=1 -h "$POLARITY_DB_HOST" -p "$POLARITY_DB_PORT" \
--username "$POLARITY_DB_SUPERUSER_USER" --dbname postgres <<-EOSQL
    CREATE ROLE "$POLARITY_DB_USER" WITH LOGIN PASSWORD '$POLARITY_DB_PASSWORD';
    GRANT "$POLARITY_DB_USER" to "$POLARITY_DB_SUPERUSER_USER";
EOSQL
echo "start create schema"
psql -v ON_ERROR_STOP=1 -h "$POLARITY_DB_HOST" -p "$POLARITY_DB_PORT" \
--username "$POLARITY_DB_SUPERUSER_USER" --dbname "$POLARITY_DB_DATABASE" <<-EOSQL
    CREATE SCHEMA polarity AUTHORIZATION "$POLARITY_DB_USER";
EOSQL
echo "start grant all"
psql -v ON_ERROR_STOP=1 -h "$POLARITY_DB_HOST" -p "$POLARITY_DB_PORT" \
--username "$POLARITY_DB_SUPERUSER_USER" --dbname "$POLARITY_DB_DATABASE" <<-EOSQL
    GRANT ALL PRIVILEGES ON SCHEMA polarity to "$POLARITY_DB_USER";
EOSQL
psql -v -U "$POLARITY_DB_SUPERUSER_USER" -d "$POLARITY_DB_DATABASE" \
-h "$POLARITY_DB_HOST" -p "$POLARITY_DB_PORT" \
-f "/app/polarity-server/deploy/sql/install/setup-schema-v3.6.sql"
echo "Polarity Server database initialization complete."

Ensure the RDS setup script is executable:

chmod a+x setup_rds_db.sh

Execute the script:

./setup_rds_db.sh

Run the Polarity database migration script

cd /app/polarity-server
npm run migrations:run

License Installation

At this point you will need to intall your Polarity License provided to you by your Polarity support or customer success team. The license file will be named polarity.lic.

Copy the polarity.lic file to the license directory

cp polarity.lic /app/polarity-server/license/polarity.lic

Ensure the license file is owned by the polarityd user

chown polarityd:polarityd /app/polarity-server/license/polarity.lic

Finally restart your server so the license is loaded

systemctl restart polarityd

You can now navigate to the polarity web interface by opening a browser (Chrome is recommended) and navigating to https://<your-polarity-server>

You can login with the default user admin and the password PolarityServer2015!

Update FQDN in Polarity Config

By default the Polarity server will assume the FQDN for your server matches the server hostname. If this is not the case, you will need to modify the Polarity Server config file to set the appropriate config.

To modify the Polarity server config begin by opening the config file in an editor.

vi /app/polarity-server/config/config.js

Find the setting rest.fullyQualifiedDomainName and set it to your FQDN (you can also set this to an IP address):

"rest": {
   "fullyQualifiedDomainName": "your.fqdn.com"
}

Save the change and restart the server:

systemctl restart polarityd

Enabling SSL between Polarity Server and RDS

By default, the underlying postgresql connector will ignore errors related to untrusted certificates. To enforce Open the .env file located at /app/polarity-server/.env and add the environment variable:

PGSSLMODE=require

This will enable SSL connections. By default, node-pg will be set to rejectUnauthorized: false and thus will allow untrusted SSL connections to happen. Both polarity-server, typeorm, and Slonik use node-pg version 7 which has this behavior.

Starting in node-pg v8+ the default behavior is for rejectUnauthorized: true which means the above env var along will not be enough to connect to most PostgreSQL servers. See release notes here https://github.com/brianc/node-postgres/blob/54b87523e29ea53379d7b9a26e45f83886f371af/CHANGELOG.md#pg800

In addition to turning SSL on via PGSSLMODE we can provide a custom CA via the NODE_EXTRA_CA_CERTS environment variable:

NODE_EXTRA_CA_CERTS=/etc/ssl/certs/rds-combined-ca-bundle.pem

This has been tested to work for node-pg and typeorm but has not been explicitly tested to work with slonik. If you are running migrations via npm run migrations:run then you need to pass this variable explicitly into the command as it is not automatically picked up by the dotenv package the same way the Polarity server is.

NODE_EXTRA_CA_CERTS=rds-combined-ca-bundle.pem npm run migrations:run