How To: Deploy and Set Up Atlas 1.5

It's your first time deploying your shiny new Atlas 1.5

Ethyca's Atlas allows customers to fulfill their subjects’ data rights by querying against data stored on their owned infrastructure.

  1. Your customer (a "Data Subject") puts in a request to download or erase their data in your Privacy Center (privacy.yourdomain.com)
  2. Ethyca stores and queues up all the requests for data that come into your Privacy Center
  3. Ethyca's Atlas, deployed in your infrastructure, polls Ethyca's web services to retrieve your organization's queue of Data Subject Requests (DSRs)
  4. Ethyca's Atlas orchestrates requests to each of your managed databases using the PII map that you will create in the Atlas UI
  5. Ethyca's Atlas returns an object of aggregated responses from the databases to Ethyca's web services. Ethyca's web services then manages the aggregation of the responses from your 3rd party SaaS products and sends the Data Subject an email containing a URL with a downloadable ZIP file to access their DSR

This guide will outline the steps to deploy and configure Atlas.

Prerequisites

  • A command-line environment (preferably Linux) to run CLI commands
  • Ability to securely receive API keys from Ethyca
  • Ability to run the docker commands/launch docker images
  • Access to a test subject’s email for the purposes of requesting data from the Privacy Center, validating Multi-Factor Authentication (MFA), and verifying the final data
  • Administrative access to customer’s Control Panel to approve the test subject’s DSR request

Atlas Resource Requirements

The machine running Atlas should have at least the following specifications:

ResourceMinimum Size
RAM4GiB
Storage1GiB

Instance Recommendations:

AWS - t2.medium or larger


Deploying Atlas

Ethyca's Atlas and the configuration tool are provided as Docker images. Instructions include details of steps needed for downloading the software images.

Your Ethyca Customer Success Manager will provide you the following information for your Environment Variable Properties file:

  • Link to Ethyca's Docker Hub Public Repo
  • API Key for your deployed Atlas to connect to Ethyca’s APIs: atlas.ethyca.api.key
  • Your unique organization name

Step 1. Pull down the latest image from the Ethyca Atlas Docker Repo.
Use the following command to pull down the latest image from the public Ethyca Atlas docker repository:

docker pull ethyca/atlas

Step 2. Create a table in a database reachable by Atlas (only if you did not previously have Atlas 1.0+ installed)

In order for Atlas 1.5 to store your database credentials locally and to support future versions of Atlas, we do require a small table to be created in one of your SQL databases.

CREATE TABLE ethyca_atlas_connection ( 
id VARCHAR(50) PRIMARY KEY, 
name VARCHAR(50) NOT NULL, 
parameters TEXT NOT NULL, 
CONSTRAINT ethyca_atlas_connection_unq UNIQUE (name) 
);

Make sure to have the connection string to this database handy for the next step!

Step 3. Deploy your Atlas instance

docker run -it \
-e atlas.organization=<ETHYCA-PROVIDED ORGANIZATION NAME> \
-e atlas.ethyca.api.key=<ETHYCA-PROVIDED API KEY> \
-e atlas.ethyca.url=https://api.ethyca.com \
-e atlas.name=<NAME OF YOUR INSTANCE> \
-e atlas.connection.db.jdbc.url=<CONNECTION STRING TO LOCAL DATABASE> \
-e atlas.connection.db.jdbc.username=<LOCAL DB USERNAME> \
-e atlas.connection.db.jdbc.password=<LOCAL DB PASSWORD> \
-e atlas.schedule.cron='0 0 * ? * * *' \
-e atlas.threads=10 \
-e atlas.log.max.files = 28 
-e atlas.log.max.dir.size = 2G
-p 8080:8080 \
--name atlas-container ethyca/atlas:latest

The following settings can be modified to customize your deployment:

  • atlas.schedule.cron='0 0 ? * *': this is the notation for the frequency by which your deployed Atlas instance will poll Ethyca's web services for pending DSRs.
  • atlas.threads: this is the maximum number of concurrent subject requests that can run in Atlas simultaneously
  • atlas.log.max.files: this is the maximum number of files that we will store in the log directory (ethyca/logs/)
  • atlas.log.max.dir.size: this is the maximum directory size for ethyca/logs/, it can be specified in the format positive integer+size . The acceptable values for size is "M" and "G", representing Megabyte and Gigabyte, respectively. (ie: 100M, 2G).
  • atlas.use.ssl: (optional) sets a "secure" flag in the cookie for the Atlas UI. Accepted values are "true" and "false", where "false" is the default value if not explicitly set.
  • atlas.name: a memorable, semantic label for you to recognize this instance once deployed

The Finishing Touches

🚨 Do not forget to safe-list or open outbound communication to Ethyca-hosted web services at api.ethyca.com


Setting up Atlas

To confirm successful deployment, you can enter the url (or IP address) that you deployed Atlas to into a browser on whatever host your deployment is located at and then proceed to the next steps.

Step 1. Log in to Atlas
Using your Ethyca Control Panel credentials, you can log into your Atlas UI.

Pass the 2FA email code challenge and you're authenticated.

Step 2: Create a new database connection

  1. Press "Setup"
  2. Select the type of database for the connection
  3. Enter the connection details in the following fields:

Name: this is a memorable label for your database connection

URL: this is the url that Atlas will be able to access your database at.

  • for MongoDB, prefix the url with mongodb+srv://
  • for Snowflake, prefix the url with jdbc:snowflake://
  • for Databricks, prefix the url with jdbc:spark://. If Databricks 7.3, affix the following to the end of the connection string ;UID=token;PWD=<yourpassword>
  • for Redshift, prefix the url with jdbc:redshift://
  • for MySQL, prefix the url with jdbc:mysql://
  • for PostgreSQL, prefix the url with jdbc:postgresql://
  • for Microsoft SQL Server, prefix the url with jdbc:sqlserver://
  • for MariaDB, prefix the url with jdbc:mariadb://
  • for GCP BigQuery, your connection string should contain a local link to your GCP credential file: jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=<YOURPROJECT>;DefaultDataset=<DBNAME>;OAuthType=0;OAuthServiceAcctEmail=<ACCOUNTNAME>@<PROJECTID>.iam.gserviceaccount.com;OAuthPvtKeyPath=/ethyca/configuration/gcp_credentials.json;Timeout=30;

You can create your credentials file following these instructions here.

Username and Password: these are the credentials for a database user that we recommend you provision specifically for Ethyca's Atlas. Please ensure that these have READ and WRITE access to the database and that the container where Atlas is deployed is able to access the database (ie: IP permissions list, security policy, etc)

"Can Access" and "Can Erase" checkboxes: this allows you additional control over what queries Atlas will construct in order to fulfill a subject request. By selecting the "Can Access" permission, Atlas will only SELECT from your database. By selecting the "Can Delete" permission, Atlas will only UPDATE your database; Atlas will never DELETE or DROP.

After entering all information, press "Save Connection", and Atlas will test the db connection for you instantly. On failure, a banner will appear stating that the connection test failed and to check your credentials.

On successful connection, the database will be added and you'll be taken to the Connections Listing page.

Step 3: Select how you'd like to map your PII

In Atlas 1.5, we can connect to your database schema so you can identify where PII lives within it. This can be accomplished in one of two ways:

  1. Your Ethyca Customer Success Manager provides a .yaml file, also referred to as an Ethyca "Structure File", for each database connection you have configured. By pressing "UPLOAD YAML", you can upload this file here.
  2. To kick off the mapping activity without a .yaml file, press "UPDATE PII MAPPING" and you'll be taken to the mapping activity.

Step 4: Review the mapping for your PII
Now that you've connected a database accessible to Atlas, we can start identifying where PII lives in the database. The mapping activity will load your database schema (both tables and their requisite columns), so you can begin to identify which columns individually contain personally identifiable information table-by-table.

To begin, follow these 3 steps:

1. Select a table from the left-side listing. You'll see the listing of tables (A) that are accessible from the database you've just connected (B).

  1. Review the columns that populate in the center of the page. We will step through these row-by-row, indicating which of the detected columns contain PII, and for any PII that's there, classifying it and applying a label, masking strategy, and relationship linkage, where necessary.
  1. Remember to save!

Validating Your Deployment and Setup

Your Ethyca Customer Success team will be here every step of the way to ensure that your infrastructure environment is ready for Atlas and DSR-S. However, you can also validate your setup with a couple basic checks, such as:

  • accessing the host/IP on port 8080 where Atlas is deployed from within the container of VPC
  • confirming that the docker image is running locally

Validating your deployment

On startup, when your deployment initializes, Atlas will verify all connection configurations and structure files. Upon successful deployment, your deployed Atlas will register its status with Ethyca's web services. When Atlas is fully deployed, you'll see the following application log line:

INFO [DD-MM-YY HH:MM:SS] com.ethyca.atlas.job.DsrJob : DSR job started

If the startup fails, then there may be a problem with the connection or structure files. Contact your Ethyca Customer Success Manager for additional details.

If the startup succeeds, then your deployment of Atlas has successfully registered with Ethyca and you’re all ready to go!

When a subject request has been approved from your Ethyca Control Panel, you will see an Atlas application log line confirming the request has been accepted by Atlas and then executed on. Each step of the query execution is logged, and when it's finally completed, it will log that the request has completed its work in Atlas, like this:

Validating your Atlas setup with end to end testing

  1. First, follow the instructions in our guide to testing a Download My Data request
  2. After a Download request has been tested, follow our guide to testing an Erasure request.

Support

Please contact [email protected] or your dedicated Customer Success Manager if you require any technical assistance for the installation or configuration of Ethyca's Atlas.

Updated a day ago

How To: Deploy and Set Up Atlas 1.5


It's your first time deploying your shiny new Atlas 1.5

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.