Atlas: How To Configure Webhooks

What are Atlas Webhooks?

Atlas webhooks are HTTP requests sent by Atlas before and after processing each Data Subject Request. Ethyca's Atlas allows customers to automatically fulfill their subjects' data rights by querying against data stored on their owned infrastructure. Atlas' webhooks will execute before and/or after that query.

38403840
  1. Your customer (a "Data Subject") puts in a request to download or erase their data in your Privacy Center (privacy.<yourdomain>.com)
  2. Ethyca stores and queues up all the requests for data that come into your Privacy Center
  3. Ethyca's Atlas, deployed in your infrastructure, polls Ethyca's web services to retrieve your organization's queue of Data Subject Requests (DSRs).
  4. Before proceeding to the database execution, Atlas will trigger the outbound pre-processing webhook to your service defined in the Atlas deployment. There is an optional callback response at this time.
  5. Atlas orchestrates requests to each of your managed databases using the PII map that you will create in the Atlas UI.
  6. After the database execution, Atlas will trigger the post-processing webhook action, which is a one-way call to your service.
  7. Atlas will then return an object of aggregated responses from the databases to Ethyca's web services. Ethyca's web services then manages the aggregation of the responses from your 3rd party SaaS products and sends the Data Subject an email containing a URL with a downloadable ZIP file to access their DSR.

When are Atlas webhooks useful?

We get it, your infrastructure may not need all the functionality that Atlas has to offer. Atlas webhooks allows your team the flexibility to extend the processes that Atlas is already managing and initiating, or forego using Atlas database querying functionality altogether. Utilizing Atlas Webhooks, you use HTTP requests to invoke custom business logic before and after every DSR processed by your Ethyca Atlas deployment. Because this infrastructure is unable to talk to Atlas directly, it is still important to utilize in processing access and erasure requests for many organizations.

How do I configure them on my Atlas deployment?

Step 1: Prerequisites

First, create your service and ensure that it's accessible by Atlas. This might require opening firewalls or changing security groups to allow for both ingress and egress from where Atlas resides to call your service and for your service to respond.

To configure Atlas instance to use hooks, there are 3 optional environment variables you must configure inside the ~/bc/configuration/atlas.properties file in your Atlas deployment:

atlas.hook.url  
atlas.hook.pre.response.wait.time.minutes
atlas.hook.post.response.wait.time.minutes

Example deployment configuration with webhooks configured:

docker run -it \  
-e atlas.organization=<ETHYCA-PROVIDED ORGANIZATION NAME> \  
-e atlas.ethyca.api.key=<ETHYCA-PROVIDED API KEY> \  
-e atlas.ethyca.url=https://api.ethyca.com \  
-e atlas.name=<NAME OF YOUR INSTANCE> \  
-e atlas.connection.db.jdbc.url=<CONNECTION STRING TO LOCAL DATABASE> \  
-e atlas.connection.db.jdbc.username=<LOCAL DB USERNAME> \  
-e atlas.connection.db.jdbc.password=<LOCAL DB PASSWORD> \  
-e atlas.schedule.cron='0 0 * ? * * *' \  
-e atlas.threads=10 \  
-e atlas.log.max.files = 28   
-e atlas.log.max.dir.size = 2G  
-e atlas.hook.url=’<YOUR API ENDPOINT URL>’ \  
-e atlas.hook.response.wait.time.minutes=’360’  \
-p 8080:8080 \  
--name atlas-container ethyca/atlas:2.0.0
  • while atlas.hook.url is optional for the overall Atlas deployment, webhooks will not be invoked unless present!
  • atlas.hook.response.wait.time.minutes will default to 10 minutes unless specified otherwise

Step 2: Atlas will call your service (pre-processing)

When Step 1 is complete, Atlas will trigger webhooks immediately after the request has been picked up by Atlas from Ethyca's web services.

Atlas will call your service using the following parameters:

URL -> from `atlas.hook.url` deployment property
Method -> POST
Header -> reply-to: "<Atlas-generated unique token\>"
Request Body ->
  {  
     callback_type: pre
     dsr_id: [SUBJECT REQUEST ID]
     dsr_type: [SAR or RTF]
     identity: [IDENTITY]
     halt: [true / false]
     reject: [true / false]
     reason: [text description for why halt or reject is TRUE]
    }
  • The Request Header originating from Atlas contains the reply-to with a unique token which you must use in your url path in the next step

Your service should respond with the following:

HTTP Status  -> 200 or 202
Response Body ->
{  
    // mandatory  
     callback_type: pre  
     dsr_id: [SUBJECT REQUEST ID]  
     dsr_type: [SAR or RTF]  
     identity: [IDENTITY]  
     halt: [true / false]  
       
     // optional  
     derived_identity: [new identity]  
     reject: [true / false]
     // mandatory when reject is TRUE (optional when halt is TRUE)
     reason: [message explaining reject:true or halt:true]
    }

Step 3: Your service may need to callback to Atlas

If your web service takes some time to perform its actions, Atlas will expect the following callback from your service:

URL -> http://<atlas URL>/<Token from request header>
Method -> POST
Request Body ->
{  
    // mandatory  
     callback_type: pre  
     dsr_id: [SUBJECT REQUEST ID]  
     dsr_type: [SAR or RTF]  
     identity: [IDENTITY]  
     halt: [true / false]  
       
    // optional  
     derived_identity: [new identity] 
     reject: [true / false]
     // mandatory when reject is TRUE
     reason: [message explaining reject:true]

    }

Atlas will respond with one of the following HTTP codes:

200 (OK) -> will process DSR   
410 (Gone) -> token doesn't exist or expired   
400 (Bad request) -> not all mandatory fields set  
400 (Bad request) -> token in header does not match dsr_id in body
  • The token is single redemption, and will otherwise expire in 10 minutes, or the value you've set in atlas.hook.response.wait.time.minutes
  • When your service responds with a derived_identity, you can provide a new identity that will replace the existing identity for Atlas to continue the database execution with.

Reject and Halt Flags

  • By configuring your service to respond to a request with the reject flag set to true, Atlas will contact Ethyca Services with a rejection request including the Subject Request ID and the optional rejection reason. This will be visible as an administrator in the Control Panel. The subject will also receive an email with the rejection reason.
  • By configuring your service to respond to a request with the halt flag set to true, Atlas will simply not continue processing the DSR request identified by the Subject Request ID. The request is stored in the database and will expire based on the token expiration time that is set ( atlas.hook.response.wait.time.minutes). Halting is only supported in a pre-processing call.
  • To un-halt, your service simply needs to send another prehook response to Atlas, similar to the previously sent halt response, but with halt set to false.
  • Reason is mandatory when reject is true.
  • If both reason and halt are set to true, the request will be rejected. Reject takes priority over halt

Derived Identity

  • A Derived Identity is an identity you've looked up as a response to Atlas' initial webhook call and responded to Atlas with, indicating that you'd like to use this new identity in the rest of the execution as the "identity" attribute. The identity attribute is used as the root of Atlas' identity lookup service.

Step 4: After Atlas has called to your connected databases, Atlas will call your service (post-processing)

After Atlas has finished executing database calls to all your connected databases, it will send one last call out to your web service:

URL -> from `atlas.hook.url` deployment property
Method -> POST
Header -> reply-to: "<Atlas-generated unique token\>"
Request Body ->
{  
     // mandatory
     callback_type: post  
     dsr_id: [SUBJECT REQUEST ID]  
     dsr_type: [SAR or RTF]  
     identity: [IDENTITY]  
     halt: [true / false]
     reject: [true / false]
     reason: [text description for why halt or reject is TRUE]
      
    // optional  
     derived_identity: [new identity]  
    }

Your service should respond with one of the following:

STATUS      202 or 200
BODY        {
               // mandatory
               callback_type: pre
               dsr_id:        [SUBJECT REQUEST ID]
               dsr_type:      [SAR or RTF]
               identity:      [IDENTITY]
               halt:          [true / false]
               reject:        [true / false]
               reason:        [text description for why halt or reject is TRUE]
               
               // optional
               derived_identity:  [new identity]
            }

After Atlas has finished executing, it will send one last call out to your web service:

URL -> from `atlas.hook.url` deployment property
Method -> POST
Request Body ->
{  
     // mandatory
     callback_type: post  
     dsr_id: [SUBJECT REQUEST ID]  
     dsr_type: [SAR or RTF]  
     identity: [IDENTITY]  
     halt: [true / false]
     reject: [true / false]
     reason: [text description for why halt or reject is TRUE]
      
    // optional  
     derived_identity: [new identity]  
    }