ETL Pipelines
ETL Pipelines integration is available for the Scale plan.
Events can be published directly to a supported message queue source of your ETL pipelines. This means you need not provision Internet reachable, unauthenticated HTTPS endpoints in order to ingest Vital events as webhooks.
Scalable
Offload the pressure on your public HTTP services.
Secure
Events are published through authenticated channels over TLS.
Compressed
Data events larger than 1 KiB are compressed before publication.
Ordered
Publication order can be preserved for select destination types.
Destinations
The following destinations are supported:
| Name | Authentication | Publication Order |
|---|---|---|
| RabbitMQ | Password | Depends on your RabbitMQ broker configuration. |
| Google Cloud Pub/Sub | IAM | Pub/Sub Message Ordering is supported. |
| Azure Event Hub | SAS Key | Depends on your Azure Event Hub configuration. |
Event Schema
Webhooks, ETL Pipelines and our API endpoints all share the same JSON object schema. Webhooks and ETL Pipelines use identical JSON event payload structure.
Check out our Event Catalog.
Features
Data compression
Vital does not compress payload blobs that are smaller than 1 KiB. Please refer to the destination-specific documentation below on how to detect a compressed blob versus an uncompressed blob.
Static outbound IPs
Vital establish connections to your ETL Pipeline destination through a static pool of IPs. Contact Vital support to obtain the up-to-date static IP list.
Double writing during transition
To ensure smooth switchover from Webhooks to your new ETL Pipelines destination, Vital supports double writing events to the combination of:
- all your Webhook endpoints; and
- your ETL Pipelines destination.
Pushed Historical Data
When you use ETL Pipelines, Vital can push all the historical data as daily.data.* data events to your ETL Pipelines
destination.
The data-less historical.data.*.created event would still be emitted to denote the completion of historical pull.
Configuring ETL Pipelines
You can change the ETL Pipelines configuration on your Teams at any time through the Set Team ETL Pipelines endpoint of the Org Management API.
Your new configuration is typically active as soon as the endpoint has acknowledged your request.
Feel free to reach out Vital support if you need assisted setup, or if you need a Vital Org Key to access the Org Management API.
Google Cloud Pub/Sub
Message Structure
Each message would be published by Vital with the following attributes:
| Type | Key | Value |
|---|---|---|
| Attribute | event_type | Vital event type, e.g., daily.data.activity.created |
| Attribute | blob_type | json : UTF8 JSON document |
| Attribute | blob_codec | gzip: The blob is compressed by gzip. null or absent: The blob is uncompressed. |
| Attribute | idempotency_key | Unique key that allows detection of subsequent retries of the same request |
| Data | - | Encoded JSON blob. This may or may not be compressed — see blob_codec. |
| Ordering Key | - | Vital User ID (If you have enabled Message Ordering) |
At this time, Vital publishes exclusively JSON blobs (blob_type == json).
Having said that, you are strongly encouraged to check for and drop any messages with unrecognized blob_type and blob_codec.
Pub/Sub message ordering
If you wish to receive message for each Vital user in their publication order, Vital supports Pub/Sub Message Ordering, and uses the Vital User ID as the message ordering key.
Vital publishes all messages through the us-central1 and europe-west1 PubSub regional endpoints.
Pub/Sub Exactly Once Delivery is discouraged to be used together with Message Ordering. Our own trial suggested that the combination would struggle to move high volume of messages.
IAM Permission
Grant the relevant IAM Principals on your topic with these pre-defined roles:
- Pub/Sub Publisher (
roles/pubsub.publisher) - Pub/Sub Viewer (
roles/pubsub.viewer)
Or more specifically these permissions if you wish to create a custom IAM role with a minimized grant:
pubsub.topics.getpubsub.topics.publish
| Environment | Region | Service Account |
|---|---|---|
| Sandbox | US | customer-topics-us@vital-sandbox.iam.gserviceaccount.com |
| Sandbox | Europe | customer-topics-eu@vital-sandbox.iam.gserviceaccount.com |
| Production | US | customer-topics-us@vital-prod-307415.iam.gserviceaccount.com |
| Production | Europe | customer-topics-eu@vital-prod-307415.iam.gserviceaccount.com |
RabbitMQ
Message Structure
Each message would be published by Vital with the following attributes:
| Key | Value |
|---|---|
| Routing Key | Vital event type, e.g., daily.data.activity.created |
| Content-Type | application/json : UTF8 JSON document |
| Content-Encoding | gzip: The blob is compressed by gzip. null/absent: The blob is uncompressed. |
| Data | Encoded JSON blob. This may or may not be compressed — see blob_codec. |
At this time, Vital publishes exclusively JSON blobs (blob_type == json).
Having said that, you are strongly encouraged to check for and drop any messages with unrecognized Content-Type and Content-Encoding.
Exchange requirements
Vital only supports password authentication at this time.
Vital publishes messages with the following settings:
- Publisher Confirm mode is required
- Messages are published with the Mandatory flag but without the Immediate flag.
- Messages are published with Event Type as the Routing Key.
Please get in touch if these need to be configured differently for your RabbitMQ exchange.
Azure Event Hub
Message Structure
Each message would be published by Vital with the following attributes:
| Type | Key | Value |
|---|---|---|
| Attribute | event_type | Vital event type, e.g., daily.data.activity.created |
| Attribute | blob_type | json: UTF8 JSON document |
| Attribute | blob_codec | gzip: The blob is compressed by gzip. null or absent: The blob is uncompressed. |
| Attribute | idempotency_key | Unique key that allows detection of subsequent retries of the same request |
| Data | - | Encoded JSON blob. This may or may not be compressed — see blob_codec. |
| Partition Key | - | Vital User ID |
At this time, Vital publishes exclusively JSON blobs (blob_type == json).
Having said that, you are strongly encouraged to check for and drop any messages with unrecognized Content-Type and Content-Encoding.