ETL Pipelines
ETL Pipelines integration is available for the Scale plan.
Events can be published directly to a supported message queue source of your ETL pipelines. This means you need not provision Internet reachable, unauthenticated HTTPS endpoints in order to ingest Vital events as webhooks.
Scalable
Offload the pressure on your public HTTP services.
Secure
Events are published through authenticated channels over TLS.
Compressed
Data events larger than 1 KiB are compressed before publication.
Ordered
Publication order can be preserved for select destination types.
Destinations
The following destinations are supported:
| Name | Authentication | Publication Order |
|---|---|---|
| RabbitMQ | Password | Depends on your RabbitMQ broker configuration. |
| Google Cloud Pub/Sub | IAM | Pub/Sub Message Ordering is supported. |
For evaluation and zero downtime transition, Vital supports double writing both to your Svix webhook endpoints, and to the message queue source of your ETL pipelines.
Event Schema
Vital ETL Pipelines integration publishes all events in the exact same JSON schema as our default Svix webhook offering. Check out our Event Catalog.
Google Cloud Pub/Sub
Message Structure
Each message would be published by Vital message queue with the following attributes:
| Type | Key | Value |
|---|---|---|
| Attribute | event_type | Vital event type, e.g., daily.data.activity.created |
| Attribute | blob_type | json : UTF8 JSON document |
| Attribute | blob_codec | gzip: The blob is compressed by gzip. null or absent: The blob is uncompressed. |
| Data | - | Encoded JSON blob. This may or may not be compressed — see blob_codec. |
| Ordering Key | - | Vital User ID (If you have enabled Message Ordering) |
At this time, Vital message queue publishes exclusively JSON blobs (blob_type == json).
Having said that, you are strongly encouraged to check for and drop any messages with unrecognized blob_type and blob_codec.
Pub/Sub message ordering
If you wish to receive message for each Vital user in their publication order, Vital message queue supports Pub/Sub Message Ordering, and uses the Vital User ID as the message ordering key.
Vital message queue publishes all messages through the us-central1 and europe-west1 PubSub regional endpoints.
Pub/Sub Exactly Once Delivery is discouraged to be used together with Message Ordering. Our own trial suggested that the combination would struggle to move high volume of messages.
IAM Permission
Grant the relevant IAM Principals on your topic with these pre-defined roles:
- Pub/Sub Publisher (
roles/pubsub.publisher) - Pub/Sub Viewer (
roles/pubsub.viewer)
Or more specifically these permissions if you wish to create a custom IAM role with a minimized grant:
pubsub.topics.getpubsub.topics.publish
| Environment | Region | Service Account |
|---|---|---|
| Sandbox | US | customer-topics-us@vital-sandbox.iam.gserviceaccount.com |
| Sandbox | Europe | customer-topics-eu@vital-sandbox.iam.gserviceaccount.com |
| Production | US | customer-topics-us@vital-prod-307415.iam.gserviceaccount.com |
| Production | Europe | customer-topics-eu@vital-prod-307415.iam.gserviceaccount.com |
Information needed by Vital
When you have completed the setup, please provide Vital the following information:
- Your Google Cloud Project ID
- Your Cloud Pub/Sub Topic name
- Whether or not you want messages to be published under Pub/Sub Message Ordering
- Whether or not you want larger blobs (≥ 1KiB) to be gzipped
RabbitMQ
Message Structure
Each message would be published by Vital message queue with the following attributes:
| Key | Value |
|---|---|
| Routing Key | Vital event type, e.g., daily.data.activity.created |
| Content-Type | application/json : UTF8 JSON document |
| Content-Encoding | gzip: The blob is compressed by gzip. null/absent: The blob is uncompressed. |
| Data | Encoded JSON blob. This may or may not be compressed — see blob_codec. |
At this time, Vital message queue publishes exclusively JSON blobs (blob_type == json).
Having said that, you are strongly encouraged to check for and drop any messages with unrecognized Content-Type and Content-Encoding.
Exchange requirements
Vital message queue only supports password authentication at this time.
Vital message queue publishes messages with the following settings:
- Publisher Confirm mode is required
- Messages are published with the Mandatory flag but without the Immediate flag.
- Messages are published with Event Type as the Routing Key.
Please get in touch if these need to be configured differently for your RabbitMQ exchange.
Information needed by Vital
When you have completed the setup, please provide Vital the following information:
- An
ampqs://connection string with username and password embedded - Whether or not you want larger blobs (≥ 1KiB) to be gzipped.