ETL Pipelines
ETL Pipelines integration is available for the Scale plan.
With ETL Pipelines, Vital streams all Vital events continuously to a supported destination over direct authenticated TLS connections. Compared to Webhooks, this simplifies your operational complexity by removing the need to operate public unauthenticated HTTPS endpoints for receiving incoming Webhooks.
Scalable
Offload the pressure on your public HTTP services.
Secure
Events are published through authenticated channels over TLS.
Compressed
Data events larger than 1 KiB are compressed before publication.
Ordered
Publication order can be preserved for select destination types.
Destinations
The following destinations are supported:
Google Cloud Pub/Sub
RabbitMQ
Azure Event Hubs
Event Schema
Webhooks, ETL Pipelines and our API endpoints all share the same JSON object schema. Webhooks and ETL Pipelines use identical JSON event payload structure.
Check out our Event Catalog.
Features
Data compression
Vital does not compress payload blobs that are smaller than 1 KiB. Please refer to the destination-specific documentation below on how to detect a compressed blob versus an uncompressed blob.
Static outbound IPs
Vital establish connections to your ETL Pipeline destination through a static pool of IPs. Contact Vital support to obtain the up-to-date static IP list.
Double writing for migration
To ensure smooth switchover from Webhooks to your new ETL Pipelines destination, Vital supports double writing events to the combination of:
- all your Webhook endpoints; and
- your ETL Pipelines destination.
This is not a proper fan-out feature, and is only intended to aid customers evaluating and one-off migrating between destinations.
There can only be one preferred destination, and the secondary destination(s) do not enjoy the same reliability guarantee as the preferred one.
Pushed Historical Data
When you use ETL Pipelines, Vital can push all the historical data as daily.data.* data events to your ETL Pipelines
destination.
The data-less historical.data.*.created event would still be emitted to denote the completion of historical pull.
Configuring ETL Pipelines
You can change the ETL Pipelines configuration on your Teams at any time through the Set Team ETL Pipelines endpoint of the Org Management API.
Your new configuration is typically active as soon as the endpoint has acknowledged your request.
Refer to the destination-specific documentation for configuration examples.
Feel free to reach out Vital support if you need assisted setup, or if you need a Vital Org Key to access the Org Management API.