Case Study: 1200 Kafka Streams to Snowflake

Mass Kafka to Snowpipe Streams & Tasks

About the Client

The Client is a global real-time information discovery company and also a source of breaking news alerts. These alerts are used by a wide variety of companies and government entities interested in being alerted by breaking news. The Client has built their own complex event processing capabilities that monitor the latest event data, bubbling up critical alerts from a sea of constantly streaming events.

About Intricity 

Intricity is a team of specialized Data Management, Data Warehousing, and Business Intelligence experts. The team members at Intricity have been handpicked over the course of 20 years and represent the top talent globally in data-oriented disciplines.

Challenge and Wins

Challenge

The Client quickly pushed the limits of traditional databases with its constant flow of over 1200 Kafka streams. The speed of storage and the ability to orchestrate the constant ingestion of data was always a wall that was difficult to traverse. The Client spent a significant amount of time evaluating the right cloud data warehousing solutions which could deliver on the promise of concurrently storing and querying data. Snowflake was the clear leader in the end, but building the orchestration framework and designing Snowflake in a way that could support their use case was something the Client team needed help to accomplish.

Navigating Constraints

1. 1. The Client lives in the world of “now” so their streams of data had to constantly be loading into their data warehouse with the lowest possible compute overhead to ensure timely arrival.
   2. The Client had 1200 Kafka streams making it difficult to ingest to a consolidated data warehouse.
   3. The Client wanted to completely own the solution with Intricity providing guidance on a regular basis.

Win 1: Ingestion Automation Using Snowpipes, Streams & Tasks and Airflow

Intricity assisted the Client with the ability to bring the data together programmatically through Airflow for complex patterns and Snowpipes with Streams & Tasks for transformation events. This made Snowflake the primary computing layer for all the transformation logic from the 1200 Kafka streams. On the front end, Looker was the BI layer of choice.

Win 2: Near Real-Time Processing of the Data Warehouse

With the deployment of Snowpipes with Streams and Tasks, the Client is able to deploy a near real-time architecture. This was accomplished by splitting the streams between real-time and scheduled streams which deliver aggregated details. Along with this real-time capability, the Client’s environment can be scaled to onboard an unlimited ingest and egress demand.

Win 3: RBAC Security for Snowflake

Intricity’s RBAC/ABAC Rollup Best Practices delivered a guided framework which the Client used to appropriately deliver access controls in their complex environment. This included diagrams of how the access controls interact with the Snowflake environment.

Win 4: CoDev Model

Intricity leveraged its CoDev Model to help their team become proficient in Snowflake. This enabled them to meet the requirement of internal ownership of their product while instilling best practices based on over 100 Snowflake deployments under Intricity’s belt.

Call to Action

To schedule a time to discuss your landscape with an Intricity specialist, go to https://www.intricity.com/contact-us and register to talk with a specialist or call the office number near you.