About Customer

This global manufacturer is one of the world’s leading makers of tissue, pulp, packaging, building products, and related chemicals. Their customers span retail, commercial, government, aerospace, mining, facilities management, and a host of other application types.

About Intricity

Intricity is a team of specialized Data Management, Data Warehousing, and Business Intelligence experts. The team members at Intricity have been handpicked over the course of 20 years, and represent the top talent globally in data-oriented disciplines.

Challenge and Wins

Summary

The CLIENT was still in the early evaluation stage of determining what platform they would be moving to. The holding company made a strategic decision that AWS would be their strategic direction moving forward. However, specific directives within AWS were still being determined. One of the data warehouses which would be impacted by this decision was their SQL server-based warehouse. The manager of this warehouse was looking for solutions besides a straight lift and shift that might modernize their SQL server-based DW.

Navigating Constraints

Snowflake had not yet been accepted as the platform of choice for Data Warehousing within the CLIENT. Thus the team that owned the SQL Server environment needed to stay within a very tight budget while proving that Snowflake would be a good target platform off of their existing SQL Server deployment.

Win 1: Refactoring SQL Server in Snowflake

Intricity entered into a co-development engagement with the CLIENT to guide their development team on how to refactor their existing SQL Server code using BIML as a code generation mechanism. The use of BIML enabled the CLIENT’s team to auto-generate the ELT specifically for Snowflake. Additionally, The ELT framework was enabled to process loads using Snowflake compute. Because the generated code was auto-generated from the transformation specification, the naming conventions and code were very consistent.

The joint development effort enabled the CLIENT team to get a jump start on redeploying their SQL Server DW in Snowflake. Intricity started full time but was able to taper off, so the CLIENT team could complete their Snowflake conversion within the budgetary constraints.

The final report to the CLIENT’s management was that Snowflake was the recommended platform of choice and that it met all the required milestones. The CLIENT ultimately chose Snowflake as their chosen solution architecture of choice within their AWS ecosystem despite massive political headwinds.

Win 2: Decoupled Compute and Storage

The CLIENT was able to load the data warehouse without tying up access to it in their PowerBI deployment. This meant that they could now consider streaming data during the day without impacting BI consumers. The transition from nightly to real-time, changed the organization's ability to act on insights while shipping logistics were occurring, enabling a deeper level of control over their warehouses and production lines.

Not only did the decoupled compute impact the BI footprint, but it also had a huge impact on the data integration footprint. The ability to dynamically assign compute resources to the ELT commands made it possible to dramatically shorten the compute window, without committing to a massive hardware footprint. The produced SQL statements contained a command to wake up a large cluster of Snowflake compute resources, run the ELT jobs, then shut down the allocated resources. From a billing perspective, the CLIENT only paid for the segment of time where compute was used during integration. Because Snowflake managed the resource pool, the “wake up” period was near-instantaneous (less than 4 seconds, usually 1.5 seconds) which made for highly cost-effective processing.