Right around 2003 the United States went through a bandwidth revolution. And suddenly it got cheaper to use bandwidth. However, in the IT space, most solutions and vendors were still carefully designing their solutions to closely guard bandwidth. But some organizations were already embracing this newfound bandwidth like salesforce.com. In their tow, other disruptive vendors began chipping at the heels of once very established business models. This has been going on for more than a decade, and the cloudification trend is now penetrating some of the most traditional IT practices.
One of these IT practices that have held back from adopting cloud replacements is data integration and analytics. Today I’m going to share what the restraining forces have been in keeping IT practices from adopting cloud-based Integration and Analytic architectures. But then we’ll share what some of the driving forces are, and how they are turning managed IT on its head.
So the first problem with cloudified architectures is with the very nature of what data integration involves. See, unlike salesforce.com which is saving and dishing up single records at a time. Data Integration seeks to move and process millions and billions of records at a time. So even though bandwidth is plentiful, data integration could quickly gobble it up.
The second problem has been the on-premise history of most applications. If your data resides behind your firewall, it makes little sense to have that raw data shipped into the cloud to do the data processing. Yes, bandwidth is cheap but you have to take into account the time cost of moving large quantities of raw data into the cloud. As a rule of thumb, you want to process the data where it lives to avoid bottlenecks, and if you have to move the data only ship the processed data, and not the raw stuff.
The other issue that comes up with any web-based application has to do with network uptime. See corporate networks expect a mostly constant uptime within their firewall. The web, however, has a very erratic uptime. You can experience a hiccup for just a few seconds on a fairly regular basis. So systems that are moving and processing millions of records have to be very tolerant of these constant hiccups.
These barriers also created a vacuum because few vendors even existed that could support a modern data warehouse in the cloud. There were plenty of “private cloud” methods, but that just means that the same stuff you would buy on-premise is being thrown into a rented server rack without the software as a service benefit. A true multiparty software as a service data warehouse was still a very raw idea.
It turns out that the same is true for Business Intelligence. There just weren’t effective solutions available for dealing with the fact that data queries would happen with the quirks of an internet-based source.
Last but not least, data integration is a complex beast. And most organizations don’t want to even touch it unless they have to. Intricity has seen many organizations that literally don’t even know what their Integration Platforms are doing. And as long as these systems keep working, there’s no need to meddle.
So we’ve covered some of the restraining forces which have been in the way of organizations adopting cloudified integration and analytic architectures. Now I want to share some of the driving forces.
First, there are now approaches for processing data on a more real-time basis through streaming JSON and REST which make the prospect of moving records over the internet more feasible. Rather than moving big swaths of data all at once, I can stream small swaths throughout the day. Additionally, the role of data compression can’t be understated here, it has been a huge catalyst in enabling cloud data movement to occur.
Second, nimble competition has a cloud-first strategy. This means that smaller companies often don’t even have on-premise applications! This competitive advantage is driving some of the old behemoths to reconsider their hard-line stances against architecting solutions in the cloud. Additionally, some ETL vendors are delivering the capacity to control where data processing occurs. So if your data is on the premise it can get processed there, but if it’s in the cloud it can be processed there as well. So if you’re on-premise and moving data to the cloud you’ll only push the smaller processed data set.
Third, dealing with the internet’s erratic nature is really something that needs to be part of a ground-up solution designed for the web. Tracking a hiccup in the data requires tracing where processing left off and redundancy to restart the request from that point. There are now both Cloud ETL and BI vendors that have those capabilities out of the box.
An actual data warehouse in the cloud that follows a true multiparty software as a service is now a reality. Not only do these environments exist they are replacing old on-premise Data Warehousing appliances. More importantly, they are instantly scalable. So rather than scaling with an additional million-dollar forklifted server, you can just scale up and down on demand.
Data Integration is one of the final frontiers for enabling a complete cloud-based IT landscape. But here’s where I am predicting a real market shift. Over the last decade, IT organizations have been outsourcing their IT to managed service contracts. No company does this because they get better IT service. This is purely a specialization and cost issue. By pushing the management of databases, servers, and applications to a third party, they can have somebody else deal with uptime issues. However, cloudification of the IT landscape is going to open a window to IT organizations that want to reconsider the outsourcing model. This is because the menial tasks of uptime maintenance are no longer the customer's responsibility.
And there are huge wins to be had from regaining control of your IT landscape. After all, nobody will be as vested in your success as your own company.
I’ve really only just scratched the surface of this. But, I recommend that you reach out to Intricity and talk with a specialist. We can help you make this cloud transition by designing a logical solution architecture that steps through the journey that the data takes from the source systems all the way through to your data consumers, in an entirely cloud-based approach. We can also help you plan out a transition from your managed service reliance, into a new model that will put IT back into an innovation role. To learn more about this architecture planning engagement take a look at this sample-priced proposal.
