The modern data stack bears the immense responsibility of storing, protecting, analyzing, and operationalizing a resource that is constantly in flux. As data continues to increase and evolve, these tools need to make sure it is both being used effectively and kept safe from leaks.
This issue and potential solutions were the focus of a webinar hosted by TDWI, “Building a Modern Data Stack for the Future of Data Analytics,” featuring TDWI’s Senior Director of Research for BI David Stodder, Immuta’s Vice President of Global Solutions Architecture Matt Vogt, Snowflake’s Principal, Data Cloud Strategy Teddy Lewis, and Alation’s Senior Director, Partner Sales Engineering Deepak Nelli. In this blog, we’ll recap what these data-driven leaders say is essential for building a modern data stack.
The Challenges of the Modern Data Stack
According to survey research conducted by TDWI, 53% of respondents believed expanding data visualization, BI, and analytics to more users was one of the top objectives for modernizing their organization’s data integration and management. The more accessible the data, however, the more risk-related challenges a data stack will face.
In his pre-roundtable presentation, Stodder broke down the range of challenges that operational modern data stacks must surmount. He first noted the growing importance of data democratization, or making data accessible and usable by all data consumers in an organization. When data becomes more accessible, it is even more imperative to keep track of who is accessing it and for which purposes. This is increasingly relevant due to the amount of sensitive and personally identifiable information (PII) present in data sets.
With data democratization and increased accessibility, this sensitive information must be protected to both maintain personal safety and achieve regulatory compliance. Whether subject to internal company regulations, government mandates like HIPAA and CCPA, or international laws such as GDPR, organizations need their data stacks to be equipped with the ability to control access to sensitive data to avoid breaches of trust, fines, or further penalties.
The modern data stack must bridge the gap between the desire to democratize access and the need to maintain stringent privacy regulations, allowing for secure and unimpeded data use for those who have the right to access.
The Requirements of a Modern Data Stack
How can data teams overcome these challenges to build a practical, secure data stack? This was the main topic of conversation between Matt Vogt, Teddy Lewis, and Deepak Nelli. Having seen firsthand the challenges of an overburdened data supply chain, these data-focused leaders understand the need for data stacks that can handle advanced requirements.
Throughout their conversation, these speakers reiterated David Stodder’s point – modern data stacks, like organizations themselves, must be:
Resilient
A modern data stack needs to be resilient enough to weather constant change. “Things are not going to slow down,” said Alation’s Deepak Nelli of the data landscape. “This will continue to move faster and faster.” Data stacks must be built with data’s fluidity in mind, and constructed so that they do not break under the strain of change.
A large part of this resilience is based in trust; trust that the tools in the data stack are going to be able to handle the large amounts of data that they are required to store and analyze, as well as trust in the legitimacy and potency of the data. Vogt noted that data users need to be able to “trust that the data is right, [and] trust that I’m allowed to have access to this data.”
The best way to create and reinforce this trust is through the implementation of governance and data access control tools within your data stack. “It’s ever more important that we have systems and platforms that can ensure discoverability [and] governance at scale with confidence,” stated Lewis, emphasizing the essential nature of proper data governance to ensure enduring compliance and trust.
Agile
While the resilience to weather a variety of changes is key to a successful data stack, agility and flexibility are just as integral. “Organizations of all sizes move at the speed of business requirements,” noted Lewis, calling attention to the fact that businesses and other organizations can’t operate effectively if their tools and technologies aren’t able to adapt to changing requirements.
Modern data stacks must be able to bend without breaking, adjusting in the moment to shifts in data use, but not allowing these changes to weaken the stack or delay analytics. When asked about how organizations can be sure access control and governance can keep pace with change, Vogt proposed automation and orchestration, saying that ”you’ve got to be able to have a system that’s flexible and automated enough to keep pace with…how data’s changing and how uses are changing.” In other words, a static data stack is not a tenable data stack.
The tools and technologies in an organization’s data stack must be scalable and adaptable. Without this agility, the stack will bend and bend until it eventually breaks.
Future-Ready
A resilient and agile data stack is one that is prepared to operate beyond simply the immediate future. While no one is certain what lies ahead for data use, there are many trends that offer an idea of what data stacks should be ready for next.
Vogt spoke again to the need for data democratization, noting that “getting more users using more data” will be a key function of future-oriented tools. In regard to specific technological advances that should be accounted for in contemporary data stacks, Nelli noted that “automation through AI/ML will be a key component to ensure that organizations keep with the pace of growth and change.” Each of these suggestions forecasts a future where data access and use are automated, scalable, and innovative in order to deliver cutting edge business outcomes.
“All of that governance, policy creation, and discoverability and business context around data needs to be unified,” Lewis stated. “There needs to be some standard, central place where all of your business users can go to discover data.”
Ultimately, a future-ready data stack will need to be prepared for efficient and secure widespread use in order to push organizations forward.
The Power of Immuta + Snowflake + Alation
Given the challenges the modern data stack must surmount and the standards set for their success, how can platforms like Immuta, Snowflake, and Alation work together to enable organizations’ data use to prosper?
With Immuta’s data access and security capabilities, Snowflake’s advanced storage and analysis capacity, and Alation’s detailed and essential metadata, these platforms can provide a modern data stack with the tools necessary to be resilient, agile, and future-ready.
“In reality, we have to work together,” said Vogt. “You’re not going to find a single platform that covers all of these bases.” Nelli shared this sentiment, saying “I like to call it a trifecta solution, Alation, Immuta, and Snowflake. A catalog built on these three solutions will basically help users understand trust, and also empower the business users to act fast in confidence.” Lewis stated plainly that “I haven’t seen an end-to-end solution that does cover the breadth and depth of this trifecta.”
For more on how this trio can streamline your data privacy, check out our blog How to Simplify Snowflake Security with Alation and Immuta.
