Data teams are critical to the success of modern data-driven organizations. Made up of engineers, analysts, scientists, and more, these teams are responsible for a range of functions that allow data to be optimized by internal and external users.
However, the dynamic nature of data means this responsibility is not guaranteed to be easy. Today’s data engineering teams encounter a multitude of challenges that can inhibit both their day-to-day operations and long-term objectives. We went to the source and surveyed 600 data professionals to determine the most common data engineering challenges.
In this blog, we’ll delve into the most common data engineering challenges as reported in the 2023 State of Data Engineering Survey, and highlight how they can be addressed to remove barriers and enable greater organizational success.
What are The Most Common Data Engineering Challenges?
Not every data engineering team is guaranteed to face the same combination of hurdles in its operations. But by surveying hundreds of data professionals across the U.S. and Europe, we were able to discern some common threads that run across industries. Here are the three most common data engineering challenges:
The Data Security and Data Access Disconnect
Confusion around the distinction between data security and data access is not a new phenomenon.
- Data security is the combination of process and technology that helps protect sensitive digital information from malicious access, alteration, disclosure, or leakage throughout its life cycle.
- Data access, on the other hand, refers to the specific controls that allow authorized users to acquire and modify data for legitimate purposes.
So data access measures are likely part of an organization’s data security initiatives, but the terms themselves are not interchangeable.
While these measures are meant to cooperate, there is a noticeable lack of understanding about their relationship. Our survey found that 90% of respondents agree that they could improve their understanding of the correlation between data access and data security. What’s more, only 39% of these professionals believe that data access is even a component of data security.
When data security initiatives and data access controls are not aligned, effective sensitive data protection is much harder to achieve. Neglecting to include data access in security measures leaves organizations vulnerable to leaks, breaches, insider threats, and more. This is especially true as more companies are migrating data to the cloud, storing it on platforms that are still evolving and often relegating security to an afterthought. More than half (54%) of our respondents report that securing data with appropriate access rights is among their biggest challenges in the modern cloud data landscape. This presents a substantial challenge that, if not addressed, will proliferate alongside continued rapid cloud migration.
Data Access Policies That Cannot Scale
As organizations aim to democratize data access, more users require access to data resources. For every new user added to a growing cloud data ecosystem, there needs to be access and security policies governing just how much data they are allowed to see. Depending on which access control methods are being enforced, this responsibility can become incredibly burdensome for data engineering teams.
If relying on role-based access control (RBAC), new policies must be created and applied for every single new user who requires data access. This can cause role explosion and create an excess of hyper-specific and static access control policies that must all be manually maintained. Role explosion is only one of many access-based data engineering challenges that must be dealt with manually. Over two-thirds (69%) of survey participants report spending an average of 6–10 hours per week responding to, managing, and resolving data access issues – up to 480 hours per year.
[Read More]: RBAC vs. ABAC for Data Access Control Use Cases
Continuing to operate with manual access control policies is simply not scalable. More than half (51%) of those surveyed state that their current policies limit their ability to securely scale data access. This challenge not only burdens engineering teams with excessive work, but it limits how effectively data users can optimize their organization’s data. A whopping 89% of professionals say their organization has missed business opportunities due to data access obstacles. If nothing else, this statistic alone should encourage today’s companies to find a solution for their unscalable data access policies.
A Lack of Data Security Resources
Securing and managing access in the modern data stack is clearly not a simple task. Data engineering teams need to be enabled with the right personnel and tools in order to successfully manage this process. Unfortunately, the lack of these resources–skilled, robust teams and effective technologies–have become a common data engineering challenge.
While organizations move massive amounts of data to cloud platforms, the teams required to secure and manage it aren’t being scaled to handle the increased responsibilities. Almost half (41%) of survey respondents claim that their data and IT teams are understaffed and do not have enough people to manage their data. Beyond this, over a third (36%) believe that their teams have too much data to handle. Data’s potential cannot be maximized without the personnel to keep it safe and in line with compliance and regulations.
When engineers are saddled with excessive data access responsibilities and lack the right support, it becomes even harder to build and maintain steady data engineering teams. Over a third (39%) of respondents report feeling burnt out by their data access management responsibilities to the point that they’d consider finding another job. And with only 26% of companies employing automated data access tools to lighten this load, it’s likely that burnout will only get worse.
How to Address Data Engineering Challenges
Whether it’s a disconnect in data access policy implementation, policy frameworks that cannot scale to meet business needs, or high levels of manual management leading to burnout, today’s top data engineering challenges can largely be traced back to one thing: inefficient data security.
To address these challenges, organizations need to streamline their data access and security processes. It is essential that all stakeholders across teams are aligned under policies that are implemented and managed efficiently. Adopting an automated data security platform can help close any gaps and unify data access and security initiatives.
Immuta’s Data Security Platform discovers, secures, and monitors all sensitive data within a data ecosystem. Utilizing plain language data policy authoring, Immuta allows any user to write and comprehend a policy, regardless of technical expertise. Understandable policies can help clarify the relationship between data security and data access for any data user. Policies are also built and applied using attribute-based access control (ABAC). This provides engineering teams with a much more flexible policy authoring and implementation system, as ABAC policies are written based on user subject attributes like geography, time and date, clearance level, and purpose, instead of singular roles like the traditional RBAC policies used by many today. Most importantly, policies determine context-aware decisions at query time, meaning that data engineers no longer need to serve as manual gatekeepers for each and every user access request.
By lessening the burden on engineers, clarifying access initiatives, and securing data in compliance with regulations, the right data security platform can enable teams to overcome these common data engineering challenges. To explore more of the insights from data professionals across industries, download our 2023 State of Data Engineering Survey. For a hands-on look at how easy it is to create an access policy in Immuta, try our self-guided demo.
2023 State of Data Engineering Survey
Data security has never been more top of mind. The problem is, it’s not necessarily well understood. Immuta’s 2023 State of Data Engineering Survey asked 600 data professionals about their responsibilities, challenges, and outlooks on data security.
Download the Report
