About the company
WorldQuant Predictive’s mission is to deliver rapid and highly accurate AI-powered prediction products that give organizations a decision-making edge over the competition. To do this, WQP brings in thousands of data sources and builds accurate and resilient predictive models based on features and insight contained in those data sources. Its customers use the predictions to make better business decisions.
WQP’s powerful predictive capability using all available data has been invaluable to organizations during the COVID-19 pandemic when traditional forecasting based on a small number of historical data sets quickly fell apart.
Immuta ensures that we don’t get distracted from our core mission. As a market-leading product, Immuta gives us the easiest, shortest, safest path to do exactly what our customers need. And, by the way, Immuta’s support has been great.
Challenge
To help its customers gain an edge, WQP’s CTO Slava Frid knew that they had to build AI models based on a broader array of datasets that hold up better in a complex, changing world. According to Slava, “WQP’s customers rely on our predictions to understand how changing world and market conditions will impact decisions to be made. Speed is critical, and so is accuracy and resilience.”
To build their AI features and predictive models, WQP’s team has worked hard to build a modern, automated MLops data flow. They use core data sets such as sales data, and also manage many other secondary data sets such as foot traffic, consumer spending, real estate prices, and COVID-19 infection rates. Together, these combined data sets enable them to create a more accurate picture of the future.
The challenge is that more data brings more concerns around how that data is managed and governed. WQP needed to ensure the datasets did not inadvertently include any sensitive information in order to adhere to data use and sharing agreements, while also ensuring model accuracy. WQP also needed to restrict access to certain data that is reserved for validation and testing. And, they needed to quickly grant/revoke access and ensure full auditability.
Solution
WQP’s team needed the best of the best for a modern cloud data stack and are always looking for ways to optimize their MLOps data flow. According to Slava’s colleague Ivan Bondarenko, Data Tech Lead at WQP, “With the large number of incoming data sources and the speed at which we need to operate, we need best-of-breed tools – that means using Databricks for what it does best, Snowflake for what it does best, and Immuta to automate data security checks and policy enforcement.”
According to Ivan, Databricks is the WQP team’s “swiss army knife” and they rely on its Spark compute engine for their ETL pipeline and Notebooks for much of their core analysis and machine learning work. But, they also load data frames to Snowflake and use Snowflake as a centralized data store where they can run fast SQL queries and data transformations.
Immuta operates as the data security and privacy layer across their environment, running data checks on incoming data to ensure that no PII or PHI is included. They also use Immuta to mask and anonymize data as necessary. According to Ivan, “We use Immuta tags and groups, and everything is automated. Immuta is really useful for organizing projects and keeping your data transparent. We can quickly answer questions about where data is stored (e.g. what is the name identifier, and for which database?), and see when a person was added to a project.”
When requests for access come in, Immuta makes it possible to grant access in just a few minutes. If a user’s view of the data needs to be limited in any way, such as by certain dates or geographies, Immuta makes it easy to enforce those policies. Immuta also makes it easy to run quick audit checks to ensure policies are adhered to according to data use agreements.
Results
The addition of Immuta to WQP’s modern data stack helps WQP maximize query performance, data protection, and ease of repeated execution for its data engineering and data science teams. By using Immuta, WQP is able to get the “Easiest, shortest,and safest path to do exactly what our customers need,” said Slava.
Immuta ensures that we don’t get distracted from our core mission. As a market-leading product, Immuta gives us the easiest, shortest, and safest path to do exactly what our customers need. And, by the way, Immuta’s support has been great.
