Building a Fast Universal Data Access Platform

Chapter 2. Building a Framework for Data Diversity and Universal Access

We’ve established three primary goals for our data ecosystem. First, we need it to handle data of varying types and sources while also being able to rapidly adapt as new sources become available. Second, we need to be able to handle large quantities of data. Our solution needs to be large enough to handle all our existing data and flexible enough to expand as more data gets added. Finally, we need our solution to be able to return queries as quickly as possible to give our business the ability to get rapid insights. Knowing these challenges, what are the solutions that will help us achieve our goals?

Federated Query System

The underlying goal of pulling in larger volumes and varieties of data is to create insights from multiple combinations of sources, expanding your knowledge of how your company runs and identifying ways to improve. The challenge is that many of these data sources require different connection types, languages, or even translations to make them usable. A federated query system does that work behind the scenes and provides a single point of connection for users attempting to utilize the data.

Federated query systems resolve the challenges of diverse data. Using an overlying system to connect to diverse data sources solves many data storage and reporting challenges. The most obvious problem this solves is the need for multiple drivers, multiple logins, and complex blends associated with connecting ...