In today’s digital world, large amounts of multiple types of data have amassed in multiple federal agencies in multiple data sources, which is a major problem for many organizations, including federal agencies – how to effectively and efficiently use the data for day-to-day operations, and analyze it to gain insights to improve operations and drive strategy. This problem is compounded further by federal, state and local agencies needing to access and share data from and across these agencies.
One of the three conventional approaches is to copy data from multiple disparate sources to centralized data warehouses or “Big Data Lakes,” but it has proven unsuccessful for several reasons: (1) many data source owners are reluctant to send, or are prohibited from sending, data or certain types of data to federal agencies apart from that dictated by law, (2) the onus has always been on the data source owners to provide clean, high quality data without them necessarily having the resources and tools to do so, and (3) distrust caused as “sharing” data and information is often a one-way arrangement in that federal agencies will not share back with other federal, state and local agencies, and, in many cases, will take credit for the data and information provided.
{mosads}The second of the three conventional approaches is to federate queries on multiple disparate sources, but it too has not been successful for several reasons: (1) poor data quality, (2) difficult-to-implement standards, (3) limited ability of source systems to process queries and (4) limited support of local IT personnel.
The third of the three conventional approaches is an unstructured text search engine, but it does not address the structured data requirements of typical reporting, BI and analytics applications.
Now, there are technologies available that combine the best and overcome the worst of these conventional approaches by leaving and securing data in sources, but enabling a powerful and transparent query processing or data virtualization layer on top. The real future of data federation and data security within federal agencies lies in data virtualization technologies that are gaining ground. But technology is overcoming one of the most significant challenges that government agencies face today: providing high quality and high performance access to both external and internal data, while supporting multiple agencies whether at the federal, state or local level, and integrating that data with non-government, commercial and other public sources.
Expensive and difficult-to-find data analysts and scientists spend around 80 percent of their time gathering and preparing data, and it will only become a larger mountain to climb as time moves on and ever more information is collected. IDC’s annual Digital Universe study predicts that by the year 2020 the amount of data globally will grow 10-fold and that 10 percent of that data will be accounted for just by the information produced by the Internet of Things. Managing this massive stockpile of data is a task advanced technologies should be able to perform, freeing analysts to be able to evaluate and gain valuable insight to improve organizational outcomes, lower costs and increase stakeholder satisfaction.
Each federal agency has their own set of policies for data management and sharing, which slows down innovation in the field; especially since most of the information needs to be secure. Yet data sharing between agencies is an important part of each organization’s duties. Thus, cybersecurity and data security will play an increasingly important role in safely managing, storing and exchanging data for the U.S. government. In a 2017 annual Forbes survey, 69 percent of senior executives say they are already readjusting their cybersecurity efforts for the upcoming years.
The future of technology solutions in federal agencies is to enable secure integrated storage of all data, while updating in real-time. The future is a virtual data management layer that stays transparent to both applications and data sources to keep the data easily accessible and secure. The future is having a technology that addresses fundamental data management processes such as data discovery, profiling, standardization, linking and master data management when building and maintaining indexes and reading results data from sources – allowing data analysts to focus on the analysis to gain important insights. One of the keys to virtual data management is the seamless and automatic integration of, and updates to, all-important master data to provide single person and other entity views of all data.
The world of developing technology to support federal agencies in keeping their data managed and secure, regardless of where it is, is an exciting one. The future is here – technologies like these exist and are already being implemented by government contractors. The prospect of further refinement and implementation of secure data management technologies in federal agencies holds promise for a very real and exciting future that benefits all stakeholders.
Gavin Robertson is the CTO and SVP at WhamTech. He holds a BSc in chemical and process engineering and a MEng in Petroleum Engineering from Heriot-Watt University in Edinburgh, Scotland.
The views expressed by contributors are their own and are not the views of The Hill.