We need an open source intelligence center

In April 2018, Dutch authorities caught four Russian intelligence officers red-handed as they attempted to hack into the network of the Organization for the Prohibition of Chemical Weapons (OPCW) from the hotel across the street. The OPCW was investigating the substances used in the poisoning of a Russian defector living in the United Kingdom, and a chemical attack by Assad-backed forces in Douma, Syria. After the Dutch government publicly identified the four officers, open source researchers at Bellingcat — an independent investigative journalism group — leveraged the personal details of the four to identify 305 additional agents of the GRU, Russia’s principal military intelligence unit. Bellingcat’s researchers made the discovery using the agents’ car registrations, which were linked to the service’s training academy.

This kind of open source discovery — once thought to be within the exclusive purview of governments — has become somewhat routine for skilled researchers in the private sector. Their accomplishments include validating U.S. government claims about the Kremlin’s military build-up around Ukraine, exposing Beijing’s efforts to acquire foreign technology on a vast scale, uncovering the Chinese military’s investments in artificial intelligence, and identifying several Russian assassination and surveillance teams behind the attempts on the lives of Russian dissidents and defectors.

The discoveries point to a broader trend that the U.S. government would be wise to harness — the exponential growth of publicly and commercially available data, and the emergence of technologies that allow for its rapid processing and dot connecting.

The roughly 6.6 billion smartphone users, 5 billion internet users, and 5,500 active satellites in orbit, as well as countless other apps, surveillance cameras, and sensors have created unprecedented opportunities to observe patterns of human life based on their digital exhaust. By some estimates, human engagement with these devices and sensors is expected to generate some 463 exabytes of data daily by 2025 — roughly equivalent to 22,047 copies of the Library of Congress’ current digital collection (for reference, Library of Congress’s digital collection was 21 petabytes in 2022). As this data is being generated, new AI tools are becoming increasingly available and essential aides for researchers to sift through for patterns and trends. Large language and computer vision models can now generate insights from massive data sets of text, images, and video. AI-enabled tools could also build out predictive models to analyze open source data.

The imperative for the U.S. government to harness these trends is not only technological. The United States is engaged in a geopolitical rivalry with the People’s Republic of China (PRC) that is wide in scope, multilayered in complexity, and fierce in intensity. We compete militarily in the Indo-Pacific and across domains. We compete diplomatically at the United Nations and across the world. We compete technologically for leadership over a wide tech stack. In an all-encompassing, constant competition, open source intelligence is an essential component of any winning strategy.

The White House and Congress appear to recognize the importance of open source. The Biden administration’s National Security Strategy released last October highlighted better exploitation of “open source materials” as one of the key areas where the intelligence community could help sharpen U.S. statecraft. The State Department last year announced the Strategic Open Source Coordination Office, to serve as a focal point for open source policy, tradecraft, and training. Congressmen Joaquin Castro (D-Texas) and Mike Gallagher (R-Wis.) introduced a bill in the last Congress to create an Open Translation and Analysis Center with the mission of conducting open source work and making translations available to the public. Similarly, each Intelligence Authorization Act over the last five years has carried a number of open source provisions.

However, these proposed initiatives and programs have either not come to pass or have fallen short. A government of multiple small efforts is insufficiently prepared to harness the current open source revolutionary potential. The volume and variety of open and commercial source materials, urgency of the geopolitical rivalry, and continued development of tools to exploit the data all necessitate a systematic effort to harness open and commercial source to support decision making. The answer, we believe, rests on standing up a standalone open source entity.

This government entity would have a national mission to collect, process, analyze, and disseminate publicly and commercially available data related to foreign rivals. To be successful, it would need to have several key attributes. First, the entity would need a voice in the intelligence community to influence and plug into the intelligence cycle. Second, it must serve as a gateway between the Intelligence Community, the rest of the U.S. Government, and outside partners in business, academia, and civil society. Third, the entity would need to employ a hybrid workforce of both cleared and uncleared personnel. Meaningful connections to the national security apparatus require security clearances; however, many routine open source tasks are not sensitive and do not need to be concealed. Throughout this entire process, this entity would need to ensure that it collects, processes, and distributes data in comformity with U.S. privacy laws.

Where would it reside and what would be the objectives of such an entity have long been debated. There are four options to address the need immediately. One option would be to create an independent agency or one housed under an existing department, such as Commerce, Defense, State, or the General Services Administration, to name a few potential candidates. The second option would be the creation of a new intelligence agency, because the IC needs an agency with the budget, hiring power, and seat at the table to champion open source. Third, is an Open Source Coordination Office reporting to the Director of National Intelligence. The fourth is focusing on the normalization of open source use in all-source analytic products across the IC.

Apart from the benefits of harnessing open and commercial source information, such an entity could also serve as a training ground for future AI algorithms. The less classified environment would make it easier to experiment with commercially-available AI tools, streamlining adoption and — in the process — enabling the organization to become operational faster. In a race for actionable insight to compete with the PRC, an open source agency would prepare the IC to sprint. And the better it performs, the more it would encourage other IC agencies to push the pace.

The imperative for change is clearly here. Competition, conflict, and national security in a data-driven, technologically-enabled world must account for public facing data. The time is now.

Rodney Faraon is a partner at Crumpton Global, a management consultancy. He is a former senior intelligence officer at the Central Intelligence Agency.

Peter Mattis is Director for Intelligence at the Special Competitive Studies Project and a former counterintelligence analyst at the Central Intelligence Agency.