DISA’s updated tech watchlist gets specific about AI capabilities

As the entire Pentagon works to better understand artificial intelligence and machine learning, the Defense Information Systems Agency is taking a more comprehensive approach to analyzing AI capabilities and their potential use cases.

In its so-called “Tech Watchlist” for fiscal 2024, DISA now has several items covering specific capabilities and issues related to artificial intelligence — from the use of large language models to what guardrails are needed when deploying AI. The new categories were split from a single item dedicated to tracking overarching AI/ML technologies, according to Steve Wallace, DISA’s chief technology officer and director of emerging technologies.

The update comes as the Pentagon pushes for improved and responsible deployment of AI. Last week, the Department of Defense released its new Data, Analytics and AI Adoption Strategy. For DISA, the specific areas will help inform how it leverages the technology in the future, Wallace said Monday as he unveiled the new watch list at DISA’s annual forecast for industry event.

“There is nothing more important than the ability to operate a system that you deliver. You can deliver the most spectacular, outstanding capability in the world, but if it’s too difficult to operate, you really haven’t done much because it’s probably more down than it is up,” he said. “So how are we going to put artificial intelligence into that chain?”

DISA’s watch list—updated regularly at least every fiscal year, but often more frequently—is a broad inventory of more than two dozen new technologies that the agency has identified as ones it is interested in pursuing. The items are divided by maturity levels, from technologies that are just being explored to actual prototypes that the agency is preparing to deploy, such as the recently awarded Thunderdome sero-trust cybersecurity effort.

One new category in the “monitor” subsection tackles AI operations and was born out of discussions the agency is having with the Defense Innovation Unit (DIU) about a prototype it has in the works, Wallace said. The AI-enabled prototype “takes logs of given infrastructure components and given systems and then uses a model to train itself over the course of weeks or months to do predictive analytics or predictive reliability analytics,” Wallace said.

DISA hopes to learn some lessons from the prototype work to improve how it addresses vulnerabilities in Defense Department networks by using artificial intelligence to predict them far in advance.

The agency also added an item in the “monitor” subsection of the watch list dedicated to trust, risk and security management of artificial intelligence. Wallace noted that this category will examine how the Pentagon can keep up with its adversaries’ use of AI, while also identifying technology barriers that DISA will need to implement.

Wallace pointed to some capabilities that serve as a proxy for publicly available large language models as an area of ​​concern. These tools will take queries from users, run them against a contained dataset, and ultimately send them to public large language models like OpenAI’s ChatGPT or Google’s Bard, he said.

Broadly speaking, large language models are deep learning algorithms trained with massive datasets to recognize, summarize, translate, predict and generate persuasive conversational text and other forms of media in a manner that mimics a real human.

“The department has a number of concerns about inquiring about those commercial, large language models,” Wallace said. “How are we going to protect ourselves? How are we going to handle it until these models exist in a [government] cloud or higher environment?”

Despite the concerns, DISA did not swear off major language models, as the fiscal 2024 technology watch list renamed its generative AI category, which was added earlier in 2023, to “concierge AI.” The technology currently sits on the “planning” subsection of the watch list — meaning the agency has an understanding of how it could impact and integrate into missions.

“What we’re looking for out of a concierge AI … is the idea of ​​having for the business side of the house that ChatGPT-like experience of interacting and asking questions about datasets,” Wallace said. “The other side of that coin that we bundle here, although it’s a slightly separate effort, is how we help the analysts. On the security platforms, how do you help the analyst diagnose threats faster [and] do the pieces come together much faster?”

It is possible that DISA will award another transaction agreement for a prototype of the capability in fiscal 2024, he added.

Although the watch list provides a snapshot of DISA’s technology efforts, the agency has a technical strategy in the works that will serve as a more specialized counterpart to DISA’s overall strategy, Wallace said. Although the strategy is still in its first draft, Wallace teased three main points it will cover: simplification of systems, integration across the agency and iterative delivery of new capabilities.

Written by Mikayla Easley

Mikayla Easley reports on the Pentagon’s acquisition and use of emerging technologies. Before joining DefenseScoop, she covered national security and the defense industry for National Defense Magazine. She received a BA in Russian language and literature from the University of Michigan and an MA in journalism from the University of Missouri. You can follow her on Twitter @MikaylaEasley