What is an AI Black Box? A Computer Scientist Offers an Insight

To some, the term 'black box' brings to mind the flight data recorders in airplanes, crucial for post-crash investigations if the unimaginable occurs. For others, it recalls small, sparsely furnished theaters. However, the term also holds significant meaning in the realm of artificial intelligence.

AI black boxes are systems where the internal processes remain hidden from the user. You can provide them with input and receive output, but the system's code or the reasoning behind the result remains inaccessible.

Machine learning is a key branch of artificial intelligence. It serves as the foundation for generative AI systems like and DALL-E2. Machine learning involves three main components: an algorithm or set of algorithms, training data, and a model. An algorithm is a set of procedures, and in machine learning, it identifies patterns after being trained on a large dataset — the training data. Once the algorithm has been trained, it produces a machine-learning model, which is then used by people.

For instance, a machine-learning algorithm might be created to detect patterns in images, with training data consisting of images of dogs. The resulting model would act as a dog detector. You would input an image, and the output would indicate whether and where in the image a set of pixels represents a dog.

Any of the three elements of a machine-learning system can be concealed, or placed within a black box. Often, the algorithm is publicly available, which reduces the effectiveness of hiding it in a black box. To safeguard their intellectual property, AI developers often place the model within a black box. Another strategy is to obscure the training data, effectively putting the data into a black box.

The counterpart to a black box is sometimes known as a glass box. An AI glass box is a system where the algorithms, training data, and model are all transparent and accessible to anyone. However, even these systems are sometimes described as having black box elements.

This is because researchers don't fully understand how machine-learning algorithms, particularly deep-learning algorithms, function. The field of explainable AI is dedicated to developing algorithms that, while not necessarily fully transparent, can be more easily understood by humans.

Why AI Black Boxes Matter

In many cases, there is good reason to be wary of black box machine-learning algorithms and models. Suppose a machine-learning model has made a diagnosis about your health. Would you want the model to be black box or glass box? What about the physician prescribing your course of treatment? Perhaps she would like to know how the model arrived at its decision.

What if a machine-learning model that determines whether you qualify for a business loan from a bank turns you down? Wouldn't you like to know why? If you did, you could more effectively appeal the decision, or change your situation to increase your chances of getting a loan the next time.

Black boxes also have important implications for software system security. For years, many people in the computing field thought that keeping software in a black box would prevent hackers from examining it and therefore it would be secure. This assumption has largely been proved wrong because hackers can reverse-engineer software — that is, build a facsimile by closely observing how a piece of software works — and discover vulnerabilities to exploit.

If software is in a glass box, then software testers and well-intentioned hackers can examine it and inform the creators of weaknesses, thereby minimizing cyberattacks.

Saurabh Bagchi is a professor in the Department of Electrical and Computer Engineering at Purdue University. He receives research support from a wide range of entities, including federal and state governments as well as private companies.

This article is republished from The Conversation under a Creative Commons license. You can read the original article here.