Different definitions of interpretability

• Ability to explain in understandable terms

• Explanation from input data to output decisions

• Explanation of function without describing internal structure

The operative part of the provisions sets a legal threshold— sufficiently transparent to enable users to interpret the system’s output and use it appropriately. If the proposal were to become law in its current form, its implementation would be complicated. Based on how we answer multiple questions about the nature of transparency required the implementation could vary. Article 13 ties transparency to ‘interpretability’, but there is little consensus on what interpretability means in XAI literature. One source defines it as the AI system’s ability to explain or to provide the meaning in ‘understandable’ terms to an individual. In XAI literature, understandability is often defined as the ability of a model to make a human understand its function without any need for explaining its internal structure or the algorithmic means by which the model processes data internally. [2]  Another source defines interpretability in a contrasting manner, requiring traceability from input data to the output.

It is worth noting that the standard of ‘interpretability’ is comparable to the administrative law standard of duty to give reasons that we discussed in detail here. Before delving into the XAI literature for suitable examples of methods that can deliver interpretability, it would be worth our while to first establish the contours of interpretability by tying it to the duty to give reasons.

Depending on the nature of algorithms and the functions that it performs, there could be multiple ways in which the ‘duty to give reasons’ test can be applied here. Let us begin with reminding ourselves that algorithms exist within a socio-technical context, and any form of evaluation for suitability must take that into account. Much like socio-technical systems in safety-critical fields such as aviation, medical devices, weapons systems, large scale manufacturing, which must clearly respond to and conform to a design problem that must always meet well-defined outcomes, when algorithms are used in high-risk systems, they need to deliver ways in which they can be validated for correspondence to key goals and requirements. [3] The duty to give reasons in an administrative context does not merely explain why a decision was taken or not taken in a vacuum but also records the knowledge of the context within which the decision can be understood. Thus, algorithm audits which do not technically serve as explanatory tools or interpretable models, can aid the delivery of the duty to provide reasons by explaining the broader context of the outcomes a system leads in its socio-technical context. 

If we have to look for a comparable XAI technique to the output interpretability requirement, it is perhaps best served by the idea of global model explanations. One prominent way to classify XAi methods is by drawing a distinction between explaining a model locally (i.e., a single prediction) or globally which is the whole model. The right to explanation is analogous to global explanations except it is perhaps even a lower standard in that information about the algorithmic logic need not include information included in several global methods such as TCAV. The transparency duty in Article 13 of the AI Act most decidedly goes beyond it to also include output interpretability. Thus, it entitles a user to receive information about an output or a single prediction aside from global explanations.  

Human Supervision and Administrative Discretion

Another provisions of the AI Act that will help us anchor our discussion is Article 14. It mandates that high-risk AI systems are designed with appropriate human-machine interface tools for persons overseeing them to be able to understand the capacities and limitations of the system fully; to remain aware of potential automation biases in (over)relying on the decision-making outputs of an AI system, and to decide or intervene in the operation thereof when safety and fundamental rights are at risk. 

I have discussed earlier how Article 14 notably takes a different route from Article 13. Instead of tying transparency to an XAI term, it merely states what this transparency must achieve. Therefore, the implementation under Article 14 is agnostic to any specific form of transparency so long as it achieves human supervision in the manner described. This approach is useful in the absence of academic or legal consensus. Article 14’s scope is limited only to human oversight in the implementation of the AI system and does not apply to any other stakeholders including both users and adjudicators. Article 14 primarily deals with human oversight to be exercised by domain experts. Sub-clause (4) does a reasonable job of spelling out the level of oversight and consequently the nature of transparency that is contemplated. It sets several metrics which can be used to test the transparency design of the oversight system including output interpretability, ability to monitor anomalies and dysfunction and kill-switch options to reverse or override the outputs. To satisfy these tests, the design of the high risk AI/ML system needs to rely on XAI techniques suitable to the context such as local and global model explanations, examination of gradients in case of neural networks and meta explanations. Importantly, the human oversight agent needs to also be empowered to apply their minds to an output and take decisions to override them. 

Yet again, we may find some guidance in the rich jurisprudence of public administration. A key aspect of public administration is the responsibility on actors to exercise administrative discretion. The structure of public administration presupposes that laws and regulations will provide guidance, yet for effective government, it is essential that discretionary power is exercised appropriately. In the previous section, we dealt with one aspect of improper exercise of discretionary power—when it is based on irrelevant consideration. Here, we are concerned with a requirement that precedes it—that public bodies exercise discretion where needed. It is expected that where an element of discretion is called for, the decision-making authority is required to apply its mind to the decision and not follow policy or any other diktat, in this case an algorithmic feed blindly.

While attempting to define the contours of human supervision as envisaged under Article 14, we may find it instructive to draw from the idea of administrative discretion and ensure that the degree of transparency afforded by the AI system to the persons overseeing them does not ‘fetter administrative discretion’ as Oswald puts it or lead to ‘judgmental atrophy’ as Hildebrandt puts it.

Several real-world situations, particularly in high-risk systems will require the evaluation of multiple factors. When this analysis is delegated to algorithms, what follows is a datafication of factors which would otherwise have been analysed by humans. The first real risk is the possibility of their being some form of ‘loss in translation’ during this process of datafication as is to be expected in any such process. The human-in-the-loop must be trained to exercise the same level of judgment as expected from administrative discretion, which is this situation would include applying their mind and recognising factors that may have been lost in translation, or other relevant factors on which the model was not trained or irrelevant factors that do not correspond to the purpose.

Even outside of the purview of public law, several decisions are made by private platforms where they play arbiters of free speech, privacy and data protection, financial and health services, and access to information, news, and entertainment. The tried and tested concepts of reasonableness, necessity, risk-assessment, and foreseeability which are well-entrenched in public law can suitably inform the nature of supervisions that humans can exercise in high-risk systems employing algorithms. The above concepts do not render themselves well to datafication or feature extraction. Therefore, creating suitable points for intervention of human discretion is critical.  

Regulatory Prescriptions

When high-risk systems employ algorithmic systems, their deployment must be preceded by the following conditions being met.

• The algorithmic systems must be designed to be interpretable to users such that the high-risk system may discharge the duty to give reasons for its decisions, as understood in administrative law.

• Where the technical nature of the algorithmic system poses fundamental interpretability challenges, it needs to be designed to flag sufficient information for independent human assessment to verify the machine’s inferences. Sufficient information may include the input data, the nature of model in use and the likely factors which informed the decision.

• The algorithmic system must be designed for supervision by a human agent within the high-risk system.

• The nature of transparency afforded to the human agent in (c) must permit them to independently verify or validate decision made by the algorithmic system.

[2] See detailed exposition at Grégoire Montavon, Wojciech Samek and Klaus-Robert Müller. 2018. Methods for interpreting and understanding deep neural networks. (February 2018) Digital Signal Processing 73. 1-15. DOI: https://doi.org/10.1016/j.dsp.2017.10.011.