Proposal for Major Change to the DPG Standard under Indicator 8

[AmreenTaneja]

Mandatory AI Risk Assessment Submission for AI Digital Public Goods

Overview

This proposal introduces a significant change to the Digital Public Goods Standard under Indicator 8 by mandating the submission of an AI Risk Assessment for all applicants applying for DPG Status for their AI systems. Applicants will be required to complete the AI Risk Assessment prepared by the DPGA Secretariat (mentioned below) or submit an equivalent industry-recognized template, such as, but not limited to:

• Google Secure AI Framework (SAIF)
• NIST AI Risk Management Framework (AI RMF)
• Microsoft Responsible AI Impact Assessment Template
• ISO 23894 - AI Guidance on Risk Management
• TMC3 AI Impact Assessment Template

Previously, the DPG Standard encouraged adherence to standards and best practices. With this proposal, compliance with the AI Risk Assessment requirement will become mandatory for AI systems seeking DPG recognition. This represents the first occasion where we have had a unique criteria for a DPG-type.

Proposed Change

1. New Requirement:

• All applicants applying for DPG status for their AI systems must submit an AI Risk Assessment.

• The applicant has the option to:
  - Complete the AI Risk Assessment template provided by the DPGA Secretariat.
  - Submit an equivalent AI Risk Assessment template recognized by industry standards.

2. Scope:

• This requirement applies only to applicants seeking DPG status for AI systems.
• Non-AI-related DPG applications remain unaffected by this mandatory submission.

3. Rationale:

• To ensure AI systems recognized as DPGs provide information about their adherence to ethical, transparent, and responsible AI practices.
• To align the DPG Standard with global best practices for mitigating AI-related risks, such as bias, privacy violations, and misuse.
• To promote accountability and trustworthiness in AI systems seeking DPG recognition.

4. Enforcement:

• Applications for AI systems will not proceed to the review stage without a completed AI Risk Assessment.

Governance Process

1. Community Engagement:

• As per the DPG Standard governance processes, this proposal is being shared with the DPGA community and stakeholders on GitHub for public comment.
• The public comment period will remain open for three weeks till March 3rd 2025 to ensure opportunity for feedback and suggestions.

2. Review and Iteration:

• After the public comment period, the DPG Standards Lead will consolidate and review all inputs.
• The updated proposal, incorporating feedback, will be presented to the DPG Standard Council for further deliberation and decision-making.

3. Implementation Timeline:

• If approved by the DPG Standard Council, the mandatory AI Risk Assessment submission requirement will take effect immediately for all new AI DPG applications.

Call to Action

We invite the DPGA community and stakeholders to review this proposal and share their feedback through comments on this GitHub issue. Your input is critical to ensure the effective and equitable implementation of this important change. Please also review and provide your input on the AI Risk Assessment template mentioned below, which DPG applicants can complete as an alternative to industry-recognised templates.

AI Risk Assessment Template

(Proposed to be mandatory under Indicator 8 of the DPG Standard- requiring major change proposal)
Please make a copy of this template and answer the following questions to the best of your ability, or share your risk assessment using another template.

1. Proportionality

• Describe the potential impact of the AI system on people, including vulnerable groups and those affected indirectly, in the context for which it was developed? (Consider how the model/system could affect these groups positively or negatively.)
• Describe if and how stakeholders were engaged in the design and development of the solution. Indicate whether the system was designed based on human-centric principles, e.g. principles for digital development, FAIR principles, OECD Principles on AI, UNESCO Recommendation on the Ethics of AI, Google AI principles, etc.

Proportionality risks refer to the potential for AI systems to exhibit disproportionate or excessive responses, actions, or impacts relative to the intended purpose. For instance, disproportionality against a particular language, geography, population, or other characteristics that might only represent certain groups of people.

2. Bias and fairness

• What potential biases could be encoded in the AI system?

(Consider that biases could occur in the data used to train and test the AI system, in the AI model, or the functioning and outcomes of the AI system.)

• What steps are taken to monitor, mitigate, and address biases?

(Document the results of relevant fairness assessments and further measures to monitor and reduce remaining bias.)

• With which metrics have you decided to evaluate the performance and fairness of the model? Please indicate why you have selected these metrics?
• What are the thresholds used by the model (if any) and how have these been selected?
• What approach do you have in place to evaluate the model’s performance and fairness against new and/or unseen data (e.g. via cross-validation or bootstrapping)?
• How does the model handle outliers, over- and underrepresentation, and bias towards certain populations?

Examples include but are not limited to:

• Data Bias: AI trained on biased data (e.g., historical criminal justice data) can perpetuate discriminatory practices.
• Algorithmic Bias: AI systems (e.g., facial recognition) trained on unrepresentative data can lead to misidentification and harm.
• Outcome Bias: AI algorithms that rely on factors correlated with socioeconomic status can indirectly discriminate against minority groups.
• Unintended Bias: AI trained on data from a company with a predominantly male workforce may inadvertently favor male candidates.
• Representation Bias: AI models trained on text data that underrepresent certain dialects or languages may limit access to information for those groups.

3. Mitigations

• What steps are taken to evaluate the accuracy of the AI system outputs and make adequate decisions?
• What processes exist to implement (and document) thorough model validation and quality assurance?
• What guardrails are implemented to mitigate the consequences of erroneous outputs and to prevent future errors?
• What are the technical measures taken to ensure the robustness and security of your system against potential attacks throughout its life cycle? (eg. AI Safety Good Practices linked in the wiki)
• Does the AI system leave meaningful opportunities for human decision-making, oversight, and control?
• Are there measures in place to effectively appeal the model’s decision? If so, how is this process managed and communicated to the users?

Examples of risk mitigation practices include but are not limited to:

• Diverse data used for training.
• Human review for high-risk or borderline applications.
• Clear appeals process with human oversight.
• Regular audits for bias detection.
• Continuous performance monitoring in real-world settings.
• Adversarial training to make the model more robust.

4. Risks and harms / Use cases

• Did you perform validation experiments in the deployment context(s)?
• Could any potential harm arise from misuse or unintended use of the AI system?
• Please describe the guardrails implemented to prevent misuse and limit harmful unintended uses and an assessment of how effective the guardrails are in preventing unintended use.

5. Transparency

• What measures are taken to ensure transparency and explainability, including using tools for responsible AI development or explainability?
• Does the AI system describe how the system works and/or explain the logic it uses to arrive at a particular output?
• If users interact directly with the system, have they been informed that they are interacting with an AI system, not a human?
• Is AI-generated content clearly labeled as such (see AI Safety Good Practices to be published in the DPG wiki)

Examples of transparency measures include but are not limited to:

• Explainable AI (XAI), AI systems that can provide clear explanations for their decisions.
• Data transparency, document the source and characteristics of the training data, including potential biases and limitations.
• Model transparency, provide information about the model's architecture, parameters, and training process.
• Performance transparency, report on the AI system's performance using clear metrics, including accuracy, fairness, and robustness.
• Auditability, design AI systems with audit trails that record key decisions and actions.
• User-facing transparency, provides clear and concise information to users about how the AI system works, what data it uses, and how it impacts their experience.

[kjetilk]

I have become rather despondent with regards to such approaches to technology governance. Let me detail why.

There is a spectrum to consider between honest and dishonest actors. Let us discuss two classes:

Honest actors

I think Risk Assessment guidelines could be good to guide honest developers to help bring to mind the issues they need consider. An honest developer would naturally seek to do their best to reduce the risk of deploying systems, and would find it welcome to have this kind of guidance.

However, they are subject to all kinds of pressures, not to speak of constrains of time and effort they are able to put into it. Therefore, there is an opportunity cost to prioritize writing a good Risk Assessment, so careful consideration should by put into whether this requirement would replace something more important.

I believe that governance practices and standards are much more important, ensuring that multiple parties with differing interests are involved, that there are automated tests, that compliance to technical standards can be verified, overall, that ecosystems are healthy. Such internal governance practices are much more important to ensure that risks can be understood and mitigated than reports.

Dishonest actors:

The Alliance must consider the possibility that dishonest actors will try to get commons-washed systems approved as a DPG.

So, what else would be used to write an AI Risk Assessment than Generative AI? A GenAI system would be trained on other "successful" Risk Assessments, to ensure that it ticks the right boxes. It could be trained on texts that are written by the auditors, so that it will appeal to them. Reports that are written to clear a bar is, in my rather despondent world, likely to be written not to be honest documentation to bring light, but to be marketing material. I'm equally critical towards similar requirements in EU AI Act, EU DSA, the Biden-administration's efforts (which will never see the light), and so on.

There is a very real risk that it will result in a weapons race kind of economic malevolent circle: In a weapons race, if your measures are more expensive than your opponents counter-measures, you loose. The best way to avoid a weapons race is to not get into conflict in the first place.

Consider the case where a dishonest submitter will use 2 minutes to generate a Risk Assessment report, while the Alliance's evaluators will require 10 hours to read, understand and examine the claims of the report, and even that may not be enough to determine that the report is bullshit, in the sense of Harry G. Frankfurt.

Again, I do not think such reports are the way to do it. For closed systems, it is perhaps the only thing you can have, and post-disaster, enforces may use it to fine, but that will be a meager comfort for those affected. Closed systems are basically ungovernable.

Digital Public Goods are governable, but the Open Source community hasn't managed to go from meritocracies to practices that can be anchored in democratic processes. That, together with the increased popularity and role of Open Source creates increased tension. It should rather be the focus to fix this, not increase the red tape that dishonest actors will easily work around.

[AmreenTaneja]

Dear @kjetilk,

Thank you for your thoughtful input. With the Standard, we are actively working to ensure that the application process remains light for applicants while maintaining meaningful safeguards. We will be providing checklists to guide submissions and reduce unnecessary complexity while ensuring that the process remains robust enough to prevent loopholes that could be exploited by dishonest actors.

That said, the DPG application process also inherently relies on a trust-based system with elements of self-declaration, meaning compliance depends on a balance of verification and good faith participation.

Additionally, I wanted to share that we have introduced another significant change to the Standard by explicitly including "misleading content" under Indicator 9B. This update strengthens responsible content management policies and aligns with global concerns around misinformation, particularly in AI-generated contexts.

It would be great to get your insights on possible solutions—especially regarding the assessment of AI risks for AI DPG accreditation. Looking forward to your thoughts.

Best,
Amreen Taneja