7 Limitations and Responsible AI

What AI can get wrong

AI systems learn patterns from data and apply them to new situations, but their outputs are not automatically correct, fair, or safe. They can produce plausible answers or predictions that are inaccurate, and their performance can decline when real-world data differs from training data or changes over time.

A pattern or correlation does not necessarily explain why something happened or what action is appropriate. Complex models may also make it difficult to identify why a particular output was produced. Outcomes are affected by data quality, system objectives, and human choices, so AI does not remove human responsibility.

For example, a hiring tool trained on past hiring records may learn patterns in those records without understanding whether they reflect job-related qualifications. Evaluation should consider the system’s intended use and whether it works reliably for the people and situations it will encounter.

Bias and fairness

Bias is a systematic skew in data, system behavior, or outcomes. It can arise when groups are under-represented in training data, records contain historical discrimination, labels are inaccurate, or development choices shape the system. Information that seems neutral may also act as a proxy for sensitive characteristics. These issues can produce unequal or discriminatory results.

Reducing harmful bias involves checking data quality and representation, testing performance across relevant groups, involving people with varied perspectives, and monitoring outcomes after deployment. Fairness can mean different things in different settings, and measures of fairness may conflict. No single score or test establishes that a system is fair in every respect.

Privacy, transparency, and explanation

AI development and use may involve sensitive personal information. Risks include collecting more data than necessary, exposing data through weak safeguards, or inferring private details from other information. Responsible practice limits collection and access, protects data, and considers privacy throughout the system’s life cycle.

Transparency means providing appropriate information about a system, such as where and why it is used, its important limitations, and who is responsible for it. Explainability means making an output or decision understandable enough for affected people to assess it. The level of explanation should reflect the impact: someone affected by a consequential decision needs a meaningful way to understand and challenge it. Transparency must also be balanced with privacy and security.

Safety, security, and human oversight

AI systems can cause harm through mistakes, misuse, unexpected behavior, or security attacks. Safety focuses on preventing harmful outcomes, while security helps protect systems and data from unauthorized access or manipulation. Testing, safeguards, monitoring, and plans for responding to failures help reduce these risks.

For high-impact uses, human review should be meaningful. Reviewers need enough information and authority to question or override a system rather than simply accept its output.

An ongoing practice of responsibility

Responsible AI is an ongoing process rather than a one-time checklist. Its safeguards should match the system’s purpose and the possible consequences for people.

1. Define the purpose and assess risks. Do this before choosing to use AI, and do not use it where likely harms outweigh its benefits.

2. Evaluate before deployment. Use relevant data and realistic conditions to check accuracy, safety, privacy, and impacts on different groups.

3. Inform affected people. When AI plays a significant role, provide an appropriate way to ask questions, seek human review, or correct mistakes.

4. Assign accountability. Identify the people and organizations responsible for development, deployment, and oversight.

5. Monitor and improve. Reassess the system as data, contexts, and impacts change, and address problems when they arise.

These practices align with trustworthiness characteristics that include validity and reliability, safety, security and resilience, accountability and transparency, explainability, privacy, and fairness with harmful bias managed. They also align with an emphasis on human rights, oversight, and non-discrimination.