5. Bias in Computing and Artificial Intelligence

Understanding Bias as a System-Wide Problem

Computing systems increasingly influence what people see, which opportunities they receive, how resources are allocated, and how risks are evaluated. They include ordinary algorithms, statistical models, machine-learning systems, and artificial intelligence systems.

Bias can arise even when no individual intends to discriminate. Human choices determine which problem to solve, which data to collect, which categories to use, what outcome counts as success, and how much risk is acceptable. As a result, bias can enter during problem definition, data collection, labeling, model development, deployment, or interpretation.

A useful starting point is to distinguish three interacting sources:

• Systemic bias reflects unequal institutions, policies, infrastructure, or social conditions.

• Computational bias results from data preparation, model design, measurement, evaluation, or deployment.

• Human bias enters through judgments made by developers, managers, annotators, users, and decision-makers.

These sources can reinforce one another. Unequal social conditions may shape historical data, a model may reproduce the resulting patterns, and users may treat the model’s output as objective.

Takeaway: Bias is a system-wide issue, not merely a programming mistake or a problem that begins after a model is deployed.

How Bias Enters Through Data

AI systems learn patterns from data. If data are incomplete, inaccurate, unrepresentative, or shaped by earlier discrimination, a system may reproduce or magnify those problems. A large dataset is not automatically a fair dataset because data also reflect how institutions have measured, categorized, and treated people.

Important sources of data bias include:

• Historical bias: Past decisions or social conditions were unequal, so the data preserve those inequalities.

• Representation bias: Some groups appear too rarely or not at all.

• Sampling bias: The collection method favors some populations over others.

• Measurement bias: A recorded variable is an imperfect substitute for the concept the system is supposed to measure. For example, medical spending may not accurately represent health needs when access to care is unequal.

• Labeling bias: Human annotators apply categories inconsistently or according to subjective assumptions.

• Missing-data bias: Information is absent more often for some groups, and treating missing information as zero, no, or low risk creates unequal effects.

• Proxy variables: Apparently neutral features such as ZIP code, school attended, language, or employment gaps may indirectly encode race, wealth, disability, or another protected characteristic.

Data quality must therefore be evaluated in relation to the intended use. Questions should include who is represented, who is missing, how the data were collected, whether labels measure the intended concept, and whether important groups have enough examples for reliable evaluation.

Takeaway: Data are evidence shaped by social choices, not neutral descriptions of reality.

Bias in Algorithms and Models

An algorithm is a defined procedure for processing information or solving a problem. A machine-learning model is an algorithmic system that learns patterns from examples rather than receiving every rule directly from a programmer.

Bias can arise when designers:

1. Choose an objective that favors speed or profit over equal access, safety, or accuracy across groups.

2. Use a target that reproduces past decisions instead of measuring the underlying need or ability.

3. Select features that encode sensitive information through direct fields or proxies.

4. Optimize only for average accuracy, hiding higher error rates for a smaller group.

5. Apply thresholds that create unequal outcomes for populations with different data distributions.

6. Deploy a model outside the population, location, time period, or task for which it was developed.

Accuracy and fairness answer different questions. Accuracy measures how often a system is correct according to a selected definition. Fairness asks whether errors, benefits, burdens, and opportunities are distributed acceptably across relevant groups. A system may be accurate on average while being unfair to a minority group.

Fairness criteria can also conflict in particular settings. Improving one fairness measure may affect another, so fairness cannot be reduced to one universal number. The relevant criteria depend on the system’s purpose, affected people, legal requirements, and social context.

Takeaway: A model’s overall accuracy does not establish that its objectives, errors, or effects are fair.

Automated Decisions and Human Oversight

An automated decision-making system uses software to make or recommend decisions that may affect people. Examples include ranking job applicants, approving or pricing loans and insurance, determining eligibility for public benefits, recommending content or advertisements, assessing medical risk, detecting fraud, assigning students to courses, and identifying suspicious activity.

Automation can increase speed and scale. That benefit also creates risk: a biased pattern that affects a few people in a human process may be applied to millions of people by an automated system, with little opportunity for correction.

Human involvement does not automatically remove bias. Reviewers may:

• Trust a generated score too much because of Automation bias.

• Misunderstand what the score represents.

• Apply the system differently to different groups.

• Fill in missing information using stereotypes.

• Lack the time, authority, or information needed to challenge a result.

Human oversight is useful when reviewers are trained, empowered to question the system, given meaningful explanations, and held accountable for the final decision. High-impact decisions should also include notice, an appropriate explanation, and a meaningful way for affected people to appeal or request correction.

Takeaway: A person in the loop is not enough; oversight must be informed, empowered, and accountable.

Recognizing and Measuring Bias

Bias should be examined throughout the system life cycle rather than only after deployment. The NIST AI Risk Management Framework organizes continuing work into govern, map, measure, and manage.

Questions to ask

• Purpose: What decision is being made, and who could be helped or harmed?

• Population: Which groups will use the system or be affected by it?

• Data: Who is represented, who is missing, and how were the data collected?

• Labels: Who defined the categories, and do they validly measure the intended concept?

• Features: Could a variable act as a proxy for a sensitive characteristic?

• Performance: Does performance differ across demographic, geographic, linguistic, age, disability, or socioeconomic groups?

• Context: Does performance change across locations, time periods, or real-world conditions?

• Impact: What happens when the system is wrong, and can a person appeal or obtain a correction?

• Governance: Who is responsible for monitoring, documenting, and correcting the system?

Useful evaluation methods

• Compare accuracy, false-positive rates, false-negative rates, and other error measures across groups.

• Test on separate validation data instead of relying only on training results.

• Examine subgroup results rather than only overall averages.

• Use Intersectional analysis to study groups defined by combinations of characteristics.

• Review data documentation, model documentation, and intended-use limits.

• Conduct pre-deployment impact assessments and independent audits.

• Test with affected communities and domain experts.

• Monitor outcomes after deployment because populations, data, and behavior can change.

Measurements must be interpreted in context. A metric that is useful in one application may be inappropriate in another, and laboratory performance may differ from real-world performance.

Takeaway: Bias evaluation requires subgroup analysis, contextual judgment, and continuing monitoring.

Reducing Bias Across the System Life Cycle

No single technical adjustment can eliminate all bias. Effective mitigation combines technical methods with organizational oversight and participation by affected communities.

Before building

• Define the problem carefully and ask whether automation is necessary.

• Identify affected communities and include diverse stakeholders.

• Establish acceptable uses, prohibited uses, and escalation procedures.

• Select an outcome that reflects the real goal rather than a convenient but misleading proxy.

• Assess privacy, accessibility, civil-rights, and safety risks.

During data collection and preparation

• Improve representation of relevant populations.

• Document how data were collected, labeled, cleaned, and transformed.

• Investigate missing data and measurement errors.

• Use privacy-protective methods when collecting sensitive attributes for evaluation.

• Review labels for consistency and revise ambiguous categories.

• Consider reweighting, resampling, or adding data when appropriate, while recognizing that changes can introduce new problems.

During model development

• Compare multiple models and baselines, including simple human or rule-based approaches.

• Measure subgroup and intersectional performance.

• Test fairness constraints or group-specific error controls when appropriate.

• Check whether removing a sensitive feature actually reduces bias; proxies may remain.

• Document trade-offs among accuracy, fairness, privacy, interpretability, and other goals.

During deployment and use

• Explain the system’s purpose, limits, and relevant factors influencing its output.

• Keep a human decision-maker accountable for high-impact decisions.

• Give affected people notice, context-appropriate explanations, and a meaningful appeal process.

• Monitor for drift, changing populations, new failure patterns, and disparate impacts.

• Pause, modify, or withdraw a system when harms cannot be adequately controlled.

Takeaway: Responsible mitigation is a continuing governance process, not a one-time model adjustment or fairness certification.

Worked Example: An AI Hiring Tool

Consider an AI hiring tool trained on résumés from employees hired during the previous ten years. The model learns that certain words, career paths, or extracurricular activities are associated with past hires.

Several forms of bias may appear at once:

1. Historical bias: If earlier hiring favored men, the training labels reflect that history.

2. Representation bias: If few applicants from certain communities were considered, the model has little evidence about their potential.

3. Proxy bias: A school, address, organization, or employment gap may indirectly encode race, gender, disability, or socioeconomic status.

4. Measurement bias: “Was hired” may be used as a label even though hiring decisions were not a perfect measure of job ability.

5. Deployment bias: The tool may be used for a different job or applicant population than the one represented in its training data.

6. Human bias: Recruiters may treat the ranking as objective and stop examining qualified applicants ranked lower.

A responsible process would define job-related criteria, audit historical labels, test relevant group performance, involve employment and civil-rights experts, document limitations, monitor outcomes, and provide human review and an appeal process.

The example illustrates why removing a sensitive field alone is not enough. Other features may act as proxies, and the original target may encode unequal institutional decisions.

Takeaway: Bias analysis must examine the target, data, features, deployment context, and human use together.

Limits and Core Principles

Bias reduction does not prove that a system is fair in every relevant sense. Important limitations remain:

• A system may satisfy one fairness metric while failing another.

• Protected characteristics may be unavailable or legally restricted, making evaluation difficult.

• Aggregate results can hide severe problems for small or intersecting groups.

• Removing sensitive attributes can make bias harder to detect without removing its causes.

• A technically balanced model can still be used in an unfair institution or for an inappropriate purpose.

• Fairness can change as society, data, and system users change.

For these reasons, fairness should be treated as an ongoing responsibility. Organizations need clear accountability, documentation, independent testing where appropriate, meaningful human review, monitoring after deployment, and ways for affected people to challenge or correct decisions.

The central principle is to evaluate not only whether a system works on average, but also whom it serves, whom it burdens, how it fails, and whether its use is appropriate in context.

Final takeaway: Responsible computing requires attention to systemic, computational, and human conditions throughout the full life cycle of an automated system.

• Computing bias

• proxy variable