As AI Takes Charge, Alarming Signs of Recurring Historic Biases Surface

A recent study from Lehigh University has unveiled significant racial bias in chatbots responsible for mortgage loan recommendations. Researchers analyzed 6,000 simulated loan applications derived from the 2022 Home Mortgage Disclosure Act, highlighting disparities in treatment based on race.

The findings showed that Black applicants faced higher denial rates compared to their white counterparts with identical financial profiles. Specifically, white applicants enjoyed an 8.5% higher approval rate than Black applicants. For those with “low” credit scores of 640, the gap widened drastically, with nearly all white applicants approved, while fewer than 80% of Black applicants received approval.

The study aimed to mimic the use of AI algorithms in financial institutions, where speed and efficiency are prioritized. Donald Bowen, an assistant fintech professor and co-author of the study, remarked on the significant yet problematic potential of these “black box” systems in influencing outcomes based on flawed data or historical biases.

“If there’s a baked-in bias, that could propagate across numerous customer interactions,” Bowen explained. This underscores the risk that these technologies, despite their efficiency, can perpetuate systemic injustices in lending practices.

AI decision-making tools have increasingly infiltrated various sectors, including finance, healthcare, and education. Most algorithms analyze inputs like age, income, and credit history. The algorithms then provide binary outcomes: approved or denied, sometimes including recommended interest rates. However, without careful oversight and unbiased datasets, these systems can inadvertently reinforce existing inequalities.

Bowen’s interest in racial discrimination within AI technologies sparked from preliminary findings in a classroom exercise. To rigorously investigate the bias, Bowen and his team processed a diverse range of loan applications through various prominent large language models, including OpenAI’s GPT 3.5 Turbo and GPT 4.

In experiments that intentionally included race-related data, discrepancies in outcomes were glaring. Conversely, instructing the chatbots to avoid bias produced nearly indistinguishable results among applicants. This raises an important question: if race is ostensibly not a factor in decision-making, why do disparities persist? Bowen attributed this to systemic influences reflected in credit scores and other socio-economic indicators rooted in historical discrimination.

The integration of AI extends beyond finance, infiltrating hiring practices, where AI tools are used to screen candidates. Despite regulations against discrimination, complaints have emerged regarding AI biases against older individuals and marginalized groups. Recent legal cases illustrate the consequences when companies misapply algorithmic screening, leading to significant settlements and policy shifts.

Moreover, the judicial system has begun utilizing AI for risk assessments concerning sentencing and pretrial releases. While these tools can enhance efficiency, experts caution against their sole reliance, especially when biases may skew judgment in critical legal matters.

Legislators have started recognizing the potential discriminatory elements of AI systems. Various states have implemented laws targeting algorithmic discrimination, and there is ongoing discourse in Congress regarding the regulation of AI in the financial sector. The proposed FAIRR Act seeks to address risks and develop guidelines for the ethical use of AI technologies.

Bowen’s study aims to guide institutions in arriving at equitable decisions through AI. Addressing bias through regular audits and incorporating more human oversight are critical steps towards a fairer future for algorithmic decision-making. As discussions surrounding AI standards gain traction, the hope remains that effective frameworks will emerge to enhance the trustworthiness and integrity of these technologies.