Researchers have developed a tool using artificial intelligence (AI) to detect unfair bias in protected areas such as race or gender. The tool could be used in finding bias in AI systems or even bias by human decisions makers, according to the researchers at Penn State and Columbia University.
“Artificial intelligence systems—such as those involved in selecting candidates for a job or for admission to a university—are trained on large amounts of data, but if these data are biased, they can affect the recommendations of the AI systems,” said Vasant Honavar, professor of information sciences and technology at Penn State in a statement.
If a company has never hired a woman for a particular job, an AI system trained on such historical data wouldn’t likely recommend a woman for a new job at the company.
Honavar said in that example that there would be nothing wrong with the machine learning in the software program itself. “It’s doing what it’s supposed to do, which is to identify good job candidates…but since it was training on historical, biased data it has the potential to make unfair recommendations.”
The researchers created an AI tool to detect bias by human or AI systems that is based on causality, or cause and effect.
The question, “Is there gender-based discrimination in salaries?” can be restated as “Does gender have a causal effect on salary?” or “Would a woman be paid more if she was a man?” To know that answer directly, the researchers used counterfactual inference algorithms to arrive at a best guess.
One way of arriving at a best guess for a fair salary for a female would be to find a male with similar qualifications, productivity and experience. “We can minimize gender-based discrimination in salary if we ensure that similar men and women receive similar salaries,” said Aria Khademi, a grad student in information sciences and technology at Penn State.
Researchers used income data from the U.S. Census Bureau to spot gender-based discrimination in salaries and New York City police department data for findings of bias against people of color arrested after stop-and-frisk arrests.
With incomes, the teams used data on 50,000 people and found evidence of gender discrimination in salary, specifically that the odds of a woman having a salary greater than $50,000 a year is only one-third that of a man. For the possible racial bias, the researchers revealed evidence of possible racial bias against Hispanics and African-Americans, but found no evidence of discrimination against them on average as a group.
Honavar said the tool the researchers built can help organizations avoid becoming instruments of discrimination and threats to unfairness. “You cannot correct for a problem if you don’t know that the problem exists…Our tool can help with that,” he said.
The research was supported by the National Institutes of Health and National Science Foundation.