Gradient Institute Fairness Demo
Gradient Institute Fairness Demo

An Automated Decision System

You've been tasked by a large hypothetical financial organisation to implement a scalable automated screening process to accept or reject loan applications.

Your AI system will decide whether to accept or reject loan applications based on the applicant's ability to repay. But ability to repay is not something you can directly observe in the data. Instead, you could use machine learning to predict a risk score using attributes that you do observe in the application process, such as the applicant's income, and the size of the loan.

At the time when we decide whether to accept or reject a customer, we don't know for certain whether an applicant will repay or default. We can only make a decision based on their score. To make the fewest mistakes, we line up all the applicants from lowest score (left) to highest score (right). We then pick a cut-off threshold, and grant a loan to everybody right of a threshold, and reject everybody left of the threshold.

We've generated a synthetic dataset for this particular scenario below. Try out this decision rule by clicking on the last person you accept (rejecting everybody to their left and selecting everybody to their right):

A binary classifier such as this has four possible outcomes shown in the confusion matrix below:

Accepted Rejected
True Positives

False Negatives
False positives

True Negatives

Note that we use colour (red or green) to show whether the applicant can repay, and we use saturation (bold or faded) to show whether the applicant was accepted or rejected by the system. Try adjusting the threshold to minimise errors and you will discover a dilemma. To accept more green people, you have to also accept more red. As we tune the threshold, we can adjust the balance between mistakes where we grant a loan to somebody who will default (a false positive), or reject an application for somebody who can repay (a false negative).

Business Objective: Profitability

Let's introduce a simple model for the consequences of our actions:
{ } Customers who are given a loan and repay generate a profit of $2800.
{ } Customers who are given a loan and default generate a loss of $9000.
{ } Customers who are not given a loan incur no profit or loss (only missed opportunities).

Can you find a threshold that maximises profit by clicking the above population? You will need to balance the benefit of true positives against the cost of false positives.

Or from the point of view of a stakeholder (who you can click to reveal the solution):

Group Fairness

The organisation would also like to ensure that the system's outcomes don't systemically disadvantage groups of people. An AI system could easily make errors on a group if the data about them is poor, or they are under-represented in the training data. It might also perpetuate or amplify historical disadvantage, if this is captured in the data. Typically, we might consider groups identified in anti discrimination law such as groups based on age, gender, race or sexuality.
Suppose the stakeholders have identified a protected group that were historically disadvantaged. We won't name a particular group here, but simply call them the "disadvantaged group", and will distinguish them from other customers using a special "protected" icon:
people without a shield belong to the general population.
people with a shield belong to a potentially disadvantaged group.

If this group turns out to be (unintentionally) disadvantaged, then we say the system exhibits "indirect discrimination". Our goal, in the rest of this demo, will be to measure and mitigate disadvantage to this disadvantaged group.

Fairness Objective: Equality of Opportunity

One potential harm of the system is if it causes capable borrowers to miss out on loans. This is visualised by extracting all the green icons from the population. Things might seem unfair if one group is selected at a lower rate than another. The Equal opportunity fairness measure compares the rate that eligible applicants are selected amongst our two groups.

Explore the selection threshold to try to obtain equitable outcomes between the groups.

Controlling for fairness

By setting a single threshold for everybody above, you were making a "blind decision" because it was not considering an individual's circumstances. Under the hood of this simulation there actually no difference in ability to pay between the groups. The inequality we are seeing is resulting from our inability to distinguish red from green customers in the data. In this setting we can be confident that the observed inequality arises from our model and data. As you can see, a blind decision does not lead to equitable outcomes (unless you select everybody, or nobody, which aren't useful policies from a business perspective). One (of many) ways to actually control for group fairness is to pick a different threshold for each group. Below we have split the thresholds of the group from the group. Try using the split controls to adjust the cut-offs of each group independently until you achieve equal opportunity.

Now that you have equal opportunity, can you maintain it and also improve the profit? It turns out there are a continuum of equal opportunity selections, taking the same fraction of eligible people from each group. Try using the slider to explore a range of equal opportunity decisions, and find the setting that maximises profit.

When we define a single notion of fairness like this we have two objectives - a business objective and a fairness objective. They are somewhat at odds, but we can think about concepts like the price of fairness, or finding the most efficient fair system. But depending on the context, there are many other notions of fairness that we could also consider.

Fairness Objective: False Discovery Parity

False discovery parity is an alternative notion of fairness, in this scenario concerned with the ethics of responsible lending. Instead of focusing on those who could repay, let's now isolate the cohort that we selected in the above list ( ) and within these selected individuals examine whether the default rate is the same for each group (i.e. comparing the rate of vs ). We now have a clear trade-off between not only profit and equal opportunity, but even between the two fairness metrics.

Although we have satisfied equal opportunity above, clearly we are being unfair by this different (yet equally valid) definition of fairness. Try now adjusting the selection to align with our false discover parity objective.

You will discover that you can't have both types of fairness simultaneously. What is really going on here is that false discovery parity requires us to be more careful (exclusive) and the equal opportunity requires us to be more inclusive. In general we will have any number of objectives (fairness or otherwise) that will need to be balanced.

But that doesn't mean that we should give up. It is important to realise that there is no "right answer" here. Instead, we need to expose this tradeoff to the stakeholders to make a value judgement about how important each objective is. We can then balance the trade-off. Lets assume that we want maximum profit for a given fairness level - the profit optimisation will be automatic. Use the slider below to tip the balance between equal opportunity fairness and false discovery parity fairness.

Mapping the trade-offs

In this simplified scenario, the decision is fundamentally two dimensional because we control two aspects of the decision policy: the acceptance rate of the protected individuals, and the acceptance rate of the general population.
Until now we have been looking at the outcomes in terms of three objectives (profit, equal opportunity and default rate) independently. While this helps us understand the impact of a policy decision, it doesn't convey the relationship between them.
Below, we map the decision space on a two dimensional plot, where any combination of thresholds becomes a point on the map. The position of the point on the X-axis is the threshold for the general group, and the position on the Y-axis is the threshold for the protected group. Try dragging the black dot around on the map to see the effect on the metrics above.
Where would you go to balance the objectives? The "correct" position is not something we can calculate. Nor is it a decision a data scientist should make implicitly. We need to reveal the trade-off that was always there for everybody developing the system to see.