Artificial Intelligence seems to be everywhere these days. As we wrote last month, generative AI tools are rapidly becoming a workplace temptation for employees seeking to streamline their job duties. Similarly, AI has taken on a role in recruiting and hiring at many companies, which has drawn scrutiny from the EEOC, as well as state and local regulators concerned about the potentially discriminatory effect of AI-driven Automated Employment Decision Tools (AEDT). One important example is New York City, whose AEDT Law became effective January 1, 2023. The NYC law bans employers from using AI tools unless, among other things, they have first submitted to an independent audit bias within a year of use. NYC will begin enforcing the AEDT Law on July 5.

While employers have a wide variety of computer-based AEDT available to assist them in hiring decisions, monitoring worker performance, determining pay or promotions, and establishing the terms and conditions of employment, potentially saving significant time and effort, use of AEDT may unintentionally disadvantage job applicants and employees based on their protected characteristics. This is a reminder that employers need to carefully implement AI tools to ensure they do not violate equal employment laws.

What is AEDT Exactly?

AEDT is defined under the NYC law as any tool that applies artificial intelligence to “substantially assist or replace discretionary decision making” of an employer, such that it does any of the following: (i) scores, classifies or ranks job applicants or employees based on only one factor; (ii) gives more weight to a simplified output as one set of criteria; or (iii) uses a simplified output to overrule conclusions derived from human decision-making or other factors. Put simply, it means, at least in part, the computer is making its own decisions with respect to applicants and employees.

There are many different types of AEDT that can incorporate AI decision-making at several stages of the employment process, and common examples include: (i) resume scanners that prioritize applications using certain keywords; (ii) “virtual assistants” or “chatbots” that ask job candidates about their qualifications and reject those who do not meet pre-defined requirements; and (iii) monitoring software that rates employee performance on keystrokes or other predetermined factors.

Possible Implications

AEDTs are making things easier than ever these days, but it is not difficult to see where potential implications can arise. These tools can fail to accommodate, “screen out” individuals based on their protected classes, request inappropriate medical or other identifying information, and have a disparate impact, among other concerns. For example, a chatbot may ask a job applicant for their work history and screen out those individuals with gaps in their employment. However, if the gap was attributed to a medical condition or parental leave, then the employer would have discriminated against the job applicant based on disability or sex.

 A notable “real world” example is a recent EEOC case alleging national origin discrimination where a job-search website for technology professionals had a job posting that included key words such as “H1B,” “visa,” “only,” and “must.” While the intent was innocuous because the language was intended to ensure the job applicant was legally authorized to work in the United States, in effect, the terms excluded an entire protected class (American citizens) because they were not H1B or visa holders. This example shows that even inadvertent errors or oversights will still carry legal risk for employers.

Bias Audits and NYC’s AEDT Law Requirements

Although there are concerns with the unrestricted use of AEDT, there are steps employers can and should take to minimize legal risk. In fact, NYC’s AEDT Law requires employers to obtain an independent audit of their AEDT within a year of use and must make the results of the audit publicly available on their websites. Employers are also required to notify applicants and employees who reside in NYC of the use of AEDT and the process for requesting an alternative selection process or reasonable accommodation 10 days before using AEDT.

Under the NYC law, the bias audits must, at a minimum:

• Calculate the selection rate for each category by dividing the number of individuals in the category moving forward or assigned a classification by the total number of individuals in the category who applied for a position or were considered for promotion;
• Calculate the impact ratio for each category, which is either (1) the selection rate for a category divided by the selection rate of the most selected category or (2) the scoring rate for a category divided by the scoring rate for the highest scoring category;
• Ensure that the calculations above separately calculate the impact of the AEDT on gender, ethnicity, and race, and intersectional categories of sex, ethnicity, and race;
• Ensure that the calculations above are performed for each group; and
• Indicate the number of individuals the AEDT assessed that are not included in the required calculations because they fall into an unknown category.

The bias audit also must be conducted by an independent auditor, defined as a person or group that is capable of exercising objective and impartial judgment on all issues within the scope of a bias audit of an AEDT. A person is not an independent auditor, however, if they were involved in using, developing or distributing the AEDT, or have or had an employment or financial relationship with the employer using the AEDT or the vendor developing or distributing the AEDT.

While the AEDT Law only applies to employers with applicants and employees in NYC, because the EEOC has stated in its ADA and Title VII guidance that it will hold all employers liable for violations of equal employment laws, even when using third party AEDT, employers outside the NYC area should still consider running bias audits to ensure compliance.