Starting January 1, 2023, New York City employers that utilize artificial intelligence (“AI”) decision-making tools in their hiring practices will need to provide notice to applicants of the technology and conduct independent bias audits to ensure that these tools do not have a discriminatory impact on candidates. This new law, which is aimed at eliminating bias in automated employment decisions, is the first of its type in the United States.
The New York City Department of Consumer and Worker Protection (“DCWP”) has issued proposed regulations in connection with the law and will be holding a public forum to discuss the proposed regulations on October 24, 2022.
Employers and employment agencies should not wait until the regulations are finalized to develop a catalog of AI-driven tools they use for assisting in hiring and promotion decisions and work with vendors and technology stakeholders to develop the means for independent audits that are sufficiently linked to the jobs and job classes for which the organization anticipates hiring.
Who is affected?
The new law (Local Law Int. No. 1894-A) applies to all employers and employment agencies that use “automated employment decision tools” (“AEDT”) in connection with making “employment decisions.”
What is an “automated employment decision tool”?
The law defines the term “automated employment decision tool” to be “any computational process, derived from machine learning, statistical modeling, data analytics, or artificial intelligence, that issues simplified output, including a score, classification, or recommendation, that is used to substantially assist or replace discretionary decision making for making employment decisions that impact natural persons.”
The term “automated employment decision tool” does not include a tool that does not automate, support, substantially assist, or replace discretionary decision-making processes and that does not materially impact natural persons, including, but not limited to, a junk email filter, firewall, antivirus software, calculator, spreadsheet, database, data set, or other compilation of data.
What counts as an “employment decision”?
The term “employment decision” means to screen candidates for employment or employees for promotion within New York City.
What are the requirements of the new law?
- Prohibition on using an automated employment decision tool to screen candidates (and exceptions):
- An employer or an employment agency in NYC may not use an AEDT to screen a candidate or employee for an employment decision unless:
- the tool has been the subject of a “bias audit” conducted no more than 1 year prior to the use of such tool; and
- a summary of the results of the most recent bias audit of such tool, as well as the distribution date of the tool, has been made publicly available on the website of the employer or employment agency prior to its use.
- A “bias audit” means an impartial evaluation by an independent auditor. At a minimum, the bias audit must assess the AEDT’s disparate impact on individuals of any race/ethnicity or sex category required to be reported by employers to the U.S. Equal Employment Opportunity Commission (“EEOC”) in an EEO Component 1 report.
- Notices required:
- Employers must provide each candidate (internal or external) with 10 business days’ notice before they can screen the candidate using the tool.
- The notice must also list the “job qualifications and characteristics” used by the tool to make its assessment and advise that the candidate may request an alternative selection procedure or accommodation (although the law does not specify what these should entail).
- Employers and agencies must make the sources and types of data used by the tool, as well as the applicable data-retention policy, available publicly (or upon written request from the candidate). Such information shall be provided within 30 days of the written request.
- It is unclear whether the new law requires notices to be issued before each use of an automated tool with respect to an individual’s application, whether the use of multiple automated tools requires a separate notice process with respect to each tool, or what counts as effective service of notice (including whether an acknowledgement of receipt is necessary to start the 10-day clock ticking).
When does the new law take effect?
The new law takes effect on January 1, 2023.
Compliance
Any employer who violates the new law will be liable for a civil penalty of not more than $500 for a first violation and each additional violation occurring on the same day as the first violation, and not less than $500 nor more than $1,500 for each subsequent violation.
Each day on which an AEDT is used in violation of the new law shall give rise to a separate violation for which penalties can be imposed. Failure to provide a candidate with any requisite notice (as described above) constitutes a separate violation.
On September 23, 2022, the New York City Department of Consumer and Worker Protection (“DCWP”) published proposed effectuating regulations.
The proposed rules are intended to clarify several items, including:
- The requirements for the use of AEDTs,
- The notices to employees and candidates for employment regarding the use of AEDTs,
- Bias audit requirements, and
- The requirement that employers publish results of the bias audit(s).
Proposed AEDT Use Requirements
The proposed regulations provide insight into the terms “machine learning, statistical modeling, data analytics, or artificial intelligence” as used in the statute. This grouping of terms is intended to encompass any mathematical, computer-based techniques: (1) that generate a prediction or classification about candidates; (2) where a computer is at least partially involved in determining, weighing, or measuring those items to improve accuracy; and (3) for which the inputs and parameters are refined through cross-validation or by using training and testing data.
The proposed regulations also define “simplified outputs,” the use of which will constitute substantial assistance in, or replacement of, discretionary decision-making in the evaluation of candidates for employment. Simplified outputs include all predictions or classifications that result from the use of the AEDT. These include tags, categories, rankings, recommendations, and other outputs that decision-makers can use in making hiring decisions.
Proposed Rules Regarding Notices
Under the proposed rules, employers and employment agencies can comply with the obligation to provide notice to candidates and current employees regarding the use of an AEDT, as well as the qualifications and characteristics that will be used by the AEDT, by: (1) including notice on the careers or jobs section of its website in a clear and conspicuous manner (or providing written notice to existing employees) at least 10 business days prior to the use of an AEDT, (2) including notice in a job posting at least 10 business days prior to the use of an AEDT, or (3) providing notice to candidates for employment via U.S. mail or e-mail (or in person for current employees), at least 10 business days prior to the use of an AEDT.
The notice must also include instructions for how to request an alternative selection process or accommodation, although interestingly the proposed regulations do not require the provision of an alternative selection process or accommodation.
Additionally, the proposed regulations would require employers and employment agencies to either: (1) provide notice as to the sources and types of data used by the AEDT, as well as the applicable data-retention policy on their website(s); or (2) provide instructions on how to submit written requests for that information (and respond to those requests within 30 days).
Proposed Bias Audit Requirements
The proposed regulations discuss two scenarios for bias testing of AEDTs.
Where an AEDT selects individuals to move forward in the hiring process or classifies individuals into groups, the required bias audit must, at a minimum: (1) calculate the selection rate for each category; (2) calculate the “impact ratio” for each category; and (3) where an AEDT classifies individuals into groups, the calculations must be performed for each classification.
Where an AEDT scores applicants or candidates, the required bias audit must, at a minimum: (1) calculate the average score for individuals in each category, and (2) calculate the “impact ratio” for each category.
In this context, “impact ratio” means either (1) the selection rate for a category divided by the selection rate of the most selected category, or (2) the average score of all individuals in a category divided by the average score of individuals in the highest scoring category.
Proposed Bias Audit Publication Requirements
Finally, the proposed regulations would also require employers and employment agencies to publish the most recent bias audit results on the entity’s website in a conspicuous manner, and keep the results published for at least six months from the last use of the AEDT.
