Back to toolkit

How to Use Part A

Part A: Organization, Implementation and Management of the AI Solutions Interpreting Evaluation Toolkit provides a comprehensive, risk-informed approach for evaluating AI and hybrid AI-human language access solutions. It is designed to be a practical resource for both strategic planning and solution implementation. The checklists included in this PART A provide useful evaluation questions relevant for all languages and language combinations.

Other publications in the AI Interpreting Solutions Evaluation Toolkit series include:

  • PART B: Technical Specifications - A detailed description of User Experience / User Interface (UX/UI) controls and language model baselines, including metrics related to AI and hybrid language solutions.
  • PART C: Legalities and Practical Considerations - A detailed consideration of the contemporary legal landscape and practical considerations for early adoption of AI and hybrid language solutions.

PART A of this toolkit series has two sections: a Toolkit overview and the Toolkit checklists.

Comprehensive Overview (The "Why" of a Risk-Informed Approach)

The Overview for Part A explains the foundational approach and framework for this resource. It provides the "why" behind both the structure of the evaluation checklists and the emphasis on risk management, which are grounded in the following core principles:

  • SAFE: Ensure that solutions will not harm primary communicators and use robust safeguards for all encounters.
  • ACCOUNTABLE: Establish clear and transparent lines of responsibility for the technology's performance and impact. Patterns of user feedback and performance monitoring are publicly disclosed, including changes stemming from these metrics, on a regular, frequent basis.
  • FAIR: Promote autonomy for all primary communicators and actively work to reduce algorithmic bias to achieve the integrity of communication.
  • ETHICAL: Guarantee transparency about safety, accountability, fairness, and user consent (opt-out / prefer human controls) when integrating AI interpreting.

This Toolkit draws attention to known limitations in current AI interpreting technology which:

  • Functions optimally only under controlled conditions.
  • Requires adequate audio and visual quality for all participants.
  • Cannot replicate human interpreters' cultural competency, contextual understanding, and problem-solving abilities.
  • Performs inconsistently across different language pairs (the combination of languages used in the interpreting interaction, such as ASL and English, or English and Vietnamese).
  • May create access barriers for languages with minimal training data, which are called "low resource languages."

Toolkit Checklists: Implementation (The "How")

When you are ready to begin the evaluation process, proceed to the practical tools located in the Checklists section, which are organized in a three-step sequence:

Step 1: Assess Your Organization's Readiness

Use CHECKLIST 1: Organizational Readiness Evaluation to determine if your organization is prepared to implement and manage this technology responsibly.

Use CHECKLIST 2: Setting-Specific Considerations for AI Interpreting Implementation as appropriate to your setting and evaluation needs.

Step 2: Evaluate Risk Factors to Determine Risk Levels

Use CHECKLIST 3: Risk Factor Assessment Framework for AI Interpreting Solutions to categorize your specific use cases and determine which encounters are broadly appropriate for AI versus a human interpreter, prioritizing the ability for primary communicators to opt out of AI interpreting and switch to a human solution, if preferred.

Step 3: Select and Evaluate Vendors

Use CHECKLIST 4: Vendor Assessment Checklist to systematically evaluate risk levels posed by potential AI solutions against your specific requirements and the SAFE principles.

Use CHECKLIST 5: Guidance for Request for Proposals (RFPs) that Include AI Interpreting if you are writing RFPs for interpreting services that include AI interpreting tools.

Match the AI Interpreting Solution to the Task

Before selecting an AI interpreting solution, it is essential to understand the dynamics of using it in the real world. A successful implementation balances the benefits of technology with the needs of the primary communicators. To do this, consider the following:

  • Align Technology with the Environment. AI interpreting solutions perform best in controlled settings. They are most suited for simple, transactional tasks in highly constrained, predictable contexts.
  • Reserve Less Predictable and Nuanced Tasks for Human Interpreters. Situations involving measurable risk require cultural competency, ethical judgment, empathy and facilitation from human interpreters.
  • Validate Performance for Your Specific Communities. Always pilot an AI interpreting solution with your primary communicators, especially in non-English languages, before implementing to ensure it can be reasonably expected to produce understanding among principal communicators.
  • Prioritize Effective Understanding among Principal Communicators. Ensure that pathways to a human interpreter remain visible and easy to enact, so that technology does not become a barrier for any community member.