Please note that the details of this Challenge are no longer open. This challenge is awarded and is no longer accepting new submissions. You can:

ASpIRE – IARPA Automatic Speech recognition in Reverberant Environments Challenge

STATUS: Awarded
ACTIVE SOLVERS: 167
POSTED: Nov 17 2014
SOURCE: InnoCentive

The development of automatic speech recognition able to perform well across a variety of acoustic environments and recording scenarios on natural conversational speech represents one of the biggest challenges in speech recognition research and development. Previous work in the literature has shown that automatic speech recognition (ASR) performance degrades in microphone recordings especially when data used for training is mismatched with data used in testing. The Intelligence Advanced Research Projects Activity (IARPA) is seeking to identify approaches to mitigate the effects of this mismatch by running this Automatic Speech recognition in Reverberant Environments (ASpIRE) Challenge. 

This is a Reduction-to-Practice Challenge that requires written documentation and delivery of output from the Solver’s automatic speech recognition system applied to supplied evaluation data. The Seeker does not wish to obtain IP transfer or licensing of solutions and seeks only to identify the leading systems and Solvers in this field. Additionally, as a Prodigy Challenge a real-time online scoring utility and leaderboard will be available to track Solver performance for this Challenge. 

Evaluation will be performed with both single microphone and multiple microphone data. There will be separate monetary awards given to the best system in the single-microphone ($30,000) and the multi-microphone ($20,000) conditions. The winner in each condition must achieve a word error rate (WER) that is at least 1% lower than the performance levels attained by the second best system to win.

Challenge ID: 9933624
 
Share Challenge Share
Challenge Overview

The IARPA ASpIRE (Automatic Speech recognition in Reverberant Environments) Challenge, a spin-off of the IARPA Babel program, seeks to identify approaches and technology for automatic speech recognition (ASR) under less than ideal conditions. Previous work has shown that ASR performance degrades in microphone recordings, especially when data used for training is mismatched with data used in testing.

The ASpIRE challenge asks solvers to develop innovative speech recognition systems that can be trained on conversational telephone speech, and yet work well on far-field microphone data from noisy, reverberant rooms. Participants will have the opportunity to evaluate their techniques on a common set of challenging data that includes significant room noise and reverberation. Whereas the Babel program seeks to develop agile and robust technology that can be rapidly applied to any human language, this Challenge focuses on English language speech recognition. 

The Seeker (IARPA) asks Challenge Solvers to create innovative automatic speech recognition software that works in a variety of acoustic environments and recording scenarios without having access to matched training and development data.  There are two evaluation conditions:

  1. The Single Microphone (single-mic) Condition tests the ability to mitigate noise and reverberation given a single microphone recording (selected randomly) from speech recorded in several rooms with a variety of microphones. Single-mic evaluation data will be made available at 10:00 A.M. Eastern Standard Time 11-Feb-2015 and submissions must be received by 11:59 P.M. Eastern Standard Time 18-Feb-2015 to be eligible for award.
  2. The Multiple Microphone (multi-mic) Condition tests the ability to mitigate noise and reverberation given all of the microphone recordings of speech recorded in several rooms with a variety of microphones. Multi-mic evaluation data will be made available at 10:00 A.M. Eastern Standard Time 19-Feb-2015 and submissions must be received by 11:59 P.M. Eastern Standard Time 26-Feb-2015 to be eligible for award.

In both conditions, word error rate (WER) will be used as the objective measure of performance. Solvers can participate in either or both conditions. There will be separate monetary awards given for the best system in the single microphone ($30,000) and the multiple microphone ($20,000) conditions. The winner in each condition must achieve a WER that is at least 1% lower than the performance levels attained by the second best system to win. 

Separate submissions are required for the single microphone and multiple microphone conditions and each must include the following: 

  1. A detailed description of the system and how it addresses each Technical Requirement presented in the Detailed Description of the Challenge. The project documentation should include a well-articulated rationale for the methods adopted. See the Evaluation Plan (in Details) for more information on what should be included.
  2. System Output from the automatic speech recognition system applied to either the single or the multiple microphone evaluation sets.

If you are selected for validation in either condition, you would also be required to either:

  1. Provide a Virtual Machine implementing the algorithm. Include all dependencies, packages, databases, documentation, and information to generate test results by the Seeker,

    OR

  2. Schedule an appointment with the Seeker’s representative to validate your solution on site.

The award is contingent upon validation of the submitted Solutions by the Seeker or their representative. To receive an award, Solvers will not have to transfer their IP rights or grant a license to the Seeker – the purpose of the Challenge is to gauge how far recent advances in speech recognition have come in solving this important problem. With broad participation, this Challenge has the potential to provide IARPA with insights on the best next steps to stimulate research for solving this challenging problem.

By making a submission to this Challenge, Solver(s) are providing written consent to the Seeker to publicly disclose their identity if their submission is chosen for an award and they choose to accept the award. Disclosure will not occur until the Challenge has been fully completed, that is, all submissions have been evaluated, rejected or accepted, and any awards have been transferred to the Solver(s).

Eligibility

Federal entities or Federal employees acting within the scope of their employment are NOT eligible to participate in this Challenge. Please note that winners will have to submit an Academic Institution Acknowledgement Letter acknowledging the role of IARPA in this Challenge if you are:  (i) a U.S. Academic Institution, (ii) an employee of such institution who is participating on behalf of that institution, or (iii) an employee of such institution who is participating in their personal capacity if they are using the resources of such institution to respond to this Challenge. Entities or employees of entities from the following countries subject to U.S. economic sanctions are not eligible to participate in this Challenge: Iran, Syria, Sudan, Cuba, and North Korea.  In addition, individuals and entities listed on the U.S. Government’s Consolidated Screening List (available at http://export.gov/ecr/eg_main_023148.asp) are not eligible to participate in this Challenge. 

About the Seeker

This Challenge is sponsored by the Intelligence Advanced Research Projects Activity (IARPA). IARPA invests in high-risk, high-payoff research programs that have the potential to provide the United States with an overwhelming intelligence advantage over future adversaries. Opinions, interpretations, recommendations and conclusions are those of the authors and are not necessarily endorsed by the United States Government.

What is InnoCentive?

InnoCentive is the global innovation marketplace where creative minds solve some of the world's most important problems for cash awards up to $1 million. Commercial, governmental and humanitarian organizations engage with InnoCentive to solve problems that can impact humankind in areas ranging from the environment to medical advancements.

What is an RTP Challenge?

An InnoCentive RTP (Reduction to Practice) Challenge is a prototype that proves an idea, and is similar to an InnoCentive Theoretical Challenge in its high level of detail. However, an RTP requires the Solver to submit a validated solution, either in the form of original data or a physical sample. Also the Seeker is allowed to test the proposed solution. For details about treatment of IP rights, please see the Challenge-Specific Agreement.

Share This Challenge
InnoCentive Trust Partners