iwslt2012

Skip to content

TED Task

IWSLT proposes challenging research tasks and an open experimental infrastructure for the scientific community working on spoken and written language translation. The IWSLT 2012 Evaluation Campaign includes the TED Task, that is the translation of TED Talks, a collection of public speeches on a variety of topics. Three tracks are proposed addressing different research tasks:

ASR track : automatic transcription of talks from audio to text (in English)
SLT track: speech translation of talks from audio (or ASR output) to text (from English to French)
MT track : text translation of talks for two language pairs plus ten optional language pairs:

official: from English to French and from Arabic to English
optional: from German, Dutch, Polish, Portoguese-Brazil, Romanian, Russian, Slovak, Slovenian, Turkish and Chinese to English

Main challenges of the proposed tracks are:

Open domain ASR, clean transcription of spontaneous speech, detection and removal of non-words, and talk style and topic adaptation.
Open domain SLT, translation of speech or ASR output into true-case punctuated text, and talk style and topic adaptation.
Open domain MT between distant languages, and talk style and topic adaptation.

Training of MT systems and language models for ASR is constrained to data supplied by the organizers. As for ASR acoustic modeling no training data are distributed, participants are allowed to use any publicly available data recorded before 31 December 2010.

ASR Track (English)

Input format: pre-segmented SPHERE files
Output format: CTM (description and example here, at section ”.ctm”), no case, no punctuation
Coding: UTF-8
Submission guidelines
Evaluation: WER
Training data for language modeling:
- in-domain: monolingual English text
- out-of-domain: supplied monolingual texts
Development data:

dev2010:

test2010:

Scoring Package
Test data:

tst2011, tst2012:

audio + sentence segmentation, 273MB

SLT Track (English to French)

Input format: pre-segmented SPHERE files or supplied ASR outputs (transcript, SLF lattice)
Output format: NIST XML format, true case with punctuation (example)
Coding: UTF-8
Submission guidelines
Evaluation: BLEU and subjective ranking
Training data:

in-domain: from the WIT³ website
out-of-domain: supplied monolingual and parallel texts

Development data:

dev2010:

test2010:

Test data:

tst2011, tst2012:

automatic transcripts, 1.8MB

MT Track (English to French, Arabic to English + 10 additional language pairs)

Input format: NIST XML format, true case with punctuation
Output format: NIST XML format, true case with punctuation (example)
Coding: UTF-8
Submission guidelines
Evaluation: BLEU and subjective ranking
Training data:

in-domain: from the WIT³ website
out-of-domain: supplied monolingual and parallel texts

Development data: from the WIT³ website
Test data: from the WIT³ website

Test References

The references of tst2011 set for ASR, SLT and MT tracks are available to participants here.

Registration

To participate in one or more tracks please fill in the Registration Form and send it by e-mail to iwslt2012.ted AT gmail DOT com

Participants are requested to submit a system paper describing their work and to present it at the workshop.

Acknowledgments

TED talks data are copyright of TED Conference LLC and distributed under the Creative Commons Attribution-NonCommercial- NoDerivs 3.0 license
Google Books ngrams are copyright of Google Inc. and distributed under a Creative Commons Attribution 3.0 license
WMT12 data are kindly supplied by the NAACL 2012 7th Workshop on Statistical Machine Translation
MultiUN data are kindly supplied by the EuroMatrixPlus project

Supplied out-of-domain monolingual and parallel training data