Ethan Shen and Gabble On are "harnessing the wisdom of he crowds" to compare and evaluate the quality of three machine translation engines: Google, Bing (Microsoft), and Babelfish (Systran). As an incentive, participants will have the chance to win a new Apple iPad.
The system is very simple. You can participate and start comparing at http://www.gabble-on.com/. Participants simply enter in text, view the three resulting translations, and rank them accordingly. Check it out and rank translations in any combination of the following languages:
- Arabic
- Bulgarian
- Chinese Simplified
- Chinese Traditional
- Czech
- Danish
- Dutch
- English
- Finnish
- French
- German
- Greek
- Haitian Creole
- Hebrew
- Italian
- Japanese
- Korean
- Polish
- Portuguese
- Russian
- Spanish
- Swedish
- Thai
It would be interesting to learn the following from this research project:
- Which language pairs will draw the most participation?
- What will be the self-declared fluency of the evaluators?
- What subject matter will be entered for translation comparison? Or will that even be included in the results? This could be an important element as SMT engines and RBMT engines will each fare better with different subject matter and different types of language. It appears an additional field to identify subject matter was sacrificed for the sake of simplicity.
- Will the prize (and possibly the novelty) motivate enough crowd participation to provide useful evaluation data?
- And, of course, which engines will fare the best in each language pair?
It will also be nice if the results take into account the instances in which engines produced equal results. Currently, the survey only allows for the selection of a single "best translation" and does not allow for a "tie," so the first engine listed (Google) will naturally be favored to win.
0 comments:
Post a Comment