id	author	title	date	pages	extension	mime	words	sentences	flesch	summary	cache	txt
work_w36sbsa2pjaoxdslwl2ehwtmba	Adam Lopez	Learning to translate with products of novices: a suite of open-ended challenge problems for teaching MT	2013	13	.pdf	application/pdf	7604	827	67	For each problem we provided data, a naı̈ve solution, and an evaluation program. Three of the four challenges also included unlabeled test data (except the decoding assignment, as We provided a web-based leaderboard that displayed standings on the test data in real time, identifying each submission by a pseudonymous handle known only to the team and instructors. The third challenge was evaluation: given a test corpus with reference translations and the output of several MT systems, students were challenged to produce a ranking of the systems that closely correlated We chose the English-to-German translation systems from the 2009 and 2011 shared task at the annual Workshop for Machine Translation (CallisonBurch et al., 2009; Callison-Burch et al., 2011), providing the first as development data and the second We conceived of the assignment as one in which students could apply machine learning or feature engineering to the task of reranking the systems, so we	./cache/work_w36sbsa2pjaoxdslwl2ehwtmba.pdf	./txt/work_w36sbsa2pjaoxdslwl2ehwtmba.txt