Optimizing Computer-Assisted Transcription Quality with Iterative User Interfaces

Publication
Language Resources and Evaluation (LREC)

The results of our user study are available in the above dataset link. It contains the following fields:

  • personId
  • groupId (LING, CS, CROWD)
  • talkId
  • inputModeId (m.1: traditional from-scratch, m.2: traditional post-edit, m.3: retyping, m.4: confidence-enhanced post-edit)
  • postEdit (FS: from-scratch, PE: post-edit)
  • aided (ASRCONF feature, ON vs. OFF)
  • segId (segment id)
  • asrWER (ASR word error rate in %)
  • asrErrors (number of ASR errors)
  • refLen (reference number of words)
  • asrContentErrors (proportion of ASR errors that were not found in a list of function words)
  • asrPER (ASR phoneme error rate%)
  • asrPErrors (ASR number of phoneme errors)
  • phoneRefLen (number of phonemes in references)
  • corrWER (word error rate of correction in %)
  • corrErrors (number of errors in correction)
  • corrContentErrors (proportion of correction errors that were not found in a list of function words)
  • corrPER (phoneme error rate of correction in %)
  • corrPErrors (number of phoneme errors in correction)
  • asrCorrEditRate (edit rate between ASR and correction in %)
  • asrCorrEdits (number of edits between ASR and correction)
  • asrLen (number of words in ASR output)
  • contentEdits (proportion of edits that were not found in a list of function words)
  • asrCorrPEditRate (phoneme edit rate between ASR and correction in %)
  • asrCorrPEdits (number of edited phonemes between ASR and correction)
  • asrPLen (number of phonemes in ASR output)
  • corrLen (number of words in correction)
  • numWordsTyped (for postEdit==PE: equal to asrCorrEdits, for postEdit==FS: equal to corrLen)
  • secondsEdited (seconds spent editing this segment)
  • numReviews (number of times the “review/repeat audio” function was used for this segment)
  • audioDuration (audio duration of the segment in seconds)
  • segConf (average word confidence of the segment)

A demo setup of the four user interface approaches is available for download here (Java, works under Mac and Windows). The full version (free for academic purposes) can be obtained here.