Interaction Patterns in Computer-assisted Semantic Annotation of Text - An Empirical Evaluation

This paper examines user interface options and interaction patterns evinced in tools for computer-assisted semantic enrichment of text. It focuses on advanced annotation tasks such as hierarchical annotation of complex relations and linking entities with highly ambiguous names and explores how decisions on particular aspects of annotation interfaces influence the speed and the quality of computer-assisted human annotation processes. Reported experiments compare the 4A annotation system, designed and implemented by our team, to RDFaCE and GATE tools that all provide advanced annotation functionality. Results show that users are able to reach better consistency of event annotations in less time when using the 4A editor. A set of experiments is then conducted that employ 4A’s high flexibility and customizability to find an optimal amount of displayed information and its presentation form to reach best results in linking entities with highly ambiguous names. The last set of experiments then proves that 4A’s particular way of implementing the concept of semantic filtering speeds up event annotation processes and brings higher consistency when compared to alternative approaches.