Selected References

Most of the references I give here are for natural language and speech processing.

General / Misc

• A nice introduction is this online book: Sinclair, J. Wynne, M. (ed.) Developing Linguistic Corpora: a Guide to Good Practice Corpus and Text - Basic Principles Oxford: Oxbow Books, 2005, 1-16 (http://ahds.ac.uk/linguistic-corpora/)


• The LDC huge catalogue (too huge?): http://www.ldc.upenn.edu/annotation/


• There is an ACL workshop dedicated to the subject, the Linguistic Annotation Workshop (LAW):
  
• 2009: http://www.ling.uni-potsdam.de/acl-lab/LAW-09.html
  
• CFP for 2010: http://www.cs.brandeis.edu/~clp/LAW4/The_LAW_IV.html 
• The proceedings are worth having a look at: http://www.aclweb.org/anthology/W/W09/#3000

Corpus creation

• An overview by B. Habert: Habert, B. (2001). Des corpus représentatifs: de quoi, pour quoi, comment ? In M. Bilger (ed.), Linguistique sur corpus. Études et réflexions, Presses Universitaires de Perpignan, 11-58. (http://www.limsi.fr/Individu/habert/Projets/ATALA280401/Habert/Perpignan00/index.html)
  

• Another, more speech-oriented, by Olivier Baude: Baude, O. 2007. Contribution des corpus oraux à  la linguistique de corpus: une démarche réflexive intégrée, Communication aux Journées de Linguistique de Corpus (Lorient).
  

• The reference article by Sinclair (EAGLES): Sinclair, J. (1996). Preliminary recommendations on Corpus Typology. Technical report, EAGLES (Expert Advisory Group on Language Engineering Standards). (http://www.ilc.cnr.it/EAGLES/pub/eagles/corpora/corpustyp.ps.gz)

Annotation formats/schemes, standards

• This overview by Nancy Ide is a good starting point: Ide, N. Annotation Science: From Theory to Practice and Use. (Invited Talk) Data Structures for Linguistics Resources and Applications Proceedings of the Bienniel GLDV Conference, 2007 (http://www.cs.vassar.edu/~ide/papers/GLDV.pdf)
  

• Annotation graphs: Bird, S. & Liberman, M. A Formal Framework for Linguistic Annotation (revised version) CoRR?, 2000, cs.CL/0010033, pp 23-60 (http://arxiv.org/abs/cs/0010033)
  

• Linguistic Annotation Framework (ISO TC 37/SC4): Ide, N. Linguistic annotation framework ISO TC37/SC4/WG1 N11 - TC37 SC4 Language Resource Management, 08., 2002 (http://www.tc37sc4.org/new_doc/ISO_TC_37-4_N029_Linguistic_annotation_framework.pdf)

Methodology

• An interesting article giving results with various methodologies (pre-annotation, use of tool, training, etc): Dandapat, S.; Biswas, P.; Choudhury, M. & Bali, K. Complex Linguistic Annotation - No Easy Way Out! A Case from Bangla and Hindi POS Labeling Tasks Proceedings of the third ACL Linguistic Annotation Workshop, 2009 (http://www.aclweb.org/anthology/W/W09/W09-3002.pdf)
  

• The MEDIA corpus methodology: Bonneau-Maynard, H.; Rosset, S.; Ayache, C.; Kuhn, A. & Mostefa, D. Semantic Annotation of the French Media Dialog Corpus InterSpeech?, 2005 (ftp://tlp.limsi.fr/public/IS052010.PDF)
  

• An article we wrote last year, focusing on the importance of the guidelines: Fort, K.; Ehrmann, M. & Nazarenko, A. Towards a Methodology for Named Entities Annotation Proceeding of the 3rd ACL Linguistic Annotation Workshop (LAW III), 2009 (http://www.aclweb.org/anthology/W/W09/W09-3025.pdf)
  

• Guidelines, again: Nédellec C., Bessières P., Bossy R., Kotoujansky A., Manine A.-P., Annotation Guidelines for Machine Learning-Based Named Entity Recognition in Microbiology, In Proceedings of the Data and text mining in integrative biology workshop, joint to ECML/PKDD, M. Hilario et C. Nedellec (Eds), p. 40-54, Berlin, Germany, september 2006. (http://www.ecmlpkdd2006.org/ws-dtib.pdf)

• Annotating using ontologies: Cimiano, P. & Handschuh, S. Ontology-based linguistic annotation Proceedings of the ACL 2003 workshop on Linguistic annotation, Association for Computational Linguistics, 2003, 14-21 (http://dx.doi.org/10.3115/1119296.1119299)

Evaluation of manual annotation

• Why even bother calculating inter and intra annotator agreement? Gut, U. & Bayerl, P. S. Measuring the Reliability of Manual Annotations of Speech Corpora Proceedings of Speech Prosody , 2004, 565-568 (http://www.uni-giessen.de/germanistik/ascl/dfg-projekt/pdfs/SP2004_GutBayerl03.pdf)
  

• The reference article on inter-annotator agreement: Artstein, R. & Poesio, M. Inter-coder agreement for computational linguistics Computational Linguistics, MIT Press, 2008, 34, 555-596 (http://dx.doi.org/10.1162/coli.07-034-R2)

Annotation tools

In theory, there are many annotation tools. In practice, very few are really available and usable.

• Transcriber: Annotation (transcription) tool for speech and more generally audio and video annotation. Freely available. http://trans.sourceforge.net/
  
• GATE: well-known annotation tool, designed for automatic annotation. Manual annotation is possible, with interesting features (annotate all, for example), but difficult to get into. Freely available. http://gate.ac.uk/
  
• GLOZZ: a new tool, designed for discourse annotation, but usable for any kind of annotation, including with (complex) relations. Doc only in videos. Freely available for research. http://www.glozz.org/
  
• Knowtator (Protégé plugin): allows to annotate using ontologies. Freely available. http://knowtator.sourceforge.net/index.shtml
  
• Callisto: esp. for time annotations (does not allow for the annotation of other relations). Freely available. http://callisto.mitre.org/
  
• MMAX2: interesting tool, allowing for the annotation of relations, but UI difficult to use. Freely available. http://mmax2.sourceforge.net/
  
• Advene: a Python tool dedicated to video annotation. Many functionalities, easy to use, ergonomic. Freely available. http://liris.cnrs.fr/advene/

Last modified :  Friday, January 6th, 2017