D3206 – Ontology Engineering Group, Universidad Politécnica de Madrid, Montegancedo Campus, Madrid, Spain e-mails: mnavas@fi.upm.es, vrodriguez@fi.upm.es"/> Institute for Information Business, Vienna University of Economics and Business, Vienna, Austria e-mails: Erwin.Filtz@wu.ac.at, Axel.Polleres@wu.ac.at, Sabrina.Kirrane@wu.ac.at"/>
Search
  • All Journals
Advanced Search
Search articles by Title, Author, Keywords, DOI
Submit Manuscript
7.7CiteScore
2.0Impact Factor
Advanced Search
MAP  >  The Knowledge Engineering Review
2019 Volume 34
Article Contents
Next Previous
RESEARCH ARTICLE   Open Access    

TempCourt: evaluation of temporal taggers on a new corpus of court decisions

More Information
  • Abstract: The extraction and processing of temporal expressions (TEs) in textual documents have been extensively studied in several domains; however, for the legal domain it remains an open challenge. This is possibly due to the scarcity of corpora in the domain and the particularities found in legal documents that are highlighted in this paper. Considering the pivotal role played by temporal information when it comes to analyzing legal cases, this paper presents TempCourt, a corpus of 30 legal documents from the European Court of Human Rights, the European Court of Justice, and the United States Supreme Court with manually annotated TEs. The corpus contains two different temporal annotation sets that adhere to the TimeML standard, the first one capturing all TEs and the second dedicated to TEs that are relevant for the case under judgment (thus excluding dates of previous court decisions). The proposed gold standards are subsequently used to compare ten state-of-the-art cross-domain temporal taggers, and to identify not only the limitations of cross-domain temporal taggers but also limitations of the TimeML standard when applied to legal documents. Finally, the paper identifies the need for dedicated resources and the adaptation of existing tools, and specific annotation guidelines that can be adapted to different types of legal documents.
  • 加载中
  • Bethard , S.2013. ClearTK-TimeML: a minimalist approach to TempEval 2013. In Proceedings of the Workshop SemEval 2013, 10–14. ACL.

    Google Scholar

    Chambers , N.2013. NavyTime: event and time ordering from raw text. In Proceedings of the Workshop SemEval 2013, 2, 73–77.

    Google Scholar

    Chambers , N., Cassidy , T., McDowell , B. & Bethard , S.2014. Dense event ordering with a multi-pass architecture. Transactions of the ACL, 2, 273–284.

    Google Scholar

    Chang , A. X. & Manning , C. D. 2012. SUTime: a library for recognizing and normalizing time expressions. In Proceedings of LREC 2012.

    Google Scholar

    Chang , A. X., et al. 2014. TokensRegex: Defining Cascaded Regular Expressions Over Tokens. Technical report CSTR 2014-02, Department of Computer Science, Stanford University.

    Google Scholar

    Cohen , J.1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement20(1), 37–46.

    Google Scholar

    Cunningham , H., Tablan , V., Roberts , A. & Bontcheva , K.2013. Getting more out of biomedical documents with GATE’s full lifecycle open source text analytics. PLOS Computational Biology9(2), 1–16.

    Google Scholar

    Dell’Orletta , F., Marchi , S., Montemagni , S., Plank , B. & Venturi , G.2012. The splet–2012 shared task on dependency parsing of legal texts. In Semantic Processing of Legal Texts (SPLeT-2012) Workshop Programme, 42.

    Google Scholar

    Derczynski , L. & Gaizauskas , R.2010. USFD2: annotating temporal expressions and TLINKs for TempEval-2. In Proceedings of the Workshop SemEval, 337–340. ACL.

    Google Scholar

    Ferro , L., Gerber , L., Mani , I., Sundheim , B. & Wilson , G.2005. Tides 2005 Standard for the Annotation of Temporal Expressions. Technical report, MITRE.

    Google Scholar

    Filannino , M. & Nenadic , G.2015. Temporal expression extraction with extensive feature type selection and a posteriori label adjustment. Data & Knowledge Engineering100, 19–33. ISSN 0169-023X. 10.1016/j.datak.2015.09.002. http://www.sciencedirect.com/science/article/pii/S0169023X15000725. doi: 10.1016/j.datak.2015.09.002

    CrossRef   Google Scholar

    Freund , Y., Schapire , R. & Abe , N.1999. A short introduction to boosting. Journal-Japanese Society For Artificial Intelligence14(771–780), 1612.

    Google Scholar

    Gildea , D. & Jurafsky , D.2002. Automatic labeling of semantic roles. Computational Linguistics28(3), 245–288. 10.1162/089120102760275983. https://doi.org/10.1162/089120102760275983. doi: 10.1162/089120102760275983

    CrossRef   Google Scholar

    Han , B., Gates , D. & Levin , L.2006. From language to time: a temporal expression anchorer. In Thirteenth International Symposium on Temporal Representation and Reasoning, 2006, TIME 2006, 196–203. IEEE.

    Google Scholar

    Hellmann , S.Lehmann , J. & Auer , S.2012. NIF: an ontology-based and linked-data-aware NLP Interchange Format. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.232.1215&rep=rep1&type=pdf

    Google Scholar

    Isemann , D., Ahmad , K., Fernando , T. & Vogel , C.2013. Temporal Dependence in Legal Documents, 497–504. Springer. ISBN 978-3-642-41278-3.

    Google Scholar

    Ji , H., Cassidy , T., Li , Q. & Tamang , S.2014. Tackling representation, annotation and classification challenges for temporal knowledge base population. Knowledge and Information Systems41(3), 611–646. ISSN 0219-3116. 10.1007/s10115-013-0675-1. https://doi.org/10.1007/s10115-013-0675-1. doi: 10.1007/s10115-013-0675-1

    CrossRef   Google Scholar

    Kajiwara , T. & Komachi , M.2016. Building a monolingual parallel corpus for text simplification using sentence similarity based on alignment between word embeddings. In Proceedings of COLING 2016: Technical Papers, 1147–1158.

    Google Scholar

    Kolya , A. K., Kundu , A., Gupta , R., Ekbal , A. & Bandyopadhyay , S.2013. JU_CSE: a CRF based approach to annotation of temporal expression, event and temporal relations. In Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), 2, 64–72.

    Google Scholar

    Lafferty , J. D., McCallum , A. & Pereira , F. C. N.2001. Conditional random fields: probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001),Williams College, Williamstown, MA, USA, June 28–July 1, 2001, Brodley , C. E. & Danyluk , A. P. (eds). Morgan Kaufmann, 282–289. ISBN 1-55860-778-1.

    Google Scholar

    Lee , K., Artzi , Y., Dodge , J., & Zettlemoyer , L.2014. Context-dependent semantic parsing for time expressions. In Proceedings of the 52nd Annual Meeting of the ACL,1, 1437–1447.

    Google Scholar

    Llorens , H., Saquete , E. & Navarro , B.2010. TIPSem (english and spanish): evaluating CRFs and semantic roles inTempEval-2. In Proceedings of the Workshop SemEval, 284–291. ACL.

    Google Scholar

    Llorens , H., Derczynski , L., Gaizauskas , R. J. & Saquete , E.2012. TIMEN: an open temporal expression normalisation resource. In Proceedings of LREC 2012, 3044–3051.

    Google Scholar

    Loper , E. & Bird , S.2002. NLTK: The Natural Language Toolkit. CoRR, cs.CL/0205028. http://arxiv.org/abs/cs.CL/0205028.

    Google Scholar

    Mani , I. & Wilson , G.2000. Robust temporal processing of news. In Proceedings of the 38th Annual Meeting on ACL, 69–76. ACL.

    Google Scholar

    Manning , C., Surdeanu , M., Bauer , J., Finkel , J., Bethard , S. & McClosky , D.2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Proceedings of the 52nd Annual Meeting of the ACL 2014, System Demonstrations, 55–60.

    Google Scholar

    Mazur , P. & Dale , R.2009. The dante temporal expression tagger. In Human Language Technology. Challenges of the Information Society, Vetulani , Z. & Uszkoreit , H. (eds). 245–257. Springer. ISBN 978-3-642-04235-5.

    Google Scholar

    Mazur , P. & Dale , R.2010. Wikiwars: a new corpus for research on temporal expressions. In Proceedings of EMNLP 2010, 913–922. ACL.

    Google Scholar

    Minard , A.-L., Speranza , M., Urizar , R., Altuna , B., Van Erp , M., Schoen , A. & Van Son , C.2016. Meantime, the newsreader multilingual event and time corpus. In Proceedings of LREC 2016, European Language Resources Association.

    Google Scholar

    Naik , V., Vanitha , G. & Inturi , S.2011. Reasoning in legal text documents with extracted event information. International Journal of Computer Applications28, 8–-13.

    Google Scholar

    Nivre , J., et al. 2007. The CoNLL 2007 shared task on dependency parsing. In Proceedings of EMNLP-CoNLL 2007.

    Google Scholar

    Northwood , C.2010. TERNIP: Temporal Expression Recognition and Normalisation in Python. Master’s thesis, University of Sheffield.

    Google Scholar

    Pellow , D. & Eskenazi , M.2014. An open corpus of everyday documents for simplification tasks. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR), 84–93.

    Google Scholar

    Pustejovsky , J., Hanks , P., Sauri , R., See , A., Gaizauskas , R., Setzer , A., Radev , D., Sundheim , B., Day , D., Ferro , L. & Lazo , M.2003a. The TimeBank corpus. In Corpus Linguistics, Lancaster University (UK), 2003, 28–31.

    Google Scholar

    Pustejovsky , J., Castano , J. M., Ingria , R., Sauri , R., Gaizauskas , R. J., Setzer , A.Katz , G. & Radev , D. R.2003bb.TimeML: robust specification of event and temporal expressions intext. In New Directions in Question Answering, Papers from 2003 AAAI Spring Symposium, Maybury , M. T. (ed). AAAI Press, 28–34. ISBN 1-57735-184-3.

    Google Scholar

    Quinlan , J. R.1993. C4.5: Programs for Machine Learning. Morgan Kaufmann. ISBN 1-55860-238-0.

    Google Scholar

    Rijsbergen , C. J. V.1979. Information Retrieval, 2nd edition. Butterworth-Heinemann. ISBN 0408709294.

    Google Scholar

    Saurí, R., Littman , J., Knippen , B., Gaizauskas , R., Setzer , A. &Pustejovsky , J. TimeML Annotation Guidelines. Version 1.2.1, 2006.

    Google Scholar

    Schilder , F.2005. Event extraction and temporal reasoning in legal documents. In Annotating, Extracting and Reasoning about Time and Events,International Seminar, Dagstuhl Castle. Revised Papers, 59–71.

    Google Scholar

    Scott , W. A.1955. Reliability of content analysis: the case of nominal scale coding. Public Opinion Quarterly, 19(3), 321–325.

    Google Scholar

    Steedman , M.Jason , B.2011. Combinatory categorial grammar, In Non-Transformational Syntax: Formal and Explicit Models of Grammar, Borsley , R. & Borjars , K. (eds). Wiley-Blackwell, 181–224.

    Google Scholar

    Strötgen , J. & Gertz , M.2016. Domain-sensitive temporal tagging. Synthesis Lectures on Human Language Technologies9(3), 1–151.

    Google Scholar

    Strötgen , J. & Gertz , M.2012. Temporal tagging on different domains: challenges, strategies, and gold standards. In Proceedings of LREC 2012, 12, 3746–3753.

    Google Scholar

    Styler IV , W. F., Bethard , S., Finan , S., Palmer , M., Pradhan , S., de Groen , P. C., Erickson , B., Miller , T., Lin , C., Savova , G., et al. 2014. Temporal annotation in the clinical domain. Transactions of the Association of Computational Linguistics2(1), 143–154.

    Google Scholar

    Tabassum , J., Ritter , A. & Xu , W. TweeTime: a minimally supervised method for recognizing and normalizing time expressions in Twitter. arXiv preprint arXiv:1608.02904, 2016.

    Google Scholar

    UzZaman , N. & Allen , J. F.2010. Event and temporal expression extraction from raw text: first step towards a temporally aware system. International Journal of Semantic Computing4(04), 487–508.

    Google Scholar

    UzZaman , N., Llorens , H., Derczynski , L., Allen , J., Verhagen , M. & Pustejovsky , J.2013. SemEval-2013 Task 1: TempEval-3: evaluating time expressions, events, and temporal relations. In Proceedings of the Workshop SemEval 2013, 1–9.

    Google Scholar

    Verhagen , M., Mani , I., Sauri , R., Littman , J., Knippen , R., Jang , S.B, Rumshisky , A., Phillips , J., & Pustejovsky , J.2005. Automating temporal annotation with TARSQI. In Proceedings of the ACL 2005 on Interactive Poster and Demonstration Sessions, ACLdemo ’05, 81–84. ACL.

    Google Scholar

    Vlek , C. S., Prakken , H., Renooij , S. & Verheij , B.2013. Representing and evaluating legal narratives with subscenarios in a Bayesian network. In OASIcs-OpenAccess Series in Informatics, 32. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.

    Google Scholar

    Zhong , X., Sun , A. & Cambria , E.2017Time expression analysis and recognition using syntactic token types and general heuristic rules. In Proceedings of the 55th Annual Meeting of the ACL, 1, 420–429.

    Google Scholar

  • Cite this article

    María Navas-Loro, Erwin Filtz, Víctor Rodríguez-Doncel, Axel Polleres, Sabrina Kirrane. 2019. TempCourt: evaluation of temporal taggers on a new corpus of court decisions. The Knowledge Engineering Review 34(1), doi: 10.1017/S0269888919000195
    María Navas-Loro, Erwin Filtz, Víctor Rodríguez-Doncel, Axel Polleres, Sabrina Kirrane. 2019. TempCourt: evaluation of temporal taggers on a new corpus of court decisions. The Knowledge Engineering Review 34(1), doi: 10.1017/S0269888919000195
    shu
Download PDF

Article Metrics

Article views(41) PDF downloads(133)

Other Articles By Authors

RESEARCH ARTICLE   Open Access    

TempCourt: evaluation of temporal taggers on a new corpus of court decisions

  • Received: 25 February 2019
  • Revised: 06 November 2019
  • Accepted: 09 November 2019
  • Published online: 17 December 2019
The Knowledge Engineering Review  34,  2019, 34(1)  |  Cite this article

Abstract: Abstract: The extraction and processing of temporal expressions (TEs) in textual documents have been extensively studied in several domains; however, for the legal domain it remains an open challenge. This is possibly due to the scarcity of corpora in the domain and the particularities found in legal documents that are highlighted in this paper. Considering the pivotal role played by temporal information when it comes to analyzing legal cases, this paper presents TempCourt, a corpus of 30 legal documents from the European Court of Human Rights, the European Court of Justice, and the United States Supreme Court with manually annotated TEs. The corpus contains two different temporal annotation sets that adhere to the TimeML standard, the first one capturing all TEs and the second dedicated to TEs that are relevant for the case under judgment (thus excluding dates of previous court decisions). The proposed gold standards are subsequently used to compare ten state-of-the-art cross-domain temporal taggers, and to identify not only the limitations of cross-domain temporal taggers but also limitations of the TimeML standard when applied to legal documents. Finally, the paper identifies the need for dedicated resources and the adaptation of existing tools, and specific annotation guidelines that can be adapted to different types of legal documents.

    HTML

    Acknowledgments

    • This work was partially funded by a Predoctoral grant from the Programa Propio de la Universidad Politécnica de Madrid, and from a grant from Consejo Social de la Universidad Polit’ecnica de Madrid. This work was supported by the Republic of Austria’s Federal Ministry for Digital and Economic Affairs and the Jubilüaumsfonds der Stadt Wien

    • https://www.iso.org/iso-8601-date-and-time-format.html.

    • http://www.timeml.org.

    • http://www.akomantoso.org/.

    • https://tempcourt.github.io/TempCourt/.

    • ISO 24617-1 Language Resource Management—Semantic Annotation Framework (SemAF)—Time and Events (SemAF Time and ISO-TimeML).

    • https://www.w3.org/TR/annotation-model/.

    • http://persistence.uni-leipzig.org/nlp2rdf/ontologies/nif-core#.

    • For instance, diagnosis such as tumors or medical tests are relevant events that should appear in a timeline of a medical doctor, as stated by Styler IV et al., (2014), but not in other types of texts. Similarly, specific legal events such as preliminary rulings (explained in Section 3) in European judgments are always relevant to lawyers, although they never appear in other kinds of texts.

    • Please note that the same sentence contains two TEs which are attributed to two different temporal dimensions.

    • https://catalog.ldc.upenn.edu/docs/LDC2006T08/timeml_annguide_1.2.1.pdf.

    • http://eur-lex.europa.eu/.

    • https://hudoc.echr.coe.int

    • https://www.supremecourt.gov/.

    • https://echr.coe.int/Pages/home.aspx?p=ddisclaimerc=.

    • https://eur-lex.europa.eu/legal-content/EN/TXT/?uri%3dCELEX:32011D0833.

    • https://eur-lex.europa.eu/content/legal-notice/legal-notice.html#droits..

    • https://www.copyright.gov/title17/92chap1.html#105.

    • Regulation (EU) 2016/679.

    • http://www.timeml.org/timebank/documentation-1.2.html.

    • http://www.timeml.org/timebank/aquaint-timeml/aquaint_timeml_1.0.tar.gz.

    • https://catalog.ldc.upenn.edu/docs/LDC2006T08/timeml_annguide_1.2.1.pdf.

    • https://www.ldc.upenn.edu/sites/www.ldc.upenn.edu/files/english-timex2-guidelines-v0.1.pdf.

    • The functionality and the rules were not modified.

    • The final corpus can be downloaded at https://tempcourt.github.io/TempCourt/.

    • http://www.cs.york.ac.uk/semeval-2013/task1/data/uploads/timeml-validator-1.1a.tar.gz.

    • https://www.cs.york.ac.uk/semeval-2013/task1/.

    • https://www.cs.york.ac.uk/semeval-2013/task1/.

    • https://github.com/HeidelTime/heideltime/.

    • https://uima.apache.org.

    • https://github.com/stanfordnlp/CoreNLP/tree/master/src/edu/stanford/nlp/time.

    • https://nlp.stanford.edu/software/sutime.shtml#Extensions.

    • https://github.com/xszhong/syntime.

    • https://github.com/cnorthwood/ternip.

    • Fraction of the results identified which were correct.

    • Fraction of the results that should have been found which were correctly identified.

    • Beginning of, Inside of, Outside of a time expression.

    • https://cleartk.github.io/cleartk/docs/module/cleartk_timeml.html.

    • https://github.com/tarsqi/ttk.

    • http://timeml.org/tarsqi/index.html.

    • https://github.com/nchambers/caevo.

    • https://github.com/hllorens/otip.

    • https://github.com/leondz/usfd2, https://code.google.com/archive/p/usfd2/.

    • https://bitbucket.org/kentonl/uwtime-standalone.

    • Except USFD2.

    • https://github.com/HeidelTime/heideltime/wiki/Evaluation-Results.

    • http://www.echr.coe.int/Documents/Note_citation_ENG.pdf.

    • Some cases, such as distinctions between EQUAL_OR_LESS/LESS_THAN (for UWTime) and LATE/END and EARLY/START (for TERNIP) were counted as errors.

    • The two first authors equally contributed to this work.

    Rights and permissions
    • © Cambridge University Press 20192019Cambridge University Press
References (50)
  • About this article
    Cite this article
    María Navas-Loro, Erwin Filtz, Víctor Rodríguez-Doncel, Axel Polleres, Sabrina Kirrane. 2019. TempCourt: evaluation of temporal taggers on a new corpus of court decisions. The Knowledge Engineering Review 34(1), doi: 10.1017/S0269888919000195
    María Navas-Loro, Erwin Filtz, Víctor Rodríguez-Doncel, Axel Polleres, Sabrina Kirrane. 2019. TempCourt: evaluation of temporal taggers on a new corpus of court decisions. The Knowledge Engineering Review 34(1), doi: 10.1017/S0269888919000195
    shu

    • Catalog

      /

      DownLoad:  Full-Size Img  PowerPoint
      Return
      Return