IRIT & Université de Toulouse 2 Jean Jaurès, Toulouse, France, e-mail: elodie.thieblin@irit.fr, cassia.trojahn@irit.f"/> Wright State University, Dayton, USA, e-mail: michelle.cheatham@wright.edu"/> University of Economics, Prague, Czech Republic, e-mail: ondrej.zamazal@vse.cz"/>
Search
2020 Volume 35
Article Contents
RESEARCH ARTICLE   Open Access    

A consensual dataset for complex ontology matching evaluation

More Information
  • Abstract: Simple ontology alignments, largely studied in the literature, link one single entity of a source ontology to one single entity of a target ontology. One of the limitations of these alignments is, however, their lack of expressiveness, which can be overcome by complex alignments, which are composed of correspondences involving logical constructors or transformation functions. While most work on complex ontology matching has been dedicated to the development of complex matching approaches, there is still a lack of benchmarks on which the complex approaches can be systematically evaluated. The aim of this paper is to present the process of constructing the consensual complex Conference dataset, describing the design choices and the methodology followed for constructing it. We discuss the issues the experts were faced with during the process and discuss the lessons learned and perspectives in the field.
  • 加载中
  • Abacha , A. B. & Zweigenbaum , P.2014. Means: une approche sémantique pour la recherche de réponses aux questions médicales. TAL55(1), 71–104.

    Google Scholar

    Algergawy , A., Cheatham , M., Faria , D., Ferrara , A., Fundulaki , I., Harrow , I., Hertling , S., Jiménez-Ruiz , E., Karam , N., Khiat , A., Lambrix , P., Liand , H., Montanelli , S., Paulheim , H., Pesquita , C., Saveta , T., Schmidt , D., Shvaiko , P., Splendiani , A., Thiéblin , E., Trojahn , C., Vataščinová , J., Zamazal , O. & Zhou , L.2018. Results of the ontology alignment evaluation initiative 2018. In OM-2018: Proceedings of the Twelfth International Workshop on Ontology Matching.

    Google Scholar

    Cheatham , M. & Hitzler , P.2014. Conference v2. 0: an uncertain version of the OAEI conference benchmark. In International Semantic Web Conference, 33–48. Springer.

    Google Scholar

    Dekhtyar , A. & Hayes , J. H.2006. Good benchmarks are hard to find: toward the benchmark for information retrieval applications in software engineering. In Proceedings of the 22th International Conference on Software Maintenance.

    Google Scholar

    Dhamankar , R., Lee , Y., Doan , A., Halevy , A. & Domingos , P.2004. iMAP: discovering complex semantic matches between database schemas. In Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data, 383–394. ACM.

    Google Scholar

    Do , H.-H., Melnik , S. & Rahm , E.2002. Comparison of schema matching evaluations. In Net. ObjectDays: International Conference on Object-Oriented and Internet-Based Technologies, Concepts, and Applications for a Networked World, 221–237. Springer.

    Google Scholar

    Doan , A. (2005). Dataset of complex correspondences. Illinois Semantic Integration Archive. Computer Science Department, University of Illinois. http://pages.cs.wisc.edu/~anhai/wisc-si-archive/.

    Google Scholar

    Euzenat , J., Ferrara , A., Hollink , L., Isaac , A., Joslyn , C., Malaisé , V., Meilicke , C., Nikolov , A., Pane , J., Sabou , M., Scharffe , F., Shvaiko , P., Spiliopoulos , V., Stuckenschmidt , H., Sváb-Zamazal , O., Svátek , V., dos Santos , C. T., Vouros , G. A. & Wang , S.2009. Results of the ontology alignment evaluation initiative 2009. In Proceedings of the 4th International Workshop on Ontology Matching (OM-2009) collocated with the 8th International Semantic Web Conference (ISWC-2009) Chantilly, USA, October 25, 2009.

    Google Scholar

    Euzenat , J. & Shvaiko , P.2013. Ontology Matching. Springer.

    Google Scholar

    Faria , D., Pesquita , C., Balasubramani , B. S., Tervo , T., Carrio , D., Garrilha , R., Couto , F. M. & Cruz , I. F.2018. Results of AML participation in OAEI 2018. In OM-2018: Proceedings of the Twelfth International Workshop on Ontology Matching.

    Google Scholar

    Halpin , H., Hayes , P. J., McCusker , J. P., McGuinness , D. L. & Thompson , H. S.2010. When owl:sameAs isn’t the same: an analysis of identity in linked data. In The Semantic Web – ISWC 2010, Patel-Schneider , P. F., Pan , Y., Hitzler , P., Mika , P., Zhang , L., Pan , J. Z., Horrocks , I. & Glimm , B. (eds), 305–320. Springer.

    Google Scholar

    He , B., Chang , K. C.-C. & Han , J.2004. Discovering complex matchings across web query interfaces: a correlation mining approach. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 148–157. ACM Press.

    Google Scholar

    Hollink , L., Van Assem , M., Wang , S., Isaac , A. & Schreiber , G.2008. Two variations on ontology alignment evaluation: methodological issues. In 5th European Semantic Web Conference, 388–401.

    Google Scholar

    Isaac , A., Matthezing , H., van der Meij , L., Schlobach , S., Wang , S. & Zinn , C.2008. Putting ontology alignment in context: usage scenarios, deployment and evaluation in a library case. In 5th European Semantic Web Conference, 402–417.

    Google Scholar

    Jiang , S., Lowd , D., Kafle , S. & Dou , D.2016. Ontology matching with knowledge rules. In Transactions on Large-Scale Data-and Knowledge-Centered Systems XXVIII, Hameurlain, A., Küng, J., Wagner, R. & Chen, Q. (eds). Springer, 75–95.

    Google Scholar

    Maedche , A., Motik , B., Silva , N. & Volz , R.2002. Mafra—a mapping framework for distributed ontologies. In International Conference on Knowledge Engineering and Knowledge Management, 235–250. Springer.

    Google Scholar

    Makris , K., Bikakis , N., Gioldasis , N. & Christodoulakis , S.2012. SPARQL-RW: transparent query access over mapped RDF data sources. In 15th International Conference on Extending Database Technology, 610–613. ACM.

    Google Scholar

    Maltese , V., Giunchiglia , F. & Autayeu , A.2010. Save up to 99% of your time in mapping validation. In On the Move to Meaningful Internet Systems, OTM 2010 - Confederated International Conferences: CoopIS, IS, DOA and ODBASE, Hersonissos, Crete, Greece, October 25–29, 1044–1060.

    Google Scholar

    Mathur , S. N., O’Sullivan , D. & Brennan , R.2018. Milan: automatic generation of R2RML mappings. In Proceedings of the 26th Irish Conference on Artificial Intelligence and Cognitive Science, Dublin, Ireland, 7th December, 1–12.

    Google Scholar

    Meilicke , C.2011. Alignment Incoherence in Ontology Matching. PhD thesis, Universität Mannheim.

    Google Scholar

    Meilicke , C. & Stuckenschmidt , H.2008. Incoherence as a basis for measuring the quality of ontology mappings. In 3rd International Conference on Ontology Matching, 431, 1–12.

    Google Scholar

    Meilicke , C., Stuckenschmidt , H. & Šváb-Zamazal , O.2009. A reasoning-based support tool for ontology mapping evaluation. In European Semantic Web Conference, 878–882. Springer.

    Google Scholar

    Nunes , B. P., Mera , A., Casanova , M. A., Breitman , K. K. & Leme , L. A. P.2011. Complex matching of RDF datatype properties. In 6th ISWC Workshop on Ontology Matching.

    Google Scholar

    Oliveira , D. & Pesquita , C.2018. Improving the interoperability of biomedical ontologies with compound alignments. Journal of Biomedical Semantics9(1), 1:1–13:13.

    Google Scholar

    Parundekar , R., Knoblock , C. A. & Ambite , J. L.2010. Linking and building ontologies of linked data. In ISWC, 598–614. Springer.

    Google Scholar

    Parundekar , R., Knoblock , C. A. & Ambite , J. L.2012. Discovering concept coverings in ontologies of linked data sources. In ISWC, 427–443. Springer.

    Google Scholar

    Pinkel , C., Binnig , C., Jiménez-Ruiz , E., Kharlamov , E., May , W., Nikolov , A., Sasa Bastinos , A., Skjœveland , M. G., Solimando , A., Taheriyan , M., Heupel , C. & Horrocks , I.2017. RODI: benchmarking relational-to-ontology mapping generation quality. Semantic Web9(1), 25–52.

    Google Scholar

    Qin , H., Dou , D. & LePendu , P.2007. Discovering executable semantic mappings between ontologies. In On the Move to Meaningful Internet Systems, 832–849. Springer.

    Google Scholar

    Ritze , D., Meilicke , C., Šváb Zamazal , O. & Stuckenschmidt , H.2009. A pattern-based ontology matching approach for detecting complex correspondences. In 4th ISWC Workshop on Ontology Matching, 25–36.

    Google Scholar

    Ritze , D., Völker , J., Meilicke , C. & Šváb Zamazal , O.2010. Linguistic analysis for complex ontology matching. In 5th Workshop on Ontology Matching, 1–12.

    Google Scholar

    Scharffe , F.2009. Correspondence Patterns Representation. PhD thesis, Faculty of Mathematics, Computer Science and University of Innsbruck.

    Google Scholar

    Serpeloni , F., Moraes , R. & Bonacin , R.2011. Ontology mapping validation. International Journal of Web Portals3(3), 1–11.

    Google Scholar

    Sim , S. E., Easterbrook , S. & Holt , R. C.2003. Using benchmarking to advance research: a challenge to software engineering. In Proceedings of the 25th International Conference on Software Engineering, ICSE’03, 74–83, Washington, DC, USA. IEEE Computer Society.

    Google Scholar

    Singh , A., Debruyne , C., Brennan , R., Meehan , A. & O’Sullivan , D.2017. Extension of the M-Gov ontology mapping framework for increased traceability. In Proceedings of the 12th International Workshop on Ontology Matching co-located with the 16th International Semantic Web Conference (ISWC 2017), Vienna, Austria, October 21, 49–60.

    Google Scholar

    Solimando , A., Jimenez-Ruiz , E. & Guerrini , G.2017. Minimizing conservativity violations in ontology alignments: algorithms and evaluation. Knowledge and Information Systems51(3), 775–819.

    Google Scholar

    Solimando , A., Jiménez-Ruiz , E. & Pinkel , C.2014. Evaluating ontology alignment systems in query answering tasks. In ISWC 2014 International Conference on Posters & Demonstrations, 301–304.

    Google Scholar

    Stapleton , G., Howse , J., Bonnington , A. & Burton , J.2014. A vision for diagrammatic ontology engineering. In International Workshop on Visualizations and User Interfaces for Knowledge Engineering and Linked Data Analytics, 1–13.

    Google Scholar

    Stevens , R., Lord , P., Malone , J. & Matentzoglu , N.2018. Measuring expert performance at manually classifying domain entities under upper ontology classes. Journal of Web Semantics57, 1–13.

    Google Scholar

    Šváb , O., Svátek , V., Berka , P., Rak , D. & Tomášek , P.2005. Ontofarm: towards an experimental collection of parallel ontologies. In Poster Track of ISWC, 2005.

    Google Scholar

    Thiéblin , E., Amarger , F., Hernandez , N., Roussey , C. & Trojahn , C.2017. Cross-querying LOD datasets using complex alignments: an application to agronomic taxa. In Research Conference on Metadata and Semantics Research, 25–37. Springer.

    Google Scholar

    Thiéblin , É., Cheatham , M., dos Santos, C. T., Zamazal, O. & Zhou, L. 2018a. The first version of the OAEI complex alignment benchmark. In Proceedings of the ISWC 2018 Posters & Demonstrations, Industry and Blue Sky Ideas Tracks co-located with 17th International Semantic Web Conference (ISWC 2018), Monterey, USA, October 8–12, 2018.

    Google Scholar

    Thiéblin , É., Haemmerlé , O., Hernandez , N. & Trojahn , C.2018b. Task-oriented complex ontology alignment: two alignment evaluation sets. In The Semantic Web - 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, Proceedings, 655–670.

    Google Scholar

    Thieblin , E., Haemmerle , O., Hernandez , N. & Trojahn , C. to appear. Survey on complex ontology matching. Semantic Web Journal.

    Google Scholar

    Thiéblin , E., Haemmerlé , O. & Trojahn , C.2018. Complex matching based on competency questions for alignment: a first sketch. In Ontology Matching Workshop.

    Google Scholar

    Tordai , A., van Ossenbruggen , J., Schreiber , G. & Wielinga , B.2011. Let’s agree to disagree: on the evaluation of vocabulary alignment. In Proceedings of the Sixth International Conference on Knowledge Capture, K-CAP’11, 65–72. ACM.

    Google Scholar

    Van Hage , W. R., Isaac , A. & Aleksovski , Z.2007. Sample evaluation of ontology-matching systems. In EON, 2007, 41–50.

    Google Scholar

    Visser , P. R., Jones , D. M., Bench-Capon , T. J. & Shave , M.1997. An analysis of ontology mismatches; heterogeneity versus interoperability. In AAAI 1997 Spring Symposium on Ontological Engineering, Stanford CA., USA, 164–172.

    Google Scholar

    Walshe , B., Brennan , R. & O’Sullivan , D.2016. Bayes-recce: a bayesian model for detecting restriction class correspondences in linked open data knowledge bases. International Journal on Semantic Web and Information Systems (IJSWIS)12(2), 25–52.

    Google Scholar

    Zamazal , O. & Svátek , V.2017. The ten-year ontofarm and its fertilization within the onto-sphere. Web Semantics: Science, Services and Agents on the World Wide Web43, 46–53.

    Google Scholar

    Zhou , L., Cheatham , M., Krisnadhi , A. & Hitzler , P.2018. A complex alignment benchmark: Geolink dataset. In International Semantic Web Conference, 273–288. Springer, Cham.

    Google Scholar

  • Cite this article

    Elodie Thiéblin, Michelle Cheatham, Cassia Trojahn, Ondrej Zamazal. 2020. A consensual dataset for complex ontology matching evaluation. The Knowledge Engineering Review 35(1), doi: 10.1017/S0269888920000247
    Elodie Thiéblin, Michelle Cheatham, Cassia Trojahn, Ondrej Zamazal. 2020. A consensual dataset for complex ontology matching evaluation. The Knowledge Engineering Review 35(1), doi: 10.1017/S0269888920000247

Article Metrics

Article views(47) PDF downloads(51)

RESEARCH ARTICLE   Open Access    

A consensual dataset for complex ontology matching evaluation

Abstract: Abstract: Simple ontology alignments, largely studied in the literature, link one single entity of a source ontology to one single entity of a target ontology. One of the limitations of these alignments is, however, their lack of expressiveness, which can be overcome by complex alignments, which are composed of correspondences involving logical constructors or transformation functions. While most work on complex ontology matching has been dedicated to the development of complex matching approaches, there is still a lack of benchmarks on which the complex approaches can be systematically evaluated. The aim of this paper is to present the process of constructing the consensual complex Conference dataset, describing the design choices and the methodology followed for constructing it. We discuss the issues the experts were faced with during the process and discuss the lessons learned and perspectives in the field.

    • The authors would like to thank Lu Zhou for his help on the creation of the correspondences. Ondřej Zamazal was supported by the CSF grant no. 18-23964S and by long-term institutional support of research activities by Faculty of Informatics and Statistics, University of Economics, Prague.

    • http://oaei.ontologymatching.org/.

    • Transformation functions cannot be formalized into $\mathcal{DL}$.

    • http://www.music.tuc.gr/projects/sw/sparql-rw/.

    • http://oaei.ontologymatching.org/2016/conference/index.html, http://owl.vse.cz/ontofarm/.

    • For example, in the work here, the simple correspondence; between cmt:Reviewer and conference:Reviewer was changed from equivalence to subsumption, which then caused a new complex correspondence, conference:Reviewer$\sqsupseteq$cmt:Reviewer$\sqcup$cmt:ExternalReviewer to be added to the alignment. If the original decision were reversed, the complex correspondence would also need to be removed.

    • © Cambridge University Press, 20202020Cambridge University Press
References (50)
  • About this article
    Cite this article
    Elodie Thiéblin, Michelle Cheatham, Cassia Trojahn, Ondrej Zamazal. 2020. A consensual dataset for complex ontology matching evaluation. The Knowledge Engineering Review 35(1), doi: 10.1017/S0269888920000247
    Elodie Thiéblin, Michelle Cheatham, Cassia Trojahn, Ondrej Zamazal. 2020. A consensual dataset for complex ontology matching evaluation. The Knowledge Engineering Review 35(1), doi: 10.1017/S0269888920000247
  • Catalog

      /

      DownLoad:  Full-Size Img  PowerPoint
      Return
      Return