Filter by Type

Filter by Year

Sort by Year

Towards an Anatomy of IR System Component Performances

Nicola Ferro and Gianmaria Silvello
Journal Paper Journal of the Association for Information Science and Technology (JASIST), vol. 69 issue 2, pp. 187-200, 2018.

Abstract

Information Retrieval (IR) systems are the prominent means for searching and accessing huge amounts of unstructured information on the Web and elsewhere. They are complex systems, constituted by many different components interacting together, and evaluation is crucial to both tune and improve them. Nevertheless, in the current evaluation methodology, there is still no way to determine how much each component contributes to the overall performances and how the components interact together. This hampers the possibility of a deep understanding of IR system behaviour and, in turn, prevents us from designing ahead which components are best suited to work together for a specific search task.

In this paper, we move the evaluation methodology one step forward by overcoming these barriers and beginning to devise an “anatomy” of IR systems and their internals. In particular, we propose a methodology based on the General Linear Mixed Model (GLMM) and ANalysis Of VAriance (ANOVA) to develop statistical models able to isolate system variance and component effects as well as their interaction, by relying on a Grid of Points (GoP) containing all the combinations of the analysed components. We apply the proposed methodology to the analysis of two relevant search tasks – news search and Web search – by using standard TREC collections. We analyse the basic set of components typically part of an IR system, namely stop lists, stemmers and n-grams, and IR models. In this way, we derive insights about English text retrieval.

Theory and Practice of Data Citation

Gianmaria Silvello
Journal Paper Journal of the Association for Information Science and Technology (JASIST) (AIS Review), vol. 69 issue 1, pp. 6-20, 2018.

Abstract

Citations are the cornerstone of knowledge propagation and the primary means of assessing the quality of research, as well as directing investments in science. Science is increasingly becoming “data-intensive”, where large volumes of data are collected and analyzed to discover complex patterns through simulations and experiments, and most scientific reference works have been replaced by online curated datasets. Yet, given a dataset, there is no quantitative, consistent and established way of knowing how it has been used over time, who contributed to its curation, what results have been yielded or what value it has.

The development of a theory and practice of data citation is fundamental for considering data as first-class research objects with the same relevance and centrality of traditional scientific products. Many works in recent years have discussed data citation from different viewpoints: illustrating why data citation is needed, defining the principles and outlining recommendations for data citation systems, and providing computational methods for addressing specific issues of data citation.

The current panorama is many-faceted and an overall view that brings together diverse aspects of this topic is still missing. Therefore, this paper aims to describe the lay of the land for data citation, both from the theoretical (the why and what) and the practical (the how) angle.

Mining implicit data association from Tripadvisor hotel reviews.

Vittoria Cozza, Marinella Petrocchi, Angelo Spognardi (2018)
Workshop PaperWorkshop Proceedings of the EDBT/ICDT 2018 Joint Conference (March 26, 2018, Vienna, Austria) on CEUR-WS.org (ISSN 1613-0073). In print.

Statistical Stemmers: A Reproducibility Study

Gianmaria Silvello, Riccardo Bucco, Giulio Busato, Giacomo Fornari, Andrea Langeli, Alberto Purpura, Giacomo Rocco, Alessandro Tezza, and Maristella Agosti (2018)
Conference Paper In Pasi, G., Piwowarski, B., Azzopardi, L., and Hanbury, A., editors, Advances in Information Retrieval. Proc. 40th European Conference on IR Research (ECIR 2018). Lecture Notes in Computer Science (LNCS), Springer, Heidelberg, Germany

Abstract

Statistical stemmers are important components of Informa- tion Retrieval (IR) systems, especially for text search over languages with few linguistic resources. In recent years, research on stemmers produced relevant results, especially in 2011 when three language-independent stemmers were published in relevant venues. In this paper, we describe our efforts for reproducing these three stemmers. We also share the code as open-source and an extended version of Terrier system integrating the developed stemmers.

Modelling Randomness in Relevance Judgments and Evaluation Measures

Ferrante, M., Ferro, N., and Pontarollo, S. (2018)
Conference Paper In Pasi, G., Piwowarski, B., Azzopardi, L., and Hanbury, A., editors, Advances in Information Retrieval. Proc. 40th European Conference on IR Research (ECIR 2018). Lecture Notes in Computer Science (LNCS), Springer, Heidelberg, Germany

Digital Libraries: From Digital Resources to Challenges in Scientific Data Sharing and Re-Use

Maristella Agosti, Nicola Ferro and Gianmaria Silvello
Book Chapter A Comprehensive Guide Through the Italian Database Research Over the Last 25 Years, Volume 31 of the series Studies in Big Data, pp 27-41, 2018.

Abstract

Digital libraries and digital archives are the information management systems for storing, indexing, searching, accessing, curating and preserving digital resources which manage our cultural and scientific knowledge heritage (KH). They act as the main conduits for widespread access and exploitation of KH related digital resources by engaging many different types of users, ranging from generic and leisure to students and professionals.

In this chapter, we describe the evolution of digital libraries and archives over the years, starting from Online Public Access Catalog (OPAC), passing through monolithic and domain specific systems, up to service-oriented and component- based architectures. In particular, we present some specific achievements in the field: the DELOS Reference Model and the DelosDLMS, which provide a con- ceptual reference and a reference implementation for digital libraries; the FAST annotation service, which defines a formal model for representing and search- ing annotations over digital resources as well as a RESTful Web service imple- mentation of it; the NESTOR model for digital archives, which introduces an alternative model for representing and managing archival resources in order to enhance interoperability among archives and make access to them faster; and, the CULTURA environment, which favours user engagement over multimedia digital resources.

Finally, we discuss how digital libraries and archives are a key technology for facing upcoming challenges in data sharing and re-use. Indeed, due to the rapid evolution of the nature of research and scientific publishing which are increasingly data-driven, digital libraries and archives are also progressively ad- dressing the issues of managing scientific data. In this respect, we focus on some key building blocks of this new vision: data citation to foster accessibility to scientific data as well as transparency and verifiability of scientific claims, re- producibility in science as an exemplar showcase of how all these methods are indispensable for addressing fundamental challenges, and keyword-based search over relation/structured data to empower natural language access to scientific data.

Thirty years of digital libraries research at the University of Padua: The systems side

Maristella Agosti, Giorgio Maria Di Nunzio, Nicola Ferro and Gianmaria Silvello
Conference PaperIn Proc. 14th Italian Research Conference on Digital Libraries (IRCDL 2018).
Communications in Computer and Information Science book series (CCIS, volume 806), to appear, Springer, Heidelberg, Germany, 2018.

Thirty years of digital libraries research at the University of Padua: The users side

Maristella Agosti, Giorgio Maria Di Nunzio, Nicola Ferro, Maria Maistro, Stefano Marchesin, Nicola Orio, Chiara Ponchia and Gianmaria Silvello
Conference PaperIn Proc. 14th Italian Research Conference on Digital Libraries (IRCDL 2018).
Communications in Computer and Information Science book series (CCIS, volume 806), to appear, Springer, Heidelberg, Germany, 2018.

A Software Library for Conducting Large Scale Experiments on Learning to Rank Algorithms

Nicola Ferro, Paolo Picello and Gianmaria Silvello
Workshop PaperIn N. Ferro, C. Lucchese, M. Maistro and R. Perego eds., Proceedings of the 1st International Workshop on LEARning Next gEneration Rankers co-located with the 3rd ACM International Conference on the Theory of Information Retrieval (ICTIR 2017) (LEARNER 2017). 2017.

AWARE: Exploiting Evaluation Measures to Combine Multiple Assessors

Marco Ferrante, Nicola Ferro and Maria Maistro
Journal Paper ACM Transaction on Information Systems, (TOIS), 36(2):20:1–20:38.

Abstract

We propose the Assessor-driven Weighted Averages for Retrieval Evaluation (AWARE) probabilistic framework, a novel methodology for dealing with multiple crowd assessors, who may be contradictory and/or noisy. By modeling relevance judgements and crowd assessors as sources of uncertainty, AWARE takes the expectation of a generic performance measure, like Average Precision (AP), composed with these random variables. In this way, it approaches the problem of aggregating different crowd assessors from a new perspective, i.e. directly combining the performance measures computed on the ground-truth generated by the crowd assessors instead of adopting some classification technique to merge the labels produced by them. We propose several unsupervised estimators that instantiate the AWARE framework and we compare them with state-of-the-art approaches, i.e. Majority Vote (MV) and Expectation Maximization (EM), on TREC collections. We found that AWARE approaches improve in terms of their capability of correctly ranking systems and predicting their actual performance scores.

Overview of Special Issue - ACM SIGIR Test of Time Awardees, 1978-2001

Allan, J., Belkin, N. J., Bennett, P. N., Callan, J., Clarke, C. L. A., Diaz, F., Dumais, S., Ferro, N., Harman, D.K., Hiemstra, D., Ruthven, I., Sakai, T., Smucker, M.D., and Zobel, J. (2017)
Int. Journal Paper In Harman, D. K. and Kelly, D., editors, SIGIR Forum, 51(2):1-25

An Adaptive Cross-Site User Modelling Platform for Cultural Heritage Websites

Agosti, M., Lawless, S., Marchesin, S., and Wade, V. (2017)
Nat. Conference Paper In Grana, C., and Baraldi, L., editors, Proc. 13th Italian Research Conference on Digital Libraries (IRCDL 2017), pages 132-141. Communications in Computer and Information Science (CCIS) 733, Springer, Heidelberg, Germany

Digital Libraries and Multimedia Archives - Proc. 12th Italian Research Conference (IRCDL 2016)

Agosti, M., Bertini, M., Ferilli, S., Marinai, S., and Orio, N., editors (2017)
Editorship CCIS 701, Springer, Berlin Heidelberg, Germany, ISBN: 978-3-319-56299-5

An Approach to the Design and Evaluation of an Enterprise Search Application

Zilio, D., Agosti, M., Turato, D. (2017)
Nat. Conference Paper In Crestani, F., Di Noia, T., and Perego, R., editors, Proc. of the 8th Italian Information Retrieval Workshop, Lugano, Switzerland, pages 85-91. CEUR Workshop Proceedings, volume 1911

Data Citation: a Computational Challenge

Susan Davidson, Peter Buneman, Daniel Deutch, Tova Milo and Gianmaria Silvello
Conference Paper Proceedings of the 36th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems (PODS 2017), pp. 1-4, 2017.

Abstract

Data citation is an interesting computational challenge, whose solution draws on several well-studied problems in database theory: query answering using views, and provenance. We describe the problem, suggest an approach to its solution, and highlight several open research problems, both practical and theoretical.

What Does Affect the Correlation Among Evaluation Measures?

Ferro, N. (2017)
Int. Journal Paper ACM Transactions on Information Systems (TOIS), 36(2):19:1-19:40

Reproducibility Challenges in Information Retrieval Evaluation

Ferro, N. (2017)
Int. Journal Paper ACM Journal of Data and Information Quality (JDIQ), 8(2):8:1-8:4

Automating data citation: the eagle-i experience

Abdussalam Alawini, Leshang Chen, Susan Davidson and Gianmaria Silvello
Conference Paper Proc. of the ACM/IEEE Joint Conference on Digital Libraries (JCDL 2017), pp. 169-178, IEEE Computer Society, 2017.

Abstract

Data citation is of growing concern for owners of curated databases, who wish to give credit to the contributors and curators responsible for portions of the dataset and enable the data retrieved by a query to be later examined. While several databases specify how data should be cited, they leave it to users to manually construct the citations and do not generate them automatically.

We report our experiences in automating data citation for an RDF dataset called eagle-i, and discuss how to gen- eralize this to a citation framework that can work across a variety of different types of databases (e.g. relational, XML, and RDF). We also describe how a database administrator would use this framework to automate citation for a partic- ular dataset.

A study on text-score disagreement in online reviews.

Michela Fazzolari, Vittoria Cozza, Marinella Petrocchi, Angelo Spognardi. (2017)
Journal Paper Cognitive Computation, Springer. DOI: 10.1007/s12559-017-9496-y.

Why Snoopy loves online services: An Analysis of (lack of) Privacy in Online Services.

Vittoria Cozza, Zisis Tsiatsikas, Mauro Conti, Georgios Kambourakis. (2017)
Conference Paper Proc. 3rd International Conference on Information Systems Security and Privacy (ICISSP), pages 431-438, ISBN: 978-989-758-209-7. Porto, Portugal, February 19 - 21, 2017.

Are IR Evaluation Measures on an Interval Scale?

Ferrante, M., Ferro, N., and Pontarollo, S. (2017)
Int. Conference PaperBest Paper Award In Kamps, J., Kanoulas, E., de Rijke, M., Fang, H., and Yilmaz, E., editors, Proc. 3rd ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR 2017), pages 67-74. ACM Press, New York, USA

Sub-corpora Impact on System Effectiveness

Ferro, N. and Sanderson, M. (2017)
Int. Conference Paper In Kando, N., Sakai, T., Joho, H., Li, H., de Vries, A. P., and White, R. W., editors, Proc. 40th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2017), pages 901-904. ACM Press, New York, USA

CLEF 2017 Working Notes

Cappellato, L., Ferro, N., Goeuriot, L., and Mandl, T., editors (2017)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1866, ISSN 1613-0073

Proc. 1st International Workshop on LEARning Next gEneration Rankers (LEARNER 2017)

Ferro, N., Lucchese, C., Maistro, M., and Perego, R., editors (2017)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 2007, ISSN 1613-0073

Proc. 8th International Workshop on Evaluating Information Access (EVIA 2017)

Ferro, N. and Soboroff, I., editors (2017)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 2008, ISSN 1613-0073

Proc. 1st International Workshop on Keyword-Based Access and Ranking at Scale (KARS 2017) - Proc. of the Workshops of the EDBT/ICDT 2017 Joint Conference (EDBT/ICDT 2017)

Ioannidis, Y. E., Stoyanovich, J., Orsi, G., Song, I.-Y., Marcel, P., Martoglia, R., Penzo, W., Mandreoli, F., De Virgilio, R., De Antonellis, V., Bianchini, D., Kotzinos, D., Christophides, V., Nikolaou, C., Theodoridis, Y., Ferro, N., Guerra, F., Ives, Z., Silvello, G., and Theobald, M., editors (2017)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1810, ISSN 1613-0073

Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Eighth International Conference of the CLEF Association (CLEF 2017)

Jones, G. J. F., Lawless, S., Gonzalo, J., Kelly, L., Goeuriot, L., Mandl, T., Cappellato, L., and Ferro, N., editors (2017)
Editorship Lecture Notes in Computer Science (LNCS) 10456, Springer, Heidelberg, Germany

Report on CLEF 2017: Experimental IR Meets Multilinguality, Multimodality, and Interaction

Cappellato, L., Ferro, N., Goeuriot, L., Gonzalo, J., Jones, G. J. F., Kelly, L., Lawless, S., and Mandl, T. (2017)
Journal w/o Peer Review Paper SIGIR Forum, 51(3)

Towards open-source shared implementations of keyword-based access systems to relational data

Badan, A., Benvegnù, L., Biasetton, M., Bonato, G., Brighente, A., Cenzato, A., Ceron, P., Cogato, G., Marchesin, S., Minetto, A., Pellegrina, L., Purpura, A., Simionato, R., Soleti, N., Tessarotto, M., Tonon, A., Vendramin, F., and Ferro, N. (2017)
Int. Workshop Paper In Ioannidis, Y. E., Stoyanovich, J., Orsi, G., Song, I.-Y., Marcel, P., Martoglia, R., Penzo, W., Mandreoli, F., De Virgilio, R., De Antonellis, V., Bianchini, D., Kotzinos, D., Christophides, V., Nikolaou, C., Theodoridis, Y., Ferro, N., Guerra, F., Ives, Z., Silvello, G., and Theobald, M., editors, Proc. 1st International Workshop on Keyword-Based Access and Ranking at Scale (KARS 2017) - Proc. of the Workshops of the EDBT/ICDT 2017 Joint Conference (EDBT/ICDT 2017). CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1810, ISSN 1613-0073

Keyword-based access to relational data: To reproduce, or to not reproduce?

Badan, A., Benvegnù, L., Biasetton, M., Bonato, G., Brighente, A., Marchesin, S., Minetto, A., Pellegrina, L., Purpura, A., Simionato, R., Tessarotto, M., Tonon, A., and Ferro, N. (2017)
Nat. Conference Paper In Flesca, S., Greco, S., Masciari, E., and Saccà, D., editors, Proc. 25th Italian Symposium on Advanced Database Systems (SEBD 2017), pages 166-177.

An Interval-Like Scale Property for IR Evaluation Measures

Ferrante, M., Ferro, N., and Pontarollo, S. (2017)
Int. Workshop Paper In Ferro, N. and Soboroff, I., editors, Proc. 8th International Workshop on Evaluating Information Access (EVIA 2017). CEUR Workshop Proceedings (CEUR-WS.org), Vol. 2008, ISSN 1613-0073

A Model for Fine-Grained Data Citation

Susan Davidson, Daniel Deutch, Tova Milo and Gianmaria Silvello
Conference Paper Proc. of the biennial Conference on Innovative Data Systems Research (CIDR 2017), 2017.

Abstract

An increasing amount of information is being collected in structured, evolving, curated databases, driving the question of how information extracted from such datasets via queries should be cited. Unlike traditional research products, such books and journals, which have a fixed granularity, data citation is a challenge because the granularity varies. Different portions of the database, with varying granularity, may have different citations.

Furthermore, there are an infinite number of queries over a database, each accessing and generating different subsets of the database, so we cannot hope to explicitly attach a citation to every possible result set and/or query. We present the novel problem of automatically generating citations for general queries over a relational database, and explore a solution based on a set of citation views, each of which attaches a citation to a view of the database. Citation views are then used to automatically construct citations for general queries. Our approach draws inspiration from results in two areas, query rewriting using views and database provenance and combines them in a robust model. We then discuss open issues in developing a practical solution to this challenging problem.

Learning to Cite Framework: How to Automatically Construct Citations for Hierarchical Data

Gianmaria Silvello
Journal Paper Journal of the Association for Information Science and Technology (JASIST), Volume 68 issue 6, pp. 1505-1524, June 2017.

Abstract

The practice of citation is foundational for the propagation of knowledge along with scientific development and it is one of the core aspects on which scholarship and scientific publishing rely.

Within the broad context of data citation, we focus on the automatic construction of citations problem for hierarchically structured data. We present the “learning to cite” framework which enables the automatic construction of human- and machine-readable citations with different level of coarseness. The main goal is to reduce the human intervention on data to a minimum and to provide a citation system general enough to work on heterogeneous and complex XML datasets. We describe how this framework can be realized by a system for creating citations to single nodes within an XML dataset and, as a use case, show how it can be applied in the context of digital archives.

We conduct an extensive evaluation of the proposed citation system by analyzing its effectiveness from the correctness and completeness viewpoints, showing that it represents a suitable solution that can be easily employed in real-world environments and that reduces human intervention on data to a minimum.

Visual Analytics for Information Retrieval Evaluation Campaigns

Marco Angelini, Nicola Ferro, Giuseppe Santucci and Gianmaria Silvello
Workshop PaperIn M. Sedlmair and C. Tominski eds. EuroVis Workshop on Visual Analytics (EuroVis 2017). 2017.

A Model for Fine-Grained Data Citation

Susan Davidson, Daniel Deutch, Tova Milo and Gianmaria Silvello
Conference PaperIn Greco, S., Saccà, D., Flesca, S., and Masciari, E., editors, Proc. 25th Italian Symposium on Advanced Database Systems (SEBD 2017).

The Road Towards Reproducibility in Science: The Case of Data Citation

Nicola Ferro and Gianmaria Silvello
Conference PaperIn Grana, C. and Baraldi, L. editors, Proc. 13th Italian Research Conference on Digital Libraries (IRCDL 2017), Revised Selected Papers.
Communications in Computer and Information Science book series (CCIS, volume 733), pp. 20-31, Springer, Heidelberg, Germany, 2017.

LEARning Next gEneration Rankers (LEARNER 2017)

Nicola Ferro, Claudio Lucchese, Maria Maistro and Raffaele Perego
Conference PaperIn J. Kamps, E. Kanoulas, M. de Rijke, E. Yilmaz, H. and Fang, editors, Proceedings of the 3rd International Conference on The Theory of Information Retrieval, (ICTIR 2017), Amsterdam, The Netherlands, October 1-4, 2017, pages 331-332, ACM Press, New York, USA.

Abstract

The aim of LEARNER@ICTIR2017 is to investigate new solutions for LtR. In details, we identify some research areas related to LtR which are of actual interest and which have not been fully explored yet. We solicit the submission of position papers on novel LtR algorithms, on evaluation of LtR algorithms, on dataset creation and curation, and on domain specific applications of LtR. LEARNER@ICTIR2017 will be a gathering of academic people interested in IR, ML and related application areas. We believe that the proposed workshop is relevant to ICTIR since we look for novel contributions to LtR focused on foundational and conceptual aspects, which need to be properly framed and modeled.

Understanding User Behavior in Job and Talent Search: An Initial Investigation

Damiano Spina, Maria Maistro, Yongli Ren, Sargol Sadeghi, Wilson Wong, Timothy Baldwin, Lawrence Cavedon, Alistair Moffat, Mark Sanderson, Falk Scholer, and Justin Zobel
Workshop Paper In J. Degenhardt, S. Kallumadi, M. de Rijke, L. Si, A. Trotman, and X. Yinghui, editors, Proceedings of the 2017 SIGIR workshop On eCommerce, (eCom 2017), co-located with the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2017), Tokyo, Japan, August 11, 2017, CEUR Workshop Proceedings.

Abstract

The Web has created a global marketplace for e-Commerce as well as for talent. Online employment marketplaces provide an effective channel to facilitate the matching between job seekers and hirers. This paper presents an initial exploration of user behavior in job and talent search using query and click logs from a popular employment marketplace. The observations suggest that the understanding of users’ search behavior in this scenario is still at its infancy and that some of the assumptions made in general web search may not hold true. The open challenges identified so far are presented.

On Including the User Dynamic in Learning to Rank

Nicola Ferro, Claudio Lucchese, Maria Maistro and Raffaele Perego
Conference PaperIn N. Kando, T. Sakai, H. Joho, H. Li, A. P. de Vries, and R. White, editors, Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2017), Tokyo, Japan, August 7-11, 2017, pages 1041-1044, ACM Press, New York, USA.

Abstract

Ranking query results effectively by considering user past behaviour and preferences is a primary concern for IR researchers both in academia and industry. In this context, LtR is widely believed to be the most effective solution to design ranking models that account for user-interaction features that have proved to remarkably impact on IR e ectiveness. In this paper, we explore the possibility of integrating the user dynamic directly into the LtR algorithms. Specifically, we model with Markov chains the behaviour of users in scanning a ranked result list and we modify LambdaMart, a state-of-the-art LtR algorithm, to exploit a new discount loss function calibrated on the proposed Markovian model of user dynamic. We evaluate the performance of the proposed approach on publicly available LtR datasets, finding that the improvements measured over the standard algorithm are statistically significant.

A Game of Lines: Developing Game Mechanics for Text Classification.

Giorgio Maria Di Nunzio, Maria Maistro and Daniel Zilio
Workshop Paper In F. Crestani, T. Di Noia, and R. Perego, editors, Proceedings of the 8th Italian Information Retrieval Workshop, (IIR 2017), Lugano, Switzerland, June 5-7, 2017, CEUR Workshop Proceedings.

Abstract

In this paper, we describe a set of experiments that turn the machine learning classification task into a game, through gamification techniques, and let non expert users to perform text classification without even knowing the problem. The application is implemented in R using the Shiny package for interactive graphics. We present the outcome of three different experiments: a pilot experiment with PhD and post-doc students, and two experiments carried out with primary and secondary school students. The results show that the human aided classifier performs similarly and sometimes even better than state of the art classifiers.

Adapting Information Retrieval to User Signals via Stochastic Models

Maria Maistro
Conference Paper In M. de Rijke, M. Shokouhi, A. Tomkins, and M. Zhang, editors, Proceedings of the Tenth ACM International Conference on Web Search and Data Mining (WSDM 2017), Cambridge, United Kingdom, February 6-10, 2017, pages 843-843, ACM Press, New York, USA.

Abstract

To address the challenge of adapting Information Retrieval (IR) to the constantly evolving user tasks and needs and to adjust it to user interactions and preferences we develop a new model of user behavior based on Markov chains. We aim at integrating the proposed model into several aspects of IR, i.e. evaluation measures, systems and collections. Firstly, we studied IR evaluation measures and we propose a theoretical framework to describe their properties. Then, we presented a new family of evaluation measures, called Markov Precision (MP), based on the proposed model and able to explicitly link lab-style and on-line evaluation metrics. Future work will include the presented model into Learning to Rank (LtR) algorithms and will define a collection for evaluation and comparison of Personalized Information Retrieval (PIR) systems.

Component-Based Evaluation using GLMM

Nicola Ferro and Gianmaria Silvello
Workshop PaperIn Crestani, F., Di Noia, T., and Perego, R., editors, Proc. 8th Italian Information Retrieval Workshop (IIR 2017). CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, 2017.

Measuring Dataset Impact: Data Citation as an Economic Process

Gianmaria Silvello
Workshop AbstractInformation Retrieval and Interaction Fest in Honour of Peter Ingwersen. (October 2016)

3.5K runs, 5K topics, 3M assessments and 70M measures: What trends in 10 years of Adhoc-ish CLEF?

Nicola Ferro and Gianmaria Silvello
Journal Paper Information Processing & Management (IP&M), 53(1):175-202, 2017.

Abstract

Multilingual information access and retrieval is a key concern in today global society and, despite the considerable achievements over the past years, it still presents many challenges. In this context, experimental evaluation represents a key driver of innovation and multilinguality is tackled in several evaluation initiatives worldwide, such as CLEF in Europe, NTCIR in Japan and Asia, and FIRE in India. All these activities have run several evaluation cycles and there is a general consensus about their strong and positive impact on the development of multilingual information access systems. However, a systematic and quantitative assessment of the impact of evaluation initiatives on multilingual information access and retrieval over the long period is still missing.

Therefore, in this paper we conduct the first systematic and large-scale longitudinal study on several CLEF Adhoc-ish tasks – namely the Adhoc, Robust, TEL, and GeoCLEF labs – in order to gain insights on the performance trends of monolingual, bilingual and multilingual information access systems, spanning several European and non-European languages, over a range of 10 years.

We learned that monolingual retrieval exhibits a stable positive trend for many of the languages analyzed, even though the performance increase is not always steady from year to year due to the varying interests of the participants, who may not always be focused on just increasing performances. Bilingual retrieval demonstrates higher improvements in recent years – probably due to the better language resources now available – and it also outperforms monolingual retrieval in several cases. Multilingual retrieval shows improvements over the years and performances are comparable to those of bilingual and monolingual retrieval, and sometimes even better. Moreover, we have found evidence that the rule-of-thumb of a 3-year duration for an evaluation task is typically enough since top performances are usually reached by the third year and sometimes even by the second year, which then leaves room for research groups to investigate relevant research issues other than top performances.

Overall, this study provides quantitative evidence that CLEF has achieved the objective which led to its establishment, i.e. making multilingual information access a reality for European languages. However, the outcomes of this paper not only indicate that CLEF has steered the community in the right direction, but they also highlight the many open challenges for multilinguality. For instance, multilingual technologies greatly depend on language resources and targeted evaluation cycles help not only in developing and improving them, but also in devising methodologies which are more and more language-independent. Another key aspect concerns multimodality, intended not only as the capability of providing access to information in multiple media, but also as the ability of integrating access and retrieval over different media and languages in a way that best fits with user needs and tasks.

Semantic Representation and Enrichment of Information Retrieval Experimental Data

Gianmaria Silvello, Georgeta Bordea, Nicola Ferro, Paul Buitelaar and Toine Bogers
Journal Paper International Journal on Digital Libraries, 18(2):145-172, 2017.

Abstract

Experimental evaluation carried out in international large-scale campaigns is a fundamental pillar of the scientific and technological advancement of Information Retrieval (IR) systems. Such evaluation activities produce a large quantity of scientific and experimental data, which are the foundation for all the sub- sequent scientific production and development of new systems. In this work, we discuss how to semantically annotate and interlink this data, with the goal of enhancing their interpretation, sharing, and reuse. We discuss the underlying evaluation workflow and propose a Resource Description Framework (RDF) model for those workflow parts. We use expertise retrieval as a case study to demonstrate the benefits of our semantic representation approach. We employ this model as a means for exposing experimental data as Linked Open Data (LOD) on the Web and as a basis for enriching and automatically connecting this data with expertise topics and expert profiles.

In this context, a topic-centric approach for expert search is proposed, addressing the extraction of expertise topics, their semantic grounding with the LOD cloud, and their connection to IR experimental data. Several methods for expert profiling and expert finding are analysed and evaluated. Our results show that it is possible to construct expert profiles starting from automatically extracted expertise topics and that topic-centric approaches outperform state-of-the-art language modelling approaches for expert finding.

The CLEF Monolingual Grid of Points

Nicola Ferro and Gianmaria Silvello
Conference PaperInformation Access Evaluation. Multilinguality, Multimodality, and Interaction - Seventh International Conference of the Cross-Language Evaluation Forum, CLEF 2016: Evora, Portugal, September 5-8, 2016. pp. 16-27. In Lecture Notes in Computer Science 9822, Springer International Publishing Switzerland. .

Abstract

In this paper we run a systematic series of experiments for creating a grid of points where many combinations of retrieval methods and components adopted by MultiLingual Information Access (MLIA) systems are represented. This grid of points has the goal to provide insights about the effectiveness of the different components and their interaction and to identify suitable baselines with respect to which all the comparisons can be made.

We publicly release a large grid of points comprising more than 4K runs obtained by testing 160 IR systems combining different stop lists, stem- mers, n-grams components and retrieval models on CLEF monolingual tasks for eight European languages. Furthermore, we evaluate such grid of points by employing four different effectiveness measures and provide some insights about the quality of the created grid of points and the behaviour of the different systems.

"Data Citation is Coming". Introduction to the Special Issue on Data Citation

Gianmaria Silvello and Nicola Ferro (2016)
Journal Paper w/o prBulletin of IEEE Technical Committee on Digital Libraries, Volume 12 Issue 1, May 2016.

Abstract

This is the introduction to the special issue on data citation of the Bulletin of IEEE Technical Committee on Digital Libraries. In this introduction we state the “lay of the land” of research on data citation, we discuss some open issues and possible research directions and present the main contributions provided by the papers of the special issue.

Data-Driven Information Retrieval

Agosti, M., Alonso, O., de Rijke, M., and Perego, R. (2016)
Int. Journal w/o Peer Review Paper SIGIR Forum, 50(2):10-14

10 Years of IRCDL: 2005-2014 (Invited Paper)

Agosti, M. (2016)
Nat. Conference Paper In Calvanese, D., De Nart, D., and Tasso, C., editors, Digital Libraries on the Move - Proc. 11th Italian Research Conference on Digital Libraries (IRCDL 2015), pages 3-7. Communications in Computer and Information Science (CCIS) 612, Springer, Heidelberg, Germany

A Content-Based Approach to Social Network Analysis: A Case Study on Research Communities

De Nart, D., Degl'Innocenti, D., Basaldella, M., Agosti, M., and Tasso. C. (2016)
Nat. Conference Paper In Calvanese, D., De Nart, D., and Tasso, C., editors, Digital Libraries on the Move - Proc. 11th Italian Research Conference on Digital Libraries (IRCDL 2015), pages 142-154. Communications in Computer and Information Science (CCIS) 612, Springer, Heidelberg, Germany

ACHS'16: First International Workshop on Accessing Cultural Heritage at Scale

Clough, P. D., Goodale, P., Agosti, M., Lawless, L. (2016)
Int. Conference Paper In Adam, N. R. , Cassel, L. (Boots) , Yesha, Y. , Furuta, R., Weigle, M. C., editors, Proc. 16th ACM/IEEE-CS on Joint Conference on Digital Libraries (JCDL 2016), pages 289-290. ACM Press, New York, USA, ISBN 978-1-4503-4229-2

Proceedings of the First International Workshop on Accessing Cultural Heritage at Scale (ACHS 2016)

Clough, P. D., Goodale, P., Agosti, M., Lawless, L., editors (2016)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, Vol. 1611

Designing A Long Lasting Linguistic Project: The Case Study of ASIt

Agosti, M., Di Buccio, E., Di Nunzio, G. M., Poletto, C., Rinke, E. (2016)
Int. Conference Paper In Calzolari, N., Choukri, K., Declerck, T., Grobelnik, M., Maegaard, B., Mariani, J., Moreno, A., Odijk, J., Piperidis, S. editors, Proc. 10th International Conference on Language Resources and Evaluation (LREC 2016). ELRA, Paris, France, ISBN 978-2-9517408-9-1

L'eredità scientifica di Ada Lovelace

Agosti, M. (2016)
Nat. Journal Paper Atti e Memorie dell'Accademia Galileiana di Scienze Lettere ed Arti, già dei Ricoverati e Patavina. Parte II: Memorie della Classe di Scienze Matematiche, Fisiche e Naturali, CXXVIII:91-103

From Users to Systems: Identifying and Overcoming Barriers to Efficiently Access Archival Data

Nicola Ferro and Gianmaria Silvello (2016)
workshop paper 1st International Workshop on Accessing Cultural Heritage at Scale (ACHS'16), 22nd June 2016, Newark, NJ, USA.

Abstract

Digital archives are one of the pillars of our cultural heritage and they are increasingly opening up to end-users by focusing on accessibility of their resources. Moreover, digi- tal archives are complex and distributed systems where interoperability plays a central role and efficient access and exchange of resources is a challenge. In this paper, we investigate user and interoperability requirements in the archival realm and we discuss how next generation archival systems should operate a paradigm shift bringing a new model of access to archival resources which allows to better address these needs. To this end, we employ the data structures and query primitives based on the NEsted SeTs for Object hieRarchies (NESTOR) model to efficiently access archival data overcoming the identified barriers and limitations.

A General Linear Mixed Models Approach to Study System Component Effects

Nicola Ferro and Gianmaria Silvello
Conference Paper 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2016), pages 25-34, ACM Press, New York, NY, USA, 2016.

Abstract

Topic variance has a greater effect on performances than system variance but it cannot be controlled by system developers who can only try to cope with it. On the other hand, system variance is important on its own, since it is what system developers may affect directly by changing system components and it determines the differences among systems.

In this paper, we face the problem of studying system variance in order to better understand how much system components contribute to overall performances. To this end, we propose a methodology based on General Linear Mixed Model (GLMM) to develop statistical models able to isolate system variance, component effects as well as their interaction. We apply the proposed methodology to the analysis of TREC Ad-hoc data in order to show how it works and discuss some interesting outcomes of this new kind of analysis. Finally, we extend the analysis to different evaluation mea- sures, showing how they impact on the sources of variance.

A Visual Analytics Approach for What-If Analysis of Information Retrieval Systems

Marco Angelini, Nicola Ferro, Giuseppe Santucci and Gianmaria Silvello
Conference Paper 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2016), pages 1081-1084, ACM Press, New York, NY, USA, 2016

Abstract

We present the innovative visual analytics approach of the VATE2 system, which eases and makes more effective the experimental evaluation process by introducing the what-if analysis. The what-if analysis is aimed at estimating the possible effects of a modification to an IR system to select the most promising fixes before implementing them, thus saving a considerable amount of effort. VATE2 builds on an analytical framework which models the behavior of the systems in order to make estimations, and integrates this analytical framework into a visual part which, via proper interaction and animations, receives input and provides feedback to the user.

System And User Centered Evaluation Approaches in Interactive Information Retrieval (SAUCE 2016)

O'Brien, H., Ferro, N., Joho, H., Lewandowski, D., Thomas, P., and van Rijsbergen, C. J. (2016)
Int. Conference Paper In Kelly, D., Capra, R., Belkin, N., Teevan, J., and Vakkari, P., editors, Proc. 1st ACM on Conference on Human Information Interaction and Retrieval (CHIIR 2016), pages 337-340. ACM Press, New York, USA

The University of Padua (IMS) at TREC 2016 Total Recall Track

Giorgio Maria Di Nunzio, Maria Maistro and Daniel Zilio
Conference Paper In E. M. Voorhees and A. Ellis, editors, Proceedings of the 25th Text REtrieval Conference (TREC 2016), Gaithersburg, Maryland, USA, November 15-18, 2016, National Institute of Standards and Technology (NIST).

Abstract

The participation of the Information Management System (IMS) Group of the University of Padua in the Total Recall track at TREC 2016 consisted in a set of fully automated experiments based on the two-dimensional probabilistic model. We trained the model in two ways that tried to mimic a real user, and we compared it to two versions of the BM25 model with different parameter settings. This initial set of experiments lays the ground for a wider study that will explore a gamification approach in the context of high recall situations.

Gamification for IR: The Query Aspects Game

Giorgio Maria Di Nunzio, Maria Maistro and Daniel Zilio
Workshop PaperIn P. Basile, A. Corazza, F. Cutugno, S. Montemagni, M. Nissim, V. Patti, G. Semeraro, and R. Sprugnoli, editors, Proceeding of the 3rd Italian Conference on Computational Linguistics, (CLiC-it 2016), Napoli, Italy, December 5-6, 2016, CEUR Workshop Proceedings, Volume 1749.

Abstract

The creation of a labelled dataset for Information Retrieval (IR) purposes is a costly process. For this reason, a mix of crowd-sourcing and active learning approaches have been proposed in the literature in order to assess the relevance of documents of a collection given a particular query at an affordable cost. In this paper, we present the design of the gamification of this interactive process that draws inspiration from recent works in the area of gamification for IR. In particular, we focus on three main points: i) we want to create a set of relevance judgements with the least effort by human assessors, ii) we use interactive search interfaces that use game mechanics, iii) we use Natural Language Processing (NLP) to collect different aspects of a query.

DIRECT and LOD-DIRECT

Ferro, N. (2016)
Journal w/o Peer Review Paper In Freire, J., Fuhr, N., and Rauber, A., editors, Report from Dagstuhl Seminar 16041: Reproducibility of Data-Oriented Experiments in e-Science, Dagstuhl Reports, Volume 6, Number 1, page 115. Schloss Dagstuhl--Leibniz-Zentrum für Informatik, Germany

Managing and Curating IR Experimental Data

Ferro, N. (2016)
Journal w/o Peer Review Paper In Freire, J., Fuhr, N., and Rauber, A., editors, Report from Dagstuhl Seminar 16041: Reproducibility of Data-Oriented Experiments in e-Science, Dagstuhl Reports, Volume 6, Number 1, pages 120-121. Schloss Dagstuhl--Leibniz-Zentrum für Informatik, Germany

CLEF 2016 Working Notes

Balog, K., Cappellato, L., Ferro, N., and Macdonald, C., editors (2016)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1609, ISSN 1613-0073

Advances in Information Retrieval. Proc. 38th European Conference on IR Research (ECIR 2016)

Ferro, N., Crestani, F., Moens, M.-F., Mothe, J., Silvestri, F., Di Nunzio, G. M., Hauff, C., and Silvello, G., editors (2016)
Editorship Lecture Notes in Computer Science (LNCS) 9626, Springer, Heidelberg, Germany

Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Seventh International Conference of the CLEF Association (CLEF 2016)

Fuhr, N., Quaresma, P., Gonçalves, T., Larsen, B., Balog, K., Macdonald, C., Cappellato, L., and Ferro, N., editors (2016)
Editorship Lecture Notes in Computer Science (LNCS) 9822, Springer, Heidelberg, Germany

Gamification for Machine Learning: The Classification Game

Giorgio Maria Di Nunzio, Maria Maistro and Daniel Zilio
Workshop Paper In F. Hopfgartner, G. Kazai, U. Kruschwitz, and M. Meder, editors, Proceedings of the the Third International Workshop on Gamification for Information Retrieval, (GamifIR 2016), co-located with the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2016), Pisa, Italy, July 21, 2016, CEUR Workshop Proceedings, Volume 1642, pages 45-52.

Abstract

The creation of a labelled dataset for machine learning purposes is a costly process. In recent works, it has been shown that a mix of crowd-sourcing and active learning approaches can be used to annotate objects at an affordable cost. In this paper, we study the gamification of machine learning techniques; in particular, the problem of classification of objects. In this first pilot study, we designed a simple game, based on a visual interpretation of probabilistic classifiers, that consists in separating two sets of coloured points on a two-dimensional plane by means of a straight line. We present the current results of this first experiment that we used to collect the requirements for the next version of the game and to analyze i) what is the 'price' to build a reasonably accu- rate classifier with a small amount of labelled objects, ii) and compare the accuracy of the player to the state-of-the-art classification algorithms.

User Studies in IR

Ferro, N., Fuhr, N., Järvelin, K., Kando, N., and Lippold, M. (2016)
Journal w/o Peer Review Paper In Freire, J., Fuhr, N., and Rauber, A., editors, Report from Dagstuhl Seminar 16041: Reproducibility of Data-Oriented Experiments in e-Science, Dagstuhl Reports, Volume 6, Number 1, pages 152-157. Schloss Dagstuhl--Leibniz-Zentrum für Informatik, Germany

Increasing Reproducibility in IR: Findings from the Dagstuhl Seminar on "Reproducibility of Data-Oriented Experiments in e-Science"

Ferro, N., Fuhr, N., Järvelin, K., Kando, N., Lippold, M., and Zobel, J. (2016)
Journal w/o Peer Review Paper SIGIR Forum, 50(1):68-82.

Proposal for an Evaluation Framework for Compliance Checkers for Long-term Digital Preservation

Ferro, N. (2017)
Nat. Conference Paper In Agosti, M., Bertini, M., Ferilli, S., Marinai, S., and Orio, N., editors, Proc. 12th Italian Research Conference on Digital Libraries (IRCDL 2016), pages 125-136. Communications in Computer and Information Science (CCIS) 701, Springer, Heidelberg, Germany

PRIMAD - Information gained by different types of reproducibility

Rauber, A., Braganholo, V., Dittrich, J., Ferro, N., Freire, J., Fuhr, N., Garijo, D., Goble, C., Järvelin, K., Ludäscher, B., Stein, B., and Stotzka, R. (2016)
Journal w/o Peer Review Paper In Freire, J., Fuhr, N., and Rauber, A., editors, Report from Dagstuhl Seminar 16041: Reproducibility of Data-Oriented Experiments in e-Science, Dagstuhl Reports, Volume 6, Number 1, pages 128-132. Schloss Dagstuhl--Leibniz-Zentrum für Informatik, Germany

Basis of a Formal Framework for Information Retrieval Evaluation Measurements

Marco Ferrante, Nicola Ferro and Maria Maistro
Workshop Paper In G. M. Di Nunzio, F. M. Nardini, and S. Orlando, editors, Proceedings of the 7th Italian Information Retrieval Workshop, (IIR 2016), Venezia, Italy, May 30-31, 2016, CEUR Workshop Proceedings, Volume 1653.

Abstract

In this paper we present a formal framework, based on the representational theory of measurement and we define and study the properties of utility-oriented measurements of retrieval effectiveness like AP, RBP, ERR and many other popular IR evaluation measures.

Descendants, Ancestors, Children and Parent: A Set-Based Approach to Efficiently Address XPath Primitives

Nicola Ferro and Gianmaria Silvello
Journal Paper Information Processing & Management (IP&M) , 52(3):399-429, 2016.

Abstract

XML is a pervasive technology for representing and accessing semi-structured data. XPath is the standard language for navigational queries on XML documents and there is a growing demand for its efficient processing.

In order to increase the efficiency in executing four navigational XML query primitives, namely descendants, ancestors, children and parent, we introduce a new paradigm where traditional approaches based on the efficient traversing of nodes and edges to reconstruct the requested subtrees are replaced by a brand new one based on basic set operations which allow us to directly return the desired subtree, avoiding to create it passing through nodes and edges.

Our solution stems from the NEsted SeTs for Object hieRarchies (NESTOR) formal model, which makes use of set-inclusion relations for representing and providing access to hierarchical data. We define in-memory efficient data structures to implement NESTOR, we develop algorithms to perform the descendants, ancestors, children and parent query primitives and we study their computational complexity.

We conduct an extensive experimental evaluation by using several datasets: digital archives (EAD collections), INEX 2009 Wikipedia collection, and two widely-used synthetic datasets (XMark and XGen). We show that NESTOR-based data structures and query primitives consistently outperform state-of-the-art solutions for XPath processing at execution time and they are competitive in terms of both memory occupation and pre-processing time.

38th European Conference on IR Research, ECIR 2016

Nicola Ferro, Fabio Crestani, Marie-Francine Moens, Josiane Mothe, Fabrizio Silvestri, Giorgio Maria Di Nunzio, Claudia Hauff, and Gianmaria Silvello
Editorship Proceedings of the Advances in Information Retrieval, Lecture Notes in Computer Science 9626, Springer 2016.

Keyword-based Search over Databases: A Roadmap for a Reference Architecture Paired with an Evaluation Framework

Sonia Bergamaschi, Nicola Ferro, Francesco Guerra and Gianmaria Silvello
Journal Paper Transactions on Computational Collective Intelligence (TCCI), LNCS 9630, vol. 21, pp. 1-20, 2016

Abstract

Structured data sources promise to be the next driver of a significant socio-economic impact for both people and companies. Nevertheless, accessing them through formal languages, such as SQL or SPARQL, can become cumbersome and frustrating for end-users. To overcome this issue, keyword search in databases is becoming the technology of choice, even if it suffers from efficiency and effectiveness problems that prevent it from being adopted at Web scale.

In this paper, we motivate the need for a reference architecture for keyword search in databases to favor the development of scalable and effective components, also borrowing methods from neighbor fields, such as information retrieval and natural language processing. Moreover, we point out the need for a companion evaluation framework, able to assess the efficiency and the effectiveness of such new systems and in the light of real and compelling use cases.

The Twist Measure for IR Evaluation: Taking User’s Effort into Account

Nicola Ferro, Gianmaria Silvello, Heikki Keskustalo, Ari Pirkola and Kalervo Jӓrvelin
Journal Paper Journal of the Association for Information Science and Technology (JASIST), vol. 67, num. 3, pp. 620-648, March 2016.

Abstract

In this paper we present a novel measure for ranking evaluation, called Twist (τ). It is a measure for informational intents, it handles both binary and graded relevance, and it shares the scene mainly with Average Precision (AP), cumulated-gain family of metrics as Discounted Cumulated Gain (DCG), and Rank-Biased Precision (RBP).

The above mentioned metrics adopt different user models but share a common approach: they measure the “utility” of a ranked list for the user and this “utility” is the user motivation for continuing to scan the result list when non-relevant documents are retrieved. The different user models adopted account for the way in which this “utility” (or gain) is computed.

τ stems from a different observation: searching is nowadays a commodity, like water, electricity and the like, and it is natural for users assume that it is available, it fits their needs, it works well. In this sense, they may not perceive the “utility” they have in finding relevant documents but rather they may perceive that the system is just doing what it is expected to do. On the other hand, they may feel uneasy when the system returns non-relevant documents in wrong positions since they are then forced to do additional work to get the desired information, work they would not have expected to do when using a commodity. Thus, τ tries to grasp the avoidable effort caused to the user by the actual ranking of the system with respect to an ideal ranking.

We provide a formal definition of τ as well as a demonstration of its properties. We introduce the notion of effort-gain plots, which allow us to easily spot those systems that look similar from a utility/gain perspective but are actually different in terms of the effort required of their users to attain that utility/gain. Finally, by means of an extensive experimental evaluation with TREC collections, τ is proven not to be highly correlated with existing metrics, to be stable when shallow pools are employed, and to have a good discriminative power.

In short, τ grasps different aspects of system performances with respect to traditional metrics, it does not require extensive and costly assessments, and it is a robust tool for detecting differences between systems.

Digital Library Interoperability at High Level of Abstraction

Maristella Agosti, Nicola Ferro and Gianmaria Silvello
Journal PaperFuture Generation Computer Systems, Volume 55, Pages 129–146, February 2016.

Abstract

Digital Library (DL) are the main conduits for accessing our cultural heritage and they have to address the requirements and needs of very diverse memory institutions, namely Libraries, Archives and Museums (LAM). Therefore, the interoperability among the Digital Library System (DLS) which manage the digital resources of these institutions is a key concern in the field.

DLS are rooted in two foundational models of what a digital library is and how it should work, namely the DELOS Reference Model and the Streams, Structures, Spaces, Scenarios, Societies (5S) model. Unfortunately these two models are not exploited enough to improve interoperability among systems.

To this end, we express these foundational models by means of ontologies which exploit the methods and technologies of Semantic Web and Linked Data. Moreover, we link the proposed ontologies for the foundational models to those currently used for publishing cultural heritage data in order to maximize interoperability.

We design an ontology which allows us to model and map the high level concepts of both the 5S model and the DELOS Reference Model. We provide detailed ontologies for all the domains of such models, namely the user, content, functionality, quality, policy and architectural component domains in order to make available a working tool for making DLS interoperate together at a high level of abstraction. Finally, we provide a concrete use case about digital annotation of illuminated manuscripts to show how to apply the proposed ontologies and illustrate the achieved interoperability between the 5S and DELOS Reference models.

Report on ECIR 2016: 38th European Conference on Information Retrieval

Ferro, N., Crestani, F., Moens, M.-F., Mothe, J., Silvestri, F., Kekäläinen, J., Rosso, P., Clough, P., Pasi, G., Lioma, C., Mizzaro, S., Di Nunzio, G. M., Hauff, C., Alonso, O., Serdyukov, P., and Silvello, G. (2016)
Journal Paper w/o prSIGIR Forum, Volume 50 Issue 1, 2016. ACM New York, NY, USA.

Fast Access to XML Data: A Set-based Approach

Nicola Ferro and Gianmaria Silvello (2016)
Conference Paper In Paolini, P., Bochicchio, M. A., and Mecca, G., editors, Proc. 24th Italian Symposium on Advanced Database Systems (SEBD 2016)

What-If Analysis: A Visual Analytics Approach to Information Retrieval Evaluation

Marco Angelini, Nicola Ferro, Giuseppe Santucci and Gianmaria Silvello (2016)
Workshop PaperProceedings of the 7th Italian Information Retrieval Workshop, IIR 2016. S. Orlando, Di Nunzio, G. M. and Nardini, F. M. Eds., 2016, CEUR Workshop Proceedings.

An Ontology to Make the DELOS Reference Model and the 5S Model Interoperable

M. Agosti, N. Ferro and G. Silvello (2016)
Nat. Conference Paper In Marinai, S., Bertini, M., Orio, N., and Ferilli, S., editors, Proc. 12th Italian Research Conference on Digital Libraries (IRCDL 2016), Communications in Computer and Information Science (CCIS), Springer, Heidelberg, Germany.

IR Scientific Data: How to Semantically Represent and Enrich Them

T. Bogers, G. Bordea, P. Buitelaar, N. Ferro and G. Silvello (2016)
Extended Abstract In Corazza, A., Montemagni, S., and Semeraro, G., editors, Proc. 3rd Italian Conference on Computational Linguistics (CLiC-it 2016).

Descendants, Ancestors, Children and Parent: A Set-Based Approach to Efficiently Address XPath Primitives

Nicola Ferro and Gianmaria Silvello (2015)
Journal Paper Information Processing & Management, accepted for publication.

Keyword-based Search over Databases: A Roadmap for a Reference Architecture Paired with an Evaluation Framework

Sonia Bergamaschi, Nicola Ferro, Francesco Guerra and Gianmaria Silvello (2015)
Journal Paper LNCS Transactions on Computational Collective Intelligence (TCCI), accepted for publication.

The Twist Measure for IR Evaluation: Taking User's Effort Into Account

Ferro, N., Silvello, G., Keskustalo, H., Pirkola, A., and Järvelin, K. (2015)
Journal Paper Journal of the American Society for Information Science and Technology (JASIST)

A Methodology for Citing Linked Open Data Subsets

Gianmaria Silvello
Journal PaperD-Lib Magazine 21 (1/2), 2015, available on-line at the URL: http://www.dlib.org/dlib/january15/silvello/01silvello.html

Visual Analytics for Information Retrieval Evaluation (VAIRË 2015)

Angelini, M., Ferro, N., Santucci, G., and Silvello, G. (2015)
Conference Paper In Fuhr, N., Rauber, A., Kazai, G., and Hanbury, A., editors, Advances in Information Retrieval. Proc. 37th European Conference on IR Research (ECIR 2015), pages 809-812. Lecture Notes in Computer Science (LNCS) 9022, Springer, Heidelberg, Germany

Towards a Formal Framework for Utility-oriented Measurements of Retrieval Effectiveness

Ferrante, M., Ferro, N., and Maistro, M. (2015)
Conference Paper In Allan, J., Croft, W. B., de Vries, A. P., Zhai, C., Fuhr, N., and Zhang, Y., editors, Proc. 1st ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR 2015), pages 21-30. ACM Press, New York, USA

Rank-Biased Precision Reloaded: Reproducibility and Generalization

Ferro, N. and Silvello, G. (2015)
Conference Paper In Fuhr, N., Rauber, A., Kazai, G., and Hanbury, A., editors, Advances in Information Retrieval. Proc. 37th European Conference on IR Research (ECIR 2015), pages 768-780. Lecture Notes in Computer Science (LNCS) 9022, Springer, Heidelberg, Germany

Targeted Query Expansions as a Method for Searching Mixed Quality Digitized Cultural Heritage Documents

Keskustalo, H., Kettunen, K., Kumpulainen, S., Ferro, N., Silvello, G., Järvelin, A., Kekäläinen, J., Arvola, P., Sormunen, E., Järvelin, K., and Saastamoinen, M. (2015)
Conference Paper In Olson, G., Bailey, D., Reddy, M., Caswell, M., Sarcevic, A., Carroll, J. C., Olson, J., Farnham, S. D., Monroy-Hernandez, A., and Semaan, B., editors, Proc. iConference 2015

SVETPET: A Surveillance System of Diseases of Companion Animals in The Veneto Region (Italy)

Martini, M., Agosti, M., Berlanda, M., Candaten, M., Cassini, R., Dotto, G., Drigo, M., Fenati, M., Ferro, N., Finotti, L., Franzo, G., Guglielmini, C., Masiero, I., Mazzotta, E., Menandro, M. L., Pasotto, D., Pietrobelli, M., Signorini, M., Veladiano, I., and Busetto, R. (2015)
Conference Paper In Fejzic, N., Ferizbegovic, J., Besirovic, H., and Grieco, V., editors, Proc. International Congress "One World - One Health - One Vision"

CLEF 2015 Working Notes

Cappellato, L., Ferro, N., Jones, G. J. F., and SanJuan, E., editors (2015)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1391, ISSN 1613-0073

Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Sixth International Conference of the CLEF Association (CLEF 2015)

Mothe, J., Savoy, J., Kamps, J., Pinel-Sauvagnat, K., Jones, G. J. F., SanJuan, E., Cappellato, L., and Ferro, N., editors (2015)
Editorship Lecture Notes in Computer Science (LNCS) 9283, Springer, Heidelberg, Germany

Proc. 1st International Workshop on Multilingual Web Access (MWA 2015)

Steichen, B., Ferro, N., Lewis, D., and Chi, E. H., editors (2015)
Editorship ACM Press, New York, USA

Unfolding Off-the-shelf IR Systems for Reproducibility

Di Buccio, E., Di Nunzio, G. M., Ferro, N., Harman, D. K., Maistro, M., and Silvello, G. (2015)
Workshop Paper In Arguello, J., Diaz, F., Lin, J., and Trotman, A., editors, Proc. SIGIR Workshop on Reproducibility, Inexplicability, and Generalizability of Results (RIGOR 2015)

A Perspective Look at Keyword-based Search Over Relation Data and its Evaluation

Bergamaschi, S., Ferro, N., Guerra, F., and Silvello, G. (2015)
Conference Paper In Atzeni, P., Lenzerini, M., Lembo, D., and Torlone, R., editors, Proc. 23rd Italian Symposium on Advanced Database Systems (SEBD 2015)

The PREFORMA Project: Federating Memory Institutions for Better Compliance of Preservation Formats

Cappellato, L., Ferro, N., Fresa, A., Geber, M., Justrel, B., Lemmen, B., Prandon, C., and Silvello, G. (2015)
Conference Paper In Calvanese, D. and Tasso, C., editors, Proc. 11th Italian Research Conference on Digital Libraries (IRCDL 2015)

Linked Open Data Framework for Serendipity in History of Art Research

Gianmaria Silvello
Workshop Paper1st AI*IA Workshop on Intelligent Techniques At LIbraries and Archives, IT@LIA 2015. S. Ferilli and N. Ferro Eds., 2015.

Markov Precision: Modelling User Behaviour over Rank and Time

Ferrante, M., Ferro, N., and Maistro, M. (2015)
Conference Paper In Boldi, P., Perego, R., and Sebastiani, F., editors, Proc. 6th Italian Information Retrieval Workshop (IIR 2015). CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1404, ISSN 1613-0073

CLEF 2000 - 2014: Lessons Learnt from Ad Hoc Retrieval

Ferro, N. and Silvello, G. (2015)
Conference Paper In Boldi, P., Perego, R., and Sebastiani, F., editors, Proc. 6th Italian Information Retrieval Workshop (IIR 2015). CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1404, ISSN 1613-0073

Improving Information Retrieval Evaluation via Markovian User Models and Visual Analytics

Maria Maistro
Workshop PaperIn L. Azzopardi, M. L. Wilson, I. Kompatsiaris, S. Papadopoulos, T. Tsikrika, S. Vrochidis, editors, Proceedings of the 6th Symposium on Future Directions in Information Access (FDIA 2015), pages 16 - 19, Electronic Workshops in Computing (eWiC).

A Graphical View of Distance Between Rankings: The Point and Area Measures

Giorgio Maria Di Nunzio and Gianmaria Silvello
Workshop PaperProceedings of the 6th Italian Information Retrieval Workshop, IIR 2015. P. Boldi, R. Perego, F. Sebastiani Eds., 2014, CEUR Workshop Proceedings, Volume 1404.

Towards a Semantic Web Enabled Representation of DL Foundational Models: The Quality Domain Example

Ferro, N. and Silvello, G. (2015)
Conference Paper In Calvanese, D. and Tasso, C., editors, Proc. 11th Italian Research Conference on Digital Libraries (IRCDL 2015)

Elementi di Basi di Dati, 3 edizione

Agosti, M. and Ferro, N. (2015)
Book Edizioni Libreria Progetto, Padova, Italia

VIRTUE: A visual tool for information retrieval performance evaluation and failure analysis

Angelini, M., Ferro, N., Santucci, G., and Silvello, G. (2014)
Journal Paper Journal of Visual Languages & Computing (JVLC), 25(4):394-413

CLEF 2014 - Information Access Evaluation meets Multilinguality, Multimodality, and Interaction

Cappellato, L., Clough, P., Ferro, N., Hall, M., Halvey, M., Hanbury, A., Kanoulas, E., Kraaij, W., Lupu, M., Sanderson, M., Toms, E., and Villa, R. (2014)
Journal w/o Peer Review Paper SIGIR Forum, 48(2):55--62

Interaction, Measures and Models

Gianmaria Silvello, Leif Azzopardi, Charles Clarke, Matthias Hagen, and Robert Villa
Journal Paper w/o pr In "Evaluation Methodologies in Information Retrieval", M. Agosti, N. Fuhr, E. Toms and P. Vakkari eds. Dagstuhl Seminar 13441, Dagstuhl Reports 3(10):123–126. Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany. ISSN 2192-5283. 2014.

CLEF 15th Birthday: Past, Present, and Future

Ferro, N. (2014)
Journal w/o Peer Review Paper SIGIR Forum, 48(2):31-55

Comparing Methodologies: Linked Open Data and Digital Libraries

Karen Coyle Gianmaria Silvello and Anna Maria Tammaro
Conference PaperProceedings of the Third AIUCD Annual Conference on Humanities and Their Methods in the Digital Ecosystem (AIUCD '14), Selected Papers. Francesca Tomasi, Roberto Rosselli Del Turco, and Anna Maria Tammaro (Eds.). ACM Press, New York, NY, USA. ISBN: 978-1-4503-3295-8.

A Linked Open Data Approach for Geolinguistics Applications

Emanuele Di Buccio, Giorgio Maria Di Nunzio and Gianmaria Silvello
Journal PaperInternational Journal on Metadata, Semantics and Ontologies (IJMSO), Vol. 9, No. 1, 2014.

Reliability and Validity - A Guide To Best Practices in IR Evaluation

Ferro, N., Joho, H., Kelly, D., Lewandowski, D., Lioma, C., O'Brien, H., Potthast, M., van Rijsbergen, C. J., Thomas, P., Tran, V., and de Vries, A. P. (2014)
Journal w/o Peer Review Paper In Agosti, M., Fuhr, N., Toms, E., and Vakkari, P., editors, Report from Dagstuhl Seminar 13441: Evaluation Methodologies in Information Retrieval, Dagstuhl Reports, Volume 3, Number 10, pages 111-113. Schloss Dagstuhl--Leibniz-Zentrum fuer Informatik, Germany

Making it Easier to Discover, Re-Use and Understand Search Engine Experimental Evaluation Data

Ferro, N. and Silvello, G. (2014)
Journal w/o Peer Review Paper ERCIM News, 96:26-27

A Visual Interactive Environment for Making Sense of Experimental Data

Angelini, M., Ferro, N., Santucci, G., and Silvello, G. (2014)
Conference Paper In de Rijke, M., Kenter, T., de Vries, A. P., Zhai, C., de Jong, F., Radinsky, K., and Hofmann, K., editors, Advances in Information Retrieval. Proc. 36th European Conference on IR Research (ECIR 2014), pages 767-770. Lecture Notes in Computer Science (LNCS) 8416, Springer, Heidelberg, Germany

Injecting User Models and Time into Precision via Markov Chains

Ferrante, M., Ferro, N., and Maistro, M. (2014)
Conference Paper In Geva, S., Trotman, A., Bruza, P., Clarke, C. L. A., and Järvelin, K., editors, Proc. 37th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2014), pages 597-606. ACM Press, New USA

Rethinking How to Extend Average Precision to Graded Relevance

Ferrante, M., Ferro, N., and Maistro, M. (2014)
Conference Paper In Kanoulas, E., Lupu, M., Clough, P., Sanderson, M., Hall, M., Hanbury, A., and Toms, E., editors, Information Access Evaluation - Multilinguality, Multimodality, and Interaction. Proceedings of the Fifth International Conference of the CLEF Initiative (CLEF 2014), pages 19-30. Lecture Notes in Computer Science (LNCS) 8685, Springer, Heidelberg, Germany

CLEF 15th Birthday: What Can We Learn From Ad Hoc Retrieval?

Ferro, N. and Silvello, G. (2014)
Conference Paper In Kanoulas, E., Lupu, M., Clough, P., Sanderson, M., Hall, M., Hanbury, A., and Toms, E., editors, Information Access Evaluation - Multilinguality, Multimodality, and Interaction. Proceedings of the Fifth International Conference of the CLEF Initiative (CLEF 2014), pages 31-43. Lecture Notes in Computer Science (LNCS) 8685, Springer, Heidelberg, Germany

CLEF 2008 Working Notes

Borri, F., Nardi, A., Peters, C., and Ferro, N., editors (2014)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1174, ISSN 1613-0073

CLEF 2009 Working Notes

Borri, F., Nardi, A., Peters, C., and Ferro, N., editors (2014)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1175, ISSN 1613-0073

CLEF 2004 Working Notes

Borri, F., Peters, C., and Ferro, N., editors (2014)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1170, ISSN 1613-0073

CLEF 2010 Working Notes

Braschler, M., Harman, D.~K., Pianta, E., and Ferro, N., editors (2014)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1176, ISSN 1613-0073

CLEF 2014 Working Notes

Cappellato, L., Ferro, N., Halvey, M., and Kraaij, W., editors (2014)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1180, ISSN 1613-0073

Bridging Between Cultural Heritage Institutions - Proc. 9th Italian Research Conference on Digital Libraries (IRCDL 2013)

Catarci, T., Ferro, N., and Poggi, A., editors (2014)
Editorship Communications in Computer and Information Science (CCIS) 385, Springer, Heidelberg, Germany

Bridging Between Information Retrieval and Databases - PROMISE Winter School 2013, Revised Tutorial Lectures

Ferro, N., editor (2014)
Editorship Lecture Notes in Computer Science (LNCS) 8173, Springer, Heidelberg, Germany

CLEF 2012 Working Notes

Forner, P., Karlgren, J., Womser-Hacker, C., and Ferro, N., editors (2014)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1178, ISSN 1613-0073

CLEF 2013 Working Notes

Forner, P., Navigli, R., Tufis, D., and Ferro, N., editors (2014)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1179, ISSN 1613-0073

CLEF 2007 Working Notes

Nardi, A., Peters, C., and Ferro, N., editors (2014)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1173, ISSN 1613-0073

CLEF 2006 Working Notes

Nardi, A., Peters, C., Vicedo, J.~L., and Ferro, N., editors (2014)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1172, ISSN 1613-0073

CLEF 2000 Working Notes

Peters, C. and Ferro, N., editors (2014)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1166, ISSN 1613-0073

CLEF 2001 Working Notes

Peters, C. and Ferro, N., editors (2014)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1167, ISSN 1613-0073

CLEF 2002 Working Notes

Peters, C. and Ferro, N., editors (2014)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1168, ISSN 1613-0073

CLEF 2003 Working Notes

Peters, C. and Ferro, N., editors (2014)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1169, ISSN 1613-0073

CLEF 2005 Working Notes

Peters, C., Quochi, V., and Ferro, N., editors (2014)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1171, ISSN 1613-0073

CLEF 2011 Working Notes

Petras, V., Forner, P., Clough, P., and Ferro, N., editors (2014)
Editorship CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1177, ISSN 1613-0073

Enhancing Digital Cultural Heritage Collections with Social Network Capabilities

Agosti, M., Ferro, N., Porat, S., and Rabinovich, E. (2014)
Workshop Paper In Oomen, J., Aroyo, L., Gena, C., and Wecker, A.~J., editors, Proc. 7th International Workshop on Personalized Access to Cultural Heritage: The Future of Experiencing Cultural Heritage (PATCH 2014)

CULTURA Outcomes for Improving the User's Engagement with Cultural Heritage Collections

Agosti, M., Ferro, N., Orio, N., and Ponchia, C. (2014)
Conference Paper In Agosti, M., Catarci, T., and Esposito, F., editors, Proc. 10th Italian Research Conference on Digital Libraries (IRCDL 2014), Procedia Computer Science, 38:34-39

Measuring and Analyzing the Scholarly Impact of Experimental Evaluation Initiatives

Angelini, M., Ferro, N., Larsen, B., Müller, H., Santucci, G., Silvello, G., and Tsikrika, T. (2014)
Conference Paper In Agosti, M., Catarci, T., and Esposito, F., editors, Proc. 10th Italian Research Conference on Digital Libraries (IRCDL 2014), Procedia Computer Science, 38:133-137

A Vector Space Model for Syntactic Distances Between Dialects

Emanuele Di Buccio and Giorgio Maria Di Nunzio and Gianmaria Silvello
Conference PaperIn Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC '14). European Language Resources Association (ELRA), 2486-2489. ISBN 978-2-9517408-8-4

Enriching Digital Cultural Heritage Collections via Annotations: The CULTURA approach

Agosti, M., Conlan, O., Ferro, N., Hampson, C., Munnelly, G., Ponchia, C., and Silvello, G. (2014)
Conference Paper In Greco, S. and Picariello, A., editors, Proc. 22nd Italian Symposium on Advanced Database Systems (SEBD 2014)

A Curated and Evolving Linguistic Linked Dataset

Emanuele Di Buccio, Giorgio Maria Di Nunzio and Gianmaria Silvello
Journal PaperSemantic Web Journal, 4(3): 265-270, 2013.

NESTOR: A Formal Model for Digital Archives

Ferro, N. and Silvello, G. (2013)
Journal Paper Information Processing & Management, 49(6):1206-1240

PROMISE Winter School 2013: Bridging between Information Retrieval and Databases

Agosti, M., Ferro, N., and Silvello, G. (2013)
Journal w/o Peer Review Paper SIGIR Forum, 47(1):46-52

A Geolinguistic Web Application Based on Linked Open Data

Emanuele Di Buccio, Giorgio Maria Di Nunzio and Gianmaria Silvello
Conference PaperIn Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval (SIGIR '13). ACM, New York, NY, USA, 1101-1102.

PROMISE Winter School 2013 on Bridging between Information Retrieval and Databases

Ferro, N. (2013)
Journal w/o Peer Review Paper ERCIM News, 93:56-57

CLEF 2013: Information Access Evaluation meets Multilinguality, Multimodality, and Visualization

Ferro, N., Forner, P., Müller, H., Navigli, R., Paredes, R., Rosso, P., Stein, B., and Tufis, D. (2013)
Journal w/o Peer Review Paper SIGIR Forum, 47(2):15-20

An Open Source System Architecture for Digital Geolinguistic Linked Open Data

Emanuele Di Buccio, Giorgio Maria Di Nunzio and Gianmaria Silvello
Conference PaperResearch and Advanced Technology for Digital Libraries - International Conference on Theory and Practice of Digital Libraries (TPDL 2013): T. Aalberg, C.Papatheodorou, M. Dobreva, G. Tsakonas, C. J. Farrugia Eds., Lecture Notes in Computer Science 8092, pp. 438-441. Springer Berlin Heidelberg, Germany.

CLEF 2013 and Beyond: Evolution of the CLEF Initiative

Ferro, N. and Rosso, P. (2013)
Journal w/o Peer Review Paper ERCIM News, 95:51-52

PROMISE Technology Transfer Day: Spreading the Word on Information Access Evaluation at an Industrial Event

Forner, P., Bentivogli, L., Braschler, M., Choukri, K., Ferro, N., Hanbury, A., Karlgren, J., and Müller, H. (2013)
Journal w/o Peer Review Paper SIGIR Forum, 47(1):53-58

Interacting with Digital Cultural Heritage Collections via Annotations: The CULTURA Approach

Agosti, M., Conlan, O., Ferro, N., Hampson, C., and Munnelly, G. (2013)
Conference Paper In Marinai, S. and Marriot, K., editors, Proc. 13th ACM Symposium on Document Engineering (DocEng 2013), pages 13-22. ACM Press, New York, USA

Improving Ranking Evaluation Employing Visual Analytics

Angelini, M., Ferro, N., Santucci, G., and Silvello, G. (2013)
Conference Paper In Forner, P., Müller, H., Paredes, R., Rosso, P., and Stein, B., editors, Information Access Evaluation meets Multilinguality, Multimodality, and Visualization. Proceedings of the Fourth International Conference of the CLEF Initiative (CLEF 2013), pages 29-40. Lecture Notes in Computer Science (LNCS) 8138, Springer, Heidelberg, Germany

Formal Models for Digital Archives: NESTOR and the 5S

Ferro, N. and Silvello, G. (2013)
Conference Paper In Aalberg, T., Papatheodorou, C., Dobreva, M., Tsakonas, G., and Farrugia, C. J., editors, Proc. 17th International Conference on Theory and Practice of Digital Libraries (TPDL 2013), pages 192-203. Lecture Notes in Computer Science (LNCS) 8092, Springer, Heidelberg, Germany

The FAST-CAT: Empowering Cultural Heritage Annotations

Munnelly, G., Hampson, C., Ferro, N., and Conlan, O. (2013)
Conference Paper In Walter, K., Price, K., and Nowviskie, B., editors, Proc. Digital Humanities 2013, pages 320-322. Center for Digital Research in Humanities, USA

Cultural Heritage in CLEF (CHiC) 2013

Petras, V., Bogers, T., Hall, M., Savoy, J., Malak, P., Pawlowski, A., Ferro N., and Masiero, I. (2013)
Conference Paper In Forner, P., Müller, H., Paredes, R., Rosso, P., and Stein, B., editors, Information Access Evaluation meets Multilinguality, Multimodality, and Visualization. Proceedings of the Fourth International Conference of the CLEF Initiative (CLEF 2013), pages 192-211. Lecture Notes in Computer Science (LNCS) 8138, Springer, Heidelberg, Germany

Digital Libraries and Archives - Proc. 8th Italian Research Conference (IRCDL 2012)

Agosti, M., Esposito, F., Ferilli, S., and Ferro, N., editors (2013)
Editorship Digital Libraries and Archives - Proc. 8th Italian Research Conference (IRCDL 2012)

Information Retrieval Meets Information Visualization - PROMISE Winter School 2012, Revised Tutorial Lectures

Agosti, M., Ferro, N., Forner, P., Müller, H., and Santucci, G., editors (2013)
Editorship Lecture Notes in Computer Science (LNCS) 7757, Springer, Heidelberg, Germany

Keyword Search and Evaluation over Relational Databases: an Outlook to the Future

Bergamaschi, S., Ferro, N., Guerra, F., and Silvello, G. (2013)
Workshop Paper In Koutrika, G. and Theobald, M., editors, Proc. 7th International Workshop on Ranking in Databases (DBRank 2013), pages 1-3. ACM Press, New York, USA

Cultural Heritage in CLEF (CHiC) 2013 - Multilingual Task Overview

Petras, V., Bogers, T., Ferro, N., and Masiero, I. (2013)
Workshop Paper In Forner, P., Navigli, R., Tufis, D., and Ferro, N., editors, CLEF 2013 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1179, ISSN 1613-0073

Cumulated Relative Position: A Metric for Ranking Evaluation

Angelini, M., Ferro, N., Järvelin, K., Keskustalo, H., Pirkola, A., Santucci, G., and Silvello, G. (2013)
Conference Paper In Basili, R., Sebastiani, F., and Semeraro, G., editors, Proc. 4th Italian Information Retrieval Workshop (IIR 2013), pages 57-60. CEUR Workshop Proceedings (CEUR-WS.org), Vol. 964, ISSN 1613-0073

Visual Interactive Failure Analysis: Supporting Users in Information Retrieval Evaluation

Angelini, M., Ferro, N., Santucci, G., and Silvello, G. (2013)
Conference Paper In Basili, R., Sebastiani, F., and Semeraro, G., editors, Proc. 4th Italian Information Retrieval Workshop (IIR 2013), pages 61-64. CEUR Workshop Proceedings (CEUR-WS.org), Vol. 964, ISSN 1613-0073

A Visual Analytics Tool for Experimental Evaluation

Angelini, M., Ferro, N., Santucci, G., and Silvello, G. (2013)
Conference Paper In Buccafurri, F. and Saccà, D., editors, Proc. 21st Italian Symposium on Advanced Database Systems (SEBD 2013), pages 139–150

(Formal) Models for systems, infrastructures, communities

Ferro, N. (2014)
Conference Paper In Agosti, M. and Tomasi, F., editors, Collaborative Research Practices and Shared Infrastructures for Humanities Computing – Atti del convegno annuale dell’Associazione per l'Informatica Umanistica e la Cultura Digitale (AIUCD 2013), pages 129-134. CLEUP, Padova, Italia, ISBN 978-88-6787-2602

Fostering Interaction with Cultural Heritage Material via Annotations: The FAST-CAT Way

Ferro, N., Munnelly, G., Hampson, C., and Conlan, O. (2014)
Conference Paper In Catarci, T., Ferro, N., and Poggi, A., editors (2014). Bridging Between Cultural Heritage Institutions - Proc. 9th Italian Research Conference on Digital Libraries (IRCDL 2013), pages 41-52. Communications in Computer and Information Science (CCIS) 385, Springer, Heidelberg, Germany

Digital Archives: Extending the 5S Model through NESTOR

Ferro, N. and Silvello, G. (2014)
Conference Paper In Catarci, T., Ferro, N., and Poggi, A., editors (2014). Bridging Between Cultural Heritage Institutions - Proc. 9th Italian Research Conference on Digital Libraries (IRCDL 2013), pages 130-135. Communications in Computer and Information Science (CCIS) 385, Springer, Heidelberg, Germany

PROMISE Retreat Report - Prospects and Opportunities for Information Access Evaluation

Agosti, M., Berendsen, R., Bogers, T., Braschler, M., Buitelaar, P., Choukri, K., Di Nunzio, G. M., Ferro, N., Forner, P., Hanbury, A., Friberg Heppin, K., Hansen, P., Järvelin, A., Larsen, B., Lupu, M., Masiero, I., Müller, H., Peruzzo, S., Petras, V., Piroi, F., de Rijke, M., Santucci, G., Silvello, G., and Toms, E. (2012)
Journal Paper SIGIR Forum, 46(2):60-84

Frontiers, Challenges, and Opportunities for Information Retrieval - Report from SWIRL 2012, The Second Strategic Workshop on Information Retrieval in Lorne, February 2012

Allan, J., Aslam, J., Azzopardi, L., Belkin, N., Borlund, P., Bruza, P., Callan, J., Carman, M. Clarke, C., Craswell, N., Croft, W. B., Culpepper, J. S., Diaz, F., Dumais, S., Ferro, N., Geva, S., Gonzalo, J., Hawking, D., Järvelin, K., Jones, G., Jones, R., Kamps, J., Kando, N., Kanoulas, E., Karlgren, J., Kelly, D., Lease, M., Lin, J., Mizzaro, S., Moffat, A., Murdock, V., Oard, D. W., de Rijke, M., Sakai, T., Sanderson, M., Scholer, F., Si, L., Thom, J., Thomas, P., Trotman, A., Turpin, A., de Vries, A. P., Webber, W., Zhang, X., and Zhang, Y. (2012)
Journal Paper SIGIR Forum, 46(1):2-32

PROMISE Winter School 2012 Information Retrieval meets Information Visualization

Agosti, M., Catarci, T., Ferro, N., Müller, H., and Santucci, G. (2012)
Journal w/o Peer Review Paper SIGIR Forum, 46(1):65-70

A System for Exposing Linguistic Linked Open Data

Emanuele Di Buccio, Giorgio Maria Di Nunzio and Gianmaria Silvello
Conference PaperResearch and Advanced Technology for Digital Libraries - International Conference on Theory and Practice of Digital Libraries (TPDL 2012): Paphos, Cyprus, September 23-27,2012. Springer, Lecture Notes in Computer Science 7489, ISBN: 978-3-642-33289-0, pages 173-178.

DESIRE 2011 Workshop on Data infrastructurEs for Supporting Information Retrieval Evaluation

Agosti, M., Ferro, N., and Thanos, C. (2012)
Journal w/o Peer Review Paper SIGIR Forum, 46(1):51-55

CLEF 2012: Information Access Evaluation meets Multilinguality, Multimodality, and Visual Analytics

Catarci, T., Ferro, N., Forner, P., Hiemstra, D., Karlgren, J., Peñas, A., Santucci, G., and Womser-Hacker, C. (2012)
Journal w/o Peer Review Paper SIGIR Forum, 46(2):29-33

PROMISE Retreat 2012 report on Prospects and Opportunities for Information Access Evaluation

Ferro, N. (2012)
Journal w/o Peer Review Paper ERCIM News, 91:39

CLEF 2012 and beyond: Perspectives for the Conference and Labs of the Evaluation Forum

Ferro, N. (2012)
Journal w/o Peer Review Paper ERCIM News, 91:36-37

The CLEF Initiative: Conference and Labs of the Evaluation Forum

Ferro, N. (2012)
Journal w/o Peer Review Paper ERCIM News, 88:63-64

La gestione e conservazione dei dati scientifici prodotti dalla valutazione sperimentale dei motori di ricerca

Ferro, N. (2012)
Journal Paper Atti e Memorie dell'Accademia Galileana di Scienze Lettere ed Arti in Padova, già Dei Ricoverati e Patavina. Parte II: Memorie della Classe di Scienze Matematiche, Fisiche e Naturali, CXXIV:41-52

DIRECTions: Design and Specification of an IR Evaluation Infrastructure

Agosti, M., Di Buccio, E., Ferro, N., Masiero, I., Peruzzo, S., and Silvello, G. (2012)
Conference Paper In Catarci, T., Forner, P., Hiemstra, D., Peñas, A., and Santucci, G., editors, Information Access Evaluation. Multilinguality, Multimodality, and Visual Analytics. Proceedings of the Third International Conference of the CLEF Initiative (CLEF2012), pages 88-99. Lecture Notes in Computer Science (LNCS) 7488, Springer, Heidelberg, Germany

Information Retrieval Failure Analysis: Visual analytics as a Support for Interactive What-If Investigation

Angelini, M., Ferro, N., Granato, G. L., Santucci, G., and Silvello, G. (2012)
Conference Paper In Santucci, G. and Ward, M., editors, Proc. IEEE Conference on Visual Analytics Science and Technology (VAST 2012), pages 204-206. IEEE Computer Society, Los Alamitos, CA, USA

Cumulated Relative Position: A Metric for Ranking Evaluation

Angelini, M., Ferro, N., Järvelin, K., Keskustalo, H., Pirkola, A., Santucci, G., and Silvello, G. (2012)
Conference Paper In Catarci, T., Forner, P., Hiemstra, D., Peñas, A., and Santucci, G., editors, Information Access Evaluation. Multilinguality, Multimodality, and Visual Analytics. Proceedings of the Third International Conference of the CLEF Initiative (CLEF2012), pages 112-123. Lecture Notes in Computer Science (LNCS) 7488, Springer, Heidelberg, Germany

Visual Interactive Failure Analysis: Supporting Users in Information Retrieval Evaluation

Angelini, M., Ferro, N., Santucci, G., and Silvello, G. (2012)
Conference Paper In Kamps, J., Kraaij, W., and Fuhr, N., editors, Proc. 4th Symposium on Information Interaction in Context (IIiX 2012), pages 195-203. ACM Press, New York, USA

PROMISE Retreat Report - Prospects and Opportunities for Information Access Evaluation

Ferro, N., Berendsen, R., Hanbury, A., Lupu, M., Petras, V., de Rijke, M., and Silvello G., editors (2012)
Editorship SIGIR Forum, 46(2):60-84

Proc. 20th Italian Symposium on Advanced Database Systems (SEBD 2012)

Ferro, N. and Tanca, L., editors (2012)
Editorship Edizioni Libreria Progetto, Padova, Italy, ISBN 978-88-96477-23-6

PROMISE Retreat Report: Prospects and Opportunities for Information Access Evaluation

Agosti, M., Berendsen, R., Bogers, T., Braschler, M., Buitelaar, P., Choukri, K., Di Nunzio, G. M., Ferro, N., Forner, P., Hanbury, A., Friberg Heppin, K., Hansen, P., Järvelin, A., Larsen, B., Lupu, M., Masiero, I., Müller, H., Peruzzo, S., Petras, V., Piroi, F., de Rijke, M., Santucci, G., Silvello, G., and Toms, E. (2012)
book PROMISE network of excellence, ISBN 978-88-6321-039-2

Enciclopedia della Scienza e della Tecnica

AA.VV. (2012)
book Volume VII-VIII. Istituto dell'Enciclopedia italiana fondata da Giovanni Treccani

Cultural Heritage in CLEF (CHiC) Overview 2012

Petras, V., Ferro, ., Gäde, M., Isaac, A., Kleineberg, M., Masiero, I., Nicchio, M., and Stiller, J. (2012)
Workshop Paper In Forner, P., Karlgren, J., Womser-Hacker, C., and Ferro, N., editors, CLEF 2012 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org), Vol. 1178, ISSN 1613-0073

Biblioteche digitali tra modellazione, gestione e valutazione

Agosti, M., Ferro, N., and Silvello, G. (2014)
Conference Paper In Ciotti, F., editor, Digital Humanities: progetti italiani ed esperienze di convergenza multidisciplinare – Atti del convegno annuale dell'Associazione per l’Informatica Umanistica e la Cultura Digitale (AIUCD 2012), pages 33–50. Quaderni DigiLab, Sapienza Università Editrice, Roma, Italy

Empowering Archives Through Annotations

Ferro, N. and Silvello, G. (2013)
Conference Paper In Agosti, M., Esposito, F., Ferilli, S., and Ferro, N., editors, Digital Libraries and Archives - Proc. 8th Italian Research Conference (IRCDL 2012), pages 57-68. Communications in Computer and Information Science (CCIS) 354, Springer, Heidelberg, Germany

Modeling Archives by means of OAI-ORE

Ferro, N. and Silvello, G. (2013)
Conference Paper In Agosti, M., Esposito, F., Ferilli, S., and Ferro, N., editors, Digital Libraries and Archives - Proc. 8th Italian Research Conference (IRCDL 2012), pages 216-227. Communications in Computer and Information Science (CCIS) 354, Springer, Heidelberg, Germany

Per il sistema archivistico regionale

Ferro, N. and Silvello, G. (2012)
Conference Paper In Regione del Veneto, editor, Memoria e innovazione. Nuovi strumenti / Nuove esigenze. Atti della Prima Giornata regionale degli Archivi, pages 91-101. Canova Edizioni, Treviso

CLEF 2011 Conference on Multilingual and Multimodal Information Access Evaluation

Clough, P., Ferro, N., Forner, P., Gonzalo, J., Huurnink, B., Kekäläinen, J., Lalmas, M., Petras, V., and de Rijke, M. (2011)
Journal w/o Peer Review Paper SIGIR Forum, 45(2):32-37

DIRECT: the First Prototype of the PROMISE Evaluation Infrastructure for Information Retrieval Experimental Evaluation

Ferro, N. (2011)
Journal w/o Peer Review Paper ERCIM News, 86:54-55

PROMISE: Advancing the Evaluation of Multilingual and Multimedi Information Systems

Ferro, N. (2011)
Journal w/o Peer Review Paper ERCIM News, 84:49

La rete di eccellenza europea PROMISE

Ferro, N. (2011)
Journal Paper DigItalia, Rivista del digitale nei beni culturali, 6(1):113-122

DESIRE 2011: First International Workshop on Data infrastructurEs for Supporting Information Retrieval Evaluation

Agosti, M., Ferro, N., and Thanos, C. (2011)
Conference Paper In Ounis, I., Ruthven, I., Berendt, B., de Vries, A. P., and Wenfei, F., editors, Proc. 20th International Conference on Information and Knowledge Management (CIKM 2011), pages 2631-2632. ACM Press, New York, USA

To Re-rank or to Re-query: Can Visual Analytics Solve This Dilemma?

Di Buccio, E., Dussin, M., Ferro, N., Masiero, I., Santucci, G., and Tino, G. (2011)
Conference Paper In Forner, P., Gonzalo, J., Kekäläinen, J., Lalmas, M., and de Rijke, M., editors, Multilingual and Multimodal Information Access Evaluation. Proceedings of the Second International Conference of the Cross-Language Evaluation Forum (CLEF 2011), pages 119-130. Lecture Notes in Computer Science (LNCS) 6941, Springer, Heidelberg, Germany

Harnessing the Scientific Data Produced by the Experimental Evaluation of Search Engines and Information Access Systems

Ferro, N., Hanbury, A., Müller, H., and Santucci, G. (2011)
Conference Paper In Sato, M., Matsuoka, S., Sloot, P. M., van Albada, G. D., and Dongarra, J., editors, Proc. 11th International Conference on Computational Science (ICCS 2011), Procedia Computer Science, 4:740-749

Handling Hierarchically Structured Resources Addressing Interoperability Issues in Digital Libraries

Agosti, M., Ferro, N., and Silvello, G. (2011)
Int. Book Chapter In Biba, M. and Xhafa, F., editors, Learning Structure and Schemas from Documents, pages 17-49. Springer-Verlag, Heidelberg, Germany

Proc. Workshop on Data infrastructurEs for Supporting Information Retrieval Evaluation (DESIRE 2011)

Agosti, M., Ferro, N., and Thanos, C., editors (2011)
Editorship ACM Press, New York, USA

Interactive Analysis and Exploration of Experimental Evaluation Results

Di Buccio, E., Dussin, M., Ferro, N., Masiero, I., Santucci, G., and Tino, G. (2011)
Workshop Paper In Wilson, M. L., Russell-Rose, T., Larsen, B., and Kalbach, J., editors, Proc. 1st European Workshop on Human-Computer Interaction and Information Retrieval (EuroHCIR 2011), pages 11-14. CEUR Workshop Proceedings (CEUR-WS.org) Vol. 763, ISSN 1613-0073

Experimental Evaluation and Hypertexts

Ferro, N. (2011)
Workshop Paper In Agosti, M., De Luca, E. W., Lawless, S., and Leveling, J., editors, Proc. First Workshop on Personalised Multilingual Hypertext Retrieval (PMHR 2011), pages 8-9. ACM Press, New York, USA

Visual Comparison of Ranked Result Cumulated Gains

Ferro, N., Sabetta, A., Santucci, G., and Tino, G. (2011)
Workshop Paper In Miksch, S. and Santucci, G., editors, Proc. 2nd International Workshop on Visual Analytics (EuroVA 2011), pages 21-24. Eurographics Association, Goslar, Germany

CHiC 2011 - Cultural Heritage in CLEF: From Use Cases to Evaluation in Practice for Multilingual Information Access to Cultural Heritage

Gäde, M., Ferro, N., and Lestari Paramita, M. (2011)
Workshop Paper In Petras, V., Forner, P., Clough, P., and Ferro, N., editors, CLEF 2011 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) Vol. 1177, ISSN 1613-0073

SIAR: A User-Centric Digital Archive System

Agosti, M., Ferro, N., Rigon, A., Silvello, G., Terenzoni, E., and Tommasi, C. (2011)
Conference Paper In Agosti, M., Esposito, F., Meghini, C., and Orio, N., editors. Digital Libraries and Archives - Proc. 7th Italian Research Conference (IRCDL 2011), pages 87-99. Communications in Computer and Information Science (CCIS) 249, Springer, Heidelberg, Germany

EuropeanaConnect

Crivellari, F., Deambrosis, G., Di Nunzio, G. M., and Ferro, N. (2011)
Conference Paper In Agosti, M., Esposito, F., Meghini, C., and Orio, N., editors. Digital Libraries and Archives - Proc. 7th Italian Research Conference (IRCDL 2011), pages 130-131. Communications in Computer and Information Science (CCIS) 249, Springer, Heidelberg, Germany

PROMISE - Participative Research labOratory for Multimedia and Multilingual Information Systems Evaluation

Di Buccio, E., Dussin, M., Ferro, N., Masiero, I., and Silvello, G. (2011)
Conference Paper In Agosti, M., Esposito, F., Meghini, C., and Orio, N., editors. Digital Libraries and Archives - Proc. 7th Italian Research Conference (IRCDL 2011), pages 140-143. Communications in Computer and Information Science (CCIS) 249, Springer, Heidelberg, Germany

The NESTOR Model: Properties and Applications in the Context of Digital Archives

Ferro, N. and Silvello, G. (2011)
Conference Paper In Mecca, G. and Greco, S., editors, Proc. 19th Italian Symposium on Advanced Database Systems (SEBD 2011), pages 274-285. Università della Basilicata, Italy

CLEF 2010 Conference on Multilingual and Multimodal Information Access Evaluation

Agosti, M., Braschler, M., Choukri, K., Ferro, N., Harman, D. K., Peters, C., Pianta, E., de Rijke, M., and Smeaton, A. (2010)
Journal w/o Peer Review Paper SIGIR Forum, 44(2):8-12

CLEF 2010: Innovation, Science, Experimentation

Ferro, N. (2010)
Journal w/o Peer Review Paper ERCIM News, 83:60

Conferenza CLEF 2010 - Conference on Multilingual and Multimodal Information Access Evaluation

Agosti, M. and Ferro, N. (2010)
Journal Paper DigItalia, Rivista del digitale nei beni culturali, 5(2):176-180

Interoperabilità tra sistemi di biblioteche digitali

Agosti, M. and Ferro, N. (2010)
Journal Paper DigItalia, Rivista del digitale nei beni culturali, 5(1):95-112

A PROMISE for Experimental Evaluation

Braschler, M., Choukri, K., Ferro, N., Hanbury, A., Karlgren, J., Müller, H., Petras, V., Pianta, E., de Rijke, M., and Santucci, G. (2010)
Conference Paper In Agosti, M., Ferro, N., Peters, C., de Rijke, M., and Smeaton, A., editors, Multilingual and Multimodal Information Access Evaluation. Proceedings of the International Conference of the Cross-Language Evaluation Forum (CLEF 2010), pages 140-144. Lecture Notes in Computer Science (LNCS) 6360, Springer, Heidelberg, Germany

CLEF 2009: Grid@CLEF Pilot Track Overview

Ferro, N. and Harman, D. (2010)
Conference Paper In Peters, C., Di Nunzio, G. M., Kurimo, M., Mandl, T., Mostefa, D., Peñas, A., and Roda, G., editors. Multilingual Information Access Evaluation Vol. I Text Retrieval Experiments - Tenth Workshop of the Cross-Language Evaluation Forum (CLEF 2009). Revised Selected Papers, pages 552-565. Lecture Notes in Computer Science (LNCS) 6241, Springer, Heidelberg, Germany

CLEF 2009 Ad Hoc Track Overview: TEL & Persian Tasks

Ferro, N. and Peters, C. (2010)
Conference Paper In Peters, C., Di Nunzio, G. M., Kurimo, M., Mandl, T., Mostefa, D., Peñas, A., and Roda, G., editors. Multilingual Information Access Evaluation Vol. I Text Retrieval Experiments - Tenth Workshop of the Cross-Language Evaluation Forum (CLEF 2009). Revised Selected Papers, pages 13-35. Lecture Notes in Computer Science (LNCS) 6241, Springer, Heidelberg, Germany

Multilingual and Multimodal Information Access Evaluation. Proceedings of the International Conference of the Cross-Language Evaluation Forum (CLEF 2010)

Agosti, M., Ferro, N., Peters, C., de Rijke, M., and Smeaton, A., editors (2010)
Editorship Lecture Notes in Computer Science (LNCS) 6360, Springer, Heidelberg, Germany

Quality and Interoperability: The Quest for the Optimal Balance

Ferro, N. (2010)
Int. Book Chapter In Iglezakis, I., Kapidakis, S., and Synodinou, T., editors, E-Publishing and Digital Libraries: Legal and Organizational Issues, pages 48-68. IGI Global, USA

10 Years of CLEF Data in DIRECT: Where We Are and Where We Can Go

Agosti, M., Di Nunzio, G. M., Dussin, M., and Ferro, N. (2010)
Int. Workshop Chapter In Sakay, T., Sanderson, M., and Webber, W., editors, Proc. 3rd International Workshop on Evaluating Information Access (EVIA 2010), pages 16-24. National Institute of Informatics, Tokyo, Japan

CLEF, CLEF 2010, and PROMISEs: Perspectives for the Cross-Language Evaluation Forum

Ferro, N. (2010)
Int. Workshop Chapter In Kando, N. and Kishida, K., editors, Proc. 8th NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Question Answering and Cross-Lingual Information Access, pages 2-12. National Institute of Informatics, Tokyo, Japan

Quality Interoperability within Digital Libraries: the DL.org Perspective

Vullo, G., Clavel, G., Ferro, N., Higgins, S., van Horik, R., Horstmann, W., Kapidakis, S., and Ross, S. (2010)
Int. Workshop Chapter In Castelli, D., Ioannidis, Y., and Ross, S., editors, Pre-proceedings of the Second DL.org Workshop - Making Digital Libraries Interoperable: Challenges and Approaches, pages 12-24

The NESTOR Framework: Manage, Access and Exchange Hierarchical Data Structures

Agosti, M., Ferro, N., and Silvello, G. (2010)
Conference Paper In Martoglia, R., Bergamaschi, S., Lodi, S., and Sartori, C., editors, Proc. 18th Italian Symposium on Advanced Database Systems (SEBD 2010), pages 242-253. Società Editrice Esculapio, Bologna, Italy

FAST and NESTOR: How to Exploit Annotation Hierarchies

Ferro, N. and Silvello, G. (2010)
Conference Paper In Agosti, M., Esposito, F., and Thanos, C., editors, Digital Libraries. Proceedings of the Sixth Italian Research Conference (IRCDL 2010), pages 55-66. Communications in Computer and Information Science (CCIS) 91, Springer, Heidelberg, Germany

Applying the DIKW Hierarchy to Large-Scale Evaluation Campaigns

Dussin, M. and Ferro, N. (2009)
Journal w/o Peer Review Paper Bulletin of IEEE Technical Committee on Digital Libraries (IEEE-TCDL), 5(1). ISSN 1937-7266

The Role of the DIKW Hierarchy in the Design of a Digital Library System for the Scientific Data of Large- Scale Evaluation Campaigns

Dussin, M. and Ferro, N. (2009)
Journal w/o Peer Review Paper Bulletin of IEEE Technical Committee on Digital Libraries (IEEE-TCDL), 5(1). ISSN 1937-7266

CLEF 2008: Ad Hoc Track Overview

Agirre, E., Di Nunzio, G. M., Ferro, N., Mandl, T., and Peters, C. (2009)
Conference Paper In Peters, C., Deselaers, T., Ferro, N., Gonzalo, J., Jones, G. J. F., Kurimo, M., Mandl, T., and Peñas, A., editors, Evaluating Systems for Multilingual and Multimodal Information Access: Ninth Workshop of the Cross-Language Evaluation Forum (CLEF 2008). Revised Selected Papers, pages 15-37. Lecture Notes in Computer Science (LNCS) 5706, Springer, Heidelberg, Germany

Access and Exchange of Hierarchically Structured Resources on the Web with the NESTOR Framework

Agosti, M., Ferro, N., and Silvello, G. (2009)
Conference Paper In Baeza-Yates, R., Berendt, B., Bertino, E., Lim, E.-P., and Pasi, G., editors, Proc. 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology, pages 659-662. IEEE Computer Society, Los Alamitos, CA, USA

Enabling Cross-Language Access to Archival Metadata

Agosti, M., Ferro, N., and Silvello, G. (2009)
Conference Paper In Cirinnà, C. and Lunghi, M., editors, International Conference on Cultural Heritage Online - Empowering Users: an Active Role for User Communities, pages 179-183. Fondazione Rinascimento Digitale, Firenze, Italy

Managing the Knowledge Creation Process of Large-Scale Evaluation Campaigns

Dussin, M. and Ferro, N. (2009)
Conference PaperBest Student Paper Award In Agosti, M., Borbinha, J., Kapidakis, S., Papatheodorou, C., and Tsakonas, G., editors, Proc. 13th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2009), pages 63-74. Lecture Notes in Computer Science (LNCS) 5714, Springer, Heidelberg, Germany

Annotation Search: The FAST Way

Ferro, N. (2009)
Conference Paper In Agosti, M., Borbinha, J., Kapidakis, S., Papatheodorou, C., and Tsakonas, G., editors, Proc. 13th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2009), pages 15-26. Lecture Notes in Computer Science (LNCS) 5714, Springer, Heidelberg, Germany

The NESTOR Framework: How to Handle Hierarchical Data Structures

Ferro, N. and Silvello, G. (2009)
Conference Paper In Agosti, M., Borbinha, J., Kapidakis, S., Papatheodorou, C., and Tsakonas, G., editors, Proc. 13th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2009), pages 215-226. Lecture Notes in Computer Science (LNCS) 5714, Springer, Heidelberg, Germany

Evaluating Systems for Multilingual and Multimodal Information Access: Ninth Workshop of the Cross–Language Evaluation Forum (CLEF 2008). Revised Selected Papers

Peters, C., Deselaers, T., Ferro, N., Gonzalo, J., Jones, G. J. F., Kurimo, M., Mandl, T., and Peñas, A., editors (2009)
Editorship Lecture Notes in Computer Science (LNCS) 5706, Springer, Heidelberg, Germany

Towards an Evaluation Infrastructure for DL Performance Evaluation

Agosti, M. and Ferro, N. (2009)
Int. Book Chapter In Tsakonas, G. and Papatheodorou, C., editors, Evaluation of Digital Libraries: An Insight to Useful Applications and Methods, pages 93-120. Chandos Publishing, Oxford, UK

CLEF 2009: Grid@CLEF Pilot Track Overview

Ferro, N. and Harman, D. (2009)
Workshop Paper In Borri, F., Nardi, A., Peters, C., and Ferro, N., editors, CLEF 2009 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) Vol. 1175, ISSN 1613-0073

CLEF 2009 Ad Hoc Track Overview: TEL & Persian Tasks

Ferro, N. and Peters, C. (2009)
Workshop Paper In Borri, F., Nardi, A., Peters, C., and Ferro, N., editors, CLEF 2009 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) Vol. 1175, ISSN 1613-0073

Design of an Information Retrieval System Based on the Peer-to-Peer Paradigm: An Application to Music Retrieval

Di Buccio, E., Ferro, N., Melucci, M., Miotto, R., and Orio, N. (2009)
Conference Paper In Agosti, M., Esposito, F., and Thanos, C., editors. Post-proceedings of the Fifth Italian Research Conference on Digital Library Systems (IRCDL 2009), pages 100-107. DELOS Association and Department of Information Engineering of the University of Padua

The FAST Annotation Service

Ferro, N. (2009)
Conference Paper In De Antonellis, V., Castano, S., Catania, B., and Guerrini, G., editors, Proc. 17th Italian Symposium on Advanced Database Systems (SEBD 2009), pages 169-176. Seneca Edizioni, Torino, Italia

CLEF Ad-hoc: A Perspective on the Evolution of the Cross-Language Evaluation Forum

Ferro, N. and Peters, C. (2009)
Conference Paper In Agosti, M., Esposito, F., and Thanos, C., editors. Post-proceedings of the Fifth Italian Research Conference on Digital Library Systems (IRCDL 2009), pages 72-79. DELOS Association and Department of Information Engineering of the University of Padua

Design and Development of the Data Model of a Distributed DLS Architecture for Archive Metadata

Ferro, N. and Silvello, G. (2009)
Conference Paper In Agosti, M., Esposito, F., and Thanos, C., editors. Post-proceedings of the Fifth Italian Research Conference on Digital Library Systems (IRCDL 2009), pages 12-21. DELOS Association and Department of Information Engineering of the University of Padua

Metodologie e percorsi interdisciplinari per la ideazione di un Sistema Informativo Archivistico

Agosti, M., Bonfiglio-Dosio, G., Ferro, N., and Silvello, G. (2008)
Journal Paper Atti e Memorie dell'Accademia Galileana di Scienze Lettere ed Arti in Padova, già Dei Ricoverati e Patavina, CXX:261-287

Annotations: a Way to Interoperability in DL

Agosti, M. and Ferro, N. (2008)
Conference Paper In Christensen-Dalsgaard, B., Castelli, D., Lippincott, J. K., and Ammitzbøll Jurik, B., editors, Proc. 12th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2008), pages 291-295. Lecture Notes in Computer Science (LNCS) 5173, Springer, Heidelberg, Germany

CLEF 2007: Ad Hoc Track Overview

Di Nunzio, G. M., Ferro, N., Mandl, T., and Peters, C. (2008)
Conference Paper In Peters, C., Jijkoun, V., Mandl, T., Müller, H., Oard, D. W., Peñas, A., Petras, V., and Santos, D., editors, Advances in Multilingual and Multimodal Information Retrieval: Eighth Workshop of the Cross-Language Evaluation Forum (CLEF 2007). Revised Selected Papers, pages 13-32. Lecture Notes in Computer Science (LNCS) 5152, Springer, Heidelberg, Germany

DIRECT: Applying the DIKW Hierarchy to Large-Scale Evaluation Campaigns

Dussin, M. and Ferro, N. (2008)
Conference Paper In Larsen, R., Paepcke, A., Borbinha, J. L., and Naaman, M., editors, Proc. 8th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL 2008), page 424. ACM Press, New York, USA

The Role of the DIKW Hierarchy in the Design of a Digital Library System for the Scientific Data of Large-Scale Evaluation Campaigns

Dussin, M. and Ferro, N. (2008)
Conference Paper In Larsen, R., Paepcke, A., Borbinha, J. L., and Naaman, M., editors, Proc. 8th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL 2008), page 450. ACM Press, New York, USA

Design of a Digital Library System for Large-Scale Evaluation Campaigns

Dussin, M. and Ferro, N. (2008)
Conference Paper In Christensen-Dalsgaard, B., Castelli, D., Lippincott, J. K., and Ammitzbøll Jurik, B., editors, Proc. 12th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2008), pages 400-401. Lecture Notes in Computer Science (LNCS) 5173, Springer, Heidelberg, Germany

A Methodology for Sharing Archival Descriptive Metadata in a Distributed Environment

Ferro, N. and Silvello, G. (2008)
Conference Paper In Christensen-Dalsgaard, B., Castelli, D., Lippincott, J. K., and Ammitzbøll Jurik, B., editors, Proc. 12th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2008), pages 268-279. Lecture Notes in Computer Science (LNCS) 5173, Springer, Heidelberg, Germany

GeoCLEF 2007: the CLEF 2007 Cross-Language Geographic Information Retrieval Track Overview

Mandl, T., Gey, F., Di Nunzio, G. M., Ferro, N., Larson, R., Sanderson, M., Santos, D., Womser-Hacker, C., and Xie, X. (2008)
Conference Paper In Peters, C., Jijkoun, V., Mandl, T., Müller, H., Oard, D. W., Peñas, A., Petras, V., and Santos, D., editors, Advances in Multilingual and Multimodal Information Retrieval: Eighth Workshop of the Cross-Language Evaluation Forum (CLEF 2007). Revised Selected Papers, 745-772. Lecture Notes in Computer Science (LNCS) 5152, Springer, Heidelberg, Germany

An Evaluation Resource For Geographic Information Retrieval

Mandl, T., Gey, F., Di Nunzio, G. M., Ferro, N., Sanderson, M., Santos, D., and Womser-Hacker, C. (2008)
Conference Paper In European Language Resources Association (ELRA), editor, Proc. 6th International Language Resources and Evaluation Conference (LREC 2008)

How Robust are Multilingual Information Retrieval Systems?

Mandl, T., Womser-Hacker, C., Di Nunzio, G. M., and Ferro, N. (2008)
Conference Paper In Wainwright, R. L., and Haddad, H. M., editors, Proc. 2008 ACM Symposium on Applied Computing (SAC 2008), pages 1132-1136. ACM Press, New York, USA

From Research to Application in Multilingual Information Access: the Contribution of Evaluation

Peters, C., Braschler, M., Di Nunzio, G. M., Ferro, N., Gonzalo, J., and Sanderson, M. (2008)
Conference Paper In European Language Resources Association (ELRA), editor, Proc. 6th International Language Resources and Evaluation Conference (LREC 2008)

Workshop on Novel Methodologies for Evaluation in Information Retrieval

Sanderson, M., Braschler, M., Ferro, N., and Gonzalo, J. (2008)
Conference Paper In Macdonald, C., Ounis, I., Plachouras, V., Ruthven, I., and White, R. W ., editors, Advances in Information Retrieval. Proc. 30th European Conference on IR Research (ECIR 2008), page 750. Lecture Notes in Computer Science (LNCS) 4956, Springer, Heidelberg, Germany

A Statistical and Graphical Methodology for Comparing Bilingual to Monolingual Cross-Language Information Retrieval

Crivellari, F., Di Nunzio, G. M., and Ferro, N. (2008)
Int. Book Chapter In Agosti, M., editor, Information Access through Search Engines and Digital Libraries, pages 171-188. Springer-Verlag, Heidelberg, Germany

Digital Annotations: a Formal Model and its Applications

Ferro, N. (2008)
Int. Book Chapter In Agosti, M., editor, Information Access through Search Engines and Digital Libraries, pages 113-146. Springer-Verlag, Heidelberg, Germany

CLEF 2008: Ad Hoc Track Overview

Agirre, E., Di Nunzio, G. M., Ferro, N., Mandl, T., and Peters, C. (2008)
Workshop Paper In Borri, F., Nardi, A., Peters, C., and Ferro, N., editors, CLEF 2008 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) Vol. 1174, ISSN 1613-0073

The Design of a DLS for the Management of Very Large Collections of Archival Objects

Agosti, M., Ferro, N., and Silvello, G. (2008)
Workshop Paper In Manghi, P., Pagano, P., and Zezula, P., editors, Proc. 1st Workshop on Very Large Digital Libraries (VLDL 2008). DELOS: an Association for Digital Libraries.

From CLEF to TrebleCLEF: the Evolution of the Cross-Language Evaluation Forum

Ferro, N. and Peters, C. (2008)
Workshop Paper In Kando, N. and Sugimoto, M., editors, Proc. 7th NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Question Answering and Cross-Lingual Information Access, pages 577-593. National Institute of Informatics, Tokyo, Japan

GeoCLEF 2008: the CLEF 2008 Cross-Language Geographic Information Retrieval Track Overview

Mandl, T., Carvalho, P., Gey, F., Larson, R., Santos, D., Womser-Hacker, K., Di Nunzio, G. M., and Ferro, N. (2008)
Workshop Paper In Borri, F., Nardi, A., Peters, C., and Ferro, N., editors, CLEF 2008 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) Vol. 1174, ISSN 1613-0073

Adding Advanced Annotation Functionalities to an Existing Digital Library System

Agosti, M. and Ferro, N. (2008)
Conference Paper In D'Atri, A., De Marco, M., and Casalino, N., editors, Interdisciplinary Aspects of Information Systems Studies, pages 279-286. Physica-Verlag, Heidelberg, Germany

Towards a Superimposed Peer Infrastructure for iNformation Access

Di Buccio, E., Ferro, N., and Melucci, M. (2008)
Conference Paper In Gaglio, S., Infantino, I., and Saccà, D., editors, Proc. 16th Italian Symposium on Advanced Database Systems (SEBD 2008), pages 398-405. Fotograf, Palermo, Italy

Content-based Information Retrieval in SPINA

Di Buccio, E., Ferro, N., and Melucci, M. (2008)
Conference Paper In Agosti, M., Esposito, F., and Thanos, C., editors, Post-proceedings of the Fourth Italian Research Conference on Digital Library Systems (IRCDL 2008), pages 89-92. ISTI-CNR at Gruppo ALI, Pisa, Italy

Building a Distributed Digital Library System Enhancing the Role of Metadata

Gianmaria Silvello
Workshop PaperBCS-IRSG Symposium: Future Directions in Information Access - BCS-IRSG FDIA 2008, in Published as part of the eWiC Series, pp. 46-53, 2008.

The Design of the User Interface of a Scientific DLS in the context of the Data, Information, Knowledge and Wisdom Hierarchy

Dussin, M. and Ferro, N. (2008)
Conference Paper In Agosti, M., Esposito, F., and Thanos, C., editors, Post-proceedings of the Fourth Italian Research Conference on Digital Library Systems (IRCDL 2008), pages 105-113. ISTI-CNR at Gruppo ALI, Pisa, Italy

Dealing with MultiLingual Information Access: Grid Experiments at TrebleCLEF

Ferro, N. and Harman, D. (2008)
Conference Paper In Agosti, M., Esposito, F., and Thanos, C., editors, Post-proceedings of the Fourth Italian Research Conference on Digital Library Systems (IRCDL 2008), pages 29-32. ISTI-CNR at Gruppo ALI, Pisa, Italy

A Distributed Digital Library System Architecture for Archive Metadata

Ferro, N. and Silvello, G. (2008)
Conference Paper In Agosti, M., Esposito, F., and Thanos, C., editors, Post-proceedings of the Fourth Italian Research Conference on Digital Library Systems (IRCDL 2008), pages 99-104. ISTI-CNR at Gruppo ALI, Pisa, Italy

A Formal Model of Annotations of Digital Content

Agosti, M. and Ferro, N. (2007)
Journal Paper ACM Transactions on Information Systems (TOIS), 26(1):3:1-3:57

A Historical and Contemporary Study on Annotations to Derive Key Features for Systems Design

Agosti, M., Bonfiglio-Dosio, G., and Ferro, N. (2007)
Journal Paper International Journal on Digital Libraries, 8(1):1-19

Proposta metodologica e architetturale per la gestione distribuita e condivisa di collezioni di documenti digitali

Agosti, M., Ferro, N., and Silvello, G. (2007)
Journal Paper Archivi, 2(2):49-73

DelosDLMS - the Integrated DELOS Digital Library Management System

Agosti, M., Berretti, S., Brettlecker, G., del Bimbo, A., Ferro, N., Fuhr, N., Keim, D., Klas, C.-P., Lidy, T., Milano, D., Norrie, M., Ranaldi, P., Rauber, A., Schek, H.-J., Schreck, T., Schuldt, H., Signer, B., and Springmann, M. (2007)
Conference Paper In Thanos, C., Borri, F., and Candela, L., editors, Digital Libraries: Research and Development. First International DELOS Conference. Revised Selected Papers, pages 36-45. Lecture Notes in Computer Science (LNCS) 4877, Springer, Heidelberg, Germany

Roadmap for MultiLingual Information Access in The European Library

Agosti, M., Braschler, M., Ferro, N., Peters, C., and Siebinga, S. (2007)
Conference Paper In Fuhr, N., Kovacs, L., and Meghini, C., editors, Proc. 11th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2007), pages 136-147. Lecture Notes in Computer Science (LNCS) 4675, Springer, Heidelberg, Germany

Annotations and Digital Libraries: Designing Adequate Test-beds

Agosti, M., Coppotelli, T., Ferro, N., and Pretto, L. (2007)
Conference Paper In Goh, D. H. L., Cao, T. H., Sølvberg, I., and Rasmussen, E., editors, Proc. 10th International Conference on Asian Digital Libraries. Digital Libraries - Asian Digital libraries: Looking Back 10 Years and Forging New Frontiers (ICADL 2007), pages 150-159. Lecture Notes in Computer Science (LNCS) 4822, Springer, Heidelberg, Germany

An Approach for the Construction of an Experimental Test Collectionto Evaluate Search Systems that Exploit Annotations

Agosti, M., Coppotelli, T., Ferro, N., and Pretto, L. (2007)
Conference Paper In Thanos, C., Borri, F., and Candela, L., editors, Digital Libraries: Research and Development. First International DELOS Conference. Revised Selected Papers, pages 167-176. Lecture Notes in Computer Science (LNCS) 4877, Springer, Heidelberg, Germany

Scientific Data of an Evaluation Campaign: Do We Properly Deal With Them?

Agosti, M., Di Nunzio, G. M., and Ferro, N. (2007)
Conference Paper In Peters, C., Clough, P., Gey, F. C., Karlgren, J., Magnini, B., Oard, D. W., de Rijke, M., and Stempfhuber, M., editors, Evaluation of Multilingual and Multi-modal Information Retrieval: Seventh Workshop of the Cross-Language Evaluation Forum (CLEF 2006). Revised Selected Papers, pages 11-20. Lecture Notes in Computer Science (LNCS) 4730, Springer, Heidelberg, Germany

The Importance of Scientific Data Curation for Evaluation Campaigns

Agosti, M., Di Nunzio, G. M., and Ferro, N. (2007)
Conference Paper In Thanos, C., Borri, F., and Candela, L., editors, Digital Libraries: Research and Development. First International DELOS Conference. Revised Selected Papers, pages 157-166. Lecture Notes in Computer Science (LNCS) 4877, Springer, Heidelberg, Germany

The Future of Large-scale Evaluation Campaigns for Information Retrieval in Europe

Agosti, M., Di Nunzio, G. M., Ferro, N., Harman, D., and Peters, C. (2007)
Conference Paper In Fuhr, N., Kovacs, L., and Meghini, C., editors, Proc. 11th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2007), pages 509-512. Lecture Notes in Computer Science (LNCS) 4675, Springer, Heidelberg, Germany

An Architecture to Share Metadata among Geographically Distributed Archives

Agosti, M., Ferro, N., and Silvello, G. (2007)
Conference Paper In Thanos, C., Borri, F., and Candela, L., editors, Digital Libraries: Research and Development. First International DELOS Conference. Revised Selected Papers, pages 56-65. Lecture Notes in Computer Science (LNCS) 4877, Springer, Heidelberg, Germany

Adding MultiLingual Information Access to The European Library TEL

Braschler, M. and Ferro, N. (2007)
Conference Paper In Thanos, C., Borri, F., and Candela, L., editors, Digital Libraries: Research and Development. First International DELOS Conference. Revised Selected Papers, pages 218-227. Lecture Notes in Computer Science (LNCS) 4877, Springer, Heidelberg, Germany

How to Compare Bilingual to Monolingual Cross-Language Information Retrieval

Crivellari, F., Di Nunzio, G. M., and Ferro, N. (2007)
Conference Paper In Amati, G., Carpineto, C., and Romano, G., editors, Advances in Information Retrieval. Proc. 29th European Conference on IR Research (ECIR 2007), pages 533-540. Lecture Notes in Computer Science (LNCS) 4425, Springer, Heidelberg, Germany

CLEF 2006: Ad Hoc Track Overview

Di Nunzio, G. M., Ferro, N., Mandl, T., and Peters, C. (2007)
Conference Paper In Peters, C., Clough, P., Gey, F. C., Karlgren, J., Magnini, B., Oard, D. W., de Rijke, M., and Stempfhuber, M., editors, Evaluation of Multilingual and Multi-modal Information Retrieval: Seventh Workshop of the Cross-Language Evaluation Forum (CLEF 2006). Revised Selected Papers, pages 21-34. Lecture Notes in Computer Science (LNCS) 4730, Springer, Heidelberg, Germany

GeoCLEF 2006: the CLEF 2006 Cross-Language Geographic Information Retrieval Track Overview

Gey, F., Larson, R., Sanderson, M., Bischoff, K., Mandl, T., Womser-Hacker, K., Santos, D., Rocha, P., Di Nunzio, G. M., and Ferro, N. (2007)
Conference Paper In Peters, C., Clough, P., Gey, F. C., Karlgren, J., Magnini, B., Oard, D. W., de Rijke, M., and Stempfhuber, M., editors, Evaluation of Multilingual and Multi-modal Information Retrieval: Seventh Workshop of the Cross-Language Evaluation Forum (CLEF 2006). Revised Selected Papers, pages 852-876. Lecture Notes in Computer Science (LNCS) 4730, Springer, Heidelberg, Germany

DelosDLMS - the Integrated DELOS Digital Library Management System

Agosti, M., Berretti, S., Brettlecker, G., del Bimbo, A., Ferro, N., Fuhr, N., Keim, D., Klas, C.-P., Lidy, T., Norrie, M., Ranaldi, P., Rauber, A., Schek, H.-J., Schreck, T., Schuldt, H., Signer, B., and Springmann, M. (2007)
Workshop Paper In Thanos, C. and Borri, F., editors, DELOS Conference 2007 Working Notes, pages 71-80. ISTI-CNR, Gruppo ALI, Pisa, Italy

A Study of Web Logs for Personalizing the MultiLingual Information Access to The European Library

Agosti, M., Coppotelli, T., Di Nunzio, G. M., Ferro, N., and van der Meulen, E. (2007)
Workshop Paper In Christodoulakis, S., editor, Workshop on Cross-Media and Personalized Learning Applications on top of Digital Libraries (LADL 2007), pages 19-28

Exploiting Relevance Assessment for the Creation of an Experimental Test Collection to Evaluate Systems that Use Annotations

Agosti, M., Coppotelli, T., Ferro, N., and Pretto, L. (2007)
Workshop Paper In Thanos, C. and Borri, F., editors, DELOS Conference 2007 Working Notes, pages 195-202. ISTI-CNR, Gruppo ALI, Pisa, Italy

The Importance of Scientific Data Curation for Evaluation Campaigns

Agosti, M., Di Nunzio, G. M., and Ferro, N. (2007)
Workshop Paper In Thanos, C. and Borri, F., editors, DELOS Conference 2007 Working Notes, pages 185-193. ISTI-CNR, Gruppo ALI, Pisa, Italy

A Proposal to Extend and Enrich the Scientific Data Curation of Evaluation Campaigns

Agosti, M., Di Nunzio, G. M., and Ferro, N. (2007)
Workshop Paper In Sakay, T. and Sanderson, M., and Evans, D. K., editors, Proc. 1st International Workshop on Evaluating Information Access (EVIA 2007), pages 62-73. National Institute of Informatics, Tokyo, Japan

CLEF: Ongoing Activities and Plans for the Future

Agosti, M., Di Nunzio, G. M., Ferro, N., and Peters, C. (2007)
Workshop Paper In Kando, N. and Evans, D. K., editors, Proc. 6th NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Question Answering and Cross-Lingual Information Access, pages 493-504. National Institute of Informatics, Tokyo, Japan

Towards a Reference Quality Model for Digital Libraries

Agosti, M., Ferro, N., Fox, E. A., Gonçalves, M. A., and Lagoeiro, B. (2007)
Workshop Paper In Castelli, D. and Fox, E. A., editors, Pre-proceedings of the First International Workshop on Foundations of Digital Libraries, pages 37-42.

Modelling DL Quality: a Comparison between Approaches: the DELOS Reference Model and the 5S Model

Agosti, M., Ferro, N., Fox, E. A., and Gonçalves, M. A. (2007)
Workshop Paper In Thanos, C. and Borri, F., editors, Second DELOS Conference on Digital Libraries Working Notes. ISTI-CNR, Gruppo ALI, Pisa, Italy

Design and Development of a Digital Archive of Illuminated Manuscripts

Agosti, M., Ferro, N., and Orio, N. (2007)
Workshop Paper In del Bimbo, A., Cucchiara, R., and Boujemaa, N., editors, Proc. International Workshop on Visual and Multimedia Digital Libraries (WMDL 2007), pages 113-118. IEEE Computer Society, Los Alamitos, CA, USA

An Architecture to Share Metadata among Geographically Distributed Archives

Agosti, M., Ferro, N., and Silvello, G. (2007)
Workshop Paper In Thanos, C. and Borri, F., editors, DELOS Conference 2007 Working Notes, pages 231-239. ISTI-CNR, Gruppo ALI, Pisa, Italy

From CLEF to TrebleCLEF: promoting Technology Transfer for Multilingual Information Retrieval

Braschler, M., Di Nunzio, G. M., Ferro, N., Gonzalo, J., Peters, C., and Sanderson, M. (2007)
Workshop Paper In Thanos, C. and Borri, F., editors, Second DELOS Conference on Digital Libraries Working Notes. ISTI-CNR, Gruppo ALI, Pisa, Italy

Adding MultiLingual Information Access to The European Library TEL

Braschler, M. and Ferro, N. (2007)
Workshop Paper In Thanos, C. and Borri, F., editors, DELOS Conference 2007 Working Notes, pages 39-49. ISTI-CNR, Gruppo ALI, Pisa, Italy

CLEF 2007: Ad Hoc Track Overview

Di Nunzio, G. M., Ferro, N., Mandl, T., and Peters, C. (2007)
Workshop Paper In Nardi, A., Peters, C., and Ferro, N., editors, CLEF 2007 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) Vol. 1173, ISSN 1613-0073

Design of the User Interface of a Scientific Digital Library System for Large-Scale Evaluation Campaigns

Dussin, M. and Ferro, N. (2007)
Workshop Paper In Thanos, C. and Borri, F., editors, Second DELOS Conference on Digital Libraries Working Notes. ISTI-CNR, Gruppo ALI, Pisa, Italy

GeoCLEF 2007: the CLEF 2007 Cross-Language Geographic Information Retrieval Track Overview

Mandl, T., Gey, F., Di~Nunzio, G.~M., Ferro, N., Larson, R., Sanderson, M., Santos, D., Womser-Hacker, C., and Xie, X. (2007)
Workshop Paper In Nardi, A., Peters, C., and Ferro, N., editors, CLEF 2007 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) Vol. 1173, ISSN 1613-0073

The DELOS Digital Library Reference Model. Foundations for Digital Libraries

Candela, L., Castelli, D., Ferro, N., Koutrika, G., Meghini, C., Pagano, P., Ross, S., Soergel, D., Agosti, M., Dobreva, M., Katifori, V., and Schuldt, H. (2007)
book ISTI-CNR at Gruppo ALI, Pisa, Italy

Extending the DelosDLMS by the FAST Annotation Service

Agosti, M., Brettlecker, G., Ferro, N., Ranaldi, P., and Schuldt, H. (2007)
Conference Paper In Agosti, M., Esposito, F., and Thanos, C., editors, Post-proceedings of the Third Italian Research Conference on Digital Library Systems (IRCDL 2007), pages 7-12. ISTI-CNR at Gruppo ALI, Pisa, Italy

Construction of an Annotation Test Collection for Search Evaluation on Digital Libraries

Agosti, M., Coppotelli, T., and Ferro, N. (2007)
Conference Paper In Agosti, M., Esposito, F., and Thanos, C., editors, Post-proceedings of the Third Italian Research Conference on Digital Library Systems (IRCDL 2007), pages 15-17. ISTI-CNR at Gruppo ALI, Pisa, Italy

Distributed Information Retrieval and Automatic Identification of Music Works in SAPIR

Agosti, M., Di Buccio, E., Di Nunzio, G. M., Ferro, N., Melucci, M., Miotto, R., and Orio, N. (2007)
Conference Paper In Ceci, M., Malerba, D., and Tanca, L., editors, Proc. 15th Italian Symposium on Advanced Database Systems (SEBD 2007), pages 479-482. Sagraf srl, Z.I. Capurso, Bari, Italia

Annotations as a Tool for Disclosing Hidden Relationships between Illuminated Manuscripts

Agosti, M., Ferro, N., and Orio, N. (2007)
Conference Paper In Basili, R. and Pazienza, M. T., editors, Proc. 10th Congress of the Italian Association for Artificial Intelligence. Artificial Intelligence and Human-Oriented Computing (AI*IA 2007), pages 662-673. Lecture Notes in Computer Science (LNCS) 4733, Springer, Heidelberg, Germany

An Effective User Interface for a Scientific DLS to Support Information Retrieval Evaluation

Dussin, M. and Ferro, N. (2007)
Conference Paper In Agosti, M., Esposito, F., and Thanos, C., editors, Post-proceedings of the Third Italian Research Conference on Digital Library Systems (IRCDL 2007), pages 18-21. ISTI-CNR at Gruppo ALI, Pisa, Italy

Annotations: What We Learned from DELOS

Ferro, N., Marcante, A., Martin, A., and Panizzi, E. (2007)
Conference Paper In Agosti, M., Esposito, F., and Thanos, C., editors, Post-proceedings of the Third Italian Research Conference on Digital Library Systems (IRCDL 2007), pages 13-14. ISTI-CNR at Gruppo ALI, Pisa, Italy

Elementi di Basi di Dati, 2 edizione

Agosti, M. and Ferro, N. (2007)
Book Edizioni Libreria Progetto, Padova, Italia

Search Strategies for Finding Annotations and Annotated Documents: the FAST Service

Agosti, M. and Ferro, N. (2006)
Conference Paper In Legind Larsen, H., Pasi, G., Ortiz-Arroyo, D., Andreasen, T., and Christiansen, H., editors, Proc. 7th International Conference on Flexible Query Answering Systems (FQAS 2006), pages 270-281. Lecture Notes in Artificial Intelligence (LNAI) 4027, Springer, Heidelberg, Germany

Graph-based Automatic Suggestion of Relationships among Images of Illuminated Manuscripts

Agosti, M., Ferro, N., and Orio, N. (2006)
Conference Paper In Haddad, H. M., Liebrock, K. M., Chbeir, R., Palakal, M. J., Ossowski, S., Yetongnoon, K., Wainwright, R. L., and Nicolle, C., editors, Proc. 21st ACM Symposium on Applied Computing (SAC 2006), pages 1063-1067. ACM Press, New York, USA

Annotation as a Support to User Interaction for Content Enhancement in Digital Libraries

Agosti, M., Ferro, N., Panizzi, E., and Trinchese, R. (2006)
Conference Paper In Celentano, A. and Mussio, P., editors, Proc. Working Conference on Advanced Visual Interfaces (AVI 2006), pages 151-154. ACM Press, New York, USA

Scientific Evaluation of a DLMS: a service for evaluating information access components

Di Nunzio, G. M. and Ferro, N. (2006)
Conference Paper In Gonzalo, J., Thanos, C., Verdejo, M. F., and Carrasco, R. C., editors, Proc. 10th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2006), pages 536-539. Lecture Notes in Computer Science (LNCS) 4172, Springer, Heidelberg, Germany

CLEF 2005: Ad Hoc Track Overview

Di Nunzio, G. M., Ferro, N., Jones, G. J. F., and Peters, C. (2006)
Conference Paper In Peters, C., Gey, F. C., Gonzalo, J., Jones, G. J. F., Kluck, M., Magnini, B., Müller, H., and de Rijke, M., editors, Accessing Multilingual Information Repositories: Sixth Workshop of the Cross-Language Evaluation Forum (CLEF 2005). Revised Selected Papers, pages 11-36. Lecture Notes in Computer Science (LNCS) 4022, Springer, Heidelberg, Germany

DiLAS: a Distributed Service for the Management of Annotations

Agosti, M., Albrechtsen, H., Ferro, N., Frommholz, I., Hansen, P., Panizzi, E., Pejtersen, A. M., and Thiel, U. (2006)
Workshop Paper In Thanos, C., editor, DELOS Research Activities 2006, pages 119-120. ISTI-CNR at Gruppo ALI, Pisa, Italy

A Study on how to Enhance TEL with Multilingual Information Access

Agosti, M., Braschler, M., and Ferro, N. (2006)
Workshop Paper In Thanos, C., editor, DELOS Research Activities 2006, pages 119-120. ISTI-CNR at Gruppo ALI, Pisa, Italy

Scientific Data of an Evaluation Campaign: Do We Properly Deal With Them?

Agosti, M., Di Nunzio, G. M., and Ferro, N. (2006)
Workshop Paper In Nardi, A., Peters, C., Vicedo, J. L., and Ferro, N., editors, CLEF 2006 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) Vol. 1172, ISSN 1613-0073

A Data Curation Approach to Support In-depth Evaluation Studies

Agosti, M., Di Nunzio, G. M., and Ferro, N. (2006)
Workshop Paper In Gey, F. C., Kando, N., Peters, C., and Lin, C.-Y., editors, Proc. International Workshop on New Directions in Multilingual Information Access (MLIA 2006), pages 65-68

Integration of the DiLAS Annotation Service into Digital Library Infrastructures

Agosti, M., Ferro, N., Frommholz, I., Panizzi, E., Putz, W., and Thiel, U. (2006)
Workshop Paper In Blandford, A. and Gow, J., editors, Proc. 1st International Workshop on Digital Libraries in the Context of Users Broader Activities (DL-CUBA 2006), pages 1-4

Implementing MLIA in an existing DL system

Braschler, M., Ferro, N., and Verleyen, J. (2006)
Workshop Paper In Gey, F. C., Kando, N., Peters, C., and Lin, C.-Y., editors, Proc. International Workshop on New Directions in Multilingual Information Access (MLIA 2006), pages 73-76

Queries and Relevance Assessments: The Right Context for the Right Topic

Di Nunzio, G. M. and Ferro, N. (2006)
Workshop Paper In Joho, H., Urban, J., Villa, R., Jose, J. M., and van Rijsbergen, C. J., editors, First International Workshop on Adaptive Information Retrieval (AIR 2006), pages 26-27

CLEF 2006: Ad Hoc Track Overview

Di Nunzio, G. M., Ferro, N., Mandl, T., and Peters, C. (2006)
Workshop Paper In Nardi, A., Peters, C., Vicedo, J. L., and Ferro, N., editors, CLEF 2006 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) Vol. 1172, ISSN 1613-0073

A Study on the User Interface Design for TEL, and on Additional User Services

Ferro, N., Fuhr, N., Panizzi, E., Santucci, G., and Rauber, A. (2006)
Workshop Paper In Thanos, C., editor, DELOS Research Activities 2006, pages 119-120. ISTI-CNR at Gruppo ALI, Pisa, Italy

GeoCLEF 2006: the CLEF 2006 Cross-Language Geographic Information Retrieval Track Overview

Gey, F., Larson, R., Sanderson, M., Bischoff, K., Mandl, T., Womser-Hacker, K., Santos, D., Rocha, P., Di Nunzio, G. M., and Ferro, N. (2006)
Workshop Paper In Nardi, A., Peters, C., Vicedo, J. L., and Ferro, N., editors, CLEF 2006 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) Vol. 1172, ISSN 1613-0073

Uniform Methodology for Evaluating Information Access Components of Digital Libraries

Di Nunzio, G. M. and Ferro, N. (2006)
Conference Paper In Agosti, M. and Thanos, C., editors, Proc. 2nd Italian Research Conference on Digital Library Management Systems (IRCDL 2006), pages 25-28. ISTI-CNR at Gruppo ALI, Pisa, Italy

A Probabilistic Model for Stemmer Generation

Bacchin, M., Ferro, N., and Melucci, M. (2005)
Journal Paper Information Processing & Management, 41(1):121-137

Annotations as Context for Searching Documents

Agosti, M. and Ferro, N. (2005)
Conference PaperBest Student Paper Award In Crestani, F. and Ruthven, I., editors, Proc. 5th International Conference on Conceptions of Library and Information Science - Context: nature, impact and role (Colis 5), pages 155-170. Lecture Notes in Computer Science (LNCS) 3507, Springer, Heidelberg, Germany

A System Architecture as a Support to a Flexible Annotation Service

Agosti, M. and Ferro, N. (2005)
Conference Paper In Türker, C., Agosti, M., and Schek, H.-J., editors, Peer-to-Peer, Grid, and Service-Orientation in Digital Library Architectures: 6th Thematic Workshop of the EU Network of Excellence DELOS. Revised Selected Papers, pages 147-166. Lecture Notes in Computer Science (LNCS) 3664, Springer, Heidelberg, Germany

Annotating Illuminated Manuscripts: an Effective Tool for Research and Education

Agosti, M., Ferro, N., and Orio, N. (2005)
Conference Paper In Marlino, M., Sumner, T., and Shipman, F., editors, Proc. 5th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL 2005), pages 121-130. ACM Press, New York, USA

CLEF 2004: Ad Hoc Track Overview and Results Analysis

Braschler, M., Di Nunzio, G. M., Ferro, N., and Peters, C. (2005)
Conference Paper In Peters, C., Clough, P., Gonzalo, J., Jones, G. J. F., Kluck, M., and Magnini, B., editors, Multilingual Information Access for Text, Speech and Images: Fifth Workshop of the Cross-Language Evaluation Forum (CLEF 2004) Revised Selected Papers, pages 10-26. Lecture Notes in Computer Science (LNCS) 3491, Springer, Heidelberg, Germany

DIRECT: a System for Evaluating Information Access Components of Digital Libraries

Di Nunzio, G. M. and Ferro, N. (2005)
Conference Paper In Rauber, A., Christodoulakis, S., and Min Tjoa, A., editors, Proc. 9th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2005), pages 483-484. Lecture Notes in Computer Science (LNCS) 3652, Springer, Heidelberg, Germany

Experiments on Statistical Approaches to Compensate for Limited Linguistic Resources

Di Nunzio, G. M., Ferro, N., and Orio, N. (2005)
Conference Paper In Peters, C., Clough, P., Gonzalo, J., Jones, G. J. F., Kluck, M., and Magnini, B., editors, Multilingual Information Access for Text, Speech and Images: Fifth Workshop of the Cross-Language Evaluation Forum (CLEF 2004) Revised Selected Papers, pages 60-72. Lecture Notes in Computer Science (LNCS) 3491, Springer, Heidelberg, Germany

DiLAS: a Digital Library Annotation Service

Agosti, M., Albrechtsen, H., Ferro, N., Frommholz, I., Hansen, P., Orio, N., Panizzi, E., Pejtersen, A. M., and Thiel, U. (2005)
Workshop Paper In Boujut, J.-F., editor, Proc. International Workshop on Annotation for Collaboration - Methods, Tools, and Practices (IWAC 2005), pages 91-101. CNRS - Programme société de l'information

Design, Implementation, and Evaluation of the Use of Annotations in Interactive and Collaborative DL Access

Agosti, M., Ferro, N., Albrechtsen, H., Frommholz, I., Panizzi, E., and Thiel, U. (2005)
Workshop Paper In Thanos, C., editor, DELOS Research Activities 2005, pages 51-52. ISTI-CNR, Gruppo ALI, Pisa, Italy

Annotations as a Support to Research Users

Agosti, M., Ferro, N., and Orio, N. (2005)
Workshop Paper In Catarci, T., Christodoulakis, S., and Del Bimbo, A., editors, Proc. 7th International Workshop of the EU Network of Excellence DELOS on Audio-Visual Content and Information Visualization in Digital Libraries (AVIVDiLib'05), pages 117-120. Centromedia, Viareggio, Italy

DIRECT: a Distributed Tool for Information Retrieval Evaluation Campaigns

Di Nunzio, G. M. and Ferro, N. (2005)
Workshop Paper In Ioannidis, Y., Schek, H.-J., and Weikum, G., editors, Proc. 8th DELOS Thematic Workshop on Future Digital Library Management Systems: System Architecture and Information Access, pages 58-63

CLEF 2005: Ad Hoc Track Overview

Di Nunzio, G. M., Ferro, N., Jones, G. J. F., and Peters, C. (2005)
Workshop Paper In Peters, C., Quochi, V., and Ferro, N., editors, CLEF 2005 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) Vol. 1171, ISSN 1613-0073

Design Choices for a Flexible Annotation Service

Ferro, N. (2005)
Conference Paper In Agosti, M. and Thanos, C., editors, Post-proceedings of the First Italian Research Conference on Digital Library Management Systems (IRCDL 2005), pages 101-110. ISTI-CNR at Gruppo ALI, Pisa, Italy

Annotations in Digital Libraries and Collaboratories - Facets, Models and Usage

Agosti, M., Ferro, N., Frommholz, I., and Thiel, U. (2004)
Conference Paper In Heery, R. and Lyon, L., editors, Proc. 8th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2004), pages 244-255. Lecture Notes in Computer Science (LNCS) 3232, Springer, Heidelberg, Germany

Experiments to Evaluate Probabilistic Models for Automatic Stemmer Generation and Query Word Translation

Di Nunzio, G. M., Ferro, N., Melucci, M., and Orio, N. (2004)
Conference Paper In Peters, C., Braschler, M., Gonzalo, J., and Kluck, M., editors, Comparative Evaluation of Multilingual Information Access Systems: Fourth Workshop of the Cross-Language Evaluation Forum (CLEF 2003) Revised Selected Papers, pages 220-235. Lecture Notes in Computer Science (LNCS) 3237, Springer, Heidelberg, Germany

Evaluation of a Digital Library System

Agosti, M., Di Nunzio, G. M., and Ferro, N. (2004)
Workshop Paper In Agosti, M. and Fuhr, N., editors, Notes of the DELOS WP7 Workshop on the Evaluation of Digital Libraries, pages 73-78

An Information Service Architecture for Annotations

Agosti, M. and Ferro, N. (2004)
Workshop Paper In Agosti, M., Schek, H.-J., and Türker, C., editors, Digital Library Architectures: Peer-to-Peer, Grid, and Service-Orientation, Pre-proceedings of the 6th Thematic Workshop of the EU Network of Excellence DELOS, pages 115-126. Edizioni Libreria Progetto, Padova, Italy

The University of Padua at CLEF 2004: Experiments on Statistical Approaches to Compensate for Limited Linguistic Resources

Di Nunzio, G. M., Ferro, N., and Orio, N. (2004)
Workshop Paper In Borri, F., Peters, C., and Ferro, N., editors, CLEF 2004 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) Vol. 1170, ISSN 1613-0073

Formal Model and Conceptual Architecture of the Annotation Service for Dynamic Ubiquitous Knowledge Environments

Ferro, N. (2004)
PhD Thesis PhD thesis, Department of Information Engineering. University of Padua, Italy

A Probabilistic Model for Stemmer Generation

Bacchin, M., Ferro, N., and Melucci, M. (2004)
Conference Paper In Agosti, M., Dessì, N., and Schreiber, F. A., editors, Proc. 12th Italian Symposium on Advanced Database Systems (SEBD 2004), pages 230-237. LITHOSgrafiche, Cagliari, Italy

Improving the Automatic Retrieval of Text Documents

Agosti, M., Bacchin, M., Ferro, N., and Melucci, M. (2003)
Conference Paper In Peters, C., Braschler, M., Gonzalo, J., and Kluck, M., editors, Advances in Cross-Language Information Retrieval: Third Workshop of the Cross-Language Evaluation Forum (CLEF 2002) Revised Papers, pages 279-290. Lecture Notes in Computer Science (LNCS) 2785, Springer, Heidelberg, Germany

Annotations: Enriching a Digital Library

Agosti, M. and Ferro, N. (2003)
Conference Paper In Koch, T. and Sølvberg, I. T., editors, Proc. 7th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2003), pages 88-100. Lecture Notes in Computer Science (LNCS) 2769, Springer, Heidelberg, Germany

Managing the Interactions between Handheld Devices, Mobile Applications, and Users

Agosti, M. and Ferro, N. (2003)
Int. Book Chapter In Lim, E. P. and Siau, K., editors, Advances in Mobile Commerce Technologies, pages 204-233. Idea Group, Hershey, USA

The University of Padova at CLEF 2003: Experiments to Evaluate Probabilistic Models for Automatic Stemmer Generation and Query Word Translation

Di Nunzio, G. M., Ferro, N., Melucci, M., and Orio, N. (2003)
Workshop Paper In Peters, C. and Ferro, N., editors, CLEF 2003 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) Vol. 1169, ISSN 1613-0073

Elementi di Basi di Dati

Agosti, M. and Ferro, N. (2003)
Book Edizioni Libreria Progetto, Padova, Italia

The Effectiveness of a Graph-based Algorithm for Stemming

Bacchin, M., Ferro, N., and Melucci, M. (2002)
Conference Paper In Lim, E. P., Foo, S., Khoo, C. S. G., Chen, H., Fox, E. A., Urs, S. R., and Thanos, C., editors, Proc. 5th International Conference on Asian Digital Libraries. Digital Libraries: People, Knowledge, and Technology (ICADL 2002), pages 117-128. Lecture Notes in Computer Science (LNCS) 2555, Springer, Heidelberg, Germany

University of Padua at CLEF 2002: Experiments to Evaluate a Statistical Stemming Algorithm

Bacchin, M., Ferro, N., and Melucci, M. (2002)
Workshop Paper In Peters, C. and Ferro, N., editors, CLEF 2002 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) Vol. 1168, ISSN 1613-0073