The core of this experiment is the use of the entity-fishing algorithm, as created and deployed by DARIAH. In the most simple terms: it scans texts for terms that can be linked to Wikipedia pages. Based on the algorithm, new keywords are added to the book descriptions, plus a list of relevant Wikipedia pages.
For this experiment, the full text of 4125 books and chapters – available in the OAPEN Library – is scanned, resulting in a data file of over 25 million entries. In other words, on average the algorithm found roughly 6,100 ‘hits’ for each publication. When only the most common terms per publication are selected, does this result in a useful description of its content?
The data file OK_Computer_results contains a list of open access books and chapters descriptions found in the OAPEN Library, combined with Wikipedia entries found using the entity-fishing algorithm, plus several actions to filter out only the terms which describe the publication best. Each book or chapter is available in the OAPEN Library (www.oapen.org), see the column HANDLE/
The data file nerd_oapen_response_database contains the complete data set. The other text files contain R code to manipulate the file nerd_oapen_response_database.
Description of nerd_oapen_response_database:
The data is divided into the following columns:
Data\tDescription
OAPEN_ID\tUnique ID of the publication in the OAPEN Library
rawName\tThe entity as it appears in the text
nerd_score\tDisambiguation confidence score
nerd_selection_score\tSelection confidence score, indicates how certain the disambiguated entity is actually valid for the text mention
wikipediaExternalRef\tID of the Wikipedia page
wiki_URL\tURL of the Wikipedia page
type\tNER class of the entity
domains\tDescription of subject domain
Each book may contain more than one occurrence of the same entity. The nerd_score and the nerd_selection_score may vary. This allows researchers to count the number of occurrences and use this as an additional method to assess the contents of the book. The OAPEN_ID refers to the identifier of the title in the OAPEN Library.
For more information about the entity-fishing query processing service see https://nerd.readthedocs.io/en/latest/restAPI.html#response.
"}},{"dsDescriptionValue":{"typeName":"dsDescriptionValue","multiple":false,"typeClass":"primitive","value":"Date: 2020-06-03"}}]},{"typeName":"subject","multiple":true,"typeClass":"controlledVocabulary","value":["Arts and Humanities"]},{"typeName":"keyword","multiple":true,"typeClass":"compound","value":[{"keywordValue":{"typeName":"keywordValue","multiple":false,"typeClass":"primitive","value":"Open access books"}},{"keywordValue":{"typeName":"keywordValue","multiple":false,"typeClass":"primitive","value":"entity fishing"}},{"keywordValue":{"typeName":"keywordValue","multiple":false,"typeClass":"primitive","value":"wikipedia"}},{"keywordValue":{"typeName":"keywordValue","multiple":false,"typeClass":"primitive","value":"classification"}},{"keywordValue":{"typeName":"keywordValue","multiple":false,"typeClass":"primitive","value":"Library and information science"}}]},{"typeName":"language","multiple":true,"typeClass":"controlledVocabulary","value":["English"]},{"typeName":"productionDate","multiple":false,"typeClass":"primitive","value":"2020-07-07"},{"typeName":"distributionDate","multiple":false,"typeClass":"primitive","value":"2020-07-07"},{"typeName":"dateOfDeposit","multiple":false,"typeClass":"primitive","value":"2020-07-07"}]},"dansRights":{"displayName":"Rights Metadata","name":"dansRights","fields":[{"typeName":"dansRightsHolder","multiple":true,"typeClass":"primitive","value":["Dr. R. Snijder (OAPEN Foundation)","Dr. R. Snijder (OAPEN Foundation)"]},{"typeName":"dansPersonalDataPresent","multiple":false,"typeClass":"controlledVocabulary","value":"Unknown"},{"typeName":"dansMetadataLanguage","multiple":true,"typeClass":"controlledVocabulary","value":["English"]}]},"dansRelationMetadata":{"displayName":"Relation Metadata","name":"dansRelationMetadata","fields":[{"typeName":"dansAudience","multiple":true,"typeClass":"primitive","value":["https://www.narcis.nl/classification/D30100"],"expandedvalue":[{"@id":"https://www.narcis.nl/classification/D30100","termName":[{"lang":"nl","value":"Digital humanities"},{"lang":"en","value":"Digital humanities"}],"vocabularyUri":"https://www.narcis.nl/classification/"}]}]},"dansTemporalSpatial":{"displayName":"Temporal and Spatial Coverage","name":"dansTemporalSpatial","fields":[{"typeName":"dansTemporalCoverage","multiple":true,"typeClass":"primitive","value":["1954-2019 - Year of publication of the books and chapters listed"]}]},"dansDataVaultMetadata":{"displayName":"Data Vault Metadata","name":"dansDataVaultMetadata","fields":[{"typeName":"dansDataversePid","multiple":false,"typeClass":"primitive","value":"doi:10.17026/DANS-2Z4-MRGM"},{"typeName":"dansDataversePidVersion","multiple":false,"typeClass":"primitive","value":"2.2"},{"typeName":"dansBagId","multiple":false,"typeClass":"primitive","value":"urn:uuid:40431827-68aa-4360-8170-3a39253b614f"},{"typeName":"dansNbn","multiple":false,"typeClass":"primitive","value":"urn:nbn:nl:ui:13-0z-nxq8"}]}},"files":[{"description":"","label":"Code_open_NERD_File_Select_Wikipedia_IDs.txt","restricted":false,"version":1,"datasetVersionId":29962,"dataFile":{"id":16304,"persistentId":"","filename":"Code_open_NERD_File_Select_Wikipedia_IDs.txt","contentType":"text/plain","friendlyType":"Plain Text","filesize":1677,"description":"","storageIdentifier":"surf://store:186dfadfc15-37ce95c37bf9","rootDataFileId":-1,"checksum":{"type":"SHA-1","value":"382bef6d3875aa56a48d2a758b327d8d0b74506e"},"tabularData":false,"creationDate":"2023-03-14","publicationDate":"2020-07-09","lastUpdateTime":"2026-07-19T13:21:10Z","fileAccessRequest":true}},{"description":"","label":"Code_select_German_wikipedia-tile_via_curid.txt","restricted":false,"version":1,"datasetVersionId":29962,"dataFile":{"id":16303,"persistentId":"","filename":"Code_select_German_wikipedia-tile_via_curid.txt","contentType":"text/plain","friendlyType":"Plain Text","filesize":2227,"description":"","storageIdentifier":"surf://store:186dfadfae5-c1cd8c1022df","rootDataFileId":-1,"checksum":{"type":"SHA-1","value":"2005648851af9b7ee05d98fd1fe985220aa42622"},"tabularData":false,"creationDate":"2023-03-14","publicationDate":"2020-07-09","lastUpdateTime":"2026-07-19T13:21:10Z","fileAccessRequest":true}},{"description":"","label":"Code_select_wikipedia-title_via_curid.txt","restricted":false,"version":1,"datasetVersionId":29962,"dataFile":{"id":16306,"persistentId":"","filename":"Code_select_wikipedia-title_via_curid.txt","contentType":"text/plain","friendlyType":"Plain Text","filesize":2798,"description":"","storageIdentifier":"surf://store:186dfae04c9-f5b26c573106","rootDataFileId":-1,"checksum":{"type":"SHA-1","value":"a0590fe56b4b8442ecf16a37d2aef6b191c29308"},"tabularData":false,"creationDate":"2023-03-14","publicationDate":"2020-07-09","lastUpdateTime":"2026-07-19T13:21:10Z","fileAccessRequest":true}},{"description":"","label":"easy-migration.zip","restricted":false,"version":1,"datasetVersionId":29962,"dataFile":{"id":16307,"persistentId":"","filename":"easy-migration.zip","contentType":"application/zip","friendlyType":"ZIP Archive","filesize":18922,"description":"","storageIdentifier":"surf://store:186dfae7028-aa87ce5ed00f","rootDataFileId":16301,"previousDataFileId":16301,"checksum":{"type":"SHA-1","value":"dc832ca37941a1fb42ed2f1268b2ae1ad7ba5eb7"},"tabularData":false,"creationDate":"2023-03-14","publicationDate":"2020-07-09","lastUpdateTime":"2026-07-19T13:21:10Z","fileAccessRequest":true}},{"description":"","label":"nerd_oapen_response_database.csv","restricted":false,"version":1,"datasetVersionId":29962,"dataFile":{"id":16308,"persistentId":"","filename":"nerd_oapen_response_database.csv","contentType":"text/csv","friendlyType":"Comma Separated Values","filesize":2586236965,"description":"","storageIdentifier":"surf://store:186dfaea5ce-b144030068ee","rootDataFileId":-1,"checksum":{"type":"SHA-1","value":"3932671367963df51125d038fb32b404b5aa02be"},"tabularData":false,"creationDate":"2023-03-14","publicationDate":"2020-07-09","lastUpdateTime":"2026-07-19T13:21:10Z","fileAccessRequest":true}},{"description":"","label":"OK_Computer_results.tab","restricted":false,"version":1,"datasetVersionId":29962,"dataFile":{"id":16302,"persistentId":"","filename":"OK_Computer_results.tab","contentType":"text/tab-separated-values","friendlyType":"Tab-Delimited","filesize":11224592,"description":"","storageIdentifier":"surf://store:186dfadf9b8-ab909f40805c","originalFileFormat":"text/csv","originalFormatLabel":"Comma Separated Values","originalFileSize":10416908,"originalFileName":"OK_Computer_results.csv","UNF":"UNF:6:58rEHVhc/OysQoLyXJFviA==","rootDataFileId":-1,"checksum":{"type":"SHA-1","value":"97d0b13ca45180b86779726182bf21f832252f2c"},"tabularData":true,"creationDate":"2023-03-14","publicationDate":"2020-07-09","lastUpdateTime":"2026-07-19T13:21:10Z","fileAccessRequest":true}}]}}}