Missing links

Permalink: https://purl.org/cpl/1479

---

The challenges of generating linked data from legacy databases

 

Some time ago I wrote a blog post about the linked data proof of concept project I am involved in, connecting bibliographic metadata from the OPAC of the Library of the University of Amsterdam with the theatre performances database maintained by the Theatre Institute of The Netherlands.
I ended that post with a list of next steps to take:

• select/adapt/create a vocabulary for the Production/Performance subject area
• select/adapt/create vocabularies for Persons (FOAF?) and Subjects (SKOS?)
• add internal relationships with the other entities (Play, Production, etc.) in the JSON structure (implement RDF in JSON)
• Add RDF/XML as output option, besides JSON
• add external relationships (to other linked data sources like DBPedia, etc.)
• extend the number of possible URI formats (for Play, Production, etc.)
• add content negotiation to serve both human and machine readable redirects
• extend the options on the OPAC side
• publish UBA bibliographic data as linked open data (probably an entirely new project)

So, what have we achieved so far? I can be brief about all the ‘real’ linked data stuff (RDF, vocabularies, external links, content negotiation): we are not there yet. This will be dealt with in the next phase.
Instead, we have focused on getting the simple JSON implementation right, both on the data publishing side and on the data using side. We have added more URIs and internal relationships, and we are using these in the OPAC user interface.
But we have also encountered a number of crucial problems that are in my view inherent to the type of legacy data models used in libraries and cultural heritage institutions.

 

Progress

First let me describe the improvements we have added so far.

The URI for ‘person’ <baseurl>/person/<personname> now also returns a link to all the ‘titles’ that person is connected to (not only with the ‘author’ role, but for all roles, like director, performer, etc.): <baseurl>/gettitles/<personname>. This link will return a set of URIs of the form <baseurl>/title/<personname>/<title>. The /<personname>/<title> bit is at the moment the only way that a more or less unique identifier can be constructed from the OPAC metadata for the ‘play’ in the TIN database. There are a number of really important problems related to this that I will discuss below.

The URI:

<baseurl>/person/Beckett, Samuel

returns among others:

/title/Beckett, Samuel/Waiting for Godot
/title/Beckett, Samuel/En attendant Godot
/title/Beckett, Samuel/Endgame
etc.

The URI for a ‘play’ <baseurl>/title/<personname>/<title> now returns a set of ‘production’ URIs of the form <baseurl>/production/<personname>/<title>/<openingdate>/<idnr>.
The ‘production’ URI returns information about ‘theatre company’, ‘venue‘ and all persons connected to that production, including their URIs, and when available also a link to an image of a poster, and a video.

The URI

<baseurl>/title/Beckett, Samuel/Waiting for Godot

returns:

/production/Beckett, Samuel/Waiting for Godot/1988-07-28/5777
/production/Beckett, Samuel/Waiting for Godot/1988-11-22/6750
/production/Beckett, Samuel/Waiting for Godot/1992-04-16/10728
/production/Beckett, Samuel/Waiting for Godot/1981-02-18/43032

The last ‘production’ URI returns:

“name”:”Beckett, Samuel”,
“title”:”Waiting For Godot”,
“opening”:”1981-02-18″,
“people”:
“description”:”Beckett, Samuel (auteur: toneelspel van)”,
“uri”:”/person/Beckett, Samuel”

“description”:”Hartnett, John (regie)”,
“uri”:”/person/Hartnett, John”

“description”:”Muller, Frans (decor: ontwerp)”,
“uri”:”/person/Muller, Frans”

“description”:”Newell, Kym (licht: ontwerp)”,
“uri”:”/person/Newell, Kym”

“description”:”Zaal, Kees (geluid)”,
“uri”:”/person/Zaal, Kees”

“description”:”Tolstoj, Alexander (uitvoerende: Lucky)”,
“uri”:”/person/Tolstoj, Alexander”

“description”:”Weeks, David (uitvoerende: Estragon)”,
“uri”:”/person/Weeks, David”

“description”:”Coburn, Grant (uitvoerende: Vladimir)”,
“uri”:”/person/Coburn, Grant”

“description”:”Evans, Rhys (uitvoerende: Pozzo)”,
“uri”:”/person/Evans, Rhys”

“description”:”Geiringer, Karl (uitvoerende: A Boy)”,
“uri”:”/person/Geiringer, Karl”

“description”:”Guidi, Peter (uitvoering muziek)”,
“uri”:”/person/Guidi, Peter”

“description”:”Kimmorley, Roxanne (uitvoering muziek)”,
“uri”:”/person/Kimmorley, Roxanne”

“description”:”Vries, Hessel de (uitvoering muziek)”,
“uri”:”/person/Vries, Hessel de”

“description”:”Phillips, Margot (uitvoering muziek)”,
“uri”:”/person/Phillips, Margot”

 

Challenges/problems

Now, the problems (or challenges) that we are facing here are essential to the core concept of linked data:

• we don’t have actual matching unique identifiers (URIs)
• we don’t have explicit internal relations with a common entity in both sources
• part of the data consists of literal strings in a specific language

These three problems are interrelated, they are linked problems, so to speak.

 

Missing identifiers

To start with the identifiers. Of course we have internal system identifiers in our local Aleph catalogue database. Because we contribute to the Dutch Union Catalogue (originally a PICA system, now OCLC), our bibliographic records also have national Dutch PICA identifiers. And because the Dutch Union Catalogue records are copied to WorldCat, these records in WorldCat also have OCLC numbers.
Also the Theatre Institute has internal system identifiers in their Adlib database. But at the moment we do not have a match between these separate internal identifier schemes. The Theatre Production database records are not in WorldCat because they’re not bibliographic records.
We are more or less forced to use the string values of the title and author fields to construct a usable URI, on both sides. Clearly this is the basis of lots of errors, because of the great number of possible variations in author and title descriptions.
But even if the Theatre Institute’s records were in the Union Catalogue or WorldCat as well, then we still would not have an automatic match without some kind of broker mechanism ascertaining that the library catalogue record describes the same thing as the theatre production database record. The same applies to the author, which of course should be a relation of the type “written by” between the play and a person record instead of string values. Both systems do have internal author or person authority files, but there is no direct matching. For authors this could theoretically be achieved by linking to an online person authority file like VIAF. But in the current situation this is not available.

 

Missing relations

This brings me to the second problem. The fact that we are using the string values of title instead of unique identifiers, means that we connect plays and productions with a specific title variety or language. In our current implementation this means that we are not able to link to all versions of one specific play.
For instance, from our OPAC the following URIs are constructed (two in English, one in French, one in Dutch):

/title/Beckett, Samuel/Waiting for Godot
/title/Beckett, Samuel/Waiting for Godot : a tragicomedy in two acts
/title/Beckett, Samuel/En attendant Godot : pièce en deux actes
/title/Beckett, Samuel/Wachten op Godot

In the Theatre Production database (two in English, four in Dutch, one in French, one in German):

/title/Beckett, Samuel/Waiting for Godot
/title/Beckett, Samuel/Waiting For Godot
/title/Beckett, Samuel/Wachten op Godot
/title/Beckett, Samuel/Wachtend op Godot
/title/Beckett, Samuel/Wachten op Godot (De favorieten)
/title/Beckett, Samuel/Wachten op Godot (eerste bedrijf)
/title/Beckett, Samuel/En attendant Godot
/title/Beckett, Samuel/Warten auf Godot

Only the first and fourth URI from the OPAC will find corresponding titles in the Theatre Production database. The second and third one, using a subtitle within the main title, don’t even have equivalents. And only two of the eight entries from the Theatre Production database have a match in the catalogue.
In a library catalogue environment we are used to this problem, because catalogues are used for describing physical objects in the form of editions and copies. Unfortunately, also the Theatre Production database just contains records describing productions of a specific ‘edition’ or translation of a play, with only the opening performance information attached.

This is where I need to talk about FRBR. Basically in a library catalogue environment this means that we should describe the relations between the ‘work’ (original text), the ‘expression’ (the version or translation), the ‘manifestation’ (edition, format, etc.) and the ‘items’ (the physical copies). Via the relations with higher level expression and work, the physical copy could be linked to the unifying work level, and then ideally through some universally valid unique identifier to, in our case, the theatre plays.
Although FRBR is a publication centered schema used only in libraries, the same concepts can be applied to theatre performances: the original work (which is the same as the work in a bibliographical sense) has expressions (adaptations, translations, etc.), which have manifestations (productions), and in the end the individual items (actual performances on a specific date, time and location).

If both the library catalogue and the theatre production database were FRBRised, we could in theory link on the Work level and cover all individual versions. But we would still need a matching mechanism on that Work level of course.

In reality however we can only try to link on the Manifestation level in an imperfect way.

At the moment, in our project, on the catalogue side we extract the title and author from the generated OPAC HTML. It could be an option to get available linking information form the internal MARC records (like the 240, 246, 765, 767, 775 tags), but that is not easy because of a number of reasons. Something similar could be done in the theatre production database, making implicit links explicit. But all this makes the effort to get something sensible out there much bigger.

 

Literal strings

The third problem, the literal strings in Dutch both in the library catalogue and in the theatre production database, prevents the effective use of the data in multilingual environments, equally in the traditional native interfaces and as linked data. Obviously for English speaking end users the Dutch terms mean nothing. And in a linked data environment the Dutch strings can’t easily be linked to other data, in Dutch, English, or any language, without unique identifiers.

 

Implicit to explicit

People calling on institutions to publish their data as linked open data tend to say it’s easy once you know how to do it . And of course it must be done. But if the published datasets have a flat internal structure designed to fulfill a specific local business objective, then they just don’t provide sufficient added value for third party use. In order to make your published open data useful for others, you have to make implicit relations explicit. And this requires something more than just making the data available in RDF ‘as is’, it requires a lot of processing.