Methods for designing vocabularies for data on the Web

Over the past year and a half I have been working on a project, in which we were tasked with producing a vocabulary for describing job postings.¹ In doing so, we were expected to write down what worked, so that others can avoid our mistakes. Apart from our own experience, the write-up I prepared took into account the largest public discussion on designing vocabularies for data on the Web. Perusing its archive, I have read every email on the public-vocabs mailing list since its start in June 2011 until April 2014. The following text distills some of what I have learnt from the conversations on this mailing list, especially from the vocabulary design veterans including Dan Brickley or Martin Hepp, coupled with research of other sources and our own experiments in data modelling for the Web.

This work was supported by the project no. CZ.1.04/5.1.01/77.00440 funded by the European Social Fund through the Human Resources and Employment Operational Programme and the state budget of the Czech Republic.

---

“All models are wrong, but some are useful.”
George E. P. Box

The presented text offers a set of recommendations for designing ontologies and vocabularies for data on the Web. The motivation for creating it was to collect relevant advice for data modelling scattered in various sources into a single resource. It focuses on the intersection of vocabularies defined using the RDF Schema (Brickley, Guha, 2014) and those that are intended to be used in RDFa Lite syntax (Sporny, 2012) in HTML web pages. It specifically aims to support vocabularies that aspire to large-scale adoption.

The vocabularies in question in this text are domain-specific, unlike upper ontologies that span general aspects of many domains. Therefore, it is necessary to delimit the domain to be covered by the developed vocabulary to restrict its scope. The target domain can have a broad definition, which may be further clarified by examples of data falling into the domain and examples of data that is out of the domain’s scope. Particular details of the vocabulary’s specialization may be made more specific during the initial research or vocabulary’s design.

Related work research

“Do not reinvent the wheel.”
HTML design principles
(Kesteren, Stachowiak, 2007)

It is appropriate to devote the initial stage of vocabulary development to research and preparation. One may consider three principal kinds of relevant resources that can be pooled when designing a vocabulary. These resources comprise existing data models, knowledge of domain experts, and domain-specific texts.

Existing data models

Research of existing data models helps to prevent unnecessary work by answering two main questions:

1. Is there an available data model that can be reused as a whole instead of developing a new data model?
2. What parts of existing data models can be reused in design of a new data model?

There are two main types of data models that are relevant for reuse in vocabulary development. The first type covers ontological resources that consist of available vocabularies and ontologies. If one finds such resource that describes the target domain and fits the envisioned use cases, it can be directly reused as a whole, provided that its terms of use permit it. If there is a suitable vocabulary that addresses only some of the foreseen uses, it can be extended to cover the others as well. Otherwise, a new vocabulary may be composed of elements that are cherry-picked from the available ontological resources, which forms a basis for the reuse-based development of vocabularies (Poveda-Villalón, 2012). One of the best places to look for these resources is Linked Open Vocabularies, which provides a full-text search engine for the publicly available vocabularies formalized in RDF Schema or OWL (Motik, Patel-Schneider, Parsia, 2012).

The second kind of resources to consider encompasses non-ontological resources, such as XML schemas or data models in relational databases. As these resources cannot be reused directly for building vocabularies, they need to be re-engineered into ontological resources, which is a process that is also referred to as ‘semantic lifting’. Taking non-ontological resources into account may complement the input from ontological sources well. Special attention should be paid to industry standards produced by standardization bodies such as ISO. An alternative approach is to analyze what schemas are employed in public datasets from the given domain, for which data catalogues, such as Datahub, may be used.

Knowledge elicitation with domain experts

“Role models are important.”
Officer Alex J. Murphy / RoboCop

Domain experts constitute a source of implicit knowledge that is not yet formalized in conceptualizations documented in data models (Schreiber et al., 2000). Knowledge elicited from experts who have internalized a working knowledge of the domain of interest can feed in the conceptual distinctions captured by the developed vocabulary. The choice of experts to consult depends on the domain in question. The interviewed experts can range from academic researchers to practitioners from the industry. Similarly, the selection of knowledge elicitation methods should be motivated by the intended use cases for the developed vocabulary. Common methods that serve the purpose of knowledge acquisition include discussion of a glossary, manual simulation of tasks to automate, and competency questions.

Glossary is a useful aid that may guide interviews with domain experts. It can be either manually prepared or constructed automatically from the developed vocabulary. Glossary can be written down as a table in which each vocabulary term is listed together with its label, working definition and broadly described type (e.g., class, property or individuum). It can then serve as a basis for discussion about the established terminology in the domain covered by the developed vocabulary.

Collaboration with domain experts is an opportunity to conduct manual simulation of tasks that are intended to be performed automatically using data described by the developed vocabulary. Such simulation can provide a practical grounding for the vocabulary design with respect to its planned use cases. The simulation should reveal what kinds of data are important for carrying out the envisioned tasks successfully. It can indicate what data can be added to aid in such tasks and what data makes a difference in deciding how to proceed in the chosen tasks. For example, if the target domain is the job market, a simulation task may set about matching sample CVs of job seekers to actual job offers, which can suggest what properties are important to tell a likely successful candidate.

A classical approach to eliciting knowledge from domain experts is to discuss competency questions. These are the questions that data described with the developed vocabulary should be able to answer. As such, competency questions can serve as tests that examine if a vocabulary is sufficiently capable to support its planned use cases. For example, these questions may specify what views on data must be possible, what are the users’ needs that data must be able to answer in a single query, or what level of data granularity and detail is needed.

Analysis of domain-specific corpora

“Pave the cowpaths.”
HTML design principles
(Kesteren, Stachowiak, 2007)

While eliciting knowledge from domain experts concentrates on implicit knowledge, analyses of domain-specific corpora seek for common patterns in explicit, yet unstructured, natural-language text. Textual analysis can be considered a data-driven approach to schema discovery. Its key purpose is to ensure that the designed vocabulary can express the most common kinds of data that are published in the target domain. The approaches to processing domain-specific textual corpora can be divided into qualitative, manual analyses and quantitative, automated analyses.

Qualitative analysis

Manual qualitative analysis can be performed with a smaller domain-specific corpus, which can consist of tens of sample documents. The corpus should be analysed by knowledge engineer to spot common patterns and identify the most important types of data in the domain. Qualitative analysis may result in clusters of similar types of data grouped into a hierarchical tree, in which the most frequently occurring kinds of data are highlighted. The identified clusters may then serve as precursors for classes in the developed vocabulary.

Quantitative analysis

A corpus of texts prepared for quantitative analysis can be sampled from sources on the Web that publish semi-structured data describing the domain of the vocabulary. Producers of these sources can be projected as potential adopters of the developed vocabulary. The texts need to be written in a single language, so that translation is not necessary. Contents of the corpus ought to be sampled from a wide array of diverse sources in order to avoid sampling bias. The corpus needs to be sufficiently large, so that the findings based on analysing it may be taken as indicative of general characteristics of the covered domain. Establishment of such extensive corpus typically requires automated harvesting of texts via web crawlers or scripts that access data through APIs.

Quantitative analysis of domain-specific corpora can be likened to ‘distant reading’. Its aim is to read through the corpus and discover patterns of interest to the vocabulary creator. A typical task of this type of analysis is to extract the most frequent n-grams, indicating common phrases in the established domain terminology, and map their co-occurrences. Quantitative analyses on textual corpora may be performed using dedicated software, such as Voyant Tools or CorpusViewer.

Abstract data model

The results of the performed analyses and knowledge elicitation should provide a basis for development of an abstract data model. At this stage, data model of the designed vocabulary is abstract because it is not mapped to any concrete vocabulary terms in order to avoid being closely tied to particular implementation. Abstract data model may start to be formalized as a mind map, hierarchical tree list or table. Vocabulary creators can base the model on the clusters of the most commonly found terms from domain corpora and sort them into a glossary table. Such proto-model should pass through several rounds of iteration based on successive reviews by vocabulary creators. Key classes and properties in the data model should be identified and equipped with both preferred and non-preferred labels (i.e. synonyms) and preliminary definitions. To get an overview of the whole model and the relationships of its constitutive concepts it may be visualised as UML class diagram or using a generic graph visualization.

Data model’s implementation

“One language’s syntax can be another’s semantics.”
Brian L. Meek

When the abstract data model is deemed to be sound from the conceptual standpoint, it can be formalized in a concrete syntax. The primary languages that should be employed for formalization of the abstract data model are RDF and RDF Schema.As simplicity should be a key design goal, the use of more complex ontological restrictions expressed via OWL ought to be limited to a minimum. The implementation should map the elements of the abstract data model to concrete vocabulary terms that may be either reused from the available ontological resources or newly created.² At this stage, the expressive RDF Turtle (Prud’hommeaux, Carothers, 2014) syntax may be used conveniently to produce a formal specification of the developed vocabulary.

The implementation process should follow an iterative development workflow, using examples of data in place of software prototypes. During each iteration samples of existing data from the vocabulary’s domain may be modelled using the means provided by the vocabulary, so that it can be assessed how the proposed data model fits its intended uses by seeing it being applied in the context of real examples.

General design principles

Implementation of a vocabulary may be guided by several general principles recommended for vocabularies targeting data written in markup that is embedded in HTML web pages. The goal of widespread adoption of the vocabulary on the Web puts an emphasis on specific design principles. Instead of focusing on conceptual clarity and expressivity as in traditional ontologies, the driving principles of design of lightweight web vocabularies accentuate simplicity, ease of adoption, and usability. This section then further discusses some of the key concerns in vocabulary development, including conceptual parsimony, vocabulary’s coverage that is driven by existing data and the like.

Simplicity

Vocabulary should avoid complex ontological axioms and subtle conceptual distinctions. Instead, it ought to seek simplicity for the data producer, rather than the data consumer.³ It is advisable that vocabulary design tries to strike a fine balance between expressivity and implementation complexity cost. Following the principle of minimal ontological commitment (Gruber, 1995), vocabularies should limit the number of ontological axioms (and especially restrictions) to improve their reusability. The developed vocabulary should thus be as simple as possible without sacrificing the leverage its structure gives to data consumers. Nevertheless, not only it should make simple things simple, it should also make complex things possible. Practical vocabulary design can reflect this guideline by focusing on solving simpler problems first and complex problems later.

Ease of adoption

Adoption of a vocabulary may be made easier if the vocabulary builds on common idioms and established terminology that is already familiar to data publishers. Vocabulary design should strive for intuitiveness. In line with the principle of least astonishment, vocabulary users should rather be exposed largely to things that can be expected.

Usability

Vocabulary design should focus on documentation rather than specification. That being said, neither specification nor documentation can ensure correct use of a vocabulary. Even though vocabulary terms may be precisely defined and documented, their meaning is largely established by their use in practice. Nonetheless, correct application of vocabulary terms may be supported by providing good examples showing the vocabulary in use. As Guha (2013) emphasizes, the default mode of authoring structured data on the Web is copy, paste and edit; for which the availability of examples is essential. Usability of vocabularies can be also improved by following the recommendations of cognitive ergonomics (Gavrilova, Gorovoy, Bolotnikova, 2010), such as readable documentation or vocabulary with narrow width and shallow depth.

Conceptual parsimony

Vocabulary design should introduce as few conceptual distinctions as possible, while still producing a useful conceptualization. Vocabulary does not need to include means of expressing data that can be computed or inferred from data expressed by other means. For example, it is not necessary to include a :numberOfOffers property because its value may be computed if there already is a :hasOffer property, which may have its distinct objects counted to arrive to the same data. An exception to this rule is warranted if it is expected that data producers may only have the computed data, but not the primary data from which it was derived from. For example, the number of offers may not available in disaggregated form as the list of individual offers. There is also no need to define inverse properties, such as :isOfferOf for the :hasOffer property. In a similar manner, vocabulary should not require explicit assertion of data that can be recovered from implicit context, such as data types for literal values. On the other hand, it is important to recognize that this approach shifts the burden from data publishers to clients consuming data that need to execute additional computation, such as inference, to materialize implicit data.

In general, additional conceptual distinctions are useful only if vocabulary users are able to apply them consistently. It is important to realize that valuable conceptual distinctions, justified from experts’ perspective, may not lead to more reliable data. Vocabulary creators should mainly concentrate on offering means for describing data that can be reliably provided by a large number of parties. A key reason for adding conceptual distinct is enabling to publish more data.

The merits of conceptual distinctions should be judged based on their discriminatory value. In other words, the value of distinction is in how it differs from the rest of the vocabulary. The more finely or ambiguously a vocabulary term is defined, the more likely it will be used incorrectly. Complex designs are a subject to misinterpretation. If vocabulary terms cannot be understood by data producers with ease and reliably, they will not be used (resulting in less data) or will be used inconsistently (resulting in lower data quality). Therefore, vocabulary should only use conceptual distinctions that matter and are well understood in the target domain.

Data-driven coverage

Since enabling to publish existing data in a structured form is an essential goal of vocabulary development, it ought to be driven by the available data. Data-driven approach implies that vocabularies should not use conceptualizations that do not match well to common database schemas in their target domains. If this is not the case, then data producers do not have a way of providing their data described using the vocabulary unless they alter their database’s schemas and change the way how they collect data. Vocabulary should be rather descriptive than prescriptive. Vocabulary design should be driven by existing data rather than prescribing what data should be published.

Communication interface

Vocabularies should accurately represent the domain they cover only to the degree it improves consistency of vocabulary use. Shared reality mirrored by a vocabulary may serve as a common referent improving shared understanding. However, the prime goal of a vocabulary is not to model the world, but to enable communication that gets a message across, which means that the prime aim of vocabulary is communication instead of representation. For example, structured values such as postal addresses, do not represent reality but they help formalize communication.

Vocabulary defines a communication interface between data producers and data consumers. Data producers are typically people, whereas data consumers are typically machines. Therefore, vocabulary design should balance usability for people with usability for machines. Vocabularies ought to be designed for people first and machines second (The microformats process, 2013). Thus vocabulary design should reflect the trade-off between consistent understanding of vocabulary among people and the degree to which it makes data machine readable.

Syntax limitations

Vocabulary should be aligned with the syntax, in which it is intended to be used. The design of a vocabulary is constrained by expressivity of its intended syntax. For example, HTML5 Microdata’s lack of mechanism for expressing inverse properties, such as with RDFa rev attribute, may warrant adding inverse properties into a vocabulary. Syntax of data can be considered a medium for the vocabulary. In case of vocabularies made for data embedded in web pages, such as Schema.org, their design should correspond to simpler markup. For example, vocabulary should require less nesting.

Tolerant specification

Vocabulary specification should be tolerant about data that it can express. It should not impose a fixed schema. No properties should be required, so that not providing some data is not invalid. On the other hand, vocabulary should allow additional data to be expressed, so that superfluous data is also not invalid, unless it raises a contradiction. It is advisable to use cardinality restrictions for properties only sparingly, as it is difficult to make them generally valid in the broad context of the multicultural Web. Vocabulary should support dynamic data granularity and varying level of detail, so that unstructured text values are allowed to be used in place of structured values if the structure cannot be reconstructed from the source data. On the other hand, specific consumers of data may add specific requirements that may be negotiated on a case by case basis with particular data producers. Overall, data consumers should be expected to follow the spirit of “some data is better than none” (Schema.org: data model, 2012) and accept even broken or partial data.

Vocabulary evolution

If a vocabulary aims for mass adoption, backwards incompatible changes need to be avoided. It is therefore advisable not to remove or deprecate any vocabulary terms, but rather list them as non-preferred with a link to their preferred variant. Large-scale use of a vocabulary raises the cost of changes, because more vocabulary users (both data producers and consumers) need to react to the changes. Widespread adoption increases the difficulty of propagating the changes, because updates about vocabulary changes need to reach a larger audience.

Conclusion

“It’s probably better to allow volcanoes to have fax machines than try to define everything ‘correctly’. Usage will win out in the end.”
Martin Hepp

The methods for designing vocabularies for data on the Web introduced in this text do not form a coherent methodology but instead compile and synthesize recommendations proposed in related work. Guiding principles manifested in the presented methods shall not be considered as hard-and-fast rules but rather as suggestions based on experience of seasoned vocabulary designers. These include both practical advice on researching state of the art in vocabulary’s target domain and concerns to keep in mind when implementing a formal conceptualization for a vocabulary. Moreover, the presented methods do not involve the notion of vocabulary being “right” but instead aim for developing vocabularies that are useful. Therefore, it is only by practical use on the Web in the long-term that these methods and recommendations may be “proved” of being useful themselves.

References

• BRICKLEY, Dan; GUHA, R.V. (eds.). RDF Schema 1.1 [online]. W3C Recommendation 25 February 2014. W3C, 2004-2014 [cit. 2014-04-29]. Available from WWW: http://www.w3.org/TR/rdf-schema/
• GAVRILOVA, T. A.; GOROVOY, V. A.; BOLOTNIKOVA, E. S. Evaluation of the cognitive ergonomics of ontologies on the basis of graph analysis. Scientific and Technical Information Processing. December 2010, vol. 37, iss. 6, p. 398-406. Also available from WWW: http://link.springer.com/article/10.3103%2FS0147688210060043. ISSN 0147-6882. DOI 10.3103/S0147688210060043.
• GRUBER, Thomas R. Toward principles for the design of ontologies used for knowledge sharing? International Journal of Human-Computer Studies. November 1995, vol. 43, iss. 5-6, p. 907-928. Also available from WWW: http://tomgruber.org/writing/onto-design.pdf
• GUHA, R. V. Light at the end of the tunnel [video]. Keynote at 12th International Semantic Web Conference. Sydney, 2013. Also available from WWW: http://videolectures.net/iswc2013_guha_tunnel
• KESTEREN, Anne van; STACHOWIAK, Maciej (eds.). HTML design principles [online]. W3C Working Draft 26 November 2007. W3C, 2007 [cit. 2014-04-29]. Available from WWW: http://www.w3.org/TR/html-design-principles/
• MOTIK, Boris; PATEL-SCHNEIDER, Peter F.; PARSIA, Bijan (eds.). OWL 2 Web Ontology Language: structural specification and functional-style syntax [online]. W3C Recommendation 11 December 2012. 2^nd ed. W3C, 2012 [cit. 2014-04-29]. Available from WWW: http://www.w3.org/TR/owl2-syntax/
• POVEDA-VILLALÓN, María. A reuse-based lightweight method for developing linked data ontologies and vocabularies. In Proceedings of the 9^th Extended Semantic Web Conference, Heraklion, Crete, Greece, May 27-31, 2012. Berlin; Heidelberg: Springer, 2012, p. 833-837. Lecture notes in computers science, vol. 7295. Also available from WWW: http://link.springer.com/chapter/10.1007%2F978-3-642-30284-8_66. ISSN 0302-9743. DOI 10.1007/978-3-642-30284-8_66.
• PRUD’HOMMEAUX, Eric; CAROTHERS, Gavin (eds.). RDF 1.1 Turtle: terse RDF triple language [online]. W3C Recommendation 25 February 2014. W3C, 2008-2014 [cit. 2014-04-30]. Available from WWW: http://www.w3.org/TR/turtle/
• Schema.org: data model [online]. June 6^th, 2012 [cit. 2014-04-29]. Available from WWW: http://schema.org/docs/datamodel.html
• SCHREIBER, Guus [et al.] (eds.). Knowledge elicitation techniques. In Knowledge engineering and management: the CommonKADS methodology. Cambridge (MA): MIT, 2000, p. 187-214. ISBN 0-262-19300-0.
• SPORNY, Manu. RDFa Lite 1.1 [online]. W3C Recommendation 07 June 2012. W3C, 2012 [cit. 2014-04-29]. Available from WWW: http://www.w3.org/TR/rdfa-lite/
• The microformats process [online]. April 28^th, 2013 [cit. 2014-04-29]. Available from WWW: http://microformats.org/wiki/process

Footnotes

1. The result of this endeavour can be found here: https://github.com/OPLZZ/data-modelling ↩

2. Those may be in turn mapped to other vocabularies’ terms; e.g., via rdfs:subClassOf. ↩

3. However, it must be possible to reconstruct the main data structures; at least from its context and without out-of-band knowledge. ↩

Vocabularies for the web of data and principles of least markup

I want to share a few thoughts about markup vocabularies that I pondered upon in the past months when developing Schema.org extension proposal targetting the long tail of job market. Schema.org is a prime example of markup vocabulary. In fact, if you search Google for “markup vocabulary”, most results will associate the term with Schema.org. Throughout this post, I’ll use this vocabulary as an example to illustrate the points made.

So, how does a markup vocabulary differ from, say, an ontology? Markup vocabularies serve different purposes than traditional ontologies, albeit their uses overlap. While the distinction between vocabularies and ontologies is blurry, it can be said that ontologies are based on logic, whereas vocabularies are based on convention. Ontologies are typically used for tasks such as inferring additional data, whereas vocabularies serve rather as structures for easier parsing when exchanging data. Alex Shubin likened Schema.org, as an example of markup vocabulary, to a set of “sitemaps for content” (source). Whereas sitemaps serve to machines to find pages within a web site, Schema.org serves to machines to find the bits of content within a web page.

In practice, vocabularies are used for the less orderly data. As Dan Brickley said in one of his talks: “Schema.org is for the rest of the Web; for that big sprawling chaos.” To reach wide adoption vocabularies need to be generic and application-agnostic so that they can be applied at the largest scale possible. So, as asked at the Schema.org panel discussion, “how does schema design at a planetary scale work, in practice?” I think the answer to this question may be approached from two complementary angles: the recommended vocabulary design patterns and markup guidance.

Vocabulary design

While there is a lot of methodologies for developing ontologies (such as NeOn Methodology or METHONTOLOGY), it seems that similar instructions are lacking for vocabularies. It is unclear what such instructions should be based on. Whereas design of artefacts is frequently shaped by their anticipated uses in practice, so that their form follows function, successful vocabularies are often those that don’t anticipate any particular uses. Their function is defined in terms of a broad goal of supporting the widest possible use. And this isn’t an easy goal to provide workable recommendations for.

I think one common rule of thumb is that vocabulary designers should strive to the lower cognitive overhead users face when working with vocabularies and focus on improving vocabulary usability. However, how do these nebulous goals translate into practice?

One (slightly less vague) advice is that vocabulary design shouldn’t require users to make difficult conceptual distinctions. In order to achieve that, make the differences between vocabulary terms clear (using clear labels and descriptions) in order to avoid ambiguity. If users regularly mix up two distinct concepts, either drop one of the concepts or provide the concepts with better definition. As the Zen of Python states on a similar note, “there should be one— and preferably only one —obvious way to do it.”

Another (slightly less vague) advice is to avoid object proliferation in the vocabulary you develop. In his talk from May 2013 Richard Cyganiak mentioned that vocabularies are typically built from bottom to top, based on usage evidence, so that unused object aren’t included. Richard reiterated the claim asserting that successful vocabularies for the web of data are small and simple (such as Dublin Core Terms), which was already presented by Martin Hepp in his account of Possible ontologies.

One practical technique in line with this advice is to avoid intermediate resources, which are typically represented with blank nodes, and are often needed for object properties. Schema.org labels such intermediate objects as “embedded items”. If your vocabulary contains an object property that points to an intermediate object further described with other properties and all these properties have 0…1 cardinality, then you may consider redefining them as direct properties of the object property’s subject. For example, the class schema:JobPosting is used with properties schema:baseSalary and schema:salaryCurrency. These properties could have been associated with an intermediate schema:Salary object, or even with 2 intermediate objects schema:JobPosition and schema:Salary, however, they are instead attached as direct properties of the schema:JobPosting class. Be careful though not to take this as a catch-all rule. Object properties that usually link to URIs, such as schema:hiringOrganization links to schema:Organization, don’t need to be treated in this way.

Markup guidance

Judging from the documentation of markup vocabularies, much of the presented guidance revolves around markup rather than vocabulary design. For instance, the guidance on designing vocabularies for HTML provided by W3C focuses on issues of markup syntax. I think a lot of recommendations concerning markup for the web of data can be considered extensions of the principle of least effort, so calling them the principles of least markup sounds about right.

A practical realization of such principles might advise to omit data that can be computed automatically. This guidance might encompass omitting inferrable types, including class instantiations and literal datatypes, when there’s only one valid option. Note that this approach doesn’t apply in cases when “type” or “unit” needs to be provided to serve as a value reference; for example when describing price and its currency. A more controversial extension of this principle might recommend to avoid forcing users to mint their own URIs unless necessary. For many purposes of data on the Web anonymous nodes represented with blank nodes are sufficient, given that they may be transformed to URIs and linked deterministically during data ingest and subsequent processing.

Besides decreasing the number of characters that users need to type to add vocabulary markup, there are few recurrent issues frequently mentioned in markup advice.

A thorny issue is the single namespace policy, which proposes that users should be able to create markup with a single vocabulary only. This recommendation is based on the assumption that having multiple vocabulary namespaces requires users to shift between multiple contexts of different vocabularies, which is held to be cognitively demanding. For example, Schema.org aims to provide this single all-encompassing namespace, from which every necessary vocabulary term may be drawn. Single namespace policy is also reflected in RDFa’s vocab attribute that enables to specify a single namespace, which is then applied to all unqualified names used in markup.

When looking for the source of errors in markup, unclear scoping rules are often to blame. Scoping is governed by rules prescribing what subject should the properties in markup be attached to, based on positioning of attribute-value pairs in the hierarchical structure of HTML and semantic context as set by other markup. The scoping rules are notoriously difficult to grasp, which might have contributed to Microdata having the itemscope attribute that sets the scope explicitly.

A related issue to scoping is directionality, which prescribes whether the current scope should be used as subject or object of marked up properties. To reverse the default directionality RDFa offers the rev attribute and previously, it used reverse direction for src attribute as well. Directionality, among other issues, is described by Gregg Kellogg in his list of common pitfalls when marking up HTML with RDFa. Microdata, on the other hand, avoids this issue by being uni-directional.

Tolerance

Markup guidelines for data publishers should have a counterpart on the side of data consumers. That counterpart is the principle of tolerance. Schema.org documentation of its data model states: “In the spirit of ‘some data is better than none’, we will accept this markup and do the best we can.” Even though markup may be broken in many different ways, data consumers should try to be fault-tolerant. This attitude is in line with Postel’s principle of robustness that states: “Be conservative in what you send, be liberal in what you accept” And so I think that until we know better about vocabulary design, we better be tolerant and liberal about data on the Web.

Towards usability metrics for vocabularies

As far as I know, there are no established measures for evaluating vocabulary usability. To clarify, when I use the term “vocabularies”, what I mean simple schemas and lightweight lexical ontologies that are used primarily for marking up content embedded in web pages, using syntaxes such as Microdata or RDFa. A good example of such vocabulary is Schema.org, an overarching, yet simple schema of things and relations that four big search engines (Google, Microsoft Bing, Yahoo! and Yandex) deem to be important to their users.

The closest to the topic seems to be the paper Ontology evaluation through usability measures by Núria Casellas. With regards to syntactical usability of markup, there was a usability study of Microdata done by Ian Hickson, the minimalistic settings of which was a subject of numerous rants, such as the one by Manu Sporny. I presume more thought needs to be spent on discovering how existing usability research relates to vocabularies and which standard usability principles apply. Nevertherless, borrowing from usability testing used for web sites or software or in libraries, three metrics relevant to vocabularies crossed my mind.

The first is error rate when using a vocabulary. It is based on the assumption that the more usable vocabulary is the fewer errors should its users make. Vocabulary validators may be used to automate this technique. Such tools may execute fine-grained rules, which may help to discern the most problematic parts of vocabularies, where users make the most errors. An example of a study testing error rate was conducted by Yandex. Note, however, that it focused more on markup syntaxes, rather than vocabularies themselves. It reported 10 % error rate in RDFa (4 % share in the sample), 10 % error rate in hCard (20 % share) and almost no errors in Facebook’s Open Graph Protocol (1.5 % share), which is also RDFa.

A broader feature that may serve as input to usability testing is data quality. A metric based on data quality should primarily take into account valid data, since invalid data should be caught by error rate testing. Recognizing that data quality as a relevant feature is based on the assumption that more usable vocabularies support creating data of better quality. However, the relation between vocabulary usability and data quality should not be considered as causation, but rather correlation, which might pinpoint weak parts of vocabulary where data quality suffers. Transforming data quality into a discrete metric is tricky, but there already are data quality assessment methodologies, some of which are documented in this paper (PDF), on which test procedures for usability of vocabularies may be derived.

The remaining metric I propose is adopted from library and information science, in which quality of indexing (much like mark-up) can be evaluated in terms of inter-indexer and intra-indexer consistency. Reframing that to usability testing of vocabularies inter-user and intra-user consistency could be more suitable labels. Inter-user consistency is the degree of agreement among users in describing the same content. On the other hand, intra-user consistency is the extent to which one is consistent in marking up content with oneself. Consistent use of a vocabulary may be taken as a sign that the vocabulary terms are not ambiguously defined, so that users do not confuse them. It may also show that there is a documentation providing clear guidance on the ways in which the vocabulary may be used. These metrics might help test if vocabularies can be “easily understood by the users, so that it can be consistently applied and interpreted” (source).

These metrics have a long history in the field of libraries and are already deployed in practice on the Web. For example, Google Image Labeler (now defunct) was a game that asked pairs of users (mutually unknown to each other) to label the same image and rewarded them if they agreed on a label. A similar service that works on the same principle rewarding consistency is LibraryThing’s CoverGuess. A naïve approach to implementing these metrics could compute the size of the diff, so that, for example, markups produced by 2 users given the same web page and instruction to use the same vocabulary can be compared. A more complex implementation might involve distance metrics that measure similarity of patterns in data, such as with the metrics offered by Silk. Finally, when applying the consistency metrics, as observed previously, you should keep in mind is that high consistency may be achieved at the expense of low overall quality. Therefore, these metrics are best complemented with data quality testing.

I believe adopting usability testing as a part of vocabulary design is a step forward for data modelling as a discipline. To start we will first need to find out what usability metrics apply to vocabularies or develop new specific approaches to usability testing. So let’s get user-centric, shall we?

Publishing the vocabulary of the types of grey literature as linked data

This blog post is based on the poster presentation delivered at the Grey Literature 12 conference.

The aim of this post is to introduce the typology of grey literature we have started to develop at the National Technical Library. The vocabulary of the types of grey literature is a controlled vocabulary that is meant to be used to express that a document belongs to a certain document type. The design of the vocabulary is based on an analysis of six existing grey literature typologies. Thus, it can be seen as a formalization of the outcomes of the systematic examination done during this analysis.

It has a loose structure with hierarchical relationships between the types' concepts. Each type has a unique identifier (a URI, in this case) and a preferred label. Some types feature labels in multiple languages and links to other types, both from the vocabulary itself and from external datasets. In the vocabulary's documentation each type will be provided with a definition and a prototypical example of a document for which it can be used.

I will briefly mention the technologies that we have used in the vocabulary's development. The vocabulary is expressed in the RDF data format as a SKOS concept schema. RDF (which stands for Resource Description Framework) is a data format for expressing data with a graph structure and Simple Knowledge Organisation System (abbreviated as SKOS) is an ontological language for representing knowledge organisation systems, such as thesauri, codelists, or systematic classifications. 

The vocabulary will be published as linked data. Linked data is a publication model for exposing structured data on the Web in a way that uses links between datasets to create a network of interlinked data. The vocabulary includes links to other vocabularies and datasets, such as the Biblioontology, Dublin Core Metadata Initiative Types, or DBPedia, which represents the structured information extracted from Wikipedia.

The vocabulary is supposed to be a product of a co-operative development. The project of the grey literature typology is hosted at the Google Code website. The reason for using Google Code is that it has the functionality to support collaborative development. At its core there is a distributed version control system that enables to track the different versions of the vocabulary submitted by the members of the development team. It makes possible to incorporate feedback by commenting on the individual changes of the vocabulary and by reporting issues that should be fixed in the future versions. The Google Code website also includes a wiki that serves as the vocabulary's documentation.

For the purposes of this vocabulary's development the Working Group for Grey Literature Typology was established. The aim of this informal group is to bring together the experts from various fields related to grey literature, knowledge organisation systems, or semantic web technologies in order to work collaboratively on the further evolution of the vocabulary. If you are interested in participating in this vocabulary's development or becoming its user I encourage you to check out the project's website at Google Code.