But that is just the relation type. To get the full relation out, we first have to know the entity on the left-hand side (subject) and right-hand side (object) of the relation. We can look at a different task of the BioCreative workshop – in this paper (PDF) – to get a feeling of how well extracting these subjects and objects work. The task focused on extracting chemicals, and this is done it a two-stage process. First the entities are recognized in the text, with an F-score of the best system of 0.8672, and then the recognized chemicals are linked to the corresponding formal identifiers, with the best F-score being 0.8136 (Figure 2). 

Figure 2: Determining the task focused F-score 

A very rough back-of-the-envelope calculation can give us an estimate of the quality of mining such entire relations (as shown in Figure 3).

Figure 3: Estimating the quality of text mining for entire relations 

An overall F-score of 0.40, as resulting from this calculation, roughly means that around 60% of retrieved relations are wrong and 60% of existing relations are not retrieved. This is clearly not even close to good enough for most types of possible applications. And mind you, these numbers come from the best performing systems when world-leading research groups compete and probably put months of effort into optimizing their systems for the specific problem. Moreover, we are talking here only about the simplest possible kind of relations of the form subject–relation–object.

"Why bury it first and then mine it again?"

This seems to point to a deeper problem. Text mining is just a work-around, and the real problem is elsewhere. As Barend Mons rhetorically asked: "Why bury it first and then mine it again?".[3]. Instead of seeing text mining as the ultimate solution, we should just stop burying our knowledge in the first place.

Representing scientific findings in logic

In the work I will explain below, we wanted to find out how we can practically publish findings without burying them.
As a first step to experiment with such a new way of publishing, we needed to find a general way of how to represent high-level scientific findings in some sort of formal logic. Even though such findings are arguably the most important outcome of science, there was no prior work on practically mapping (most of) these findings to formal logic across domains. To better understand the logical structure of such high-level scientific findings (e.g. that a gene tends to have a certain effect on the course of a given disease), we selected a random sample of 75 research articles from Semantic Scholar.

Studying the high-level findings from these random articles, we managed to elicit a wide-spread logical pattern, which we then turned in the super-pattern model. It consists of five slots to be filled, and translates directly to a logical formula. Our paper [1] explains the details, and I give here just one example. The finding of the article, entitled "P‐glycoprotein expression in rat brain endothelial cells: evidence for regulation by transient oxidative stress," can be expressed with the super-pattern as follows:

• Context class: rat brain endothelial cell
• Subject class: transient oxidative stress
• Qualifier: generally
• Relation: affects
• Object class: Pgp expression

The three class slots take any class identifier (here shown by class name for simplicity), whereas the qualifier and the relation come from closed lists of a few possible values. Informally, the example above means that whenever there is an instance of "transient oxidative stress" in the context of an instance of "rat brain endothelial cell" then generally (being defined as in at least 90% of the cases) it has a relation of type "affects" to an instance of "Pgp expression" that is in the same context. Formally, it corresponds to this logical formula (in a bit of a non-standard notation using conditional probability) shown in Figure 4.

Figure 4: Logical formula 
 

Once we discovered this pattern we tried to formalize the 75 high-level findings, and the result is shown in Figure 5, i.e., 68 of the 75 findings, shown in blue on the left, could be represented with the super-pattern. And the remaining seven findings, shown in red on the right, are in fact easier to represent, as they can be captured with a simple subject–relation–object structure. (So these latter simpler structures are the ones that existing text mining approaches struggle with, and it is safe to assume that they would perform even worse on the more complicated statements here shown in blue.)

Figure 5: Pattern obtained after formalization of findings

So, it seems that we have found a logical pattern that allows us to represent most high-level scientific findings from different disciplines.

Stepping into the new era of scientific publishing

Next, we wanted to make a serious practical step into the new era where findings are machine-interpretable from the start. We designed this as a kind of field study with a special issue of machine-interpretable papers at an existing journal. We wanted these papers to look like regular papers to those who want to look at them in that way, but they should also come with representations in formal logic for anyone or anything that knows how to deal with that. For that special issue, we chose the journal Data Science, of which I am co-Editor-Chief.

We also had to make a practical concession though: While the whole setup could be used to publish novel findings, we restricted ourselves to findings from existing publications. For that, we introduced the concept of a formalization paper whose novel contribution is the formalization of an existing finding. So, authors of a formalization paper take credit for the formalization of the finding, but not for the finding itself.

To represent these findings and thereby the formalization papers, we used the nanopublication format and the Nanobench tool. Researchers who contributed to this special issue filled in a form to express and submit their formalization that looked like that shown in Figure 6.

Figure 6: Requirements for expressing and submitting formalization of papers

Apart from the actual formalization in the assertion part of the nanopublication (blue), this also involved specifying the provenance of that formalization (red) and further metadata (yellow). The provenance in this case states that the assertion was derived by a formalization activity taking an existing paper as input. This interface provides auto-completing drop-down menus and maps the result to the logic language RDF in the back. I won't go into the details of defining new classes and reviewing (both done with nanopublications and Nanobench too), but you can have a look at the preprint of our paper [2] on that.

We ended up with 15 formalization papers in our special issue, as summarized in Table 1. 

Table 1: Resulting formalization papers (see original post to access the embedded links)

Each row of this table corresponds to a formalization, and can be written out as a logic formula. So, these papers look here quite different from what we are used to. But if you go to the official page for the special issue on the publisher's website, they look like regular papers. Besides the regular button to download the paper as a PDF, there is also a link that points to the nanopublication representation, thereby connecting the two ways of looking at the paper as shown in Figure 7.

Figure 7: Resulting special issue paper with nanopublication link

So, for the first time, software can reliably interpret the main high-level findings of scientific publications. This special issue is a just a small first step, but it could prove to be the first step into a new era of scientific publishing. The practical immediate consequences of this might seem limited, but I think the longer-term potential of making scientific knowledge accessible to the interpretation by machines is hard to overstate.

References

1. “Expressing high-level scientific claims with formal semantics“ by Cristina-Iulia Bucur, Tobias Kuhn, Davide Ceolin, and Jacco van Ossenbruggen, in Proceedings of K-CAP '21: Proc. 11th Int. Conf. Knowledge Capture, Anna Lisa Gentile and Rafael Gonçalves (Eds.), pp. 233–240 (2021; ACM); link: https://dl.acm.org/doi/10.1145/3460210.3493561 (last accessed: 31 August 2022). 
2. “Nanopublication-Based Semantic Publishing and Reviewing: A Field Study with Formalization Papers “ by Cristina-Iulia Bucur, Tobias Kuhn, Davide Ceolin, and Jacco van Ossenbruggen, arXiv, 2203.01608 (2022); link: https://arxiv.org/abs/2203.01608 (last accessed: 31 August 2022).
3. "Which gene did you mean?” by Barend Mons, BMC Bioinformatics. Volume 6, Article 142 (2005); link: https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-… (last accessed: 31 August 2022). 

Further Reading

• Initiative: GloBI with its integrations to Plazi/Zenodo and Pensoft
• Journal: Biodiversity Data Journal
• Viewpoint paper: “Liberating host–virus knowledge from biological dark data “ by Nathan S Upham, Jorrit H Poelen, et al., The Lancet Planetary Health, Vol. 5, Iss. 10 (2021). 
• Implementation: “Implementation of TaxPub, an NLM DTD extension for domain-specific markup in taxonomy, from the experience of a biodiversity publisher” by Lyubomir Penev, Terence Catapano, Donat Agosti, Teodor Georgiev, Guido Sautter, and Pavel Stoev., in Journal Article Tag Suite Conference (JATS-Con) Proceedings (2012).
• Editorial: “Taxonomy shifts up a gear: New publishing tools to accelerate biodiversity research” by Lyubomir Penev, David Roberts, Vincent Smith, Donat Agosti, and Terry Erwin. ZooKeys, Vol. 50, Iss. 1-4 (2010).
• Position paper: “Genuine semantic publishing” by Tobias Kuhn and Michel Dumontier, Data Science, Vol. 1, Iss. 1-2 (2017).

The topic of this post was the focus of the talk given by Tobias Kuhn at the IOS 35th anniversary symposium; click for details.