#dhiha5 Panel III: Evaluation and Quality Control in the Digital Humanities

Working group: Sascha Foerster (Max Weber Foundation, Bonn), Lilian Landes (BSB Munich), Bertram Triebel (TU Bergakademie Freiberg). Thanks to Christian Höschler for the English translation!

Word of Welcome

We have been working on two main areas: evaluation and quality control. Our presentation is structured as follows: for both of the areas mentioned, we will start off with a description of the current status-quo and then suggest some practical recommendations to various target groups (to which the findings of this conference will ultimately be addressed in the form of a joint manifesto), for example granting agencies, editors and, not to be forgotten, young academics. Many aspects can only be mentioned briefly, maybe too briefly. This article is furthermore a presentation of what the blog carnival #dhiha5 has produced with regard to evaluation /quality control.

1. Evaluation

This refers to proposals submitted to granting agencies by researchers or research groups, applications for jobs at universities or other research institutions, scholarships etc.

To begin with, here are two theses:

• The internet changes the way we publish research results. As a consequence, the criteria for evaluation will also have to change.
• New forms of scholarly activities are emerging: the creation of databases, the development of technical tools, the programming of codes, visualizations, big data, dynamic bibliographies, wikis and many more. New evaluation criteria also have to be developed for these concepts. The basic question which we constantly need to ask ourselves is: how do we define academic activities or achievements?

1.0 Situation

The point most often raised in those blog posts which dealt with our topic prior to this conference was that digital media is currently not accepted by professors and (!) by students (e.g. tekninen historia, gab_log-Blog, Criticalbits).[1]

The same view was expressed in the Blog hellojed.[2]: “History remains a subject which favors books over articles, print publications over PDF files, and publishers over open access.” [„Geschichte bleibt weiterhin ein Fach, dass die Monographie über den Aufsatz stellt, das Papier über das PDF, das Verlagshaus über den offenen Zugang.“]

Generally, issues regarding evaluation and acceptance tend to be less critical with books or open access journals (the latter more often face problems regarding their impact factor; however, the significance of the impact factor in the humanities is a much-disputed issue, so this shall be disregarded for now). Monographs or journals which are published online can be assigned ISBNs or ISSNs and thus formally be regarded as more or less equal to print publications. Things become problematic when it comes to genuine web formats, e.g. blogs, microblogs, comments, and wikis (RKB blog).[3]

It is difficult, if not impossible, to mention online activities (by which, as already mentioned, we mean databases, wikis, tools, codes, big data, social media, blogs, comments etc.) in funding proposals, scholarship applications and other things of that nature. They are usually not a part of the scenarios envisioned by those institutions that fund research, even though all of these activities are crucial and formative for those working in the Digital Humanities. The same applies to the recognition of qualifications regarding programming languages and Web 2.0 competences.

Basic problems: What is considered as an academic achievement (initial step, see above) and how does evaluation take place beyond the world of ISBNs and ISSNs, in the case of formats which can (only) be published online?

1.1 Recommendations to Granting Agencies

The following can be seen as both recommendations and as questions to those responsible for funding research, such as Professor Žic-Fuchs (ESF).

We start off with a question: how does the ESF structure the process of evaluating projects in the field of Digital Humanities, and how does the fact that these are naturally based on an interdisciplinary approach affect this process?

Guidelines published by the Modern Language Association (MLA)[4] stress (amongst other things) the need to “Engage Qualified Reviewers”. How difficult is it to find reviewers who are experienced in the field of digital media? Reviewers should also consider aspects which are specifically relevant for media work. This implies, for instance, that online projects should actually be evaluated in an electronic environment, and also, that the traditional circle of reviewers needs to be broadened: computer scientists, engineers and technology experts have to be approached in addition to scientists working in the humanities.

Generally, granting agencies should be open towards new forms of publishing and constantly reflect on, as well as modify, existing guidelines – at least with regard to the evaluation of genuine DH projects (see also dhd blog)[5]. Todd Presner, director of the Center for Digital Humanities based at the University of California, has defined some specific evaluation criteria: “support by granting agencies or foundations, number of viewers or contributors to a site and what they contribute, citations in both traditional literature and online (blogs, social media, links, and trackbacks), use or adoption of the project by other scholars and institutions [criterion of prototype delevopment], conferences and symposia featuring the project, and resonance in public and community outreach (such as museum exhibitions, impact on public policy, adoption in curricula, and so forth).”[6]

Similarly, the Center for Digital Research in the Humanities based at the University of Nebraska has released detailed recommendations for evaluation, such as “Peer Review of digital research sites or tools”, “Technical innovation and sophistication of projects”, or the crucial aspect of “Long-term accessibility, viability for archival use”.[7]

In our opinion, it is also highly important to familiarize young academics with genuinely digital text genres at an early stage, and to lead them away from working “behind closed doors”. In order to facilitate this, it is necessary to create incentives.

In the case of the blog carnival, this demand naturally referred to the medium blog (hellojed).[8] Therefore, a good example one could mention is the Gerda Henkel Foundation (co-organizer of this event), or rather their platform L.I.S.A., representing an experimental environment which provides space for initial blog posts left by individuals, and also for video projects documenting the benefits of scholarships.[9]

Making use of Web 2.0 technologies awards young academics with something they need most, namely attention (which also benefits the granting agency). The slogan “Social Media as a competitive advantage”, mentioned in the editorial blog of de.hypotheses, applies to this.[10]

At the same time, this facilitates the development of “archetype” blogs. Their relevance has been described by the gab_log blog.[11]

But we do not want to focus exclusively on blogs: a central aim in the world of DH is to facilitate the co-existence of different types of media. Closely linked to this is the principle of focusing on the evaluation of content, regardless of the medium (MinusEinsEbene)[12]. Therefore, not only the spectrum of commonly accepted media and text genres has to be broadened: the characteristics of individual types of media also need to be taken into account.[13]

Finally, we would like to propose two brief recommendations to granting agencies. They are more of a visionary nature, but nonetheless important:

A significant criterion for evaluation is to ask how academics contribute to a legitimization of science in society. This is a question which affects granting agencies themselves, whereas individual academics have so far not been forced to deal with this issue. Online activities are of great significance in terms of legitimization.[14]

And finally: the aspect of „Experimentation and Risk-Taking“[15] (Presner) is crucial for the evaluation of DH projects. Practically all of these projects are at the head of a movement which is subject to incredibly rapid change. Risk is a constant theme in the field of Digital Humanities.

1.2 Recommendations to (Young) Academics

We are convinced that demanding a change of regulations alone will not solve the problem, which is why we are also deliberately addressing young academics. We ourselves have a choice: we can either lay new foundations or consolidate old structures.

A quote from the MLA guidelines: “Ask about Evaluation and Support: When candidates for faculty positions first negotiate the terms of their jobs, they should ask how credit for digital work will be considered in terms of teaching, research, and service in the reappointment, tenure, and promotion processes”.[16] We therefore advise you to inform yourself!

One of the most important recommendations is our second one: have the courage to change your opinions. This means not to be afraid of revising or modifying a view you might express online, even more since the internet offers you the means to communicate with others (tekninen historia).[17]

Our third piece of advice refers to two forms of courage (RKB blog)[18]:

• First: it is important to mention whatever form of content you have created in the digital world (see above) when writing your own applications/funding proposals. This is often neglected, in fear of the assumed views of the reviewers (see 1.0 Situation).
• Second: have the courage to dismiss the possibility of publishing your texts, including important ones, in (supposedly) renowned print media, and choose open access or even fluid text formats instead.

Closely linked to this is the explicit demand that the EU put the re-publication rights into effect by law, so that authors may be provided with the necessary security to make use of green open access models.

Work together and structure your cooperative work: “(…) authors should indicate the roles that they played (and time commitments) at each phase of the project development” is what Todd Presner has (rightly) advised.[19] It should be clear who has contributed which type of work to a joint project, since this facilitates the process of evaluation. This includes work on the respective “environment”, i.e. the structure of a database or the features of a website.

2. Quality Control 

2.0 Situation

Peer review procedures carried out by traditional journals before publication are regarded as an academic standard. Concepts moving towards open peer review procedures are viewed (with much sympathetic skepticism) as outsider approaches.[20]

A quote from the blog carnival: “The assumption that printed books are worth more than digital books can not be confirmed in any debate. Which does not change that this is the way it is seen” [„Die Wertung, dass gedruckte Bücher mehr wert seien als digitale, hält keiner Debatte stand. Was nichts daran ändert, dass es so gesehen wird“] (MinusEinsEbene).[21] Because of this, quality control in the digital world is considerably complicated: due to the fact that traditional publishing formats (journals, monographs) are usually more accepted, the better works are mostly still published the traditional way.

This skepticism, in its radical form, refers to new forms outside the common text format, i.e. videos, dynamic bibliographies, databases, podcasts and so forth. This is because the traditional procedures of quality control applied in print media do (or can) not apply in the digital world, except in the case of monographs or journals published online.

Digital forms of publishing are typically more and more independent of former hierarchies (“Ent-Hierarchisierung”, one of key words taken up by the blog carnival, DHIP blog)[22], which is often considered a danger to academic quality standards – based upon the assumption that a professor’s opinions are irrevocably more significant than the views of a young academic.

It is evident that particularly in the German-speaking academic world, traditional peer reviewing is seen as the best method to ensure quality control. Believing in neutral entities as a crucial method of filtering is a common phenomenon (RKB blog).[23] However, editors (especially those of smaller journals) make decisions based upon criteria which are not transparent (see also the comment of K. Graf, editorial blog of de.hypotheses).[24]

As a result, we are confronted with two basic questions:

• Are there more advantages to digital publishing and new forms of publication than there are to (often ineffective) peer review procedures taking place prior to publishing?
• How can quality control be carried out in genuinely digital environments?

Another quote by Todd Presner: “Peer review can happen formally through letters of solicitation but also be assessed through online forums, citations and discussions in scholarly venues, grants received from foundations and other sources of funding, and public presentations of the project at conferences and symposia”.[25]

2.1 Recommendations to Editors Using Traditional Formats

We will limit ourselves to three recommendations. The first one refers to another key word in the DHIP blog, namely “acceleration” [“Beschleunigung”][26]. The element of acceleration in the digital world is often seen as something negative. However, several aspects are indeed useful when it comes to quality improvement: for instance, the possibility of exchanging information with colleagues who work far away, or with people one did not even know before. Therefore, we recommend encouraging your authors to become active prior to the release of traditional publications, e.g. through blogging.

Be visible in social media, in order to not “lose” the interest of young academics.

Setting up an online archive: Depending on the copyright situation, discuss this point with the publisher involved, in order to avoid moving walls or to keep these as short as possible.

2.2 Recommendations to online editors

To begin with, one should reflect on whether it is advisable to transfer editorial restrictions applying to print media into the online world. The thinking behind this is linked to the illusion that texts are “fixed” (which does not do justice to the potential of online publishing). It is also influenced by the illusion that a text can either be “valuable” or “of no value” (and therefore not publishable, RKB blog).[27]

In order to counteract this basic skepticism towards online formats, more environments specifically designed for scientific use should be introduced. With regard to blogs, for example, this applies to hypotheses.org (RKB blog).[28]

Experiment! Try, for instance, to combine text, images, videos and social media. Develop concepts based upon the idea of the “prosumer”, or explore new forms of quality control such as open peer review (for example, Historyblogosphere, see above). For this, you will need to employ at least one staff member who enjoys digital technology and has the basic know how.

In a comment left in the RKB blog[29], the following steps for online publishing have been suggested (H. Hartmann): Evaluation, online publication, commenting, and finally the possibility to react to comments or even republish the text in a modified version. In our opinion, this is a suitable transition model. It includes pre-publication filtering methods applied in print media, taking into account that an extensive post-publication filtering process may not take place if not enough scientists become involved (through comments, tweets etc.). These post-publication processes will ideally allow high-quality content to become more visible, following the idea of “publish first, filter later”.[30]

It is also important to make use of the added value of electronic publishing, i.e. posting links, including non-textual elements, the effective use of metadata, long-time archival storage, URNs or the optimization of search engines and so forth.

With innovative formats, it is important to regularly ask users to provide feedback, in order to improve user rates and/or satisfaction (“mind the gap(s)”, in the case of digitized content).[31]

2.3 Recommendations to (Young) Academics

Collaborative writing is just as important for quality as it is for evaluation: explore some of the tools mentioned, don’t write “behind closed doors” and make sure you choose your companions wisely. In the early stages of an academic career everyone has different degrees of experience and this is likely to improve the quality of the work, this also makes use of the benefits of the digital world (such as multimedia / communicativity). You will also prevent yourself from merely transferring elements of print culture into the internet.

Make experiences of your own! Anne Baillot, who runs the blog Digital Intellectuals[32], has for instance stated that writing a blog has not just been useful for documenting her research, but has actually helped her move forward in terms of productivity. That can be seen as an increase in quality

• “reaching outward “ (e.g. connecting with other academics worldwide)
• “on the inside”, i.e. the structuring of ideas, the elaboration of concepts through blogging as an activity prior to writing the actual text.

The basic attitude of seeing online activities as merely “additional work” (e.g. comments in the RKB blog)[33], should be dismissed. Somebody asked who would actually read all of the blog posts, and who would possibly spend entire nights commenting on what other people had produced. Our answer to that question would be that all of this helps filtering content and saving time, and it “only” means more work in a phase of transition, as long as two different systems are operating next to each other, for example in the case of traditional reviewing and online commenting. The individual is not alone with these filtering processes: communities develop fast, which is something one can easily experience by creating a Twitter account. It is surprising how much more (valueable) information is circulated whilst everyone is saving time.

2.4 Recommendations to Granting Agencies

If applied for in the context of DH projects, make funds available for academic communication in blogs and social media.

Whilst it is not advisable to support projects “replicating editorial structures”, one should invest in research and tools regarding search engines and metadata, in order to facilitate the findability, contextualization and evaluation of texts online. The basic principle behind this is publish first, filter later (see above), and also the realization that there is virtually unlimited space online, so that even the weakest texts can provide scientists with at least some new information (RKB blog)[34].

Investing in long-time archival storage is a prerequisite for quality control in the field of academic publishing (RKB blog)[35].

It is furthermore important to support DH projects with regard to interdisciplinarity (another point mentioned in the DHIP blog[36]), since this would also result in a (long-overdue) increase in quality.

During the evaluation of applications, peer reviewed publications should not be the only ones to be considered. Instead, other academic achievements should also be taken into account.

We would like to conclude our thoughts on quality control with a further quote:

“Texts published online [implication: without traditional peer reviewing] should be approached critically and read carefully. This includes taking a look at the references provided by the author, as well as looking out for messages between the lines. But is that not how we should read any text, regardless of whether it has been released in print or on screen? Is it that the internet poses such as risk, or is it maybe that because it is new, we are merely reflecting on our methodology in general?” [„Bei den Texten im Internet [impliziert: solche ohne redaktionelle Qualitätskontrolle] sollen wir besondere Sorgfalt walten lassen, sie kritisch lesen und zum einen die Belege des Autors überprüfen und zum anderen nach eventuellen versteckten Botschaften suchen. Aber ist das nicht, wie wir jeden Text, gedruckt oder am Monitor, lesen sollten? Ist das Internet jetzt besonders gefährlich oder besinnen wir uns nur durch das Neue wieder auf die kritische Methode?“] (Criticalbits)[37]

3. Conclusion

We now look forward to a lively discussion, possibly about the limits of evaluation, for everything we have mentioned is only applicable to a certain extent in some areas, such as micro-blogging:

Is a Tweet ultimately, as suggested in a comment on the RKB blog[38], to be regarded as similar to a statement made at a conference, which itself is just an informal way of taking part in academic communication?

We would like to conclude our presentation with one last quote taken from the many rich contributions to the blog carnival:

“I think that interaction through blogging, Facebook, Twitter etc. will, in the next years (I will avoid the use of the term “decades” in a day and age that moves so fast), not replace the traditional ways we publish and communicate, but will rather represent a further, additional form.” [„Ich denke, das Interagieren durch Bloggen, Facebook, Twitter, etc. wird in den nächsten Jahren (mit “Jahrzehnten” bin ich in dieser schnelllebigen Zeit vorsichtig) nicht das Publizieren und Vernetzen in den herkömmlichen Weisen ersetzen, sondern nur eine weitere, parallele Form darstellen“] (tekninen historia)[39]. If so, it is even more important that we start thinking about this new co-existence now, and then find appropriate methods of evaluation and quality control.

---

---