• Replication and Computational Literary Studies

    Author(s):
    Maria Antoniak, Fotis Jannidis, David Mimno, Christof Schöch (see profile) , Karina van Dalen-Oskam
    Date:
    2020
    Group(s):
    DH2020, Digital Humanists
    Subject(s):
    Digital humanities, Literary informatics
    Item Type:
    Conference paper
    Conf. Title:
    Digital Humanities 2020 (DH2020)
    Conf. Org.:
    Alliance of Digital Humanities Organizations (ADHO)
    Conf. Loc.:
    Ottawa, Canada
    Conf. Date:
    July 20-25, 2020
    Tag(s):
    Bias, Computational Literary Studies, Funding, replication, typology
    Permanent URL:
    http://dx.doi.org/10.17613/ekd2-ew51
    Abstract:
    The "replication crisis" that has been raging in fields like Psychology (Open Science Collaboration 2015) or Medicine (Ioannidis 2005) for years has recently reached the field of Artificial Intelligence (Barber 2019). One of the key conferences in the field, NeurIPS, has reacted by appointing 'reproducibility chairs' in their organizing committee. In the Digital Humanities, and particularly in Computational Literary Studies (CLS), there is an increasing awareness of the crucial role played by replication in evidence-based research. Relevant disciplinary developments include the increased importance of evaluation in text analysis and the increased interest in making research transparent through publicly accessible data and code (open source, open data). Specific impulses include Geoffrey Rockwell and Stéfan Sinclair's re-enactments of pre-digital studies (Sinclair and Rockwell 2015) or the recent replication study by Nan Z. Da (Da 2019). The paper has been met by an avalanche of responses that pushed back several of its key claims, including its rather sweeping condemnation of the replicated papers. However, an important point got buried in the process: that replication is indeed a valuable goal and practice. As stated in the Open Science Collaboration paper: "Replication can increase certainty when findings are reproduced and promote innovation when they are not" (Open Science Collaboration 2015, 943). As a consequence, the panel aims to raise a number of issues regarding the place, types, challenges and affordances, both on a practical and on a policy or community level, of replication in CLS. Several impulse papers will address key aspects of the issue: recent experience with attempts at replication of specific papers; policies dealing with replication in fields with more experience in the issue; conceptual and terminological clarification with regard to replication studies; and proposals for a way forward with replication as a community task or a policy issue.
    Notes:
    A version of this submission with a range of formats is available at: https://doi.org/10.5281/zenodo.3893428
    Metadata:
    Status:
    Published
    Last Updated:
    4 weeks ago
    License:
    Attribution
    Share this:

    Downloads

    Item Name:pdf replication-and-computational-literary-studies.pdf
     Download View in browser
    Activity: Downloads: 100