Evaluating digital services: a Visitors and Residents approach
This infoKit contains advice on evaluating the services you offer to your users. The focus is primarily on digital/online services but set within the broader context of more traditional services, exploring the relationship between the two.
Much of the information presented here is derived from a longitudinal research project, which explored how individuals at various educational stages (emerging, establishing, embedding, and experiencing) engage with the digital environment for learning.
|Emerging||Last year high school/secondary school and first year undergraduate college/university students|
|Establishing||Upper division undergraduate college/university students|
The ‘emerging stage’ bridges the transition between school and university. It was designed to do this as it was felt that in terms of digital engagement or digital literacies the ‘gap’ between school and university is not as stark as the education sector might imagine.
We outline broad findings from the research and describe the methods used to gain a deeper understanding of what motivates users to engage with specific aspects of the information environment in a given context. The contents of this infokit are not presented as research findings in the traditional sense but represent themes that have emerged over the course of the research which can usefully inform our understanding of how learners are navigating the overlapping worlds of the web and formal education. In particular, this infoKit is designed to facilitate critical thinking about web-based tools and services, and more effective digital engagement on the terms that our users have come to expect.
What do we mean by Visitors and Residents?
Well, it’s perhaps best described as a continuum, with two modes of online engagement at either end.
When in Visitor mode, individuals have a defined goal or task and select an appropriate online tool to meet their needs. There is very little in terms of social visibility or trace when online in Visitor mode.
When in Resident mode the individual is going online to connect to, or to be with, other people. This mode is about social presence. We’ll discuss Visitors and Residents in more detail throughout the infoKit.
Paying attention to which services people adopt can lead to more effective engagement between individuals and institutions, and the resources/services therein. It’s not just about identifying how individuals engage with technology and how they acquire their information but also why they make the choices that they do. We need to understand the larger, more complex contexts that surround individual engagement with digital resources, spaces, and tools before we can think constructively about how to (if, indeed in some cases, we should) provide institutionally-based digital resources and services. This is also important when considering how institutional systems interface, or overlap, with the environment and culture of the wider web.
Evaluation tools such as surveys and compiled statistics are ubiquitous. For example, within a library setting, traditional quantitative data that report outputs (the number of books circulated, the number of reference questions answered or the attendance at library) and inputs (space, budget, collections, equipment and staff) create a narrow picture of ‘performance’. Such reporting is useful for describing things that are easily measured, but some important information is not measurable as direct statistics and requires a different form of investigation and assessment. Insights into the ‘how’ and ‘why’ of user engagement with technology and resources often can be most effectively gained by employing qualitative research methods.
We want to help practitioners generate their own qualitative inquiries, to inform their own policy needs and to frame the organisation (corporate, department, faculty) and its services within the institutional agendas surrounding student experience and engagement. For a detailed review of the qualitative/quantitative methods used in the research informs this infokit please see: Evaluating digital services – research methods. We have also developed a practice focused mapping tool which uses the Visitors and Residents continuum to explore online engagement.
This infoKit is a collaborative project between Jisc, the University of Oxford, and OCLC Research, and in partnership with the University of North Carolina, Charlotte. The content for the infoKit was created by David White (University of Oxford), Lynn Silipigni Connaway, Ph.D. (OCLC Research), Donna Lanclos, Ph.D. (UNC Charlotte), Erin M. Hood (OCLC Research), and Carrie Vass (OCLC Research).