Repository Evaluation

Each institution will develop criteria which will define the success or otherwise of its repository. There may well be specific local conditions and expectations which will need to be accommodated. Administrators will need to make sure that a repository has criteria for success which relate directly to its business plan and individual stakeholder needs.

In general terms, it might be said that a successful repository would be one which supports the needs of academics in its institution in making available their research outputs to the widest possible audience and facilitating their discovery and delivery. In practical terms, this means:

• the repository which has been accepted into the information environment of the institution and is integrated with other web services provided by the institution

• the repository has been accepted by academics and forms part of their working habits, as both a repository in which to deposit information, but also as part of a larger repository network which is used by the academics to search and retrieve information

• any issues relating to the user interface of a repository to deposit material have been ironed out and that there is a successful process in place within the institution to deal with ingested materials. This may be self-archiving by the academics themselves, facilitated archiving by assistants within their department, or a fully mediated service offered by the library to deposit materials on behalf of academics

Should metrics for the success of a repository focus on measurement of the absolute quantity of material in a repository? Even if every research output from an institution was captured within its repository, an absolute measure of quantity could still be misleading if a repository from a highly productive institution was compared with a repository from a less productive institution. A count of research outputs in this way is better phrased as a proportion of the total institutional output.

What of metrics to look at different research outputs? Should an institution measure relative proportions of types of research output that are held within it - ratios of preprints to post-prints to conference papers. One of the difficulties with any statistical measure is that the results are then often compared with those of other institutions, who may well have a very different basis for their repository, or for their research.

In their current form repositories are not being developed to alter research methodologies, but to reflect them. A repository is there to support whatever research output culture exists within the academic disciplines it serves. Therefore, a successful repository for an arts department may have no eprints in the form of journal articles but consist solely of multimedia outputs.

While it would be desirable if a repository took in some high percentage of the total research output of an institution, a repository could equally be described as successful in some measure if it gives open access to 'a number/a high number' of research outputs in a way which increases the use of these by 'a high percentage/a lower percentage'.

Therefore, if a repository holds a small amount of material, but this is very widely used then this might be described as successful. A repository holding a large amount of material where the usage has increased, but not necessary spectacularly, could also be described as successful.

For example, although eTheses repositories currently do not contain a high percent of the total thesis output, the increased usage which is given to the included thesis material surely makes these repositories 'successful'.

Of course, the use of quantitative metrics is often essential in building or maintaining a business case with an institution, but care should be taken for such metrics to reflect the true strategic and value-added intentions of the repository.