Pagliari C et al. 2003 "DARTS 2000 online diabetes management system: formative evaluation in clinical practice."

Pagliari C, Clark D, Hunter K, et al. DARTS 2000 online diabetes management system: formative evaluation in clinical practice. J Eval Clin Pract 2003;9(4):391-400.
"Rationale, aims and objectives: Failure to engage in user-informed evaluation of emergent health informatics tools can have negative consequences for future implementation, related both to poor usability or clinical utility and to suboptimal stakeholder buy-in. This paper describes a formative evaluation in primary care of a multifaceted, web-based resource for diabetes management. The primary aims were to assess the usability and utility of the prototype in order to inform system refinements prior to implementation, and to investigate barriers and facilitators to system use so as to aid the development of a tailored implementation plan.
Methods: A mixed-method approach involving survey, remote observation, semi-structured interviews and electronic feedback.
Context: One Scottish local health care cooperative comprising five general practice surgeries and their staff.
Results: A survey following temporary exposure to a dummy site revealed high levels of computer familiarity, welcoming attitudes and positive ratings of usability, format and utility. Comments mainly addressed content accuracy, feature suggestions and usability issues. Key barriers and facilitators to use included time and training. Remote observation following access to live clinical data enabled profiling of usage by individual and professional group. In the 3 month observation period administrators were the most frequent users, followed by GPs and nurses, with clinical and data entry screens accessed most often. Semi-structured interviews with key respondents sampled by professional group and usage frequency provided richer qualitative information on barriers and facilitators to use, patterns of system integration into work routines and system usability, content and utility. Content analysis of electronic feedback revealed mainly technical queries and general expressions of satisfaction.
Conclusions: Evaluation informed a number of important and unforeseen improvements to the prototype and helped refine the implementation plan. Engagement in the process of evaluation has led to high levels of stakeholder ownership and widespread implementation."

"To assess the usability and utility of the [Web-based diabetes management] prototype in order to inform system refinements prior to implementation, and to investigate barriers and facilitators to system use so as to aid the development of a tailored implementation plan."

Type Clinic
Primary care
Small and/or medium
Other Information
One Scottish local health care cooperative was sampled. This cooperative consisted of five general practices that had a population of approximately 600 diabetic patients.
Type of Health IT
Disease registry network
Type of Health IT Functions
The system was an "accessible, web-based source of timely, patient-specific, practice-specific and regional data that could meet both clinical and strategic decision support needs." Features included "patient-specific clinical information at the point of care (e.g. treatments, trends, outcomes), shared data from primary and secondary care (general practice, hospital clinics, laboratories, retinal screening services, podiatry, etc.), organizational information (diaries, clinics), guidelines and summaries of evidence (Tayside Diabetes Handbook), practice statistics, regional comparisons, audit facilities, patient information leaflets, [cardiovascular disease] CVD risk calculator, links to professional bodies and diabetes support agencies, information about the ... project (development information, news, personnel, local research) and multiple search facilities."
Workflow-Related Findings
"The overall website was rated useful, easy to use and well presented by all respondents (all scores < 4) [on a scale of 1 to 5]. Most individual features were rated favourably on all dimensions by all respondents, with the majority receiving scores in excess of three. Features rated less useful were those containing general information about the ... project itself ('News' and 'Research')."
"Most [end users] reported using the system during the consultation to pull patient records ... or for other features (e.g. 'for the eye van', ...), while one thought this was too time-consuming. Most clinicians used the data recording features between consultations and the educational and research facilities at other times. 'I generally use the handbook in my own time.'"
"Most [general practitioners] GPs and nurses reported sharing screens with patients and receiving positive reactions. 'They're very impressed...the patients think I'm very up-to-date'...'The risk calculator is popular with patients.'"
Some technical barriers to implementation included "Compatibility with existing systems, for instance 'double a problem'...; usability issues, such as 'passwords didn't work' [and] 'worrying warning messages.'" Technical facilitators to implementation include "ease of use, fast system, up-to-date information.'"
"No major usability problems were reported. 'I think it's quite easy, and I'm not a computer person'...[However,] a number of respondents observed that 'information from secondary care isn't always available quickly enough'...In all cases, this was attributed to less sophisticated administrative procedures at the secondary care end."
"All respondents had heard of the website and ... ninety-six per cent expressed a positive attitude towards the website.... Eighty-seven per cent thought they would use the website 'often' or 'very often' after it went live."
"Sixty-six per cent [of end users] had been able to use [the Web site] without assistance. All but one of those who required assistance had received it from a colleague."
Study Design
Only postintervention (no control group)
Study Participants
Thirty-eight staff members were eligible to participate in the study. Of these 38, 18 were general practitioners, eight were practice nurses, and 12 were administrators.