Three by Three by Three: a Model for E-learning Evaluation

This paper describes the approach adopted to carry out an evaluation of the STEEL eLearning system, developed for an Italian online university. The aim of the evaluation study was to identify the strengths and weaknesses of the system in view of innovative potential and impact. The core of the adopted approach is based on an adaptation of the Technology Acceptance Model (TAM), originally proposed by Davis (1989) as a theory to model how users of a new technological system accept it and take it on, based on two main acceptance factors: perceived usefulness and perceived ease-of-use. Although the TAM was not developed specifically for educational contexts, several authors have subsequently proposed and demonstrated how adaptations of the TAM theory can be used to evaluate the impact of technology even in these contexts. This paper is a step forward along this line of work to perform formative evaluation of eLearning systems. Though based on the TAM, our approach is also grounded on a couple of assumptions. The first is that data concerning actual use of the system and its effectiveness, in terms of concrete actions and achievements of its users, should also be considered, at least to complement and verify the data concerning users' perceptions. A second assumption is that the evaluation should consider all the phases of use of the system (course design, running and evaluation), all the users of the system (students, teachers, and eLearning management), and all the components of the system (the eLearning platform, the learning resources and, last but not least, the pedagogical approach underlying the eLearning system). The resulting model is a three-dimensional one (phases of use, users and components), with three aspects to be considered on each axis. For each of the 27 combinations of these aspects, indicators of usefulness and ease-of-use have been identified, as well as, when available, data concerning actual use (derived from the tracking functions of the platform) and data regarding effectiveness (based on teachers' adoption of new tools and students' exam results). The paper describes the model and its indicators, discusses its pros and cons based on its field test. A summary of the results of the evaluation are presented along with considerations about future research in the field.