A Critical Assessment of Open Distance Learning Materials in Selected Higher Learning Institutions in Ethiopia: Alignment with the Quality Matters (QM) Higher Education Rubric

Introduction

Open Distance Learning (ODL) is an institutional concept of education centred on Self Directed Learning that uses correspondence courses with an integrated element of communication technology being facilitated by means of tutorial sessions, seminars, and other related platforms to meet the need for more human resources and professional skills that the job market requires [1]. The ODL system provides alternative ways of gaining established qualifications of the conventional system. Minichiello discussed that ODL first and foremost, is a movement that sought not so much to challenge or change the structure of higher learning, but a movement to extend the traditional college/ university, a movement to overcome its generic problems of scarcity and exclusivity [2]. It was developed when convocation, which serves as an organisation concept for the university, had reached its natural limit (limit in size and limit in resources).

As Mishra posits, the content of correspondence courses cannot depart from the requirements of a subject [3]. The difference in the learning experience arises from the presentation of course materials and students circumstances, and therefore, different pedagogical skills are required. The term instructional material refers to the tools used in educational lessons comprising of active learning and assessment [4]. In general terms, any resource a teacher uses to enable their students learn is an instructional material.

According to Hashim, ODL, one of the most innovative concepts in the history of education relies on self-learning materials developed by the institutions for the purpose of teaching and learning [5]. Similarly, the success and effectiveness of ODL programmes depend on the design, development, and use of high-quality learning materials [6].

High-quality learning materials should be student-centred, consider the needs and interests of the student and ensure that the student derives maximum benefit from their use[7]. They underscored that customised SIM are at the heart of instructional delivery in ODL. Tezpur University affirmed that selflearning materials should not only impart knowledge to learners, but they should also be in a better position to inspire and provoke students to learn [8]. It also stated that the potential impact of ODL on all education lies in the use of SIM complemented by visual, auditory, audio-visual and multimedia content.

The above discussion highlights the importance of effective design and development of instructional materials for ODL students, and that it enhances instructional delivery and maximizes student attainment.

Experts of the field advise that the following aspects should be considered when we plan to prepare SIM: organization of the overall structure of a course; presentation of the content visà-vis the developing subject matter of a course; the importance of adopting a systems approach; acknowledging the characteristics of the target group of learners; and other important considerations to make while developing SIM. These aspects define the characteristics of self-learning materials and the principles that should be adhered to [9]. These authors highlighted the following principles:

1. The preparation of SIM should be aligned to the synthesis of learning theories (which are concerned with the process of acquiring knowledge, means of interaction between learners and teachers, guiding students on the presentation of content or discussion).

2. The preparation of SIM should be done in such a way that they arouse students interest in learning, and to attract their attention to what is being discussed [10]. There are several factors that can attract the attention of students, and these can include change, novelty, and attractiveness of the materials that they use in their learning, and according to DıÌ?Å?len, these should motivate and provide students with a purpose and direction to follow and to read them attentively so that they can achieve their set individual goals [11].

3. The development of SIM for ODL should ascertain that effective learning outcomes are attained when students learn actively and recall information purposefully [12]. Luque et al. corroborated that the information that was learned previously guides students into new learning [13].

4. Learning new principles, supported with access devices like „clues or hintsâ?? might be directed either verbally or pictorially using SIM. This is a good way of guiding students, especially at the beginning or introductory stage of a text.

5. The development of SIM should consider various techniques of providing feedback to students self-check exercises, assignments, academic counseling and tutorials left within the learning packages. Tullis and Benjamin concurred that there is a need for students to get feedback on their accomplishments as they are motivated to work with a clear direction on the path they are taking [14]. This means that self-learning materials should have clearly defined objectives of instruction so that the ODL students can become fully aware of what they are expected to do to succeed.

According to IGNOU, SIM developers need to know in advance that writing for ODL is a challenging task and requires different skills that make it different from that of books and journals written for conventional learning [15]. SIM seek to exploit various means of communication to suit to the needs of the learners. Gbenoba and Dahunsi stated that the “ODL study material is expected to imitate what the teacher does in the face-to-face learning environment, meaning that the learner should be able to learn with or without the support of the teacher” [16]. They further emphasized that the material prepared for ODL should be self-explanatory, self-contained, self-directing, self-motivating and self-evaluating. According to them, study materials satisfying these requirements make the learner not to feel isolated in the absence of the teacher and other students as they keep the learner virtually engaged with the teacher. For this reason, the Commonwealth of Learning (COL) advised that ODL materials should be prepared by a team made up of people with skills such as curriculum design, instructional design, tutorial support and print or web design skills [17].

Statement of the Problem

The lead researcher holds an M.A. degree in Distance Education and has experience in developing Open and Distance Learning (ODL) materials for various higher learning institutions in Addis Ababa, Ethiopia. Through his consultancy work on the development of Study Instructional Materials (SIMs) for ODL, he observed that a significant number of institutions offering ODL—whether as standalone programs or integrated with conventional schooling— often relied on conventional instructors who lacked the necessary knowledge and skills for effective modular preparation in ODL contexts.

This observation aligns with findings from Tessema, which indicated that many ODL modules were deficient in essential attributes needed for effective distance learning [18]. Furthermore, the Ministry of Education (MoE) confirmed that the quality of education across Ethiopia’s higher education system—including regular, continuing, and distance education programs—was inadequate [19]. The MoE specifically noted that poorly organized modular materials led to more pronounced quality issues in continuing and distance education initiatives within private institutions compared to regular and public institution programs [19].

The MoE emphasized the necessity for ongoing training of university staff to identify required competencies, develop practical modules, and implement teaching methodologies that prioritize student learning over traditional teaching methods [19]. In light of these concerns, the current study aims to critically assess how ODL materials in selected higher learning institutions in Ethiopia align with the Quality Matters (QM) Higher Education Rubric.

Theoretical Framework and Literature Review

Open Distance Learning differ from the long-lived face-to-face conventional system in certain ways [20]. For instance, the instructional content and instructional strategies in ODL can be designed and developed before a course is ever offered. Thus, when people talk about the quality of ODL, they often differentiate between how a course is designed and how it is taught. While a bad instructor can arguably find a way to ruin a well-designed ODL course material (e.g., by being non-responsive), a well-designed ODL course material is generally recognized as a hallmark of ODL course quality. With this in mind, in the following section, we briefly summarize literature about ODL course quality and the distance learning materials of selected HLIs of Ethiopia to establish a way to discuss the quality of the latter.

Quality in Teaching and Learning in Open Distance Universities

The purpose of teaching is learning, and the quality of teaching provided by educational institutions should be relevant to the aim of promoting learning. As Hénard and Roseveare explained, quality teaching takes place at three interconnected levels:

1. at the educational institutions level, which includes work plans for designing the institutions policy strategy on quality, supporting internal and external evaluation systems;

2. at the curriculum level, which includes actions aimed at evaluating and improving the design, content and application of the curriculum; and

3. at an individual level, which includes initiatives that help tutors achieve their goal, encouraging them to innovate, support and improve student learning, and adopt a student-centred approach to learning [21].

In general, quality teaching in higher institutions requires using pedagogical techniques to generate learning outcomes in students learning and is designed to create conditions that encourage the development of learners skills such as critical and creative thinking [21]. And when it comes to ODL, according to COL (Commonwealth of Learning), the following attributes require special attention for quality is to be assured by the provision:

• The programme design: should be done in such a way that course outcomes, objectives, activities and assessments should be well-aligned [22].

• Provision of support systems: an ODL system needs to have a precise profile of learners to decide on the appropriate support for the different learners, and the teaching should provide timely feedback on assignments to learners to enhance effective learning.

• Material development: ODL material is the core aspect of quality in quality assuring mechanisms. There should be proof that academics in ODL are trained in instructional materials design for print and online delivery. If active and deep learning of learners is to be promoted through engagement, learning materials should be developed based on sound instructional design principles.

• Assessment of learning: it is mandatory to establish a valid and effective assessment strategy which manifests the inclusion of a variety of assessment approaches to make the assessment rigorous and authentic.

• Infrastructure and facilities: it is necessary that appropriate technological innovations are applied in the learning transactions so that the learning experiences of students can be enriched. Similarly, using appropriate technology facilitates the institutional provision of support to students and conducting examinations and recording students results is more efficient than manual capturing of such results.

• Regarding staffing: the institution should comprise of relevant staff with appropriate qualifications that are required for an open, flexible and distance mode of delivery. Similarly technical and support staff should be adequately qualified for the job they are doing.

• Referring to the overall systems and structures: it is mandatory that the system considers the peculiarities of ODL and the needs of the learners. This means that policies and processes are in place to ensure the quality of learning materials and quality assessments (assignments, quizzes, examinations); and in protecting students grades, in handling students queries/appeals.

As Xanthopoulou and Kefis noted, research and scientific publications on quality in teaching and learning, particularly on ODE are limited [23]. Research on open universities around the world reveals the absence of a quality model that fits in every context since there are different needs and conditions though the existing schemes and models for quality assurance are designed to offer flexibility for institutions to suit national and institutional contexts [24]. They further highlighted critical aspects that an international organization like the International Council for Open and Distance Education (ICDE) should take care of to address the issue of quality in ODL, namely, provision of a register of good-quality systems, and a guide to action towards securing quality work; addressing common issues around training, best practice sharing, localization; and ensuring a harmonized regulatory environment to mention a few.

Course Quality Standards in ODL

It is a routine procedure in conventional schools to monitor a teaching-learning activity at its input, process, and output stages to ensure that the system incorporates new techniques and innovations that can address the educational goals best. This procedure has been given serious attention in ODL as it puts teamwork at the centre to reach many learners from different backgrounds and interests. Different types of expertise are combined to produce high-quality learning materials and make them accessible to learners via relevant media. Quality standards or rubrics, consisting of a set of criteria have been used to guide HEI offering ODL to evaluate course materials and improve online and blended courses [25].

Many assessment techniques have been developed to identify course quality standards (rubrics) or guidelines in ODL. The researchers reviewed five of the most basic and recent ones and realized that the standards share some common goals: advancing the standard of ODL courses by standardizing course information, provision of support to learners and accessing resources. With the parameters consisted in its framework, quality matters rubrics for higher education has been found to be relevant to this study based on its orientation as discussed in detail below.

Online Learning Consortium Standards

The Online Learning Consortium Standards (OLCS) have been specifically oriented to provide criteria useful for evaluation of online programmes and serve the USA in enhancing the quality of its HE system by internationalizing online education [25].

E-Campus Alberta E-Learning Rubric

According to Ekren, the e-Campus Alberta e-learning rubric was introduced into the education system in 2000 to support the quality of online curriculum [25]. Its objectives were to determine the standards of online courses, the standards of support being given to ODL by institutions both in providing access to courses and administrative measures in the context of post-secondary level education. It was then restructured in 2013 and referred to as “Quality Standards 2.0” [25]. The modified rubric was drafted to assess the quality of existing courses and those that are still in the development process. Ekren further mentioned that Quality Standards 2.0 has also been a guide for a faculty to manage its course provision online, and for curricular experts to design instructions.

The Chico Rubric

According to Ekren, the Chico rubric is an instrument in creating or evaluating the design of fully online or blended courses [25]. It was drafted by a group of experts and students brought together from Chico State faculty of California State University in 2003. This rubric was revised in 2009 to assess the organisational activities required in the design and delivery of online courses and support the effort of the faculty in the development of Online Instruction (OI) expertise.

The Online Course Evaluation Project Standards (OCEP)

Finally Ekren explained that OCEP is a tool for assessing and comparing the quality of online courses delivered in HEI [25]. It is different from other rubrics described above. The categories such as “course developer and distribution models” and “developer comments” are not provided by the other rubrics. The focus areas of OCEP are the approaches in instruction and communication, techniques of content presentation and the art of teaching and learning aspects of online courses.

Quality Matters Rubrics for Higher Education and Continuing and Professional Education

A quality assurance organization initiated by a small group of colleagues in the Maryland Online Inc., QMTM was set up to answer the question which has been a common problem of educational institutions: “how do scholars measure and guarantee the quality of a course?” [25]. Quality Matters (QM) provides non-profit subscription services which allow adaptation of different evaluation rubrics for HE, continuing and professional education, K-12 and educational publishing. Each rubric focuses on a unique section of the material under supervision: course overview and introduction, learning competencies, assessment and measurement, etc. The Higher Education and Continuing and Professional Education (HECPE) rubric provides criteria for evaluation on the following five standards: learning objectives (competencies), assessment and measurements, instructional materials, course activities and interaction and course technology, which should be used together to ensure that the desired outcomes of learnersâ?? attainment are met [26].

After analyzing the existing criteria determining course quality in ODL in the context of higher education, this study aligned itself with Quality Matters TM (QMTM) rubric as it has relevant measuring tools to achieve the aim of the study. QM^TM focus on andragogy (i.e., adult education principles) in terms of how instruction is delivered, and the medium used for delivery of ODL courses to learners. They call for different experts to work together on the selection of courses, choosing the instructional design aspects of courses, and evaluating the appropriateness of the subject matter and the scope of courses. This study was guided by the research question, “What is the quality of ODL materials used by the selected Ethiopian universities?”. It adapts and applies the QM^TM rubrics released during 2013-2015, to assess the quality standards of modules taken from selected HLIs in Ethiopia.

QM^TM is one of the most popular and widely used quality assurance frameworks in the United States. QM^TM began under a Department of Education Fund for Improvement of PostSecondary Education (FIPSE) grant. Quality Matters (QM) is now an international organization focused on improving the quality of online courses at the K-12, Higher Education, and Professional Education levels. There are currently more than 800 QM subscribers [27]. QM is a peer review and faculty development process that is centered on the following eight general standards:

1. Course overview and introduction

2. Learning objectives

3. Assessment and measurement

4. Instructional materials

5. Learner interaction and engagement

6. Course technology

7. Learner support

8. Accessibility

Each of these general standards has a number of related and more specific sub-standards [26]. While each subscriber arguably could use QM differently, the formal QM process involves taking a course that has been taught before and having it reviewed by three peer reviewers (which must include one master review, one subject matter expert, and one external reviewer) to see if each standard has been met and then revising the course to meet any standards that were not met. The process is not perfect, but it is a widely accepted model for designing quality online courses [27].

As it was mentioned by MOE in the problem statement, it was capitalized that poorly organized modular materials led to more pronounced quality issues in continuing and distance education initiatives within private institutions compared to regular and public institution programs [19]. It seemed, hence, logical to apply a commonly adopted quality control framework like QM to investigate the quality (at least in terms of course design) of the selected ODL modular materials. If HLIs, teaching through ODL system, are supposed to be at par with the conventional HLIs, then, generally speaking, their ODL modules ideally should score well on QM reviews.

Methods

As previously mentioned, the adapted QMTM Higher Education Rubric, which includes score attached statements for evaluating the quality of self-instructional materials designed for Open and Distance Learning (ODL), was utilized for document and content analysis. The adapted QMTM Higher Education Rubric was employed to objectively assess the access devices, also referred to as advance organizers, established within the distance learning materials. The analysis aimed to evaluate the effectiveness and relevance of these devices in promoting self-learning, utilizing score-attached statements and checklists as evaluation criteria. The modified rubric included specific parameters intended to evaluate whether the ODL materials fostered independent learning. These parameters encompassed the use of icons as access devices and were incorporated within the accessibility and usability standards, while omitting those not relevant to the framework of this study. As discussed above, the rubric is comprised of 8 General Standards being accompanied by 42 Specific Review Standards (23 of which are deemed essential) used for evaluating the design of ODL courses, complete with annotations explaining the application of the standards and their interrelations. This research adhered to the minimum standards established by Research & Development Quality Matters, which an institution must meet to fulfill Quality Matters review expectations. Accordingly, it is required for a course/module to satisfy all 23 Essential Standards and achieve an overall score of at least 85 (representing 85% of a possible 100 points). This technique was specifically utilized to address the research question aimed at determining how well the distance learning materials from Higher Learning Institutions (HLIs) participating in this study met the established ODL criteria.

Research Findings

As this study adopted a document analysis approach, the findings are presented qualitatively expressed in quantitative terms to be able to address the research question: How appropriately are the distance learning materials used by selected Ethiopian higher education institutions align with the QMTM HE rubric? Table 1 below presents the results obtained from the evaluation of the courses randomly selected from four of HLIs: Five courses from AAU, four courses from KEU, five courses from UU, and four courses from RGCVOL.

                      Table 1: Quality Matters Results (Obtained using an adapted comprehensive rubric)

Key: AAU= Addis Ababa University, KEU = Kotebe Education University, UU = Unity University, RGCOVL - Renaissance Global College of Open and Virtual Learning.

Discussion

As the findings of this study show, except for one private institution, ODL materials of three of the HLIs institutions failed to align themselves with the QMTM Rubric framework. Only Renaissance Global College of Virtual and Open Learning (RGCVOL, private HLI) scored the minimum percentage, 85.0%. It tells to the reader that the modules prepared for ODL were not properly developed which means that they couldnâ??t make the purposes they are supposed to serve, i.e., ideally to be self-instructional. As to the main factors contributing to the misalignment of the ODL modules of the HLIs included in the study, were the incompetency of the course writers (all were of the conventional school system), their feeble attitudes towards open and distance learning and insufficient professional skills they acquired while developing the course materials [18]. This trend seemed to be almost the same across the remaining distance education institutions. Regarding the expected skills of course writers, explained: “Writing is an art and writing for open and distance learning is even more difficult because you need to use certain styles and techniques that are so different from traditional writing [29]. In designing and developing distance learning course materials, we have to ensure that writers are aware of learning theories and techniques.”

The aim of this study was to critically assess the quality of distance learning materials used by selected HLIs in Ethiopia, i.e., their alignment with the a standard QMTM Rubric (adopted for content analysis design) which consists of specific criteria that are accepted by International Council for Open and Distance Education and are adopted by leading open and distance learning universities worldwide. The study also made it possible devising a model believed to improve the development of self-instructional materials.

Recommendations

It is believed that ODL should provide learners with learning materials that encourage them to be responsible for their learning, facilitated by the interaction between the learner, instructor, tutor and the institution. All these transactions keep the learner motivated and engaged and they are fundamental constructs of Mooreâ??s theory of transactional distance and the constructivist theory of learning. If higher education institutions already engaged in open and distance learning and those planning to do so could adopt both theories and address quality issues and benchmark their course materials against the standard rubric, they would benefit significantly and be in a better position to create excellent open and distance learning course materials.

As Anderson (2013) explained:

“Each of us, as responsible open and distance educators, is compelled to examine the affordances and challenges of [ODL] development and delivery methods, critically examine their effect on public education and perhaps most importantly insure that our own educational systems are making the most effective use of these very disruptive technologies [30].” (p. 8)

“Each of us, as responsible open and distance educators, is compelled to examine the affordances and challenges of [ODL] development and delivery methods, critically examine their effect on public education and perhaps most importantly insure that our own educational systems are making the most effective use of these very disruptive technologies [30].” (p. 8)

References