Educational Content in Open and Distance Learning Environments: an insight into the use of standards and

guidelines for evaluation Stefani A.

Educational Content, Methodology and Technology Laboratory, Hellenic

Open University Patron-Clauss 278, GR-26335,

Patras, Greece stefani@eap.gr

Kameas A. Educational Content, Methodology

and Technology Laboratory, Hellenic Open University

Patron-Clauss 278, GR-26335, Patras, Greece

kameas@eap.gr

ABSTRACT Educational content and its quality evaluation has been the subject of abundant research. However, models have been used mostly for the purpose of understanding, rather than improving. In this work we present a new framework for the quality evaluation of digital material mapped to its life-cycle process: quality control procedures are mapped to every phase, from initiation and design to dissemination via Learning Management Systems. Assessment is not a simple procedure, since digital material is not as simple as data is and thus ISO standards are not directly applicable. We show how educational, technical and managerial principles need to be applied in order to obtain a complete, flexible and practical model of quality life cycle. We propose instantiating quality models with the specific purpose not only to understand the current digital learning material but also to improve quality through its life cycle.

Keywords Digital Educational content, Lifecycle Model, Assessment, Quality Assurance, Open and Distance Learning, certification guidelines.

1. INTRODUCTION A new stage is set for the transition to the paperless University; International efforts plan for a full transition of University teach ing material from paper to digital form within the next few years. Battling with poor awareness, Education Institutions need now to face a dual role as publishers and distributors of educational content. Standardization of the procedures used for designing and developing such content is one of the many challenges ahead. Quality standards and techniques have been successfully used for evaluation of systems and software throughout their lifecycle. However, research on content (viewed as data), and especially digital assets used for educational purposes, is somewhat behind as far as formal quality evaluation techniques are concerned [1].

Standardization of educational content design, development and management requires not only standard business processes but a meaningful evaluation of the educational process itself. Questions on how tutors set goals and how the educational content is designed around those goals, how content is consumed by stakeholder and in which way, are some of the key research questions. Current practice relies on guidelines or the transfer of best practices.

In this paper, we address the problem of how to design digital educational content (and in particular content suitable for open and distance learning) based on formal quality and assurance practices. Based on the experience gained in the Hellenic Open University’s (HOU) regarding the use of ISO standards along with Q&A procedures for the development of educational content for over a decade, we discuss the first steps towards formalizing the procedures towards a standard. We use a two way approach: a top-down analysis of Q&A needs through the definition of a content life-cycle model; a bottom-up synthesis of a standard-like hierarchy through the definition of certification guidelines based on the significant knowledge acquired from years of quality control at HOU. Our ultimate goal is to map needs, stakeholders, and quality attributes to quality characteristics, sub-characteristics and metrics of an ISO-like standard.

To this end, we present a new framework for the quality evaluation of digital material mapped to its life-cycle process: quality control procedures are mapped included in every step, from initiation and design to dissemination via Learning Management Systems. Assessment is not a simple procedure, since digital content is not as simple as data is and thus ISO standards are not directly applicable. We argue that educational, technical and managerial principles need to be applied in order to obtain a complete, flexible and practical model of evaluation. We also present the basics of the certification guidelines and present examples that show its practical use.

The paper is organized as follows: in sections 2 and 3 we set the research questions and briefly review the literature. In section 4 we present the design method and lifecycle model itself; in sections 5 and 6 the certification guidelines and examples of use are described, respectively. Finally, in sections 7 and 8 preliminary results and future research directions are briefly presented.

2. EDUCATIONAL CONTENT Educational content can be roughly distinguished in printed

and digital material. The printed form includes study books, which are available to students in paper or electronic form. Digital content is defined as the artefact, which combines digital content, and dissemination media (digital content container) or an application system. For example, in a digital learning material, the text is the main educational content, the technological means is an application overlay (e.g. browser) while the pedagogical / didactic application is the context in which the application is used (e.g. a

Draft

training module). Digital educational content may take many forms each of which is stored in a number of different formats. The basic types of digital educational content are:

Text: Text is a coherent set of characters, words or paragraphs, which may include static and visual material. Text as educational content includes plain text (doc, pdf, odt), hypertext and wikis.

Audio: Audio includes static audio applications and audio streaming that can be used by students.

Graphic: This static digital content based on visual (and not verbal or audio) representation. It includes photographs, images, maps, diagrams.

Video: This category includes content which has been produced with the aid of recording media. It may be divided into interactive or non-interactive content and two-dimensional or three-dimensional (2D/3D animation) content.

Animation: This category includes moving images, which is not considered as audiovisual material. A common example is an animation which represents processes or experiments that are difficult to be recorded in a real environment. The final formats are similar to those of audiovisual material. Animations may be further divided into interactive or non-interactive content and two-dimensional or three-dimensional (2D/3D animation) content.

3. QUALITY AND CONTENT: RESEARCH CHALLENGES 3.1 State of the art Quality has been long recognized as a comparative advantage in the world of business and education. Quality in academic terms until recent years referred to teaching and research excellence, managerial and support services. For many countries where the educational system is mostly state-funded, quality in education is a term being debated, with some Q&A actions in partial implementation. The debate of how to implement quality assurance in higher education has posed many significant theoretic and practical questions concerning the links between quality and teaching, how to manage educational quality and how much it costs;

In practice, quality management in Higher Education is mainly based on the application of the ISO9001: 2008 standard and relates to administrative procedures. A variant of ISO 9001:2000 for educational institutions appeared as the International Workshop Agreement Part 2 [3]. On the other hand, examples of adoption of standards or guidelines in universities are found in the UK (full frame mode Universities [4]), the USA (for instance in the context of Good Operating Practices in Cornell [5]) and in many European countries [6]. Reports avoid referencing to the use of standards in functions other than those of administrative nature.

Possibly, a meaningful evaluation of the educational process itself and more precisely, of how tutors set goals and how the educational content is designed around those goals, is appropriate. Current practice relies on guidelines or the transfer of best practices. To the best of our knowledge, there is no educational-specific or formal standard which certifies the educational process or educational material except ISO19796 [2]. However, this standard is more a reference model rather than a standard in the strict sense of the term.

The ISO organization in a recent effort to limit the number of standards and to reduce duplication (and thus the complexity of

application of standards), proceeded to issue a new series of standards over the last 3 years. Of course, the design of each model takes at least 2 years even for those who are simply updates or instructions for implementing existing ones.

In the field of education and related services, ISO posted a plethora of standards until early 2011 while others are expected in 2013. These standards cover a wide range of prototyping processes, general guidance, data standards, specifications and software systems. A brief review on the new series of these models shows that the complexity and duplication has been reduced, but not smeared. Tables 1-4 present a breakdown of ISO standards relevant to/applicable to education (excluding standards clearly referring to metadata, the outcomes of earlier standardization efforts of IEEE LOM).

As far as the educational process is concerned, the most significant progress was achieved with the introduction of IWA2 (International Workshop Agreement) which is not a standard but rather a set of instructions implementing ISO9001:2001 for educational institutions (Table 1). The instructions are brief and their practical value may be questioned. The limited literature on IWA2 practical results reports both strengths (specialization, clarification of concepts) and problems (complexity, need for adjustment procedures in educational contexts [7], adaptation to legislation on education [8], adaptation to modern forms of education and learning [1]).

Table 1. ISO Process standards Standard

(Publication Date)

Description

IWA (2007) [3] The IWA 2:2007 provides guidelines for implementing 9001:2000 in educational organizations.

ISO 29990 (2010) [12] The ISO 29990:2010 establishes the basic requirements for providers of non-standard (non-formal) education services.

With respect to software and services, important though incomplete so far, is the renewal of the ISO9126 standard through the ISO25000 series. Specific parts of the series such as ISO25010 on software and ISO25012 on data can fully replace ISO9126. Being a relatively new model (first edition March 2011), ISO25010 has not been tested yet in the real world (table 2).

Table 2. ISO System standards Standard

(Publication Date)

Description

ISO9126 (2008) [13] Software Evaluation. Applies to all software systems. Replaced by ISO2510.

ISO/IEC 25010 (2011) [11]

Software Evaluation. Replaces ISO9126. Applies to all software products and computer systems. Besides evaluation, it can be used to control the software specification process. Many features can be used to evaluate a wide range of systems and services (including ICT and LMS).

ISO/IEC 24751 (2008) [14]

Specifies the requirements of learners with disabilities.

Regarding the quality of data, particular attention is paid to the description and exchange of data between applications or systems that support learning processes. For the first time in 2011, there was a focus on mobile learning applications data standardization.

Draft

The ISO organization has also just recently paid particular attention to specific categories of users (the disabled), applications for mobile devices, and to non formal modes of education (Table 3).

Table 3. ISO Data standards Standard

(Publication Date)

Description

ISO/IEC TS 29140-2 (2011) [15]

The ISO / IEC TS 29140-2 provides an information model for mobile learners. It can be used as a reference for developers of LMS/LET, designers of educational content, to ensure that the requirements of mobile users are met.

ISO/IEC 19788-3 (2011) [16]

Specifies the metadata and properties for describing educational resources.

ISO/IEC 25012 (2008) [17]

Specifies the data structures that can be used for the exchange of LET (Language, Education and Training) content between systems.

ISO/IEC 19778-2 (2008) [18]

Defines a general quality model for data kept in a structured form in a computer system. It can be used to define data quality requirements, quality measures or for designing plans of data quality assessment.

ISO / IEC 19796 can be regarded as an education-specific model, being a frame of reference associated with the management of learning processes and related information (including training materials). It requires adaptation before being applied to an educational institution [2].

Recognizing the close link between content and the means of its diffusion, ISO proceeded to issue the ISO 24725-3 standard which can be used as a dictionary for learning environments (Table 4).

Table 4. ISO Frameworks Standard

(Publication Date)

Description

ISO/IEC 19796 (2005, 2009) [9]

Information technology - Learning, education - Quality Management, Assurance and Metrics.

Part 1: General Approach

Part 3: Reference Model and Metric

ISO/IEC TR 24725-3 (2010) [19]

The ISO / IEC TR 24725-3 provides a standard dictionary, a taxonomy for media technology platforms and a process that can be used to describe bonds (bunldes) between tools and platforms that support learning and education.

ISO/IEC 2382-36 (2008) [20]

The ISO / IEC 2382-36 contains instructions (vocabulary, definitions and relationships between concepts) that facilitate internal communication between applications used for learning and education.

3.2 Research challenges It is important to stress the link between educational processes and education content that is used by them (for example during teaching a class in winter semester). Educational content and learning processes are closely interconnected. Although the design

as a process, is independent of the content, there should exist specifications that allow:

• compatibility with the educational processes,

• to achieve the maximum out of the learning outcomes,

• reduction of costs during content use (ideally without compromising educational quality),

• reuse of the content.

There is a need for a clear grouping of processes (what should be done), content (to what) and stakeholders (by whom); a quality life cycle for educational content. This life cycle should take into account the context (implementation in an academic environment) and the need for practical application. In our case, the goals were to design a lifecycle using a mix of techniques:

• adoption of (some) characteristics of quality standards in order to achieve the necessary level of formality (formality),

• eliciting good practices through the application of benchmarking methods in order to achieve the necessary level of practicality (practicality),

• coverage of all types of educational content (paper, electronic, active) in order to achieve the necessary level of completeness.

In order to design such a life cycle, one has to answer at least the following research questions:

• What to measure from phase to phase? Educational content has a diverse nature including educational, pedagogical and technical facets. Parameters such as functionality, educational suitability, educational correctness, medium of content delivery, (self-) assessment, subject coverage and many others, form one body, but they are different parts. Different aspects must be assessed (qualitatively speaking) through different methods/tools. What is the best method to asses which part?

• How to measure it? The general concept of quality is neither measurable nor strictly defined in the field of distance education. High quality educational material requires to model quality, quality assurance procedures, compliance and quality-of-use.

The development of digital educational content life cycle requires both running a project (with analysis, design and development phases) and producing a product (data view). The first step in producing the new life cycle for digital educational material is to analyze in depth the processes of these two directions that are combined: processes and data. One of the first steps to evaluate quality is to define non-functional requirements usually through quality models. The philosophy of the ISO standards is to use quality models and different evaluation approaches depending on the stage of life cycle in which learning material is evaluated. Each approach corresponds to specific, discrete quality characteristics and quality sub-characteristics.

3.3 Lifecycle views The development of educational content can be regarded as a project, an approach also followed by the ISO19796 standard (which is however more general, covering a wide range of training applications) [9]. In classic project management as defined by the PMBOK, a project is essentially the application of

Draft

knowledge, tools, skills and techniques in the execution of project activities in order to meet the requirements of the stakeholders [10]. The implementation of a project includes the following steps: un-official start project (optional), initialize project, design and implementation (figure 1).

Figure 1. The Project life-cycle as depicted by the PMBOK

[10].

The spark of an idea initiates the process, defines the objectives and scope of the project (Project Initialization). Then, the project is planned, executed and delivered (intermediate phases). In the final stages of the project, results are approved by the customer and delivered for use. The starting of a project includes demands, problems or opportunities. In the first steps before the starting of the project, the objectives are defined. The results of a project (and thus when to terminate/conclude the project) are directly linked to the objectives set before it is started. When a project is approved then the following steps involve setting goals and include identification of requirement, setting of realistic goals, trade-offs between cost, time and quality and adjust plans to stakeholder needs. The requirements are the conditions that must be satisfied by the result of work so that it is considered successful. The sum of the phases of the project form the project lifecycle.

Figure 2. The Product life-cycle as depicted by the PMBOK

[10]

The above analysis indicates two things: first of all, when a project for developing educational content is initialized, the goals (educational, institutional, managerial, economic) must be clear. Secondly, the process must follow the rules of project management (needs analysis, recruitment of appropriate staff, continuous evaluation of results vs. objectives etc.).

If the educational content is regarded as a product (data, knowledge or information) then instead of the Project Life Cycle, the term Product Lifecycle is used. The latter contains the former as illustrated in Figure 2.The life cycle of the product initially contains a business plan, the duty cycle that produces the product, and most importantly, further steps concerning operation (use) and upgrading (enrichment even redesigning if necessary, in the case of educational content).

4. THE LIFE CYCLE MODEL The top-down step to the design of the standard is the establishment of a life-cycle model that will be used to map quality assurance and quality control needs to model quality characteristics and sub-characteristics. The life cycle of digital educational content incorporates basic parameters such as data idiosygracies, knowledge representing the osmosis between the educational environment and the stakeholders, instructional design and assessment processes. The lifecycle is designed as a mixture of the two cycles mentioned in the previous section (illustrated in Table 5) and includes the main phases of the project life cycle that produces educational content, enriched with phases and processes that refer to a product life-cycle.

Table 5. Description of the digital educational content life-cycle.

The life cycle includes five basic phases: § Phase 1: Requirements Analysis. This phase includes

processes such as: ü Setting the initial idea and the overall objectives as well

as the needs that lead to the development of educational content and making initial predictions about the type of educational materials, the budget and the delivery method.

ü Defining stakeholders: who will be involved and how. ü Setting Goals: setting out the basic project parameters

such as the target group (e.g. distance learning, adult education, skills-training), complex types of educational content, duration of use and needs for educational support material (technical, scientific, clarifications).

ü Finally, defining requirements: defining the specifications of education material such as content, teaching methodology, technical specifications, interaction and interface.

§ Phase 2: Design. This phase includes four procedures: ü Definition of the educational objectives: which are the

educational objectives and which in-content educational scenarios will be mapped to them.

ü Define content type: select the type of learning materials (digital, LO, multimedia, hypertext) so as to reach the educational objectives in the best possible way.

ü Selection of didactical methods: selection of method(s) by which to implement the educational scenarios (build knowledge, knowledge transfer, etc.). At this point, a more detailed design of the scenarios defined in 2.1. takes place.

ü Level of interaction: optional process related educational content offered through LMS and/or requires special design features so as the user interacts with the system.

Draft

§ Phase 3: Development. In this phase, the authors develop and deliver the content according to the educational objectives set and target group requirements. This phase includes the pilot use of the educational content as an optional process followed by internal assessment. The latter, is used to study the satisfaction of user expectations (educational objectives, expected functions, educational environment) at an early stage (before the product reaches the customer).

§ Phase 4: Management. Stage 4 includes all processes involving the use of educational materials (although it is often ignored, it involves considerable indirect costs). In some cases, the use of educational content includes management tasks in e-environments such as an LMS. Educational content can be reused if it has the appropriate format (e.g. LO format), as a whole (for other educational purposes) or partly (to synthesize new training content).

§ Phase 5: Assessment/Improvement. The last phase includes processes involving evaluation and improvements that can be made in the content after it has been delivered for use. The external assessment involves assessing the quality of educational content as perceived by the user. Design modifications involve repetition (of all or of some) of the processes of phase 2. A meta-evaluation process is also possible as an extra feedback step: it essentially examines the effectiveness of the instructional design by taking into account the qualitative and quantitative interpretation of internal metric values. Assessment processes either lead to the acceptance of content for use as is or trigger improvement processes.

5. A BOTTOM-UP APPROACH TO STANDARDIZATION 5.1 Existing Q&A processes at HOU The University uses formal standards such as ISO9001:2008 and ELOT:1429 for the standardization of managerial procedures, including the design, production and evaluation of educational content. There is a vast amount of accumulated knowledge: more than 12 years of Q&A for the assessment of more than 300 books, 500 hours of video lectures and Gigabytes of supporting learning material included in HOU’s curricula; and of the on-going effort to produce a large amount of digital content (including Wikis, hypertext, animations and podcasts) by 2015.

There exist internal quality control ISO9001 procedures at HΟU: content evaluation (performed by evaluators) and certification (performed by specialists from the Division of Certification of the e–Comet lab). Both processes involve different versions of the same content; the accreditation concerns content that has successfully completed the evaluation procedure; they are performed sequentially (the assessment prior to certification) and on the same quality features of the content. Special attention is given to the feature of open and distance education of the content that is, to the extent that it serves the purpose of distance education. Certification, as described in the literature, is used to confirm that educational content meets quality targets set in the design stage. The certification is carried out through inspections by experts external to the production process.

However, there are some additional needs for the enhancement of existing processes. First of all, there is a need for separation between internal (evaluation) and external (certification) processes of producing educational content. Assessment and certification (and thus, Quality Control) require quality objectives. Therefore, any kind of control requires the existence of qualitative

targets or benchmarks. Otherwise it can not produce reliable results.

There should also be a separation between the quality of processes (that produce the content) and the content itself. The standardization of the design and production processes (based on ISO9001: 2008) provide a degree of control over what is done by whom and when. However, experience has shown that the general principle: ‘quality procedures may produce quality products; unquantifiable procedures usually produce non-quality products’, holds in real life. The quality of the production process affects, but does not guarantee, the quality of the product. It is therefore necessary to enhance the existing HOU processes with content-targeted control procedures.

Finally, on a practical level, the provision of evaluation measures to produce material suitable for distance education is not easy. The e-comet Lab has designed and is using specific educational and technical guidelines for the evaluation of educational content during its development process. Practice shows that HOU content evaluators have also adopted informal, personal standards and assessment measures from their personal experience. Similarly, there is significant evaluation feedback by the end users of content (tutors and learners); this experience, probably of great practical importance, is not recorded by existing processes.

5.2 A Quality Control methodology Following the establishment of the life cycle model, we proceed to the presentation of an assessment methodology for the certification of educational content based on the experience gained at HOU. This is the bottom-up step to the design of the standard that will enhance the so much needed practicality in real life situations; The experience described in section 5.1 is used to derive (Quality Control) practical rules that are used by life-cycle stakeholders (authors/evaluators/QC experts) for the assessment and certification of (mainly) printed educational content.

Experience has shown that the quality of educational content is a critical element in the quality of education. Especially in the case of distance education, it is of primary importance since there is no direct, physical presence of a tutor and content provides (besides knowledge) guidance, self-assessment and practical experimentation. The set of practical rules are combined to form a practical assessment methodology for certification initially based on the characteristics of Open and Distance Learning. The initial goals set for methodology were: • reliability, for effective operation, • completeness, covering all the necessary aspects of certification, • formality, in the sense of a well defined group of procedures based on official standards, • practicability, so as to yield realistic results based on experiences in the development of educational content.

The methodology is designed so as to give feedback for continuous improvement of both processes and content; and expandable so that it can be applied to other types of content (i.e. digital material for the new distance education system of HOU). The methodology should be simple so that it is easily understandable and usable by evaluators and directly lead to practical results.

Based on the need to define practical assessment measures, the methodology produced initial guidelines in the form of directives and later enhanced them with best practices (e.g. examples per topic), alerts and implementation guides for authors and reviewers. The certification guide includes 11 guidelines relating

Draft

to content accuracy, expression, educational feedback and self-assessment, completeness, scientific accuracy, readability, motivation for further reading and personal research, clarity of expression and up-to-dateness.

6. CERTIFICATION GUIDELINES: AN EXAMPLE In this section we present an example of a certification guideline, namely the need for feedback provision to the learner. The guideline is described as follows (Table 6):

Table 6. Example of the description part of a guideline

Guideline description

The self-assessment exercises and activities are the primary means of instruction in distance education. They should be designed very carefully in order to provide appropriate feedback due to the lack of physical contact between learner - instructor.

Goal: The main task of the evaluator is to asses the quality of feedback in the educational content.

What is needed: Each exercise (especially any self-assessment activity) should contain feedback, indicating where the learner should focus and why. There should be a way out of possible educational bottlenecks (e.g. through tips, further reading etc.).

It is important for the evaluator to understand how to implement the guideline (Table 7):

Table 7. Example of the implementation method

Guideline implementation method

Check the answers of each self-evaluation exercise and any other activity for:

• a clear statement of the educational goals of the exercise,

• analytical solution (in the self-assessment exercises),

• model of a solution or key points for solving (for activities only),

• reference to the level of difficulty,

• instructions for getting over possible educational bottlenecks.

Examples are an essential part of the guidelines. By using best practice/bad practice examples (most of them taken from real situations encountered by content evaluators), one provides insights and explanations on how to apply the guidelines and what to look for when assessing content (Table 8).

Table 8. Guideline example 1

Guideline examples of use The following example of an answer to a self-assessment question, shows how to state the educational goals, how to offer a reward to the student who has solved the exercise indicating the level of difficulty. It offers a way out of a bottleneck using the guide for further study.

Answer to Self-Assessment Exercise 2.1

This exercise is designed to familiarize you with

the design of control charts for quality control services in accordance with the standard ISO17000, taking into account the need for feedback.

The first step you should do to solve the exercise is to ...

If you managed to draw the diagram correctly, you succeeded in the most difficult point of the exercise, designing the feedback process; This means that you now are familiar with a key aspect of quality control.

This exercise was a bit difficult. If you did not succeed, do not worry, you probably need to give a little more time to develop your ability to analyze processes. You may do one repetition of section 2.1.1 emphasizing on example 2.2. In the guide for further study, you may find additional information in Chapter 2 of references 2 and 3 which analyze many similar examples.

It is important to try to understand the basic principles of diagram design because they are considered essential elements of audit processes. You will also need them in the next chapters which deal with non-deterministic quality assurance techniques.

Most guideline example sections contain 2-3 examples that complement each other (having a different focus). It is understandable that there are many ‘correct’ ways and personal styles in authoring educational content (Table 9).

Table 9. Guideline example 2

Guideline examples of use The next example is a response to an Activity. It analyzes the key points of the response and provides feedback.

Answer to Activity 5.8

In this activity you were asked to analyze briefly the vision of light according to Plato, by reading Bibliographic reference 1.

The work of Plato is of great importance, since it is considered by many....

If your answer spotted these 4 main points of Plato's vision then you are on the right track! If you did not manage to find them all, maybe you should read again more carefully section 3.1. After completing the activity, you may be interested in reading the 4th Chapter as well, for more information on how Plato laid the foundations for the Renaissance endorsements of light. A critical examination of the Platonic view and variations through time (with particular emphasis on the culture of the 20th century) can be found in the Guide for further study.

Remarks on the method used:

The last paragraph enhances the activity stirring the interest of students. It is important that activities are motivating students to further research into an object using the Guide for Further Study included in the content.

Examples are general in nature and not subject-specific, since it is nearly impossible to devise a technique that is both generally applicable to all educational subjects and specialized per subject (in terms of specific needs).

Draft

Finally, special conditions for the application of the guideline are presented (if necessary):

Table 10. Guideline remarks

Guideline Remarks

1. It is important to provide an effective way out of an educational bottleneck, particularly if this bottleneck is associated with a significant learning goal. Re-reading the same content without instructions, is usually not enough. Using the Guide for Further Reading and/or additional sources of information is more appropriate.

2. The solution should emphasize on any difficult points, analyzing them in more detail.

3. The author should design the content so that all of its components (learning objectives, main study content, evaluation and feedback mechanisms, further study) are closely linked. Any loose connection between them reduces significantly the educational impact.

7. TOWARDS A NEW QUALITY MODEL The next step of our research involves the mapping of phases and processes of the model to quality characteristics, sub-characteristics of a hierarchical standard structure (as the one used by ISO standards). The mapping of metrics is based mainly on the certification guidelines (which also provide hints on the distribution of quality characteristics and sub-characteristics). Our approach is based on the quality standards ISO / IEC 9126, ISO / IEC 25000 series and especially on ISO / IEC 25010 (Software engineering-Software product Quality Requirements and Evaluation (SQuaRE) Quality model) [11], the ISO / IEC 25012 (SQuaRE quality model-Data quality model) [17] and ISO 25020 (measurement method) [21].

The philosophy of our approach is to use the same model, but different evaluation approaches depending on the stage of the life cycle and the facet of the content that is being evaluated. Each approach corresponds to specific, discrete, characteristics and sub-characteristics.

As in other ISO standards, content evaluation, from the developer’s side of view, is mapped to internal quality characteristics. External quality is used for content assessment when the later is in use by real users or evaluators. Quality in use refers to the level of satisfaction of the learners while using the educational content and emphasizes on their behavior during use rather than the characteristics of the content per se.

According to ISO25010, quality evaluation happens in three main phases of a product life cycle: when the product is under construction, when the product is being evaluated and when phase the product is in use. Following the methodology of the series 25000 standards, Internal, External Quality and Quality in Use are associated with the phases of the content lifecycle. In this context, quality (and hence is characteristics) are connected to the content life cycle. Figure 3 depicts the process followed by ISO2520 (and by the proposed standard) for performing evaluation using the lifecycle- of a product. The requirements for Quality in Use determine the required level of quality from the users' perspective (learners, tutors). These requirements are used to validate the product by users. They are determined by specific Quality in Use metrics.

In turn, these requirements contribute to the definition of requirements for external quality. The latter are used for validation (validation & verification) of the educational content. These requirements are expressed quantitatively through external quality measures and used as criteria for evaluation.

Figure 3. Product life-cycle process according to ISO 2520 [21] External requirements contribute to the specification of internal quality requirements and which are, in turn, used to predict the requirements of Quality in Use. This creates a cycle, where the requirements of one category contribute to the specification of the requirements of next one; requirements are specified in the form of quantitative metrics - measures.

Initial results have produced a dual, data and process standard with quality characteristics, 30 sub-characteristics and 72 metrics. The model describes the quality characteristics of quality for any type of educational content used in distance education; it describes the qualitative characteristics of the educational content based on qualitative characteristics of data (data quality characteristics) and quality characteristics of software (software quality characteristics). Each of the qualitative features of the model vary in importance depending on the type of educational content being evaluated and the stage in the educational material applied.

The quality model can be applied as a guide for the design, creation and evaluation of educational content. Different perspectives for all stakeholders involved in the content lifecycle are present both internal & external evaluators and users (learners and teachers).

8. CONCLUSIONS In this work we presented the first step towards the establishment of a quality standard for educational content using a mixed approach. The next step of our research involves design and implementation of an VV(Verification and Validation) Tool that implements life-cycles’ quality evaluation with automated and semi-automated functions. We intend to validate this standard on the on-going production of digital educational material at HOU in order to assess its usefulness, practicability and make adjustments for a stable version.

9. Acknowledgements This paper has been co-financed by the European Union (European Social Fund – ESF) and Greek National funds through the Operational Program "Education and Lifelong Learning" of the National Strategic Reference Framework (NSRF) (Funding Program: “HOU” ).

Draft

10. REFERENCES

[1] Arrascaeta, F. 2007. Improved guidelines on implementing ISO 9001 in education sector. ISO Management Systems. (Nov/Dec 2007), 15-18.

[2] Pawlowski, J. M. 2007. The Quality Adaptation Model: Adaptation and Adoption of the Quality Standard ISO/IEC 19796-1 for Learning, Education, and Training. Educational Technology & Society, 10, 2, 3-16.

[3] ISO IWA 2. 2007. Quality management systems - Guidelines for the application of ISO 9001:2000 in education. ISO, Switzerland.

[4] Quality Assurance Agency for Higher Education (QAA). http://www.qaa.ac.uk/

[5] Cornell University Teaching Evaluation Handbook, 2011. http://www.cte.cornell.edu/resources/teh/teh.html

[6] The European Association for Quality Assurance in Higher Education (ΕΝQΑ), 2011. http://www.enqa.eu/

[7] Caraman, I., Lazar, G., Bucuroiu, R., Lungu, O., and Stamate, M. 2008. How IWA 2 Helps To Implement A Quality Management System In Bacau University. International Journal for Quality research. 37, 498.

[8] Roszak, M.T. 2009. Systemic approach to problems of the quality in education. Journal of achievements in materials and manufacturing technology, 37, 2, 751-758.

[9] ISO 19796-1. Information technology -- Learning, education and training -- Quality management, assurance and metrics -- Part 1: General approach. 2005. ISO Organisation.

[10] A Guide to the Project Management Body of Knowledge (PMBOK Guides).2013. Project Management Institute. ISBN 978-1-933890-51

[11] ISO 25010: Software engineering -- Software product Quality Requirements and Evaluation (SQuaRE) -- Measurement reference model and guide. 2011. Geneva, International Organization for Standarization.

[12] ISO 29990: 2010: Learning services for non-formal education and training -- Basic requirements for service providers. 2010. ISO, Switzerland.

[13] ISO9126: ISO/IEC 9126: Software Product Evaluation –Quality Characteristics and Guidelines for the User. 2008 Geneva, International Organization for Standarization.

[14] ISO/IEC 24751-1:2008. Information technology -- Individualized adaptability and accessibility in e-learning, education and training -- Part 1: Framework and reference model, 2008. Geneva, International Organization for Standarization.

[15] ISO/IEC TS 29140-2: 2011. Information technology for learning, education and training -- Nomadicity and mobile technologies -- Part 2: Learner information model for mobile learning. 2011. International Organization for Standarization.

[16] ISO/IEC 19788-3:2011. Information technology -- Learning, education and training -- Metadata for learning resources -- Part 3: Basic application profile. 2011. Geneva, International Organization for Standarization.

[17] ISO/IEC 25012: 2008. Software engineering -- Software product Quality Requirements and Evaluation (SQuaRE) --

Data quality model. 2008. Geneva, International Organization for Standarization.

[18] ISO/IEC 19778-2:2008. Information technology -- Learning, education and training -- Collaborative technology -- Collaborative workplace -- Part 2: Collaborative environment data model. 2008. Geneva, International Organization for Standarization.

[19] ISO/IEC TR 24725-3:2010. Information technology for learning, education and training -- Supportive technology and specific integration -- Part 3: Platform and Media Taxonomy (PMT). 2010. Geneva, International Organization for Standarization.

[20] ISO/IEC 2382-36:2008. 2010. Information technology -- Vocabulary -- Part 36: Learning, education and training. Geneva, International Organization for Standarization.

[21] ISO/IEC 25020:2007. Software engineering -- Software product Quality Requirements and Evaluation (SQuaRE) -- Measurement reference model and guide. 2007. Geneva, International Organization for Standarization. Draft