The Multiplication Table as an innovative Learning Analytics Application

Eltion KrajaEducational Technology, Graz University of Technology, Austria

krajael@sbox.tugraz.at

Behnam TaraghiEducational Technology, Graz University of Technology, Austria

b.taraghi@tugraz.at

Martin EbnerEducational Technology, Graz University of Technology, Austria

martin.ebner@tugraz.at

Abstract: The main topic of this paper is the development of a web-based application that helps children to learn the one-digit multiplication table. The developed application supports individual learning process of the pupils and also provides the teachers with the possibility to intervene according to the analysis of users’ answers. The application uses modern technologies in order to offer high performance and availability to the users. The system also provides an interface for mobile clients, which present the questions and the processed data in different forms. The answers of the pupils, as well as other gathered data from the application show interesting results related to the participation and learning improvement.

Introduction

Nowadays the possibility for the users to use their mobile devices, websites or different technologies for learning is growing. Also the learning process and the methodology of learning have changed, depending on the way in which the teachers present the learning material and how the pupils consume it.

Independently from the form of learning, the characteristics of learning are (Berg 2011): 1) individuality, as there are different methods and strategies how to acquire knowledge, 2) activity, considering that the process of learning is a physical and mental activity, 3) constructiveness that includes the symbols and conventions we use to learn, 4) accumulation of knowledge - new knowledge is added to the preexisting, 5) self regulation, the learning process has its individual pace (fast, slow), and 6) dependence on the situation because learning is also a product of experience, emotions etc..

Using digital and electronic material, the users produce many data. The data can be used to evaluate the behavior of the users and also to make considerations concerning a group of users i.e. a school class of pupils. Having big amount of user data, intervention strategies and predictions can be created in order to influence the process. The produced data can be used for different purposes, such as for data evaluation, for the identification of patterns, for a better use of resources, to create individual learning plans, for the identification of the strength and weakness of the group and to improve the learning materials and communication.

Learning analytics as a field of study that processes and analyzes such data helps to improve learning efficiency. The 2016 Horizon Report1 describes learning analytics as: "an educational application of web analytics aimed at learner profiling, a process of gathering and analyzing details of individual student interactions in online learning activities".

1[] https://library.educause.edu/topics/teaching-and-learning/learning-analytics (last visit 15.10.2016)

Usability is another field of study in this work. It plays a significant role in web applications. Usability (Scholtz at al. 2016) is defined by the ISO as follows: “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use”.

In this work, we introduce a learning application, which is designed for school pupils to learn one-digit multiplication problems. To make the application more interesting, the posed questions and the tasks are presented in form of a game. The idea of gamification has been applied more often in many game based learning applications in recent years. While playing a game, users can be motivated by trying to achieve more points, badges, themes etc. Gamification is defined as: “using game-based mechanics, aesthetics and game thinking to engage people, motivate action, promote learning, and solve problems” (Kapp 2012). Children tend to learn and discover new things, but most often there is a problem of knowledge presentation. Furthermore, if one asks a child “What is work?”, she would probably answer “homework”. Asking, “What is fun”, the answer could be “video games”. The motivation of the children to play games and doing homework while it is amusing, can be included in the learn processes1.

This paper describes the implementation and the features of the new “1x1 trainer” application. The created platform is aimed to apply the knowledge from the mentioned research fields above to replace the old version that was in operation till March 2016. The application’s logic is based on the previous work by (Schön et al. 2012). The implemented algorithm provides users with the multiplication tasks. The selection of tasks is based on user’s profile and the degree of her knowledge. The new 1x1 web application refractors the structure of the old application, provides some new features (gamification aspects) and bases on the newest technology stack. The application is extended so that it is able to communicate with other existing platforms (e.g. platforms for central user management and mobile apps).

The developed new web learning application keeps track of the user’s learning behavior. It presents each pupil individually the result and the level of his/her knowledge. The children can earn points while answering the questions and use them to activate or enable the game characters. The users also have the possibility to observe their game activities and thereby revise the gained knowledge presented during the game.

The teachers and the administrators can access and evaluate the data produced by the pupils. In order to have an overview over the classes and groups, the data is presented in a clustered form and can be easily scaled to a more detailed level. For a better analysis, the information is highlighted with colors. Besides that, indicators and charts are used to give the user a quick overview. The collected data is used to provide an individual learning support as well as to give feedback to the children on their performance.

State of the Art

This section covers three main areas of this work, namely Learning Analytics, Usability and Gamification.

Learning analytics

Learning Analytics is an innovative research field that uses intelligent algorithms to process (searching, filtering, mining and visualizing (Khalil & Ebner 2016)), exploit and retrieve useful and meaningful information from the data gathered from the users.

The Society for Learning Analytics Research (SoLAR) defines Learning Analytics as: “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (Siemens & d Baker 2012).

Four aspects of learning analytics are introduced by (Daniel 2016). These are: what, who, why and how to analyze. More exactly, What explains the data used for the analysis and the environment or context the date is

1[] http://yukaichou.com/gamification-examples/top-10-ecommerce-gamification-examples-revolutionize-shopping/ (last visit 20.10.2016)

retrieved from, e.g. LMS1 and PLE2 (Ebner et al. 2010). Who: involves the stakeholders the learning analytics is applied for, e.g. teachers, students, tutors, institutions, and researchers. Why: explains the objectives of the analysis. This could be monitoring the learning behavior, prediction of learning outcome or necessary interventions, mentoring, recommendations, personalization and supporting adaptive learning systems. How: is about how the data is analyzed, which patterns or methods are used; some examples are statistics, visualizations, data mining, Social Network Analysis and machine learning.

Furthermore Khalil and Ebner (Khalil & Ebner 2015) proposed the Learning Analytics Life Cycle, which reveals the relation between the four aspects mentioned above. The life cycle begins with the stakeholders within a learning environment from which the data is gathered. This might be interaction data between stakeholders and/or stakeholders and the learning environment (traces), the personal data or academic information in general. The data needs to be processed (analyzed) in the next step using different methods and techniques; some examples were mentioned above. The result of the analysis step leads to the data interpretation that is the objective.

Usability

Heuristics Evaluation (HE) is an informal method where evaluators can analyze usability aspects of a user interface (Nielsen & Molich 1990). Alsumait and Al-Osaimi (Alsumait & Al-Osaimi 2009) described a set of heuristics for e-learning applications. One aspect that is to be evaluated is the general user interface is hat it should be very intuitive and appropriate for 6-10 years old children. The pedagogical aspect as well as supporting the learners in the learning process is also very important. The sets of heuristics are divided into three categories, by which the mentioned aspects are analyzed. These are: 1) Nielsen Usability Heuristics (NUH) (Alsumait & Al-Osaimi 2009) that is concerned with general usability und design aspects; 2) Child Usability Heuristics (CUH) (Alsumait & Al-Osaimi 2009) focused on the child abilities and preferences; 3) E-learning Usability Heuristics (EUH) (Alsumait & Al-Osaimi 2009) focuses on the learner centered design. These three categories are briefly described as follows:

1) Nielsen Usability Heuristics (NUH) (Alsumait & Al-Osaimi 2009, Nielsen 1994): According to NUH, the system should keep the user informed about what is happening and give him feedback in order to motivate the user to use the application and stay active. Additionally, the user should have the possibility to see his status (points, level, etc.) in the application and also understand the terminology used. The system presented to the user should have familiar concepts from the real world and also be built in an intuitive form, in order to guarantee a clear functionality of the buttons and other interface elements. Furthermore giving the user the control, freedom and knowledge to navigate and move in the system within defined frames is a very important aspect. Characteristics of this aspect are for example the possibility to undo false input, to move easily in the application, to navigate through and filter large amount of data, to save the state of the application, etc. The application should give the user feedback and prevent errors by design. The application should provide the user with the possibility to correct his input (e.g.: by asking questions such as “Did you mean…”). Another characteristic of this heuristic is “Recognition vs. Recall”. It means that the user should not have to remember the information when navigating from one view to another. He should have the possibility to get all the information he needs quickly from the current view. Additionally the tool should integrate beginners but also help experts to speed up their interaction. Also showing the information in different way is a very important aspect.

2) Child Usability Heuristics (CUH) (Alsumait & Al-Osaimi 2009, Sim et al. 2006, Barendregt et al. 2003): According to CUH, the screen layout should be efficient, attractive and also be presented as simple and readable. All elements of the site need to be chosen suitable for the children. Game or application devices should be chosen appropriately for the target user group. Buttons with no functionality should be disabled in order to prevent input errors. The user should have enough information to use the application when he turns it on. The goals of the application should be clear. It should also challenge but not frustrate the user. The application should be easy to learn but hard to master. While playing, the application should reward the child to motivate and encourage him. The user should get involved in the program and its topics as far as possible consequently. The tool should help the

1[] Learning Management System2[] Personal Learning Environment

children to use their imagination. The user should be able to use his imagination to interpret the game; the characters are chosen appropriately for the target user and they should attract the child’s interest and curiosity.

3) E-learning Usability Heuristics (EUH) (Alsumait & Al-Osaimi 2009): The terminology and the used vocabulary should be chosen appropriately and suitable for the stakeholders. There are many examples and graphics to illustrate abstract formulas or rules. The structure of the content should be well chosen, so that the rapid understanding of the goal is possible. The application should keep the user concentrated and motivated throughout the activities, stories and games situations.

Gamification

Gamification is a concept with an enormous potential in the field of education. The Gamification technique should increase interest, curiosity, and fun. It should challenge the user, while using learning software (Martí-Parreño et al. 2016, Scholtz et al. 2016, Brull & Finlayson 2016, van Roy & Zaman 2017).

In order to motivate the children to use a game-based learning application different works about rewards and gamification were analyzed. Becker and Nicholson (Becker & Nicholson 2016) describe meaningful gamification as a personal connection to a non-game setting. The use of specific elements in the application causes the user to reflect on the situations he experienced during the game. Additionally the use of narratives helps the users connect the context to their own experiences. Giving users the freedom to make decisions, to try other alternatives and to explore the context is also very important. Some of the gamification characteristics described by (Becker & Nicholson 2016 ) are (1) Reflection, (2) Exposition, (3) Choice, (4) Information, (5) Play, and (6) Engagement – communication and engagement between the users.

Previous System

The application that is described here is based on the previous software and work made by (Schön et al. 2012). The applied algorithm in this application is based on the idea of the degree of competence (DoC), which aims to select the appropriate questions depending on the user’s knowledge. At the beginning the pre-knowledge of the user, his initial DoC respectively, is estimated from a pretest; this is done by asking two questions to estimate the DoC of the user, as illustrated in (Fig. 1).

Figure 1. Pre-DoC estimation by (Schön et al. 2012)

The DoC or the so-called learn rate will be regulated, according to the user’s answers, within the interval of [0, 1]. The questions are divided into two parts: the already-known ones (those that are below the user’s learn rate) and the ones that are to learn or to practice (those above the user’s learn rate). The algorithm is programmed to avoid giving the users very difficult or very easy questions in order to prevent frustration or boringness. The algorithm presents a mix of known and unknown questions, within the range of user’s learn rate. In order to motivate the user, the algorithm also selects questions from the so-called extended learning area, which is 25% above the user’s learn rate (see Fig. 2).

Figure 2. Distribution of the questions based on the degree of competence (Schön et al. 2012)

The questions are classified into following three categories according to the users’ answers:

- Unknown questions (identified via incorrect answers or unanswered questions) - Known questions (identified by correct answers) - Well-known questions (known questions that are answered correctly more than once. Depending on

the probability of the question, the user has to answer a question more than twice right in order to classify it as well-known).

The so-called “result type” is also a significant aspect of the categorization of the answers. It divides the answers into “NR” (a new question, correctly answered), “NW” (a new question, incorrectly answered), “KR” (a known question, correctly answered) and “KW” (a known question, incorrectly answered) categories.The collected answers and their categories have provided a data basis for this work and many other previous related works such as (Taraghi et al. 2015, Taraghi et al. 2014).

The Multiplication Table (1x1) Trainer

The so-called existing “Einmaleins-Trainer” has been rebuilt based on the new web-technologies.The challenge, while implementing the application, was the combination of the modern technologies, the flexible structure as well as applying the outcome of the research fields discussed in section “State of the art”. To make the trainer available for as many users as possible the application was written as a web-platform. The application also offers a SOAP-interface to mobile devices and other co-applications such as one responsible for the user management. The application structure consists of a client and a server component. The structure and the chosen design patterns guarantee that the application is extendable and flexible for further improvements and future work. (Fig. 3) gives an overview about the structure of the new 1x1 trainer.

Figure 3. Structure of the 1x1 application

The application offers four user roles: supervisor, administrator, teacher and pupil. The Administrator is allowed to see all registered entities and to gather grouped and clustered information, but also to zoom into the detailed view of each entity. The administrator can also regulate the system settings. The Teacher can only observe his own pupils and classes. He can also view the data in a clustered form and can scale down and filter the information to see the individual answering behavior of each pupil. Using the new 1x1 application, the teachers can quickly find an answer for many questions that could be of their interest, such as:

- Which pupils exercised mostly? - Who never played? - Which questions are/were very easy to answer by the pupils? - Which questions are/were difficult to solve?

Figure 4. Clustered answers. Teachers and administrators can view at once the answers’ status. The colors vary from “dark green = very easy” to “dark red = very difficult” symbolizing the difficulty of the question.

(Fig. 4) shows an overview of the clustered data. Through the colors and the information in the matrix, a teacher can see which questions were answered easily (the green ones) and which were very difficult (the red ones).

The Pupil can play the game and answer the questions. (Fig. 5) shows the state of the game before the user begins to play. Pupils can view their own activities and gain points for correct answers, which in turn can be used to activate the game characters (see Fig. 6). The characters bring more variety and fun in the game (through changing layouts, animations etc.).

Figure 5. Game view

Figure 6. Available characters

The answered questions will flow to the main statistic, so that the teacher can estimate the progress of the user.

The application is based on responsive design and can be accessed from modern browsers of every device. The symbols are simple and meaningful so that they are familiar for the users from other programs and everyday life (e.g.: game symbol, play and stop button etc.). The pupils get a feedback after each answered question. They have an overview over their answered questions. The characters are consciously designed to be gender neutral and be familiar to children. The pupils have also the possibility to review their exercises and learn from questions experienced in the past.

Teachers have all the information in form of a statistic with different diagrams and lists. They can quickly reveal the strength and weaknesses of the class or a group of pupils on different dimensions. The application offers the possibility to reveal the relation between question and answers, difficultness, and the relations within the classified answers in different forms.

The visualizations can be filtered and scaled down in order to better target the point of interest and eventually help teachers to intervene for improvement of their pupils.

Discussion and Outlook

We have gathered large amount of data through the 1x1 trainer application. The application is used by 18 schools, containing 83 classes and 7164 users (who have answered at least two questions). Totally 1033237 questions have been already solved1.

1[] Per 9 October 2016

The bar chart of (Fig. 7) depicts the distribution of the classified answers, described before. It is obvious that the majority of the questions are well-known, followed by known and unknown questions.

The technology and structure used for the implementation of the application allows further extensions and enhancements. There is the possibility to reach more devices and more operating systems.

The 1x1 trainer application is implemented using Zend Framewok 21 for the server component and AngularJS2 for the client component. These frameworks allow a modular setup. They are also suitable for dividing the application logic into smaller parts. The client component is suitable for building hybrid apps with platforms like cordova3 or ionic4. In this way it is for example possible to build a Windows app with the same look and feel as the web app. The server component could be extended to offer more web-services in order to provide more data to the clients. Furthermore there is still potential to improve and extend the current visualizations.

Currently the exercises can be answered by typing the result in an input form. Other methods like multiple choice options can be implemented as well. To challenge the user even further, the possible answers could be presented close to each other.

The algorithm, which is responsible to select the appropriate question to be posed to the user, can easily be exchanged. This allows the administrators to compare the results and data produced by the children via different learning algorithms. Hence the effectiveness and performance of different algorithms on the learning goal can be analyzed.

Figure 7. Classified answers. Well-kwon questions in the dark green bars, known questions in the light green bars and unknown questions (wrong answers) in the red ones

References

Alsumait, A., & Al-Osaimi, A. (2009, December). Usability heuristics evaluation for child e-learning applications. In Proceedings of the 11th International Conference on Information Integration and Web-based Applications & Services(pp. 425-430). ACM.

1[] https://framework.zend.com/ (last visit 20.11.2016)2[] https://angularjs.org/ (last visit 20.11.2016)3[] https://cordova.apache.org/ (last visit 20.11.2016)4[] http://ionicframework.com/ (last visit 20.11.2016)

Barendregt, W., Bekker, M. M., & Speerstra, M. (2003, September). Empirical evaluation of usability and fun in computer games for children. In Proc. IFIP Interact (Vol. 3, pp. 705-708).

Becker, K., & Nicholson, S. (2016). Gamification in the Classroom: Old Wine in New Badges. Learning, Education and Games, 61.

Berg, M. (2011). Wie lernen Kinder das Lernen?. GRIN Verlag.

Brull, S., & Finlayson, S. (2016). Importance of Gamification in Increasing Learning. The Journal of Continuing Education in Nursing, 47(8), 372-375.

Daniel, B. K. (Ed.). (2016). Big Data and Learning Analytics in Higher Education: Current Theory and Practice. Springer.

Ebner, M., Scerbakov, N., Taraghi, B., Nagler, W., Kamrat, I., & Maurer, H. (2010, March). Teaching and learning in higher education–an integral approach. In Proceedings of Society for Information Technology & Teacher Education International Conference (pp. 428-436).

Kapp, K.M. (2012). Using game-based mechanics, aesthetics and game thinking to engage people, motivate action, promote learning, and solve problems. In The gamification of learning and instruction: game-based methods and strategies for training and education. John Wiley & Sons. p 10

Khalil, M., & Ebner, M. (2015, June). Learning analytics: principles and constraints. In Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications (pp. 1326-1336).

Khalil, M., & Ebner, M. (2016). When Learning Analytics Meets MOOCs-a Review on iMooX Case Studies. In Innovations for Community Services: 16th International Conference, I4CS 2016, Vienna, Austria, June 27-29, 2016, Revised Selected Papers 16 (pp. 3-19). Springer International Publishing.

Martí-Parreño, J., Seguí-Mas, D., & Seguí-Mas, E. (2016). Teachers’ Attitude towards and Actual Use of Gamification. Procedia-Social and Behavioral Sciences, 228, 682-688.

Nielsen, J. (1994, April). Usability inspection methods. In Conference companion on Human factors in computing systems (pp. 413-414). ACM.

Nielsen, J., & Molich, R. (1990, March). Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 249-256). ACM.

Scholtz, B., Raga, L., & Baxter, G. (2016). Design and evaluation of a “gamified” system for improving career knowledge in computing sciences.

Schön, M., Ebner, M., & Kothmeier, G. (2012, April). It's just about learning the multiplication table. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 73-81). ACM.

Siemens, G., & d Baker, R. S. (2012, April). Learning analytics and educational data mining: towards communication and collaboration. In Proceedings of the 2nd international conference on learning analytics and knowledge (pp. 252-254). ACM.

Sim, G., MacFarlane, S., & Read, J. (2006). All work and no play: Measuring fun, usability, and learning in software for children. Computers & Education, 46(3), 235-248.

Taraghi, B., Saranti, A., Ebner, M., Müller, V., & Großmann, A. (2015). Towards a Learning-Aware Application Guided by Hierarchical Classification of Learner Profiles. J. UCS, 21(1), 93-109.

Taraghi, B., Saranti, A., Ebner, M., & Schön, M. (2014, June). Markov chain and classification of difficulty levels enhances the learning path in one digit multiplication. In International Conference on Learning and Collaboration Technologies (pp. 322-333). Springer International Publishing.

van Roy, R., & Zaman, B. (2017). Why Gamification Fails in Education and How to Make It Successful: Introducing Nine Gamification Heuristics Based on Self-Determination Theory. In Serious Games and Edutainment Applications (pp. 485-509). Springer International Publishing.