Posts in category Papers

Enabling Real-Time Adaptivity in MOOCs with a Personalized Next-Step Recommendation Framework

This paper is presented at the Fourth (2017) ACM Conference on Learning @ Scale in Cambridge (MA), USA, April 20–21, 2017.

Abstract

In this paper, we demonstrate a first-of-its-kind adaptive intervention in a MOOC utilizing real-time clickstream data and a novel machine learned model of behavior. We detail how we
augmented the edX platform with the capabilities necessary to support this type of intervention which required both tracking learners’ behaviors in real-time and dynamically adapting content based on each learner’s individual clickstream history. Our chosen pilot intervention was in the category of adaptive pathways and courseware and took the form of a navigational suggestion appearing at the bottom of every non-forum content page in the course. We designed our pilot intervention to help students more efficiently navigate their way through a MOOC by predicting the next page they were likely to spend significant time on and allowing them to jump directly to that page. While interventions which attempt to optimize for learner achievement are candidates for this adaptive framework, behavior prediction has the benefit of not requiring causal assumptions to be made in its suggestions. We present a novel extension of a behavioral model that takes into account students’ time spent on pages and forecasts the same. Several approaches to representing time using Recurrent Neural Networks are evaluated and compared to baselines without time, including a basic n-gram model. Finally, we discuss design considerations and handling of edge cases for real-time deployment, including considerations for training a machine learned model on a previous offering of a course for use in a subsequent offering where courseware may have changed. This work opens the door to broad experimentation with adaptivity and serves as a first example of delivering a data-driven personalized learning experience in a MOOC.

Keywords

Adaptivity; Personalization; Real-time intervention; MOOC; RNN; Behavioral modeling; Navigational efficiency; edX

Reference

Zachary A. Pardos, Steven Tang, Daniel Davis, and Christopher Vu Le. 2017. Enabling Real-Time Adaptivity in MOOCs with a Personalized Next-Step Recommendation Framework. In Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale (L@S ’17). ACM, New York, NY, USA, 23-32. DOI: https://doi.org/10.1145/3051457.3051471

Download

Teaching Software Engineering Principles to K-12 Students: A MOOC on Scratch

This paper is accepted for Software Engineering Education and Training @ 39th International Conference on Software Engineering (SEET-ICSE 2017) in Buenos Aires in May 2017.

Abstract

In the last few years, many books, online puzzles, apps and games have been created to teach young children programming. However, most of these do not introduce children to broader concepts from software engineering, such as debugging and code quality issues like smells, duplication, refactoring and naming. To address this, we designed and ran an online introductory Scratch programming course in which we teach elementary programming concepts and software engineering concepts simultaneously. In total 2,220 children actively participated in our course in June and July 2016, most of which (73%) between the ages of 7 and 11. In this paper we describe our course design and analyze the resulting data. More specifically, we investigate whether 1) students find programming concepts more difficult than software engineering concepts, 2) there are age-related differences in their performance and 3) we can predict successful course completion. Our results show that there is no difference in students’ scores between the programming concepts and the software engineering concepts, suggesting that it is indeed possible to teach these concepts to this age group. We also find that students over 12 years of age perform significantly better in
questions related to operators and procedures. Finally, we identify the factors from the students’ profile and their behaviour in the first week of the course that can be used to predict its successful completion.

Keywords

Programming education, MOOC, Scratch, code, smells, dropout prediction

Reference

Felienne Hermans, Efthimia Aivaloglou (2017) Teaching Software Engineering Principles to K-12 Students: A MOOC on Scratch. TUD-SERG-2017-003. ISSN 1872-5392

Download

Opening University Education to the World and Improve Education

Paper presented at the ICERI 2016 conference, 14th-16th November 2016, Seville, Spain.

Abstract

Online education can be used as a catalyst for gaining knowledge on learning and learning processes due to its generation of massive corpora of data on student behaviour. This knowledge can then be utilized to improve the quality of education.

Since 2013 Delft University of Technology (TU Delft) offers online courses for a global population of lifelong learners through its programme for Massive Open Online Courses (MOOCs). TU Delft’s MOOC programme was created with three specific goals:
(i) to deliver high quality open & online education (O2E) to the world;
(ii) to improve education; and
(iii) to grow research output. Since its inception, TU Delft’s MOOC programme has created and run over forty MOOCs, gaining nearly a million enrolled students in the process.

As TU Delft’s MOOC programme developed and expanded over time, an organisational structure was created in which educational processes and research activities were aligned and integrated. This organisational structure supports three research agendas:
(i) course evaluation, which focuses on post-course analysis;
(ii) research-driven innovation through short-cycled research projects; and
(iii) long-term experimental research with a specific focus on big data and learning analytics.

Through the integrated organisational structure, data is simultaneously collected for all three research agendas. This includes user feedback through survey data for course evaluation; user enrolment and activity data for the short-cycled research projects; and data from experiments for the long-term research. Analysing this data has resulted in dozens of course evaluation reports, business and marketing analyses, cross-course analyses, internal reports on student learning behaviour, and a substantial number of peer-reviewed academic papers about results of learning analytics and pedagogical innovations. This output has been useful for their individual research tracks, but combining the results has provided TU Delft with additional insights only attainable by careful synchronization of the three. TU Delft has benefited from these insights and has adapted to the findings both in online and on-campus course design. TU Delft’s MOOC programme provides a valuable environment for innovating educational design experience and developing new educational delivery strategies that can also be used to improve on-campus education.

Future plans build on the current organisational structure and include:
(i) using the results from learning analytics interventions experiments to build a learning analytics ‘suite’ for all online courses;
(ii) testing and validating TU Delft’s proprietary Online Learning Experience (OLE) pedagogical model for online course design; and
(iii) using these experiences to transform and improve on-campus education.

First steps in these matters are already under way with the acquisition of a new digital learning environment and its accompanying learning analytics suite for on-campus education. This way the MOOC programme serves TU Delft’s strategic goals of both educating the world and improving the quality of its online and on-campus education.

Keywords

higher education, innovation, learning analytics, massive open online courses (moocs), links between education and research.

Reference

J.P. van Staalduinen, D. Davis, S. Topolovec (2016) OPENING UNIVERSITY EDUCATION TO THE WORLD AND IMPROVING EDUCATION: USING MOOC-BASED RESEARCH AS A TOOL FOR INNOVATION, ICERI2016 Proceedings, pp. 7336-7343.

Download

Acknowledgements

We gratefully acknowledge the support of the Erasmus+ Programme of the European Union; the Erasmus+ Forward Looking Cooperation STELA Project, number 562167-EPP-1-2015-BE-EPPKA3-PI-FORWARD.

What have they done with the MOOCs?! The impact of MOOCs on Campus Education

Article published in the Conference Proceedings of The EADTU Online, Open and Flexible Higher Education Conference 19-21 October 2016 in Rome.

Abstract

In 2013 Delft University of Technology (TU Delft) started to offer Massive Open Online Courses (MOOCs) onthe EdX platform. One of the main principles in the MOOC development process was to publish alleducational resources under an open license (Creative Commons License) (Ouwehand, 2015). This alignedwith TU Delft’s Open Access Policy. Another important starting point in the MOOC development process was the aim to improve campus education by integrating MOOC content in those courses.
The impact of MOOCs concerns not only the world outside the university, but more importantly also withinthe university. Especially for a traditional brick-and-mortar research-based university, like TU Delft, this is abig gain: education has become more important. In the past three years it has become clear that developing a MOOC has led lecturers to re-think their approach towards teaching and to integrate MOOC materials in campus education, which has impact on the way they teach on campus.
MOOCs are used on campus in different ways, from a small addition to an existing course to a full integration into a completely redesigned campus course. Moreover, some teachers became conscious of the importance of educational resources under CC License and started to use material from other universities. One of the faculties created a course which uses MOOC materials to help the students to prepare for a master program.
This paper describes the way in which MOOCs have been used in campus education and the impact this has had on teaching and learning.
Keywords: Open Educational Resources, Improving, Campus Courses, MOOC, blended learning

Reference

Cabral, Pedro; Van Valkenburg, Willem; Dopper, Sofia (2016). What have they done with the MOOCs?! The impact of MOOCs on Campus  Education in Ubach, George & Konings, Lizzie (2016). Conference Proceedings The Online, Open and Flexible Higher Education Conference 2016. Published by EADTU. ISBN:978-90-79730-25-4 [Page 652-660]

Download

Slides

Can Learners be Earners? Investigating a Design to Enable MOOC Learners to Apply their Skills and Earn Money in an Online Market Place

This article is published in the journal IEEE Transactions on Learning Technologies.

Abstract

Massive Open Online Courses (MOOCs) aim to educate the world. More often than not, however, MOOCs fall short of this goal — a majority of learners are already highly educated (with a Bachelor degree or more) and come from specific parts of the (developed) world. Learners from developing countries without a higher degree are underrepresented, though desired, in MOOCs. One reason for those learners to drop out of a course can be found in their financial realities and the subsequent limited amount of time they can dedicate to a course besides earning a living. If we could pay learners to take a MOOC, this hurdle would largely disappear. With MOOCS, this leads to the following fundamental challenge: How can learners be paid at scale? Ultimately, we envision a recommendation engine that recommends tasks from online market places such as Upwork or witmart to learners, that are relevant to the course content of the MOOC. In this manner, the learners learn and earn money. To investigate the feasibility of this vision, in this paper we explored to what extent (1) online market places contain tasks relevant to a specific MOOC, and (2) learners are able to solve real-world tasks correctly and with sufficient quality. Finally, based on our experimental design, we were also able to investigate the impact of real-world bonus tasks in a MOOC on the general learner population.

Reference

Guanliang Chen, Dan Davis, Markus Krause, Efthimia Aivaloglou, Claudia Hauff, Geert-Jan Houben, “Can Learners be Earners? Investigating a Design to Enable MOOC Learners to Apply their Skills and Earn Money in an Online Market Place”, IEEE Transactions on Learning Technologies, vol. , no. , pp. 1, 5555, doi:10.1109/TLT.2016.2614302

Download

Push or Pull Students into Blended Education: a Case Study at Delft University of Technology

Paper is published in the International Journal of Engineering Education (ISSN 0949-149X) Volume 32 Number 5(A).

Abstract

Blended education, or ‘‘flipping the classroom’’ is rapidly becoming a mainstream form of teaching within universities. Within Engineering Education, it is popular as it allows more time in-class to focus on hands on activities such as demonstrations and solving complex problems. This paper discusses the effort conducted to re-structure, according to the blended learning principles, the ‘‘Propulsion and Power’’ course of the Aerospace Engineering Bachelor degree programme at Delft University of Technology (TU Delft). The redesigned course was supported by a dedicated online & blended education unit within the university, and is characterized by a very peculiar structure due to the different approach chosen by the two involved lecturers. The first lecturer decided to ‘‘pull’’ the students, by proposing a number of additional videos available in the World Wide Web as a support and complement to the material taught in class. Conversely, the second lecturer opted for a ‘‘push’’ approach, self-recording theory videos to be watched by the students at home and devoting the in-class hours to exercises and applications of the theory. This format resulted in a clear improvement of the average exam grades and pass rates. The student feedback showed enthusiasm about the new blended course, with only a very small minority still preferring the previous, more traditional approach. Although there seems to be a slight preference of students towards the ‘‘push’’ strategy, the ‘‘pull’’ approach has also been widely appreciated.
However, the objective to re-attract students to the contact hours in class was only partially achieved, since just a slight improvement in the number of attending students was observed. This paper clearly shows that the efforts to implement a blended teaching strategy has great benefits for both students and staff alike.

Keywords

blended learning; online education; student engagement; electronic assessment; aerospace engineering

Reference

Cervone, A., Melkert, J.A., Mebus, L.F.M., Saunders-Smits, G.N. (2016). Push or Pull Students into Blended Education. A Case Study at Delft University of Technology. In International Journal of Engineering Education. Volume 32 Number 5A. ISSN 0949-149X (page 1911-1921)

Download

Download paper from researchgate

MOOCs and their Effect on the Institution

Paper was published in journal Foro de Educación.

Abstract

Four years after the introduction of MOOCs – which were proclaimed to be «the end of education as we know it» in 2012 – the role and effect of these free, online courses is becoming clearer. The online means of delivery to the heterogeneous audiences of MOOCs have enabled and compelled instructors and course teams to develop innovative and flexible learning materials. We can analyse the data on the study behaviour of learners to identify which course elements are effective. In addition, the integration of elements of MOOCs in campus education has resulted in promising outcomes and positive reactions from both students and teachers. On the level of the institution, we also see the effect of MOOCs: ranging from new possibilities in communication and branding, to new needs for faculty development and the support organisation. Furthermore, MOOCs play a role in the unbundling of education, e.g. the learning experience and the assessment tasks now can be uncoupled and may be delivered by different institutions and by different means: the learning experience may be in the form of a MOOC and the assessment may be a written exam at an institution.

Reference

Kiers, J. (2016). MOOCs and their Effect on the Institution: Experiences in Course Design, Delivery and Evaluation; Research; Faculty Development; Unbundling and Credits for MOOCs. Foro de Educación, 14(21), 133-149. doi:http://dx.doi.org/10.14516/fde.2016.014.021.007

Download Paper

Augmented Learning Environment for wound care simulation

Paper presented at the EDEN Annual Global Meeting 2016 in Budapest, Hungary from 15-17 June 2016.

Introduction

Emerging technologies for teaching and learning have made it possible to create environments, scenarios and virtual patients that simulate clinical practices in order to promote the development of skills and knowledge in healthcare education (Lewis et al., 2005; Hogan, Sabri, & Kapralos, 2007). These simulations are seen as educational techniques that bring interactivity and immersion into the learning process, allowing the recreation of clinical experiences without the risk of causing harm to patients (Maran & Glavin, 2003). Other known advantage is the possibility learners have to practice an unlimited number of times a procedure or technique until correct realization, before applying it in real-world scenarios (Rey et al., 2006).
Virtual reality (VR) and Augmented reality (AR) are examples of emerging technologies for teaching and learning that allow the creation of digitally enhanced learning environments. These technologies are expected to have an impact in education, as highlighted by the Horizon report for Higher Education in 2010, 2011, and most recently in 2016 (Johnson et al., 2010, 2011, 2016). Regarding healthcare education, several studies indicate the positive effect of AR and VR in developing decision making skills and practical procedures using virtual simulators, with a higher impact on non-experienced participants (Zhu et al., 2014).
However, using VR in healthcare education can be a debatable approach since it immerses learners in a synthetic environment, enabling them to see the surrounding real-world. Acting in a different environment from which learners will have to act in real life scenarios is another concern to take into account.

Reference

Jorge, Nelson; Morgado, Lina; Gaspar, Pedro (2016). Augmented Learning Environment for wound care simulation in Teixeira, Szucs and Mazar (2016). Conference Proceedings EDEN 2016 Annual Conference. Page 256- 264. ISBN 978-615-5511-10-3. License CC-BY 4.0.

Download

The TU Delft Online Learning Experience: From Theory to Practice

Best Practice Initiative AwardPaper presented at the EDEN Annual Global Meeting 2016 in Budapest, Hungary from 15-17 June 2016. At the conference Gala Dinner the paper received the EDEN 2016 Best Practice Initiative Award.

Introduction

In 2014 TU Delft started an innovation programme to educate the world and improve quality of education based on online learning. The programme included open (OpenCourseWare and MOOCs) and online (Professional Education, Bachelor and Master) courses. Lecturers of TU Delft have shared their knowledge in MOOCs with more than 750,000 learners around the world. Next to the open courses, more than 800 learners enrolled in our online courses. Adopting new instructional strategies based on online learning elements has had a very positive impact in TU Delft’s overall education, benefiting traditional on campus education and contributing to the changing need in educating engineers (Kamp, 2014). Naturally, blended learning has arisen on campus, where online learning materials are reused in a flipped classroom approach with very positive results: higher completion rates, higher average grade, more flexibility for students to interact with the course material and more flexibility for teachers in choosing which elements to include in the interactive classroom sessions (van Valkenburg, 2015).

The development of TU Delft online courses is based on the Online Learning Experience (OLE), a pedagogical model that supports the development of our courses and strives for increasing quality. The creation of the OLE was an important step for TU Delft, contributing to the development of online courses in a more systematic and consistent way, guiding all course development teams through the realisation of several shared educational principles.

Last year, when we presented the OLE at the 2015 EDEN Conference in Barcelona, we were still at an early stage of its development, collecting fundamental background to support it and feedback from online learning experts. Although we only had a collection of ideas translated into 8 principles, it was clear that the model needed to be flexible in order to accommodate many educational scenarios that coexist among TU Delft’s Faculties, but with a clear and useful purpose to help improve the quality of our online education (Jorge, Dopper & van Valkenburg, 2015). This paper describes how the OLE is applied in practice.The main goal of the OLE is to improve the quality of our online education by setting course design and development principles to support course teams. At the same time, the OLE can be used as a tool to promote reflection before the course starts to set expectations, and in the end to evaluate and plan improvements for the next run. In the next sections we’ll describe the OLE in both ways – as course design principles (guidelines) and as a tool (the radar graph).

Reference

Jorge, Nelson; Van Valkenburg, Willem; Dopper, Sofia (2016). The TU Delft Online Learning Experience: From Theory to Practice in Teixeira, Szucs and Mazar (2016). Conference Proceedings EDEN 2016 Annual Conference. ISBN 978-615-5511-10-3. License CC-BY 4.0

Download

video of teacher’s perspective of Online Learning Experience:

Beyond the MOOC platform: gaining insights about learners from the social web

Paper presented at the 8th ACM Conference on Web Science in Hanover, Germany.

Abstract

Massive Open Online Courses (MOOCs) have enabled millions of learners across the globe to increase their levels of expertise in a wide variety of subjects. Research efforts surrounding MOOCs are typically focused on improving the learning experience, as the current retention rates (less than 7% of registered learners complete a MOOC) show a large gap between vision and reality in MOOC learning.

Current data-driven approaches to MOOC adaptations rely on data traces learners generate within a MOOC platform such as edX or Coursera. As a MOOC typically lasts between five and eight weeks and with many MOOC learners being rather passive consumers of the learning material, this exclusive use of MOOC platform data traces limits the insights that can be gained from them.

The Social Web potentially offers a rich source of data to supplement the MOOC platform data traces, as many learners are also likely to be active on one or more Social Web platforms. In this work, we present a first exploratory analysis of the Social Web platforms MOOC learners are active on — we consider more than 320,000 learners that registered for 18 MOOCs on the edX platform and explore their user profiles and activities on StackExchange, GitHub, Twitter and LinkedIn.

Reference

Guanliang Chen, Dan Davis, Jun Lin, Claudia Hauff, and Geert-Jan Houben. 2016. Beyond the MOOC platform: gaining insights about learners from the social web. In Proceedings of the 8th ACM Conference on Web Science (WebSci ’16). ACM, New York, NY, USA, 15-24. DOI=http://dx.doi.org/10.1145/2908131.2908145

Download

Download paper

© 2011 TU Delft