My eyes are closed!

https://www.opcw.org/media-centre/news/2022/07/opcw-advisory-board-education-and-outreach-advance-strategies-promote

I have just returned from The Hague, where I had the privilege to attend the 13th meeting of the Advisory Board for Education and Outreach of the #OPCW. Before the meeting, I had the great honour to work with several of the leaders of the OPCW to discuss a model for high-end e-learning to inform the OPCW's global educational initiatives. I began by asking them to describe a powerful learning experience. They shared incredible stories and identified these common elements of peak learning experiences:

* Feeling overwhelmed
* Exposure to content that deepened their understanding of the world
* Diversity of opinions within the learning environment
* Interactivity with the instructor and other students
* The growth of self-efficacy
* The opportunity to express what they were learning with family, colleagues, and friends
* Relevance to their real-world environments
* Reinforcing and strengthening their internal motivation

This describes what the best faculty and instructional designers try to achieve in any learning experience in any modality. High end e-learning requires particular attention to these elements, and this can only be achieved if the organization has strategy, infrastructure, and well-prepared instructors who can build community and connection. For an institution like the OPCW, these essentials can be supplemented by Open Education Resources and the issuance of micro-credentials that learners value.

It was a great and exciting meeting, and I am thrilled to be part of such an important mission. (Of course, my eyes are closed in the picture!).

Authentic Assessment for Online Learning MOOC - Final Report

Executive Summary

Authentic Assessment for Online Learning (AAOL) is a massive open online course (MOOC) first offered through the Commonwealth of Learning May 9 – June 3, 2022 using the mooKIT platform developed by the Indian Institute of Technology, Kanpur. The course curriculum was initially drafted by Dr. Dianne Conrad and revised by Dr. Jason Openo in the Winter of 2022. AAOL introduced the theory and practice of authentic assessments as the heart of the learning experience, with an emphasis on how authentic assessments intersect with academic integrity and various forms of human diversity.

The final curriculum for the four-week MOOC was provided by content expert Dr. Jason Openo, co-author of Assessment Strategies in Online Learning: Engagement and Authenticity (2018). AAOL explored the concept of learner-centred design for online assessment in higher education using short videos, open access readings, interactive forum discussions, and a learning portfolio assignment designed to model the characteristics and qualities of authentic assessments. Aligned with the theory of constructivism and authentic assessment, AAOL explored the changing nature of work in a digital age and the competencies and skills needed in the contemporary workplace, focusing on assessment strategies that engage and motivate learners in the e-learning environment and promote both academic integrity and deep learning. The course provided an overview of the fundamentals of creating learner-centered digital assessment through 21st century examples using the 5-dimensional framework for authentic assessment as both a tool for diagnosing existing assessments for authenticity and as a design template for new assessments.

The course was divided into 4 units. Unit 1 explored how the COVID-19 pandemic disrupted traditional forms of assessment and provided an opportunity to rethink assessment practice in postsecondary education. Unit 2 explored the intersection of authentic assessment and academic integrity. Unit 3 highlighted how authentic assessment is a values-based, context-dependent approach that can be applied in different cultural contexts, and how the values of authentic assessments overlap with and support Universal Design for Learning (UDL) principles. Unit 4 reviewed the challenges of moving towards more authentic assessments using practical examples from a wide variety of disciplines. A full outline of the curriculum is included in the final report.

2,541 individuals registered for AAOL, demonstrating the timely appeal of this learning topic. 1,522 active participants were drawn from all over the world, with the greatest concentrations in India (409), Nigeria (281), Kenya (243), Ghana (193), South Africa (132) and Malaysia (125).

An overwhelming majority of students were either postgraduates (1,192) or doctoral level educators (528). An overwhelming majority came from academic institutions (1,467) or governmental institutions (385).

There was a strong response to AAOL, and AAOL was well-received by participants. The following report documents the strong response to AAOL, suggesting that authentic assessment in online learning is a timely educational development topic, and that the instructional team provided a high-quality learning experience modeling best practices in online learning design. The report concludes with several learnings that may be used to refine this course’s development and the development of future of MOOCs, including comments on the mooKIT platform, and learner behaviour in a MOOC within the context of academic integrity.  

Participants submitted 358 pre-course surveys and 220 course evaluations. Respondents to an end-of-course survey (220) expressed a high level of satisfaction with the course, with 95.1% agreeing or strongly agreeing to a statement that learners got what they wanted out of the MOOC, and 99% agreeing or strongly agreeing to a statement that they had gained a greater understanding of authentic assessments.[JFA1]  As the raw responses to the survey indicate, the diverse array of learners valued different learning elements of the course, including the strong teaching presence of the instructional team, the curricular focus on the 9 Principles of Good Practice in Assessing Student Learning, the unit on academic integrity, and the practical steps of designing authentic assessments with UDL principles in mind. Participants specifically cited the recorded lectures, the reading materials, and the community as important elements that contributed to their learning.  

AAOL was designed for digital learning enthusiasts, senior managers, instructors, and others who are interested in building knowledge, competencies and skills that are usable in or transferable to the postsecondary workplace. The forums and the assessments offered a choice to participate in the course as a faculty member or as a leader within an academic institution (e.g., chair, dean, administrator). Strong participation in the varied pathways of the AAOL MOOC suggests learner choice held value, and that multiple levels of institutional influence are required to support the move to more authentic assessments.

Two types of certificates were available for participants of AAOL, a Certificate of Participation and a Certificate of Completion, which were granted according to participants’ level of participation and completion of assessment activities. The Certificate of Completion was awarded to participants completing a minimum of 60% on all 4 unit quizzes, the completion of the learning journal to required specifications, and meaningful participation in a minimum of 4 forum discussions. Certificates of Participation were awarded to those who received a minimum of 60% on all 4 quizzes and meaningful participation in at least 3 forum discussions.

There were a total of 376 certificates issued: 235 Certificates of Competency and 141 Certificates of Participation, for a total certification rate of 14.2%.

The following report elaborates upon the curriculum, the participants’ experience, and reflections from the instructional team for growth and improvement of the CoL’s future MOOC offerings.

Authentic assessment and academic integrity

https://www.col.org/news/registration-open-for-on-authentic-assessment-for-online-learning/

This question appeared in the MOOC forum today: “What is the best practice in curbing academic dishonesty among undergraduate students in higher institutions?”

There are at least 3 different answers to this. The first answer is we don’t know. Students commit academic integrity violations for lots of different reasons. For some, they worry they will not be able to achieve the reward they desire on their own merits; for some, cheating provides a thrill. To know what would curb academic dishonesty, we would need to know that a student was going to cheat or be dishonest but didn’t because we intervened somehow and our intervention was successful in changing their mind and behaviour. Establishing this causal chain of evidence is very difficult research. If we actually knew what worked, postsecondary institutions around the world would be in a much better place.

The second and most prominent approach involves telling students what academic integrity is, providing examples, and then telling them that if they commit an academic integrity violation and get caught, they will be punished severely. The most dominant method is punitive and threatening. This is the path of increased surveillance and the technological arms race where students get more creative in their approaches, including hacking institutional IT systems to change a grade or steal the final exam. Do a Google search for "how to cheat Respondus Monitor" or "how to cheat Proctorio." If we build a jail, there will always be someone thinking about how to escape.

The best practice, and the one this course advocates for, is the third way to answer this question. The best way to curb academic dishonesty is to create relevant and engaging learning tasks that connect with a learner’s motivation and goals. Different cultural conceptions of textual ownership, copyright, language, and educational systems all come into play when discussing academic integrity, but all students have ethical beliefs and learning goals, and if the assessments we design ask students to engage in meaningful tasks they care about, they are less likely to act in a dishonest fashion. Students are more likely to cheat in less personalized, less task-oriented learning environments. By creating more personalize and more task-oriented learning environments, authentic assessments can appeal to students’ higher natures.

Technology cannot stop academic dishonesty. Authentic assessments will not put an end to cheating. But they can help by changing the nature of the game.

Exams and authenticity

https://www.col.org/news/registration-open-for-on-authentic-assessment-for-online-learning/

Are exams authentic? Can they be authentic?

The short answer is yes. The longer answer is in certain circumstances.

There are disciplines where exams are a concrete reality. In my context, for example, if you wish to be a plumber, welder, engineer, nurse, or information technology professional, there are certain exams students must successfully complete to participate in professional practice. In those instances, educators have a professional (perhaps moral) obligation to do what they can to best prepare their learners to take those exams. Using the 5-dimensional framework:

• Task – Pass the discipline specific exam

• Physical/digital context: Online proctored exam

• Social context: Individual, private assessment

• Polished product: Test score

• Criteria and standards: Right or Wrong? (1st time pass rate)

In these certain circumstances, exams are authentic because they match what a learner must be able to achieve in a real-world context. In my context, if a nursing student doesn’t pass the NCLEX, they don’t get to be a nurse. Our Nursing program is also judged on the percentage of nursing students who pass the NCLEX their first time. In those certain circumstances, online or in-person proctoring will be required to ensure that students are honest. But during their 4-years of study, our nursing program also makes heavy use of simulation and clinical, which come as close as possible to the ideal of authentic assessment. Students are recorded doing activities, and they can’t fake it.

Remember the major point from the last post; all assessment strategies are imperfect. Because we cannot achieve perfectly authentic assessments in this imperfect world, we need to be clear about how our assessments are imperfect. There will be rare times and occasions when exams are authentic, but exams are often accompanied by other assessment strategies that provide a holistic understanding of the learning process.  

Dr. J

Authentic assessment is a political act

https://www.col.org/news/registration-open-for-on-authentic-assessment-for-online-learning/

Authentic assessment is a political act. As the 9 Principles of Good Practice in Assessing Student Learning puts it, “There is a compelling public stake in education. As educators, we have a responsibility to the publics that support or depend on us to provide information about the ways in which our students meet goals and expectations” (Astin, et al.) We have an obligation to ourselves as educators, to the disciplines we care about, and to our students. But we also have an obligation to society to improve our assessment strategies.

I just finished reading Failure to disrupt: Why technology alone can’t transform education (Reich, 2020). Reich argues that “all assessments are imperfectly designed” (p. 177), which means you won’t and can’t develop a perfectly authentic assessment. But it also means we need to pay close attention to how our assessments are imperfect. In online education, more and more people are turning towards online proctoring and/or auto-grading. There are several problems with this. First, surveillance is not good pedagogy. Secondly, autograders are good at routine tasks, but “these are exactly the kinds of task we no longer need humans to do” (p. 171). Third, as Reich states, “we need students to develop complex communication skills and take on unstructured problems,” such as problem finding and problem framing. If we want to assess people on the kinds of performance that are most worthwhile for people to learn, we will have to depend heavily on assessments evaluated individually, and this represents a challenge to large-scale learning environments. 

I’ll talk more about exams and essays tomorrow. There are times when they are necessary and occasions when they are authentic, but they are imperfect in very different ways than authentic assessment, and one of their imperfections (as noted by the International Centre for Academic Integrity and the European Network for Academic Integrity) is that they are the assessments students are most motivated to cheat on or can most easily outsource through contract cheating.

Dr. J

Astin, A. W., Banta, T. W., Cross, P., El-Khawas, E., Ewell, P. T., Hutchings, P., Marchese, T. J., McClenney, K. M., Mentkowski, M., Miller, M.A., Morgan, E. T., & Wright, B. D. Nine principles of good practice for assessing student learning. https://www.ncat.edu/_files/pdfs/campus-life/nine-principles.pdf

Reich, J. (2020). Failure to disrupt: Why technology alone can’t transform education. Harvard University Press.

Assessment & Teaching Presence

https://www.col.org/news/registration-open-for-on-authentic-assessment-for-online-learning/

Moore’s 3 forms of interaction matured and evolved in the Community of Inquiry’s 3 presences – cognitive presence, teaching presence, and social presence. Within the Community of Inquiry framework, assessment is part of teaching presence, defined as “the unifying force” that “brings together the social and cognitive processes directed to personally meaningful and educationally worthwhile outcomes” (Vaughn, et al., 2013, p. 12).  Teaching presence consists of design, facilitation, organization, delivery, and assessment.

“Assessment very much shapes the quality of learning and the quality of teaching. In short, students do what is rewarded. For this reason one must be sure to reward activities that encourage deep and meaningful approaches to learning” (Vaughn, et al., 2013, p. 42). 

In its simplest terms, deep learning is good and surface learning is bad. When adopting a surface learning approach, students consume content as a commodity to be acquired through a form of mechanistic transfer.

Students who take a deep learning approach “recognize the dynamic and interrelated structure of content to be learned, and learning is less a process of knowledge transfer than one of exploration, discovery, and ultimately, growth” (Platow, et al., 2010).  Deep learning is an approach to learning consciously or unconsciously selected by the student, but the design of the learning opportunity encourages students to adopt a particular approach.  

In short, assessment is the heart of the student experience, and assessment design will determine how students behave and engage with the learning environment. Authentic assessments change the nature of the relationship to content and can encourage students towards deep learning. Deep learning principles recommend active and interactive learning, and “graded activities that require collaboration and constructivist thought will encourage students to work toward this end” (Vaughn, et al., 2013, p. 33).

Dr. J

Platow, M. J., Mavor, K. I., & Grace, D. M. (2013). On the role of discipline-related self-concept in deep and surface approaches to learning among university students.  Instructional Science: An International Journal of the Learning Sciences, 41(2), 271-285. doi: 10.1007/s1125101292274

Vaughan, N. D., Garrison, D. R., & Cleveland-Innes, M.  (2014). Teaching in blended learning environments: Creating and sustaining communities of inquiry. Athabasca University Press. http://site.ebrary.com/id/10840609

Authentic Assessment for Online Learning - The 3 Interactions

https://www.col.org/news/registration-open-for-on-authentic-assessment-for-online-learning/

As efforts to expand online education proceed, it is critical to design more interactive educational experiences that integrate regular, direct, and meaningful contact and communication (Protopsaltis & Baum, 2019).

Moore (1989) realized this 30 years ago, and he identified three types of learner interaction.

1. Learner-content

2. Learner-instructor

3. Learner-learner

Learner-content interaction “is a defining characteristic of education. Without it, there cannot be education, since it is the process of intellectually interacting with content that results in changes in the learner’s understanding” (p. 1).  Learner-content in this course takes place with the lectures and the readings, the quizzes, and the assignments. The “tyranny of content” (Petersen, et al., 2020) can squeeze out other forms of interaction, however.

Learner-instructor interaction takes place through the curriculum , where the instructors seeks to stimulate the learner’s interest and motivation. Learner-instructor interaction takes place directly in the lectures, and indirectly in every aspect of the course.

Learner-learner interaction, as Moore writes, “is sometimes an extremely valuable resource for learning.” I would go even further; my favorite quote about teaching is probably this one: "The best answer to the question, 'What is the most effective method of teaching?' is that it depends on the goal, the student, the content and the teacher. But the next best answer is, 'students teaching other students.'" (McKeachie, et al., 1987, p. 63). Learner-learner interaction takes place in the forums, where the learners have responded to, expanded upon, and enlightened others with their knowledge and experience. It also takes place in collaborative work.

Authentic assessments have the potential to weave these three interactions into a single learning event. Learners interact with the content in relevant, real-world ways. The instructor designs the curriculum and the authentic task, and can provide coaching to the students. If the task is a collaborative one, learners interact with each other in the process of building the polished product. This is partly why authentic assessments can be such powerful learning experiences.

Dr. J

McKeachie, W. J., Pintrich, P. R., Lin, Y., and Smith, D. A. F. National Center for Research to Improve Postsecondary Teaching and Learning, A. M., & And, O. (1987). Teaching and Learning in the College Classroom. A Review of the Research Literature (1986) and November 1987 Supplement. 

Moore, Michael. (1989). Three Types of Interaction. American Journal of Distance Education. 3. 1-7. 10.1080/08923648909526659. 

Petersen, C. I., Baepler, P., Beitz, A., Ching, P., Gorman, K. S., Neudauer, C. L., Rozaitis, W., Walker, J.D., & Winger, D. (2020). The tyranny of content: "Content coverage" as a barrier to evidence-based teaching approaches and ways to overcome it. Life Sciences Education, 19(2). https://www.lifescied.org/doi/10.1187/cbe.19-04-0079

Protopsaltis, S., & Baum, S. (2019). Does online education live up to its promise: A look at the evidence and implications for federal policy. George Mason University. https://distance-educator.com/does-online-education-live-up-to-its-promise-a-look-at-the-evidence-and-implications-for-federal-policy/

Authentic Assessment for Online Learning - MOOC

https://www.col.org/news/registration-open-for-on-authentic-assessment-for-online-learning/

May 9, 2022 was one of the most exciting days of my professional life because Authentic Assessments for Online Learning launched with 2,440 dedicated educators from around the world! Thank you all for joining this course! 

One of the participants asked, "What is online learning?" This is a seemingly simple question with NO clear answer? 

Duus (2009) observes it is difficult to refer to online education as a singular entity because it is so big (like referring to an ocean as a big body of salt water). Duus makes a critical distinction between “low-end e-learning” and “high-end e-learning." Low-end e-learning is characterized by content transmission and is volume-based. Technology provides the innovation to these popular, mainstream strategies that are often erroneously made synonymous with all forms of e-learning (Duus, 2009, Figure 1). Duus argues that most of the discourse and academic research has focused on low-end e-learning, and while it is simplistic to categorize it this way, low-end e-learning is best conceived as content-heavy education with low interactivity (Protopsaltis & Baum, 2019, p. 30).

Moving towards a learning-centred instructional approach means moving towards interactivity, moving towards student engagement and student knowledge-construction. Moving towards authentic assessments is a move from low-end to high-end e-learning. 

Online education is a form of distance education where the primary delivery mechanism is via internet-based technologies, but this definition lacks nuance because the terminology struggles to keep up with the invention of new approaches and applications. Online education may include synchronous “face-to-face” technologies such as Blackboard Collaborate Ultra, Zoom, or Google Meet, asynchronous or multi-synchronous platforms such as the learning management system (LMS) and Google Docs, and/or participatory flow technologies such as Twitter, Facebook, Mentimeter, and Padlet.

Dron (2014) suggests that online instruction will grow to include “emerging systems and their capabilities for assembly and integration” that allow for a “depth of sophistication that we have not seen before.”

I love that phrase - a depth of sophistication we have not seen before - this depth of sophistication is what we are after as we explore authentic assessments.

Dr. J

Dron, J. (2014).  Innovation and change: Changing how we change. In Zawacki-Richter, O. & Anderson, T. (Eds.), Online Distance Education (pp. 237-265). Athabasca University Press.

Duus, H. J. (2009). A socioeconomic approach to the development of e-learning. E-Learning & Education (eleed) Journal, 5. https://eleed.campussource.de/archive/5/1985

Protopsaltis, S., & Baum, S. (2019). Does online education live up to its promise: A look at the evidence and implications for federal policy. George Mason University. https://distance-educator.com/does-online-education-live-up-to-its-promise-a-look-at-the-evidence-and-implications-for-federal-policy/

CIDER Session

https://www.youtube.com/watch?v=l2zILzYTA7U

I delivered this Centre for Innovation in Distance Education Rsearch (CIDER) session on April 6, 2022. It’s me reading the highlights of my dissertation in 45 minutes.

The Multiple Realities of Professional Development for Online Contingent Faculty in Canadian Strategy and Practice

The growth of contingent faculty and the growth of online education over the first two decades of the 21st century have generated an emergent but overlooked subgroup of faculty – online contingent faculty. These twin dynamics have placed the professional development of online faculty in a strategically important position for Canadian postsecondary institutions to enhance online instructional effectiveness and mature online educational quality. This presentation outlines how a two-phase multimethod research study employed Ursula Franklin’s technology as practice to explore the following research questions: How are online faculty and their professional development represented in current Canadian postsecondary academic plans? How are the professional development needs of contingent online faculty being served by Canadian teaching and learning centres?

Phase one consists of a document analysis of 17 academic plans from Canadian colleges and institutes covering the current period and immediate future to reveal how faculty development is described and prioritized in academic strategy (the projected reality of the future). The document analysis highlights important strategic purposes of professional development, such as Indigenization and internationalization, but also shows that part-time and online faculty are marginally represented. Email interviews with 12 directors of Canadian teaching and learning centres comprise phase two (the extended reality of experience) and illuminate the contested space of providing educational development services to online contingent faculty. The findings reveal formidable barriers to providing professional development opportunities to part-time faculty who teach online, but also innovative solutions to meet the needs of part-time online educators in Canada.

The expanding possibilities of professional development

The growth of contingent faculty and the growth of online education over the first two decades of the 21st century have generated an emergent but overlooked subgroup of faculty – online contingent faculty. Exposure to part-time instructors and participation in online education can both negatively impact student success, and these twin dynamics place the professional development of online contingent faculty in a strategically important position for Canadian postsecondary institutions to enhance online instructional effectiveness and mature online educational quality. This presentation details the analysis of email interviews with 12 directors of Canadian teaching and learning centres to illuminate the enduring barriers of providing educational development to online contingent faculty, including unequal participation in online education, a lack of programs targeted at part-time instructors, especially in smaller institutions, and that part-time faculty are usually uncompensated when participating in professional development. Despite these significant barriers, the interviews also show that the COVID-19 pandemic spurred innovations to meet the needs of part-time online educators in Canada. Teaching and learning centres are striving to meet the unique needs of part-time faculty with emerging programs that build a comprehensive teaching identity through faculty secondments and mentoring programs. These expanding possibilities for professional development must account for the unique needs of online contingent faculty, the frequent conflict between technology and pedagogy, and the highly political nature of quality assurance in online education. The presentation concludes by sketching a potentially darker future where professional development is minimized by unbundling, outsourcing, and the rise of microprograms.

https://youtu.be/2QGquduDigA

Openo, J. (2021, October). The expanding possibilities of professional development for online contingent faculty in Canada. [Conference session]. Athabasca University 2021 Graduate Student Research Conference, Edmonton, AB.