Category: learning

Learning While Building eLearning: Part 2 – Learning Objectives & Assessment

signsThis is the second (slow!) of four pieces reflecting on the experiences of Emilio, a subject matter expert who was tasked with converting his successful F2F training into an elearning offering. Emilio has let me interview him during the process.

This piece focuses on the thorny issue of learning objectives at the front end of an elearning project and assessment at the other end. You can find the context in part 1 here. (Disclaimer: I was an adviser to the project and my condition of participation was the ability to do this series of blog posts, because there is really useful knowledge to share, both within the colleague’s organization and more widely. So I said I’d add the blog reflections – without pay – if I could share them.)

Nancy: Looking back, let’s talk about learning objectives. You started with all of your F2F material, then had to hone it down for online. You received feedback from the implementation team along the way. What lessons came out of that process? How do we get content more precise when you have fewer options to assess learner needs and interests “in the moment” as you do face to face, and with the limited attention span for online?

Emilio: I realize now that I had not thought about being disciplined with learning objectives. I had created them with care when I first developed my F2F offering. Once I had tested the course several times, I recognized that I forgot my own initial learning objectives because in a F2F setting I adapted to student’s interest and knowledge gaps on the spot, and I was also able to clarify any doubts about the content. Therefore, over time, these learning objectives become malleable depending on the group of students, and thus lost presence in my mind.

This became apparent as  I was doing the quizzes for the online work and got comments back from Cheryl (the lead consultant).  She noted where my quiz questions were and were NOT clarified in the learning objectives and content. I realized I was asking a bunch of questions that were not crucial to the learning objectives.

With that feedback, I narrowed down the most important questions to achieve and measure the learning objectives. It was an aha moment. This is something that is not necessarily obvious or easy. You have to put your mind into it when you are developing an e-learning course especially. It applies to the F2F context as well, but in an e-learning setup you are forced to be more careful because you cannot clarify things on the spot. There is less opportunity for that online.  That was very critical. (Note: most of the course was asynchronous. There were weekly “office hours” where clarifications happened. Those learners who participated in the office hours had higher completion rates as well.)

It was clear I had to simplify the content for elearning set up – and that was super useful. While my F2F materials were expansive to enable me to adapt to local context, that became overload online.

Nancy: What was your impression of the learners’ experiences?

Emilio: It was hard to really tell because online we were  dealing with a whole different context. Your indicators change drastically. When I’m in F2F I can probe and sense if the learners are understanding the material. It is harder online to get the interim feedback and know how people are doing. For the final assessment,  we relied on a final exam with an essay question. The exam was very helpful in assessing the learner’s experience, but since it is taken at the end of the course, there are no corrective measures one can take.

Nancy: Yes, I remember talking about that as we reviewed pageviews and the unit quizzes during the course. The data gives you some insight, but it isn’t always clear how to interpret it. I was glad you were able to get some feedback from the learners during your open “office hours.”

We used the learning objectives as the basis for some learner assessment (non graded quizzes for each unit and a graded final exam which drew from the quizzes.) How did the results compare with your expectations of the learners’ acquisition of knowledge and insights? How well did we hit the objectives?

Emilio: We had 17 registered learners and 7 completed. That may sound disappointing. Before we started, I  asked you about participation rates and you warned me that they might be low and that is why I am not crying. The 7 that completed scored really well in the final exam and you could see their engagement. They went through material, did quizzes and participated in the Office Hours. One guy got 100% in all of the quizzes, and then 97% in the exam.

We had 8 people take the final exam. One learner failed to pass the 70% required benchmark, but going deep into it, Terri (one of our consultants) discovered the way Moodle was correcting the answers on the multiple choice was not programmed precisely. It was giving correct answers for partially correct answers. We need to fix that.  Still, only one failed to pass the 70% benchmark even with the error.

The essay we included in the exam had really good responses. It achieved my objective to get an in depth look at the context the learners  were coming from. Most of them described an institutional context. Then they noted what they thought was most promising from all the modules,  what was most applicable or relevant to their work. There were very diverse answers but I saw some trends that were useful. However, it would be useful to have know more of this before and during the course.

Nancy: How difficult was it to grade the essays? This is something people often wonder about online…

Emilio: I did not find it complicated, although there is always some degree of subjectivity. The basic criteria I used was to value their focus on the question asked, and the application of all possible principles taught during the course that relate to the context described in the question.

Nancy: One of the tricky things online is meaningful learner participation. How did the assessment reflect participation in the course?

Emilio: We decided not to give credit for participation in activities because we were not fully confident of how appropriately we had designed such activities for an e-learning environment in this first beta test. I think this decision was the right one.

First, I feel that I did not do a good job at creating an atmosphere, this sense of community, that would encourage participation. Even though I responded to every single comment that got posted, I don’t really feel that people responded that much in some of the exercises. So I would have penalized students for something that is not their fault.

Second, we had one learner who did every exercise but did not comment on any of the posts. He is a very good student and I would have penalized him if completion relied on participation. Another learner who failed did participate, went to the office hours and still did not pass the final exam.

We failed miserably with the group exercise for the second module.  I now realize the group exercise requires a lot of work to build the community beforehand.  I sense this is an art. You told me that it is completely doable in the elearning atmosphere, but after going through the experience I really feel challenged to make it work. Not only with respect to time, but how do you create that sense of community? I feel I don’t have a guaranteed method for it to work. It is an art to charm people in. I may or may not have it!

Nancy: The challenges of being very clear, what content you want to share with learners, how you share it, and how you assess it should not be underestimated. So often people think it is easy: here is the content! Learning design in general  is far more than content and learning design online can be trickier because of your distance from your learners – and not just geographic distance, but the social distance where there is less time and space for the very important relational aspects of learning.

Up Next: Facilitating Online

Embracing Uncertainty, Diversity and Othering for Learning

For a little over 2 years I’ve been supporting an internal learning program for my colleague Pete Cranston of Euforic Services. As we wrap up this project on “Learning about learning” in an international development context, Pete and I have debriefed and he is writing a series of comprehensive blog posts on our reflections. I want to pick up on one particular aspect of our conversation that struck me through the work, and which has also come up in a number of other contexts: the role of others in our learning. How does otherness help us learn? (Dr Zuleyka Zevallos, thanks for your great resource site on otherness!)

What we observed

Instead of starting with defining otherness, I want to share my experience. One important way to learn is with and from others. This is particularly relevant in workplace learning. In international development, regardless of the domain (agriculture, health, water/sanitation/hygiene, etc.) each investment of money, time and practice holds the potential for useful learning – useful to those implementing the work themselves, and to others interested in the same work. Often these lessons get stuck locally, or even remain invisible or unconscious because they don’t get a chance to be examined by ourselves or others. It is no surprise that many find a stark lack of reflection time, alone or with others. We learn through reflection, conversation, and the modeling and mentorship of others.

Our hypothesis in the learning to learn project (LAL for the acronym lovers out there) was that simply by creating more space and opportunity for conversational interaction between project staff, new insights would emerge and existing insights would gain wider visibility and application. We had a particular emphasis on conversational practice, both face to face and online. Learning with and from each other. While there was no explicit intention for mentorship, that possibility was always open.

Through the process, we noted the following three things that made a difference in conversational learning:

  1. Come out as a learner. Declare our intention and orientation towards learning by how we prioritize time and how we practice as learners. This includes being open about not knowing, willing to take learning risks (what, you don’t KNOW the answer!!!) and practice the art of asking questions and LISTENING to answers.
  2. Recognize what we are doing is reflective practice. We call it M&E (monitoring and evaluation), reviews, and all those business process words. It is more fundamental: Stop. Think. What did we actually learn? What do we know more about now? What new questions does that uncover? Unless we are intentional about our reflective practice, the learning drips by. Another way to view this is the idea of  catching ourselves learning. Simply noticing triggers the potential and recognition of learning.
  3. Create space and leadership for learning.  We can think about making it happens as about space. Or spaces. The fastest, but perhaps least rich is surveys. The shrink the time commitment and we know time is the biggest impediment. But what about actually making spaces for reflection? Can we commit 10 minutes in a weekly meeting? Something once a month? An annual learning review? We need to PUT that stuff in, notice when and how it becomes intentional. We must lead with it from from top by role modeling and recognizing reflection as a valid, yes, IMPORTANT use of our time. Yes, stuff t happens without leaders, but leaders make it happen better by lending legitimacy. We must lead from every direction and in every corner. Reflection and learning can be recognized by donors. Without being too literal, what about 10% for M&E, 10% for communications, and 10% for learning. That leaves 70% for the DOING and probably will increase the effectiveness and value of that doing.Don’t get me wrong. Learning is also random, chaotic, free flowing, emergent, and what we learn may often be different than what we set out to learn. The LAL project included very basic funded Learning Exchange Visits where grantees visited each others projects and went into the field. They had real time with each other and stuff just happened. The immersion triggers experiences and memories that take you conceptually to a new point. They can suggest new practices, behaviors and even ways of monitoring and evaluation. In the pursuit of learning with an open agenda, we embrace uncertainty and find new learning entry points.

Adam Frank writes about the Liberating Embrace of Uncertainty, which resonates with these three things Pete and I observed:

For science, embracing uncertainty means more than claiming “we don’t know now, but we will know in the future”. It means embracing the fuzzy boundaries of the very process of asking questions. It means embracing the frontiers of what explanations, for all their power, can do. It means understanding that a life of deepest inquiry requires all kinds of vehicles: from poetry to particle accelerators; from quiet reveries to abstract analysis.

So what about otherness?

When we discussed #3, I asked Pete, what did we notice about the culture of the organizations and individuals involved? We talked about the huge value of the leader’s openness, despite a position of positional and resource power (the funder!) We talked about the diversity of project staff – geographically, organizationally, amount of experience (age), etc. Then I was reminded about a question Bron Stuckey had recently asked on Facebook :

I am wondering how you would respond to the proposition “The best person to influence the practice of a novice is someone who was most recently a novice themselves” It’s some research bubbling up for which I have a real passion. I do have to admit it is more driven by intuition and first hand experience than a body of literature or prior research 🙂 Just wondering what my respected colleagues think of this proposition. Does it fit with your research, experience and vision?

The discussion that followed Bron’s question was diverse and fascinating. I’m sharing the thread,  Bron Stuckey Learning With Others with her permission. Through it came the thread that learning happens in a (diverse) ecosystem of people, contexts and content and that who we learn with influences our learning.

The other we know and who knows us: Sometimes we learn best from people who’s experience is close to or just advanced of ours. It is a place of relative comfort.

The other we respect (and/or fear): Sometimes we learn from people with influence and power because that causes us to pay attention. Sometimes we learn from the deep experience of a expert or long-time practitioner.

The other we aren’t so sure about: Sometimes we learn from people who are different from us, hold different opinions and experiences because that asks us to examine our own understandings, biases and assumptions. This is a powerful but often more challenging set of others when we can step across mistrust and even animosity and step curiously into learning from someone who really is the “other” from us.

How otherness feeds into #1, #2, and #1

Coming out as a learner means we have to trust ourselves, regardless of the “others” we engage with. So part of our identity has to be “learner.” Reflective practice asks us to question our own assumptions, not just those of the “others.” And creating space for learning asks us to invite in the “others.” So for me, the bottom line is a strong look inward into identity, reflection and invitation. This requires us to stop our often unwitting exclusion of others, of not inviting, not listening and not engaging. Thus the opening picture. I bet you wondered why I picked it, right?

Stuff that Inspired this Post:

Symmathesy = Learning together.(Pronounced: sym- math-a-see) A working definition of symmathesy might look like this:Symmathesy (Noun): An entity composed by contextual mutual learning through interaction. This process of interaction and mutual learning takes place in living entities at larger or smaller scales of symmathesy.Symmathesy (Verb): to interact within multiple variables to produce a mutual learning context….

The way in which we have culturally been trained to explain and study our world is laced with habits of thinking in terms of parts and wholes and the way they “work” together. The connotations of this systemic functional arrangement are mechanistic; which does not lend itself to an understanding of the messy contextual and mutual learning/evolution of the living world.

Reductionism lurks around every corner; mocking the complexity of the living world we are part of. It is not easy to maintain a discourse in which the topic of study is both in detail, and in context. The tendency is to draw categories, and to assign correlations between them. But an assigned correlation between a handful of “disciplinary” perspectives, as we often see—does not adequately represent the diversity of the learning fields within the context (s). The language of systems is built around describing chains of interaction. But when we consider a forest, a marriage, and a family, we can see that living entities such as these require another conceptual addition in their description: learning.

 

P.S. Added on Friday – Catherine mentioned an image in her comment below. Here is the image!

tempoeespacios

Resource Guide to Learning Delivery Methods

resource guide to learning

What learning delivery methods are there and how do you consider them? In 2014 Scott Leslie and I supported an analysis for decision making on training strategies for the International Labour Organization’s Microinsurance Innovation Facility, now the Impact Insurance Facility (http://www.impactinsurance.org/ ).

As I moved on to other related works, I kept drawing on these initial documents. Finally, I’ve gotten around to a) getting permission from the ILO to share (THANK YOU!) and trying to pull out some of the domain specific stuff so it might be more widely useful.

The Resource Guide to Learning Delivery Methods  (pdf for now) is a version of one of the outputs of our work, shared with permission from the Facility with the hope that it adds value to your work. (The other elements include a Glossary and a Strategic Options documents.) We hope to also put these online for crowd-sourced critique and revisions. (TBA!) In the meantime, I just want to get this OUT THERE!

From the Introduction:

This document is a version of one of the outputs of our work, shared with permission from the Facility with the hope that it adds value to your work. (The other elements include a Glossary and a Strategic Options documents.) We hope to also put these online for crowd-sourced critique and revisions. (TBA!)

The Facility team was evaluating their e-learning options to expand capacity building for microinsurance for the poor. An early identified need was to understand elearning in the wider context of delivery mechanisms.

The basic content can be useful when starting to consider capacity development, training or specifically an e-learning strategy. It is not exhaustive, and some things have become dated since it was written. Understanding that, it can help you understand the range of learning options, and where they might be most effective.

We surveyed 20 different learning delivery methods across five major categories:

  • Face-to-face delivery methods
  • Online delivery methods:
    • Traditional online courses
    • MOOCs, communities & self-directed learning
    • Synchronous methods
    • Mobile delivery methods
  • Offline delivery methods
  • Blended and hybrid methods

This guide is the detailed analysis of these methods in the context of the Facility’s domain. It includes a general description, domain-related examples, and provides insight and comparisons on these methods based the qualities the organization identified as key to assessing any proposed solution. These qualities include:

  • the implications of this method on scalability and adaptability of content
  • implications on quality control
  • any effects the method may have on motivating learners’ completion of training and achievement of learning objectives
  • the ease and costs of implementing any of these methods

Your qualities may be different, so you may wish to consider your needs and how the methods may or may not meet your needs. This document is offered as a Free Cultural License as defined by the Creative Commons https://creativecommons.org/licenses/ .

If Teaching is Relational, How Does That Inform Online Teaching?

learningI was in my car the other day, thinking about all the workplace learning projects I’m currently involved in. At times, I feel like I’m moving a giant rock up a hill, with gravity being “lets push content to our learners.” I keep pointing to the importance of practice, context, reflection, informal learning. But I realize I have not sufficiently highlighted the importance  of teaching as a practice about relationships. About being human. About, liberation and yes,  about LOVE.

On our local public radio station, KUOW, ran a story about Nate Gibbs-Bowling reflecting on segregation in Washington state schools and what he is doing about it. At the core of his thesis was that teaching was relationship centric. His success with students of color was not dependent on his subject matter expertise, but his relationship with those students. If you have five minutes, take a listen: Washington Schools Are Segregated And That’s Not OK | KUOW News and Information

Nate’s experience in K-12 education rang a bell with me in my experience with workplace learning. To get the engagement that leads to gains in the application of learning, I use two things. The first is to work as hard as I can to make sure the learning offering has real, applicable relevance to the learner in doing their job. The second is to get to know them and use that relationship to engage with the learners to co-discover ways to liberate the application of the learning.

So if we believe (and hopefully can prove) that teaching is relational, what does that imply for online learning and teaching online, especially the proliferation of self paced, content centric elearning? Or worse, online teaching as enforcement and control.

It means we have to challenge the status quo of content-centricity! This does not mean throwing out content, but it means starting with relationship in the appropriate contexts.

My friend and colleague, educator Brad Beach of Australia and I have been having a years long, very slow conversation on what unlocks learner engagement online and if it varies by domain. I have changed a lot of my thinking about online interaction over time, both as I’ve learned and the environment has evolved, but one thing has always been central. Treat people like real, human beings. Use what the Dali Lama calls being “heard, seen and loved.” We may use the word “respected,” but I think it really is about love. But suit yourself! 🙂

A week ago during one of these conversations (they usually happen very early my Friday morning, Brad’s late Friday night) Brad came back again after working with some folks trying to do more vocational education online, with, as they call them in Australia, the “tradies.” He said “Nancy, you were right.” There had been a lot of push back that tradies don’t hew to this idea of relationship building online. Well heck, they do. It might look and sound different, that’s all.

I asked him, “so what does that say about the facilitation of online learning?” Brad, smart man that he is, answered “it is about good teaching. Period. Online or offline.” And together, our experience tells us that good teaching is, among other things, relationship centric.

handsRelationship Centric Practices in Online Teaching

So let’s name some of these practices. I’ll share a few of mine. Please share some of yours in the comments.

  1. Bring your whole self. A workplace learner is juggling many things. Compartmentalization takes more energy to maintain. Bring a little bit of who you are, and find out a little bit of who they are. This helps identify opportunities to liberate learning, often in unexpected ways. For me, a little goofiness goes a long way. Just a little bit.
  2. Bring your unknowingness and curiosity along with your knowledge. As adults, we are co-learners. We learn with and from each other through our conversations, activities and reflections. If we are “know it alls” we often block this co-learning.
  3. Bring human expression into all forms of communication. Use text based body language. (I’m jumping up and down in my chair as I read your response!) Add pictures and images (even silly little sketches) that not only contextualize the content, but our engagement with it and its application in our lives.
  4. Keep the content tap turned to low. This is really, really hard for me. Look at the length of this blog post. I’ll be working on this until the day I die. But pouring more on rarely is the key to engaged teaching and learning.

What do you suggest?

Edit on 2/11: Some related links:

Learning While Building eLearning: Part 1

F2FtoONlineIntroduction

This is the first in a short blog series based on conversations with a colleague who is “learning while doing” as he is building an eLearning offering. Disclaimer: I’m an adviser to the project and my condition of participation was the ability to do this series of blog posts, because there is really useful knowledge to share, both within the colleague’s organization and more widely. So I said I’d add the blog reflections – without pay – if I could share them. So here we go! Over time it will probably include some additional comments from other members of the team working on this project. (Edit: Part 2 and Part 3.)

The reason I wanted to share this story is time after time I hear people state “Hey, let’s just convert our face-to-face (F2F) training into elearning” without a real sense of what that process might entail. What really happens when you decide to convert your face to face training to an elearning offering? What types of offerings lend themselves to conversion? What would that conversion look like? What should you consider?  I’ve done piles of research for clients in the past (and I’m working on getting permission to share some of it.) But nothing in the research is surprising. What is surprising is that it is not considered before diving in!

There are many paths to answering the questions about converting F2F learning to online or blended learning. The most important starting point is to ask some important initial questions, explore the options, and learn from others. Then, if you still want to proceed,  you can hire a firm to fully convert materials, do it yourself or work with a few others.

Regardless, the “conversion” process asks us some fundamental questions about learning: what and why we want to learn and what the impact of that learning might be. Looking at our assumptions around these fundamental questions can inform any initiative to “convert” something to elearning.

Meet Emilio  

Emilio is a technical officer at a large international development organization. He is an expert in his domain with years of knowledge and experience. Over time he has been asked to share his expertise and has developed a set of materials and practices to offer face to face (F2F) workshops around his area of expertise. He is passionate about his topic and his depth of experience brings richness to every conversation he has with people who want to learn more. Now he has been asked to reach more people by teaching online. That “elearning” thing.

The first decision Emilio faced was to understand what elearning really means. What are the options? What does it take to convert his offline materials into online opportunities? In our first learning conversation, Emilio shared his discovery that there is a large range of types of elearning courses and that they are difficult to categorize. He started out thinking that this is simply presenting his F2F lectures in a real time online space, augmented with the materials he had developed. But he discovered there was more to it than that.

Matching the Material With eLearning Options

multimediaAn early insight was that what you choose to do depends on your target audience and what they want or need to learn. (Or what YOU want them to learn!)

From conversations with other colleagues at work, Emilio learned some forms of elearning that have been produced in his organization. For example packages were created to introduce ideas, concepts and general information to government officials. Emilio now sees these can be less interactive, self-paced, and can potentially reach hundreds in unfacilitated courses. His colleagues have handed material over to consultants who have converted them into self paced offerings – quickly and efficiently.  This form fits with a goal of information dissemination. The value added to the learner, as compared to doing an internet search or picking up a book, is that the material is chunked and sequenced into digestible chunks and because of the reputation of the agency, people have confidence in what they are learning.

Emilio’s existing  training courses focus on a variety of complex policy issues and practices.  The material is more about converting concepts into practice, which is far more challenging than introducing ideas. Learners need to wrestle with the material, practice its application, consider their contextual differences and get the kind of feedback experts like Emilio have in their heads – the kinds of stuff that is rarely included in the slide deck or readings. Subtle. Contextual. Experienced. This is more challenging than converting basic or introductory materials into elearning.

As the volume of complexity of the material grows, there are other issues to consider. How do you keep the learner’s attention? How do you know if they are falling off course and what do you do? What and how do you customize for different contexts?

An early implication of these differences was that Emilio’s colleagues and bosses may have been thinking that he was doing the same thing as the people making introductory courses. The might expect it would take the same amount of time and resources to implement. Like him, his colleagues were not familiar with the range of elearning options. Add to that the fact that a strong organizational driver towards elearning is the idea of efficiency and reach, you can fall into a trap assuming that elearning is just about low-cost content distribution to many people, and that it is always easy or effective to implement.

Sometimes we can and do reach thousands with introductory material. But how do we really build capacity for more complex topics at a distance? What does “cost per student” really mean for deeper learning?

Emilio discovered that the reality is quite different than he thought. There is a mountain of jargon. Even thinking about the course learning objectives in a more complex offering is much more challenging (and we’ll talk about this with Emilio in a later blog post). All his tacit information that is easily available when he is present in a F2F training needs to be identified and reconsidered. What needs to be made explicit? What can come out through the facilitation of a course, with direct online interaction with learners? What has to be packaged into knowledge products? What is REALLY important amongst the vast possibilities of the content?

The adventure has begun. Stay tuned for the next part!