For three days last week I was participating in an advanced writing workshop. This was a very small class of eight students with two instructors. Learning in the class was dependent both on the knowledge of the instructors and the knowledge/professionalism of the students.
The format included an overview of the background tools/techniques needed to write effectively within my organization followed by practical exercises reviewing papers from both the instructors and the students. Initially we reviewed/graded the papers handed out by the instructors individually and then collaborated in teams of two on a joint review. This was followed by a open discussion of both an initial and revised version of the paper. After this we brought in our own papers and graded them while going through two other student papers. This was followed by different teams of two collaboratively discussing their review. The open discussion started with the first author giving his/her personal grade followed by the team who had evaluated that paper. Other students were then invited to add anything that they felt was missed. Once the review process was completed, students were given the time to update their papers and send them around for review in the same process by the same teams.
Although not familiar with many of the specific topics in the papers, I found both the class format and the associated resources (e.g. checklists, tradecraft materials) to be very valuable in learning to effectively evaluate both my own writing and others. And the feedback was critical in helping me focus my own writing which tended to cram too many points into one rambling mess.
But as I finished this class, I was thinking about how it compared to the two massive open online courses (MOOC) that I had just completed. In these two courses, the peer review process was key in completing the course. Each student was assigned to review the work of 5 other students and provide written grades based on instructor protocols. You could not pass the course unless you had also completed the minimum 5 evaluations per assignment. In some ways the process was very similar to that of the classroom where we reviewed student efforts. However, there was no opportunity to interact personally with the other four members of the review team or the students you were reviewing. That later created some frustration among students who felt that they were unfairly judged and given significantly lower grades with little feedback. Additionally, there was no continuity throughout the course. While this meant that we were able to see more students, it also meant that you weren’t following a student’s ideas from creation to completion.
I think that for MOOCs to be truly effective, they must provide better opportunities for interaction during the course beyond the forums. Somehow, the same type of small group interaction that occurred during my advanced writing class should also occur during a MOOC. This way learning not only occurs from knowledge shared by the instructor but knowledge shared by the students. The technology exists to permit that experience with applications like Google+ hangouts, Skype or getStudyRoom.com. The biggest drawback may be the global nature of the student body that limits real time interaction because of schedule conflicts or different timezones. It will be interesting to see if the #edcmooc group can solve this lack of interaction through the community that has been forming since November.
In Search of Knowledge 
Hi Kelcy, I agree that the global nature of our edcmooc may pose problems regarding peer interaction and review. Here’s hoping all the tools that we have experimented with so far are a benefit. This community of 150 plus has been a great help so far. I’m counting on the larger community to be helpful as well. As I’ve heard so often since November….Time will tell.
Interesting observation, Kelcy. I’m optimistic that, in time, the immersive aspects of video and telepresence will supplement MOOCs and other interactions with enough of the humanity and warmth that currently they lack. But “in time” is going to be a while, a decade or more for really powerful telepresence that begins to mimic the persistence and closeness of actual physical proximity. In the meantime, based on the distinction you’re pointing out, there appears to be something like Kahneman’s “system 1” vs “system 2” cognitive gulf between the two collaborative learning methods. Distance-collaboration might be better for system 1 tasks and vice versa. Don’t know, but it’s a very interesting observation you’ve made. It also strikes me as a complement to your point in the previous post about online closed-minded viewpoints vs. arguing face-to-face in a cafe…. The valuable challenge to shuttered opinions is also there in the face-to-face interaction with F2F writing reviewers, and there’s much more likely to be a kind of synthesis out of the debate if you’re immersively together (meaning, today, in real time/space).
In the one MOOC I completed, the forums at the end were full of complaints about unfair assessment by the peer assessment method you describe in your MOOC experience.I think your workshop method is far better, but understandably more appropriate in a smaller group.
I’m sure Willa will agree that as a teacher, assessment and grading is no light task and a skill that some people have mastered better than others. For many MOOC students, this might be their first experience in assessing work and their only reference may be their own work as well as the other 4 assignments in front of them. Bound to be subjective. The other factor is that some people are really only interested in themselves and may begrudge having to put time into reading and grading the work of others. I went to a MOOC workshop recently where the process of assessing was discussed and it was clear that this part of MOOC participation was very time consuming, so no doubt, feedback will be minimal in many instances.
I will really look forward to seeing what we all produce in edcMOOC, and it will be a pleasure for me to follow the progress and seeing the final results of our learning with those we have come to know precourse, and the others as well, But not everyone may feel the same way. Given that many are teachers,lets hope there is a higher degree of interest in assessment than perhaps in other MOOCS.
As we discussed before somewhere on fb, I think dividing a MOOC into nodes where we can experience continuity and more personal interaction would be a good thing. At least we will be able to seek out those we know for support and encouragement should it be required.
Good post Kelcy.
I don’t see how MOOC assessment can come anywhere near the experience you had on your writing course. Were you able to seek clarification during the reviews. I assume, but may be wrong, that all participatnts were of a similar cultural background. This is not possible without a lot of organisation by Coursera and cultural differences could influence the reviewers interpretation of the work. I also think that there is a danger that distance peer review may be harsher as there is little or no chance of meeting those you are reviewing. There are many other influences that will be at play during the reviews both positive and negative.
Lewis’s comments eloquently expand on your post and I agree with him that it will be a decade or more before there is parity between distance and face to face .
Angela’s point about nodes may be an answer develop and maintain a sense of belonging, a very human need, but I am not so sure.
It will be interesting to see how the course organisers deal with this and furthermore how the students react. Of course these things are part of the course \ study I suppose.
One thing is certain it will be an interesting journey.
Thanks, Nigel. It will be interesting to see if #edcmooc will even address this or provide feedback to Coursera on peer review grading. I do agree with Lewis that MOOCs will ultimately get to the same level of intimacy as a F2F course but it will take time.
In terms of the peer reviews that I went through, I believe that the computer randomly assigned you. In some cases there were language difficulties in terms of trying to translate their thoughts into English relating to an American business model (e.g. several Gamification assignments had scenarios that were based on American cultural business problems). If I couldn’t understand something, I tried to give them the benefit of the doubt. But in one evaluation of one of my assignments, I got told that not only was the paper disorganized (totally agreed with that one) but that English must not be my main language. Fortunately the other reviewers were much kinder. There is no opportunity to interact with other members of the review team and no way to understand the grading unless the reviewer took the time to write their thoughts down. I always tried to but I found many that did not on my own assignments. This could be something that Coursera or other MOOCs could require – not only would students have to do peer reviews but would have to provide written feedback on why they chose the grade they did.
I think there is a danger that non acredited courses may not take the reviews as serioulsy as they should. This commenty is related to my scepticism about the objective of those offering free courses.
I think that is a valid danger. I also read somewhere about another one where some students became ultra competitive and treated the peer review process as an opportunity to “destroy” competitors. I can actually believe that. But even instructors in F2F courses can grade unfairly (I certainly had some experience with that in one of my own college courses years ago). MOOCs will have to figure out how to handle grading for subjective assignments if they want to move toward accrediting classes for credit. I expect we will hear a lot more about that over the coming year.
This is all good stuff about ‘digital cultures’. Ultimately the human factor cannot be ignored and is still the most important ingredient and bullying, ganging up and destroying perceived competitors will still exist. The question is, is it easier and less controllable from a distance than F2F? Also how would the dynamics of the group change and what impact would this have on the learning process? A PhD for a Psychology student there perhaps.
Such fascinating conversations here! I’m glad I came back for another look. One of the goals of this course is to see how digital cultures intersect with learning cultures. As stated in the first e-mail “This course will explore how digital cultures and learning cultures connect, and what this means for e-learning theory and practice. ” It also states that we are to think critically and creatively within the framework of a supportive environment. As a teacher, I know how subjective evaluations can be. I’m hoping that there will be some kind of rubric for these evaluations and that any kind of bullying or uncivilized behavior in the forums will be taken care of by the moderator of those forums. As to F2F reviews, I think that could be arranged via a Google + hangout. I hope that we will have a say as to who we connect and collaborate with……a random computer generated group just wouldn’t fly with me. I like Angela’s idea of dividing ourselves into nodes…..I’ve made great connections here these past two months and want that connection to continue.
It is a great conversation. My last MOOC experience was not rewarding in the forum part of it. There was no moderation Willa, and I found the upvoting, downvoting, jostling to post early to get the most votes, the general lack of courtesy from some and the whole frenetic business of so many inane posts and even more inane replies just to post SOMETHING, was bewildering to say the least. I hope this will be a better experience in edcmooc. I don’t even know if we have forums this time round! Do we?
Willa your post raise a few thoughts with me:
1. Is a Google + hangout truly F2F. I don’t consider Skype F2F
2. Why would ac omputer generated group not fly? It could be that people like you, Kelcy, Angela and I need to be expose to others. If not and the groups are self selecting are we in danger of forming cliques not groups. The disadvantage of course is that the relationships we have now developed would have to start again.
3. The connections can still continue but with different modus. For instance if we were all put into new groups with new people we can still act as counsellors and sounding boards to each other.
Angela’s point of inanane posts resonates with me. I just dont’ have the time to read 10 posts to get 1 that means anything. I wish I did.
Angela I am not sure if we have forums, or is it forae, I never know.
Hi Nigel,
No, a Google + hangout is not truly f2f, but we do get the nuances of voice and facial expression to convey meaning rather than just print where meaning can easily be misconstrued.
A computer generated group implies little or no human input, just random chance. As a teacher, I do try to mix students with different abilities and interests when collaborating on a project, but always try to group with some familiarity as well. At a young age, if you are thrown into a group knowing no one, pupils tend to clam up. As Kelcy and Angela stated some peer reviewers have other issues and agendas and the evaluation may not have been fair. You are right about the formation of a self selecting group limiting our horizons and being cliquish. I was being a bit selfish and not wanting to leave my comfort zone. As Angela pointed out, we don’t even know if we will be participating in forums.
Holding on to the connections we’ve made so far and using them as sounding boards and as counselors is a great idea.
Willa
Nigel, did you watch Dave Cormier’s video, what is a MOOC? It’s terrific and well explains how we are meant to connect and form groups and connected networks within the massive. It is the best one I have seen on this.
I do consider skype has many elements of f2f. Willa, after our hangout, I know you are patient, purposeful, positive and reassuring. You have a kind demeanour and smile a lot. I know you are polite and a good listener and talk when you have something to say which you articulate in a thoughtful considered manner. So I think I learned more about you than just general body language of a conversation.
I tutor in a course run from the opposite side of Australia. Its a 5 hour flight to get there. I talk every day with the Prof and my fellow teacher on hangout (it used to be skype). We have all met in person often, and I think working this way which is really videoconferencing, is as good as f2f for the purpose. Of course, if we want to have a glass of wine and chew the fat, we need to do it in person.
The prof works on campus, but this course is fully online for on campus students and those who live anywhere in the world that want to join us. So we get feedback from students about the lack of f2f, and they much prefer this way of learning and studying but suffer terribly if we can’t have the hangout at least once a week. They see it very much as f2f with other students. Even those on campus who can meet over a coffee.
So whilst it is not f2f, it delivers many positive aspects of the experience and is one of the great tools for making connections and enhancing the experience of online learning.
Well said, Angela…just another reason why I value the connection we’ve made so much!
Angela I can see how the scenario you describe works but the important thing to note here is that you have all met in person often. I run the VC suites at Aberystwyth University and advise that if people are VCing on a regular basis, and if at all possible, they meet at least once to strengthen the connection. The simple act of shaking a hand can tell us much about a person.
I can see where Google+, Skype, VC etc. can be of use and I promote VC use on sustainability grounds alone but as I said it is not ‘truly’ F2F but is the next best thing.
Thanks for the video I will watch it later.
This is a very good discussion.
Do you intend having another hangout? If so I would like to meet you ‘in person’ so to speak.
Nigel, I posted a reply and then it vanished… now where was I , yes, in relation to f2f with my colleagues, I did not meet them in the flesh until after 18 months of skyping. When I first met the prof, we met on a busy esplanade and it didn’t occur to us that we hadn’t actually met, although he had wondered if my head would look like a computer monitor! He was taller than I imagined, but apart from that it seemed like we were old friends.
We are trying to arrange another hangout. I think Laurie’s quadbloggers hungout today, so when they get over the excitement, we will arrange our next session. Andy Mitchell and Chris also joined, it was good. Will let everyone know when its going to happen.