e-Ako taking a deliberate break

I'm hanging up my blogging spurs for a while to concentrate on my PhD. I am considering a new format when I return... which is likely to be as late as mid 2011. It's been difficult trying to keep up specialist knowledge across multiple areas. My reading on e-learning will continue, but blogging drops in priority for a season!

Mark.

5.3.2: Wikis in the latest JOLT

The latest Journal of Online Learning and Teaching (JOLT, 6[1]) is now available online (not a dedicated link - to the latest issue only). Three articles on wikis further the conversation [1 PDF] [2 PDF] [3 PDF]... but not, admittedly, by much! A pity that none of them consider learning outcomes using an experimental approach, which, I am convinced, the literature regarding the use of wikis in higher education actually requires. We have far too many of the 'What did students think?' evaluation; what we really need is an answer to how well did students learn? The studies also have small response rates and discrete surveys... which is not atypical in much e-learning research. This has the unfortunate consequence of results being highly contextualised and very fragmented.

It is perhaps indicative that we tend to give e-learning tools a go from our enthusiasm, rather than from a consideration of the lessons already learned; the literature review is performed after the evaluation to provide a context for what we did rather than what we will do. This approach is perhaps understandable from the standpoint of encouraging innovation, but it is a poor basis for research. I have no doubt that the authors of these papers learned a lot. As a reader, I'm left a little disappointed!

5.3.3: The Electronic Portfolio Student Perspective Instrument

Article: Ritzhaupt, A.D., Singh, O., & Seyferth, T. (2008). Development of the Electronic Portfolio Student Perspective Instrument: An ePortfolio integration initiative. Journal of Computing in Higher Education 19(2), 47-71.

---

Ritzhaupt et al (2008) describe the development of an instrument designed to measure student perceptions of ePortfolios. The Electronic Portfolio Student Perspective Instrument (EPSPI) was tested on a sample (n=204) college of education students. Based on the ESPSI, three subscales were found to determine student perspectives toward ePortfolios: learning, assessment, and visibility.

Some points of interest to me:

1. Ritzhaupt et al (2008) cite Meeus et al (2006) as suggesting two reasons for the popularity of ePortfolios in education: the contemporary bias toward constructivist pedagogy, and gains made in the accessibility and usability of ICT.
2. Respondents to the survey (and therefore the agents of its validation) were students using the ChalkandWire eportfolio system (at a cost of "less than" $US50, with renewal options at half the cost per annum...!)
3. The student perspective should be the basis of ePortfolio implementation (or at least an important consideration). Ritzhaupt et al (2008, p.52) remark that "no standardized instruments have been developed to capture this critical information".

The EPSPI was based on an extensive literature review, which identified four domains (learning, assessment, employment, visibility) and four primary stakeholders (students, administrators, faculty, employers). It's goal "was to capture critical information related to student perspectives and intended uses of ePortfolios to aid faculty and administrators in the decision-making process" (2008, p.54).

The inital instrument had 40 items, and underwent expert review and field testing (n=22). Each item also underwent testing through a 'cognitive interview' with three representatives of the population of interest, to ensure clarity. Eventually 34 items were decided on, and the resulting survey was made available as a Web form. Two hundred and four college of education students responded (no population number is provided).

While the survey certainly helped to validate the ESPSI, the actual impressions from students toward ePortfolios were less than impressive:

• Results indicate "that respondents from the sample may not perceive their ePortfolios as a meaningful learning device" (2008, p.56)
• "Only 19% of the respondents believed their ePortfolios would be beneficial to securing employment" (p.59)
• "Fifty-three percent of the sample indicated they would not use an ePortfolio to aid the employment process if they were the employer" (p.59)
• "More than half of the respondents suggested that they would not showcase their ePortfolios to friends (61%) or family (60%)" (p.59).
• In open ended responses, "More than 25% of the respondents [n=92 who gave open-ended feedback] mentioned that faculty did not offer help with using ChalkandWire(c) and did not appear to understand the program themselves".
  
• Other qualitative feedback included "negative comments about personal investment in time and money, comments indicating predominantly negative attitudes toward use of technology, and feelings of lack of choice and control" (p.64).
  

Phew. Not flattering - but, then again, not necessarily an inherent problem with ePortfolios as a concept. It may well be that the poor responses here were to do with implementation - and here is where the ESPSI may let itself down. It seems to measure student perceptions without a consideration of the implementation context. The authors identify this themselves (2008, p.66):

The purpose of the ePortfolio initiative appears to be a key issue for successful integration. For the administration,the system was implemented as an assessment management tool. This raises the question of student versus organizational control... the authors believe that that the low subscale means are representative of user resistance to system change and poor system integration.

The ESPSI instrument is available here. Personally, I see it as a positive step; I am in favour of standardised instruments as they enable comparative studies and can provide insight into good practice. What concerns me about the ESPSI is that it is a measure of student perspective rather than the effectiveness of implementation. It is here that the work of Joyce et al (see previous post) is particularly useful... there is scope, I believe, for an instrument based on their five roles.

5.3.3: Meta-cognitive development through ePortfolios: A school evaluation

Article: Abrami, P.C, Wade, A., Pillay, V., Aslan, O., Bures, E.M., & Bentley, C. (2008). Encouraging self-regulated learning through electronic portfolios [Full text HTML]. Canadian Journal of Learning and Technology 34(3), np.

---

Abrami et al (2008) describe an evaluation of ePEARL (the Electronic Portfolio Encouraging Active Reflective Learning) application, developed for school use. ePEARL has been specifically designed to facilitate self-regulated learning and the development of meta-cognitive skills, based on a three-part model (quoted here):

The forethought phase includes task analysis (goal setting and strategic planning) and self-motivation beliefs (self-efficacy, outcome expectations, intrinsic interest/value and goal orientation). Tasks involved in the forethought phase are: set outcome goals, set process goals, document goal values, plan strategies, and set up learning log.

The next phase, the performance phase, includes self-control (self-instruction, imagery, attention focusing and task strategies) and self-observation (self-recording and self-experimentation). Tasks involved in the performance phase are: creation of work, and learning log entries.

Finally, the self-reflection phase includes self-judgment (self-evaluation and casual attribution) and self-reaction (self-satisfaction/affect and adaptive-defensive responses). Tasks involved in the self-reflection phase are: reflection on work, reflection on process, and awareness of new goal opportunities.

The evalaution of ePEARL use involved 62 teachers and approximately 1200 students. It was hoped that teachers would apply ePEARL using the self-regulation model of forethought, performance, and self-reflection. Unfortunately results were less than optimal:

• The teacher survey (n=21 respondents) indicated that teachers were not using the application frequently enough (most using it 5 to 8 hours per month, well short of the requested 12 hours per month);
  
• There were few positive differences between pretest and posttest scores (though the positive effects were in items of interest such as "students identifying strategies for achieving their goals; students documenting the processes they used when working on tasks; teaching students to identify strategies for achieving their goals; students using portfolios to demonstrate their strengths; students using portfolios to identify areas needing improvement");
  
• Teacher focus groups indicated that teachers did not think their students capable enough to partiicpate in the self-regulation process (particularly the 'forethought' phase), not all teachers understood the term 'reflection' in the same way, and not all students were comfortable/able of providing efective feedback to one another;
  
• A sample of student portfolios (n=66) "did not reveal widespread or extensive use of the tool. The majority of the portfolio pieces were reading responses, stories and poems, language arts presentations, social science or science projects, and music and art projects".

While the findings conclude with the paragraph...

However, on occasion, there were teachers who implemented EPs extensively; in these cases, teachers used ePEARL in both creative and practical ways. As a result, student portfolios in the classroom of these teachers were often richer, and demonstrated that students can learn self-regulation skills in order to improve their work and become better learners.

...specific results for these 'occassions' are not provided.

I can't help wondering if this project suffered from implementation flaws. Reading between the lines (and from the first paragraph of the conclusion) I suggest that:

1. Teachers were asked to use ePortfolios in a way that might have been foreign to them, asked to apply a prescribed learning approach that not all thought was appropriate.
   
2. Students were asked to apply themselves to a learning methodology that they were either not capable of participating in, or else were not supported in because of 1. above.
   
3. The application itself was designed to facilitate a specific learning process (the self-regulation model). As a combined result of 1. and 2. above, this made the application itself inseperable from the process.

It seems as though to some teachers the ePortoflio was a peripheral experiment, rather than an integrated and central element of teaching and learning. It will be interesting to, as the authors conclude, "Stay tuned" with what follows.

As an addiitonal comment, the survey used instruments called the 'TLSQ' (Teaching and Learning Strategies Questionnaire) and 'SLSQ' (Student Learning Strategies Questionnaire). Both look to be pre- and post-test instruments customised to the K12 sector.

1.1, 5.3.4: Linking two posts together

Two great pieces in my Google reader account this morning, which I originally thought to comment on separately:

1. The latest VLENZ (no.166), outlining the promising development of linking an open source MUVE (OpenSim) with Second Life; and
2. a post from Tony Bates, "Will lecture capture replace asynchronous distance learning?" in which he (rightly!) criticises plans of one US provider's plan to facilitate distance education through recorded lecture.

At first I thought these were worthy of separate comment (indeed, they are!) But then I started to wonder whether it might be better to use one as the lens for commenting further on the other. The VLENZ piece is concerned with advances in technology and what it might enable technically, whereas Bates' post looks at what is possible technically and how it has been applied to learning.

In e-learning we need both advances in technology and a self-conscious approach to how that technology is applied for the purposes of teaching and learning. MUVEs emphasise interaction, 'presence', synchronicity, simulation, activity within a virtual environment where resources can also be freely shared. The 'video capture' initiative described by Bates emphasises access, the view and explanations of the expert, the ability to watch, listen, and revise.

However most academic learning takes place through the reflection and deliberation. It is only insofar as these activities are supported that either might be considered truly educative. As Bates indicates, there are volumes written on the effectiveness of asynchronous distance learning. We have substantial insight as to what rests within the 'black box' of distance education. The challenge is to draw on those lessons that they might inform how we apply new technologies; both the VLENZ and Riverside cases are both generally concerned with 'distance education', which has a considerable literature and a very mature practice. My concern is that fully immersive MUVEs might supercede the deliberate exchange that might take place through, say, online discussion boards - which are by nature more reflective: "Hey, let's all meet in the virtual forest. We don't need those old bulletin boards anymore!"

I remain convinced that MUVEs will find their place as an extremely important addition to the distance educator's toolset - but I have reservations about synchronous virtual exchange being somehow inherently better than asynchronous discourse. It is here that Bates' critique provides a good check-point for the technology; just because it can be done, doesn't mean it should be done. Maybe MUVEs could make lecture rooms redundant and lead to more collaborative and situated learning. We just need to ensure that we don't overlook any advantages that are inherent to less 'wizzy' solutions in the process of transfer, or assume that we need to learn lessons about pedagogy from scratch.

I'm excited about the activities of the VLENZ group. The link between SL and OpenSim should be celebrated. But we should also be thinking ahead about the potential contribution such technologies might make. By considering - deeply - what we already know about effective education, we can save ourselves a whole lot of unneeded learning and mistakes later.

5.3.3: Good things take time

Article: Wickersham, L.E., & Chambers, S.M. (2010). ePortfolios: Using technology to enhance and assess student learning [Full text PDF]. Education 126(4), 738-746.

---

Wickersham & Chambers (2006) report on a post first-semester evaluation of a programme ePortfolio implementation (n=26, masters level secondary teacher students). Using a 13 question Likert scale survey and two open-ended questions, they found the following:

• Students had not made the connection between the use of an ePortfolio and the potential for enhanced self-knowledge, and knowledge and skills transfer. In fact, students were almost evenly divided in terms of a positive, neutral and negative response on the ePortfolio's usefulness.
  
• Students did report an improvement in their technology skills however most commented in the open-ended questions that the technical interface was the greatest barrier for them.
  

This is a useful study in that it clearly demonstrates that ePortfolio use must be purposive, and that its benefits (realised through reflection and the gradual building of a collection of artefacts) take time to be appreciated by students. Wickersham & Chambers (2006, p.744) state that

What was learned from all results is the need to be more effective in helping students make that connection [between the ePortfolio exercise and their own development], and to integrate the ePortfolio concept firmly within the program.

Good things, says the cheese ad, take time.

5.3.3: ePortfolio implementation and the threshold concept

Article: Joyes, G., Gray, L., & Hartnell-Young, E. (2010). Effective practice with e-portfolios: How can the UK experience inform implementation? [Full text PDF]. Australasian Journal of Educational Technology 26(1), 15-27.

---

Joyes et al's (2010) paper won a best paper award - and it's not difficult to see why. It's authority and groundedness are very clear, as is its significance for ePortfolio practice.

After an overview of ePortfolio activities within the JISC (e-portfolios are one of five main areas of activity) and an overview of the importance of ePortfolios, the authors provide five key roles for the successful implementation of ePortfolios in whatever setting.

1. The role of purpose - "For successful implementation, the purpose/s behind the use of the e-portfolios must be aligned to the particular context" (p.22). The authors emphasise that ePortfolios cannot be understood outside of the context in which they are used.
   
2. The role of learning activity design - "There must be a conscious design and support of a learning activity/activities suited to the purpose and the context" (p.22).
3. The role of process - "The processes involved in the creation of the e-portfolio in the particular context must be understood and both technical and pedagogic support needs to be provided" (p.22).
   
4. The role of ownership - "The e-portfolio processes and outcomes need to be owned by the student. This view leads to considering portability, choice of tool (use of their own phone, camera, audiorecorder, Web 2.0 application, for example)" (p.23).
5. The disruptive nature of e-portfolios - "E-portfolios are potentially transformative and as a result are disruptive from a pedagogic, technological and an institutional perspective because they tend not to fit exactly within existing systems". Naturally, this means that ePortfolios are best implemented in a systematic way.
   

Joyes et al (2010) go on to note that, to many experienced ePortfolio users, the five findings above "may seem quite naive" (p.23). However they raise a very interesting point based on this observation - there is a threshold of knowing that the experienced have, but which the uninitiated lack:

The threshold concepts approach recognises that developing understanding is a developmental journey, both intellectually and experientially, but that once the threshold is achieved the perspective of an area is changed forever. Thus guidelines/ guidance will only make sense to a stakeholder if the threshold concept is understood and the preconceptions resolved. Is this why the wheel has been invented so many times in the e-portfolio area? (2010, pp.24-25).

Indeed, it may well be why the wheel is so constantly reinvented. We e-learning practitioners tend to avoid informed practice, letting our enthusiasm for the tool prompt us to action. As a result, we frequently miss the easy and important lessons lying on the sruface of literature. Articles such as this by Joyes et al (2010) demonstrate the very real importance of keeping up with developments in journals.