One thing I'm keen to do is get access to OU search analytics, both internal and relating to search engine referrals, to see whether we can gain any useful curriculum development intelligence from the data: seeing what keywords students use to find courses, for example, or whether there are any frequent searches we can't accommodate.
I'm in the process of planning a course on gaming and interactive media design, for example, so it'd be handy to see whether anyone is searching the OU courses site with terms relevant to such a course, whatever they may be. (I've posted kite flying thoughts elsewhere about how we might be able to use search as a predictive tool: Search Powered Predictions and Click Powered Predictions...?).
Analytics can be used elsewhere in the system, of course - I'm personally not that interested (at the moment...!) in "academic analytics", using data about student behaviour to predict success in a course, identify students who may be struggling/about to drop out, and so on, though I do know that there are various people around the OU who have started taking an interest in such things. After all, when students drop out, we don't get any funding for them...
What is of interest to me, though, are analytics that describe how students may be interacting with our online course materials (this data of course is likely to feed in to "academic analytics", particularly with respect to identifying struggling or disengaged students, for example).
I hope to be able to post something a little more concrete about this over the summer, but for now I'm pondering the question of what sorts of data we want to collect.
After all, web analytics of the Google kind seem to be best suited to tracking user behaviour on sites with clearly defined goals, such as successfully completed shopping transactions.
For online course materials - like the Relevant Knowledge short courses - where 10 weeks worth of material are online, organised as weekly topics - identifying relevant goals and sensible "questions" we might ask of the analytics data (with a view to optimising/improving our course materials) is something we haven't really addressed.
So - what should we track? Or indeed, to try and simplify the problem, what might we track.
The Relevant Knowledge courses all tend to have a similar structure (although each course does tend to introduce it's own innovations). A course calendar lays out the content for each study week, as does an index of the course materials. Links to online course forums are provided at the top level of navigation. Courses may or may not have a glossary and/or a resource page that contains links to external resources referenced in the course materials. A search box that allows users to search the web based course materials (but not external sites, our downloadable course content (I think...) is provided on each page, along with the option to print the page on the course content pages.
(Searching over content linked to from the course materials is easily achieved using a bounded/custom search engine. For example, see OpenLearn Unit Search Hub which describes an automated route for creating a bounded search engine over domains linked to from an OpenLearn Unit.)
The course content itself is chunked into 10 to 20 HTML pages per week. Links are often provided to external websites. Links to downloadable documents (such as PDF files, movie or audio files) may be provided. The use of embedded audio/visual material (along with download links, as well as transcripts) is increasingly likely. Course materials include self-assessment exercises as well as narrative content. So-called SAQs (formative self-assessment questions) comprise of a question along with sample answer text. The answer text is revealed by clicking a form button or link (i.e. a trackable action is taken to reveal the answer). Online summative assessment materials are also provided - a CMA (computer marked assignment) around week 5, and an ECA (end of course assessment) in week 10. All assessment materials are "seen" from the start of the course.
So - what to track?
Here are some preliminary thoughts:
Identifying sensible goals (which, if tracked, enables additional analytics, such as reverse path tracking (what links did a user follow to reach the goal).
Of course, making sense of course analytics is not just a question of collecting the data, but also interpreting it appropriately and not being misled by naive readings of it...
If you have thoughts about what else we could sensibly track, please post a comment or drop me a line. References to anyone else exploring this area would also be much appreciated...
PS one question I meant to pop down: if we were optimising online course content pages, what would we be optimising them for?Posted by ajh59 at July 12, 2007 10:36 PM