Featured post

Dead Blog?

Because of serial spam attacks which the Blogger platform seems unable to deal with (yes - people warned me about Blogger), I have moved the...

Thursday, 20 June 2013

Why I don't approve of Rate Your Lecturer

According to some lame-brains who commented upon this piece by Bill Cooke (with which I largely, though not entirely, agree), if you oppose the new Rate Your Lecturer (RYL) site you must be a really terrible lecturer who doesn't care about your teaching and is afraid of bad feedback.

So let's clear that one up first.

Feedback on pretty much all aspects of my teaching has always ranged from very good to excellent.  I care about my teaching.  I don't always get it right (as we'll see) but I do try to do it well and to keep it as fresh and innovative as I can, within the limits of what I feel comfortable with (trying 'cutting edge' teaching techniques if you're just not comfortable with them ends in miserable failure and general mockery, from what I have seen: e.g. I am not comfortable in role play so I don't use it). Student feedback has played an important (if not the only) role in keeping my teaching up to the mark, not least because we have had student feedback on teaching for a long time.  I'll repeatedly come back to that.

I care about my students (well, almost all of them); indeed I actually like them (mostly).  They make me laugh; it's a blessing to have a job that puts you amongst young, lively, intelligent people.  They also drive me up the wall sometimes, because they are smart but they aren't making enough of their abilities and the opportunities they are given.  I sound off about it in depressive moments; I get into trouble for sounding off about it, I get lambasted in the press, but most of those who saw that story (above all my own students) knew that I was ranting at them because I cared about them.  One or two, I admit, I want to punch in the face (and they loom larger in my impression of the student body than they deserve to).  That aside, I am, they tell me, 'a legend'; when it was fashionable to set up lecturer appreciation societies on Facebook (c.2006) I had the largest and most active by far in my dept - maybe across my university.  For what any of all that's worth.

I used to be competitive about being popular with the students and I won at that game in two different institutions with two very different types of students.  On that front, all bets are off.  But enough own-trumpet-blowing on my part.  If I object to RYL it's not because I am a bad/unpopular lecturer.  Hell, bring it on.  10 years ago I'd probably even have got a chilli or two; 5 years ago I'd probably still have cared.  I just want to make sure we've established that if I don't approve it's not because I am frightened of RYL.  OK?

But I do not approve of RYL.  At all.  Why?  Here's why.

The site makes the bold claim that this is 'the only way' that teaching in the UK will improve.  That is nonsense.  But let me make a positive point first.  The categories of RYL are considerably more thoughtful and useful than those on its older US cousin Rate My Professor (RMP), which include 'Easiness'....  Nonetheless the dreaded hotness chillies are apparently coming soon.  One might legitimately ask what the hell the hotness or otherwise of a lecturer has to do with educational quality or review, or furthering any of the avowedly lofty aims of RYL.

But then those lofty aims are only so much smoke and mirrors.

For one thing, anonymous feedback processes already exist and indeed have done for a long time.  It is ludicrous of Michael Bulman, creator and owner of RYL, to suggest that he is creating something new and valuable.  I have had to deal with anonymous student feedback since the mid-1990s, I think.  Those with free text options are the most useful and I have decried before the move to numerical ratings.  The problem with numerical scores is that though they look precise and scientific they are anything but either of those things.  When we give a student a grade on an essay or exam we have a whole series of grade descriptors against which we have to set the mark.  We have second-markers to check, and externals to oversee the process.  Even then it's not an exact science.  There are, however, no grade descriptors for feedback scores and no checks on the appropriateness with which they are awarded.  That's not to say they are useless; far from it.  Free text, as I said, is more valuable because 'Prof. Grumpy speaks too fast/ is always late/ has Powerpoints with the text too small/ has too many Powerpoint slides/ mumbles/ talks too much in seminars/ talks over the students/ is too negative in his comments', or whatever is more useful than '2/5'.  It is a bit as though we gave essays back with nothing other than the mark on them.  Nonetheless, as an internal check on how well teaching is being done numerical feedback is, while crude, valuable.  The problem comes in turning those scores into overall averages and then league tables.  You aren't comparing like with like.  You aren't comparing the responses of the same set of judges.  What an Oxford student, say, rates as 3 out of 5, a Poppleton student might think of as a 4, or vice versa.  There are different expectations to take into account.  An Oxbridge student might, for instance, mark a lecturer down for being too dirigiste, for not allowing her the space to think and be creative; a student in another institution might mark them highly for directing his thinking so clearly.  As ratings within a department, then, numerical scores are very crude; as the basis of comparison between institutions they are meaningless.

These are just not, as they say, robust data.

Those on RYL are even less robust, because there is no way of checking whether the raters are students who have attended the course - even indeed if they are students at all.  There is no index of what proportion of students on a course think the same way about a lecturer, or the extent to which personal motivations (positive or negative) intrude into the assessment.  Previous reactions have focused on the negative, with good reason, but there are equal and opposites.  A friend of mine (who is, by the way, actually a really good teacher) once had a RMP comment, for example, that read something like "♥ My future husband!"  So much for the intellectual value of whatever comment she wrote about his teaching.  Take a hypothetical example.  Suppose a young male lecturer talks a lot about football and is generally 'blokish' in his humour etc.  He might be rated 9 or 10/10 by some of the male students who like his laddishness, but only 1-2/10 by some of the female students who justifiably feel alienated by it (OK - I've made some problematic, stereotypical comments about gender interests of football but stay with me, fill in some other analogy of your own that might work better at yielding the outcome I suggest).  You might reasonably suppose that those two, rather polarised, subsets would be those most likely to post to RYL, and they unsurprisingly yield an average of 5-6 out of 10.  But suppose that, that one unfortunate trait aside, our man's teaching was actually really good and that, had a full sample responded, his grade would average at 8 out of 10 (or conversely that it was pretty poor and would otherwise have got 4 out of 10).  All these issues make the consumer analogy highly problematic.  The problem is not that a lecturer is not a dancing bear (in Bill Cooke's formulation), but that a lecturer is not a standard mass-produced functional commodity, like a kettle reviewed on Amazon.

This yields yet another point, and one that belies RYL's avowed (but entirely disingenuous) aims.  High or low scores have nothing necessarily to do with educational value.  This develops my point about grade descriptors or their lack.  A low score can mean 'this course was too hard for me to coast on', 'I didn't like this course', 'I didn't want to do this course but I had to and I'm still sulking', 'this lecturer made me work harder than I wanted to', 'this lecturer didn't tell me what I needed to write for the exam', 'I thought he was a bit of an arse' or any number of other things which have nothing at all to do with pedagogical quality.  A high score might mean 'this lecturer is cute', 'this lecturer is funny', 'this lecturer gives high marks', 'this course was really easy', 'this lecturer pretty much told us what was on the exam', 'this lecturer brought us chocolate/bought us all a drink at the end of the course' or any number of other things which equally have nothing to do with educational or pedagogical quality.  Again, students need to beware if they are making choices based on RYL, and in my view they shouldn't make their choices on that basis.   In fact I very much doubt that many will (at least in the quality universities), for the simple reason that most are too smart to do so.  That raises the point of what RYL is really for (on which more later).

In any case, as I have argued before, simple numerical data, league tables and competition do not raise standards.  In my view they can (and frequently do) actually lower standards.  Where these things get taken seriously (as all too often, but let me reiterate that RYL won't be taken seriously), they lead to the production of the sorts of things that generate high scores and thus higher league table positions, not necessarily things of higher actual quality.  Hypothetical analogy.  Supposing the research assessment developed its impact agenda and started to award points for readership/books sold (a concrete, objective datum at least, you might say).  That would certainly lead to Universities pressuring lecturers to write best-sellers - not necessarily (or even likely to be) works of higher intellectual quality.  I've talked before about how KIS data lowers pedagogical standards.  If making courses as easy as possible, on undemanding subjects, telling students what to write, setting easy exams, and giving free cake at each class produced high RYL feedback scores (and believe me it almost certainly would), and that led to universities compelling all teachers to make their courses like that would it raise standards?  No of course it wouldn't.  Because what yields high scores is not necessarily what is good.

As I have discussed before I decided to move away from being a 'dancing bear' (in Bill Cooke's terms) to teaching more self-consciously difficult history that I felt would stretch the students intellectually and represent more of what I thought a history degree would do, and to damn my feedback ratings.  As it happens my students responded  well to that, which is an important point.  But my feedback score did drop.  I'm happy to take a 10% cut in approval to do something more rigorous.  How does RYL acknowledge or respond to that in its disingenuously self-proclaimed crusade for better standards?

There is yet a further point.  Another one that I have made before, I admit.  The consumer analogy is fatally flawed.  Take the kettle example I used above.  Someone who buys a kettle, from Amazon or KettlesRUs.com, or wherever, has (usually) a pretty clear idea of what a kettle is and what they want it to do, frequently on the basis of having used kettles in the past.  Someone who buys a CD (if anyone still does), or whatever the young people do these days, by a particular artiste has an idea of what they like about that artiste's work, or that of others in the genre.  Certainly one hopes they at least know what a CD/download is supposed to do.  On that basis they have a good basis upon which to review the product on-line.  This kettle is rubbish; it gives off a funny smell and took an hour to boil a pint of water is perfectly valid, if true.  This CD is not as good as X's last one, or as Y's current album, is equally valid.  On that basis, a review of a course on the basis that a student didn't enjoy it as much as another course she'd taken is also fair enough - but brings us back to the point about comparing like with like.  But there are strict limits to this.  At the end of the day a lecturer knows better than a student what the important elements of a course are, what the students ought to be taking away from a course and what would be the best way of conveying all that - in the way that a kettle-producer doesn't know better than the kettle-consumer, or even in the way that (to some extent, in some ways) an artiste doesn't know better than his fans.  If you think you know what a, say, history degree should involve, how it should be taught, what subjects it should cover and what weighting different areas should be given, then really you ought not to be taking the degree, but teaching it.  If you expect a degree just to give you what you want then why have a degree?  You want a qualification.  But you want it to be worth something too, right?  To that extent you have to trust the professionals to know better than you.  Once you have a degree (or two) then maybe you're in a position to judge a degree course.  Before then, not.  As I have said before, the point of a degree is to produce someone more capable of judging a degree.  If you were knew enough to judge it at the start, or half way through, it would not be worth doing.  Ultimately it is for the lecturers and the many assurance processes they have to go through to get courses approved to determine the intellectual quality of a degree.  Student feedback has an important role in ensuring the quality of its teaching but numerical scores - as I have been at pains to point out, and have doubtless laboured the point as a result - is a blunt and potentially very misleading tool to use to that end.  And, as I have said, it is most unlikely that any university will take any account of RYL anyway.

And then there is RYL's fatuous claim to be trying to 'redress the balance' between teaching and research.  It is, I have to say, a real shame that so many students evidently think that their lecturers derive more credit for their research than for their teaching, or care so much more about their research.  Some do; that's true.  It is a minority these days, and that is one good outcome of the greater degree of quality audit that the profession has been exposed to over the past 20 years (again making the point that RYL has come too late to make any difference).  To be promoted you have to show that you are doing your job to the required standard in all three areas of the lecturer's job: teaching, research and administration.  You then choose two areas in which you think you have done more than could be expected at your level.  Most lecturers go on teaching and research, a few on teaching and admin (largely because most lecturers enjoy teaching and research more than they enjoy admin) but even those who go on research and admin have to show that they've been doing their job properly as a teacher.  In those cases, if they want to be promoted again, even if they still want to emphasise their roles in research and admin they still have to show that they are teaching effectively.  That, at least, is the case in my university and I don't think that that is atypical.  Ten years ago, once you had your senior lecturership you could indeed ignore everything but research to get promoted but that's really not the case any more.  Teaching feedback is an essential part of the promotion process.  Thus RYL's claim to be rewarding good teaching, and for the first time giving students a voice in that, is, in addition to being flawed and factually erroneous for all the reasons already set out, missing the boat by about ten years.

RYL refers to moving away from valuing lecturers simply for 'churning out research'.  Do you really think that that is all that is involved in university level research?  Churning it out?  Try actually doing some, all ye who can't get a 2,000-word undergraduate procedural essay in on time.  Leaving that aside, though, research is what universities are there to do and it is as important a role as teaching.  Do you really want UK HE to stop producing world-leading research?  Maybe more important than that, though, is the fact that active, high-quality research and good teaching are fundamentally linked.  (As an aside this is especially disheartening when set aside the claims of the purveyors of on-line degrees that students should be lectured to by 'star' media dons rather than ordinary lecturers.)  How can I teach you how to be a good historian if I am not a good historian myself?  How can I teach you about how to research if I don't research?  How can I give you cutting edge insights into historical topics if I am not myself actively involved at that cutting edge?  You can't have this both ways.  You can't demand to be taught by the best and then claim that the best spend too much time 'churning out' research (as if that would be what 'the best' did).  Lecturers only have 24 hours in a day like anyone else, and (believe it or not) have lives outside work (well, most do).  The pressures on that time are increasing all the time, frequently in ways that have little or nothing to do with either good teaching or high-quality research.

As a last point on this issue, the very same government policies that have slashed back funding, forcing Universities to charge fees (and at which you should be directing your ire, not at University teaching staff) have also forced Universities to rely ever more heavily on research funding.  And believe me, that is FAR more lucrative and significant in a departmental budget than fees.  RYL won't make any difference to that.  Most lecturers I know would far rather be teaching than filling in grant applications but in the current financial situation that is what they have to do, for very real reasons relating to the very sustainability of UK universities.  Argument about students being the sole consumers and financiers of universities are simply wrong.  Want to change this?  Pressurise the Labour Party to commit to reversing the government university funding policy and vote for them if they do, and insist they make good on that.  And realise that a top-quality education system with has fair access requires state funding, and that in turn requires taxation, especially of the rich.

Thus far I have set out why the claims made by RYL are misleading, why it won't (and indeed can't) do what it claims.  My objection to the site is so far based mostly on its essential fraudulence.  But you might say it's still pretty harmless even so, especially if no one in any position will take any notice of it (and they won't, any more than US universities take any serious notice of RMP).  It is not useful but it is not harmless.

What, then, is the point of RYL?

Michael Bulman, the owner of RYL, clearly, from his statements on the site, knows nothing about British higher education, as all the above makes clear.  But that does not matter to him.  The purpose of his creation of the site is simply to make money.  By creating a site that will get thousands, indeed millions, of hits he can charge for advertising space on it and make a tidy profit.  Let's be absolutely clear about this.

The purpose of RYL is not to improve HE; it is cynically to cash in on the obscene situation where you, the students, have to pay for your higher education.

For that reason alone, if you actually care about UKHE, you, the students, should shun it.

There are however, lastly, very serious moral or ethical issues about sites like this.  They further the pernicious and spreading culture of bullying, cowardly, anonymous abuse.  Bill Cooke (author of the 'Dancing Bears' piece linked to at the top) argues that it will stop lecturers from giving negative feedback on students.  Actually I think that most lecturers have more guts than that.  I have recently finished teaching the worst 'Barbarian Migrations' class that I have taught in 17 years. They weren't stupid (far from it), they weren't unpleasant (even further from it), they weren't even lazy, overall.  But they seemed just to have no commitment to the course and its requirements, showed no initiative, couldn't be bothered to discuss things or talk in class.  I tried with them; I put a lot of thought into the course organisation and seminar set-up; I held a discussion half way through about how I could improve the class.  To no avail.  So I have said as much on their reports.  Most are mature and self-aware enough to take it on the chin.  At least one, though, is puerile enough to write something nasty on RYL as a result.  So be it.  If I get another class like that I'll do the same.  RYL is not going to stop me calling it how it is.  But I should not, nevertheless, have to take public abuse on the internet for doing so, just for doing what I see as my job: abuse that anyone (such as colleagues in other universities) can read without knowing the context behind it.

But I am an old hand who generally (as I said) gets good feedback (even from that 'Barbarians' class).  Let me give you an example that raises more serious issues.

A few years ago we had a young lecturer who was given the task of giving a survey lecture course to the first-years.  This is a serious and daunting task, with a large audience.  Maybe we shouldn't have saddled her with it, though everyone has to start somewhere, and it's the sort of thing I had to do, back in the mists of time, in my first job.  I guess one or two lecturers are brilliant from the off but most aren't.  Even the best are a bit rubbish at first, as they'll admit - even if they tend to be rubbish in an entertaining or endearing way (as I hope may have been the case with me) rather than a boring or incompetent way.  But one thing you often do when you start is to have a script and read it.  It is a prop for confidence and it makes sure you stay to time and on the point.  But it also (usually) means you have far too much to get across.  And so it was.  Unfortunately this lecturer was also up against (in the other survey courses) experienced and popular lecturing old hands.  I had to review the feedback on this course and it was vicious.  Unfair, rude, unhelpful, destructive, sexist.  Indeed I really think that a male lecturer who did the same would not have faced the same abuse.  It was, in short, bullying; bullying a bright young woman starting out on her career.  This was bad enough in internal feedback.  Imagine what it would be like if those students had published all that on the internet for all to see.  Apart from potentially harming employment or career prospects, how would it feel to read all that about yourself?  To adapt the famous quote of (I think) George Bernard Shaw, 90% of lecturers will admit to liking reading gushing praise about themselves from students, and 10% will lie.  Similarly no one apart from the super-thick-skinned likes reading bad feedback or rude comments about themselves, even if 1% of the total return, even if manifestly unfair, even if patently written by someone who never turned up and did no work.  So this sort of thing could easily drive someone with great promise out of the entire educational sector.  Indeed some of the students started a Facebook group that had that abuse about that lecturer on it - which I shut down pronto.  The problem with the attitude that the anonymous, 'drive by' abuse culture fosters is that is is one wherein one does not think about how comments affect the recipient.  It fosters the idea that you have no responsibility.  These are all very bad things.  The fact that it goes on all over the place, from comments strands and Facebook through to Amazon does not make it right or justify the production of yet more.

The Union should provide legal assistance to back action to have lecturers removed from the site, thus making it even more meaningless.  It should also sue Bulman's ass for any even remotely bullying or offensive material.  I doubt it will do either.

But otherwise, by way of response, how about 'Rate Your Students'?  Lecturers could (anonymously of course) name the students on their courses and grade them on their punctuality, attendance, ability to write, manners, reliability, performance in class.  Maybe they could add free text comments on their personalities and defects, and maybe a chilli or two on their general 'hotness'?  There would be every bit as much justification for this.  We educate people who then go out into the wide world and are employed.  Thus the employers of our students are 'stakeholders' (in awful New Labour Speak) in UK Higher Education and deserve candid information to be made freely available about their potential employees that goes beyond anodyne results break-downs and reference-ese. The people who - though they may not pay the fees - pay much else towards students' education need also to be sure that the students are working hard enough.  Those who make the loans deserve to know whether they are backing a sound investment.  And education is a two-way street.  Students deserve to know whether their fellow students in a class are going to be collaborative and supportive or whether they are going to be the sort that sit in seminars never saying a thing, who make no contribution to class discussions or group work - in other words, who piggy-back on everyone else's hard work.  It would be the only way of raising standards among UK students.  Why not?  Additionally, it'd get loads of hits as a site - maybe more than RYL - and generate a decent advertising revenue.  [Get in touch Mr Bulman.]  But can you imagine what the students' response to that would be?!  But why the outrage?  If you're opposed to being publicly, anonymously rated it must be because you're a shit, lazy student who's only at university to drink and get laid, right?

14 comments:

  1. Curiously, history students at my place stopped posting to RMP years ago.

    ReplyDelete
  2. '...it must be because you're a shit, lazy student who's only at university to drink and get laid, right?'

    *sticks up hand* Alright, I'll admit it. Four years in it's getting a bit old, though.

    ReplyDelete
    Replies
    1. I was confessing to only being at university to drink and get laid!

      Delete
    2. And how's that working out for ya?

      Delete
    3. I imagine it'll wear thin by the time I submit my doctoral thesis.

      But ignoring my dreadful attempts at humour and in the interest of providing something of value to the discussion: it's so refreshing to hear commentary against the notion of the 'student-as-consumer'. I'm certainly not going to dispute that my accumulated debts from the past four years have made me think about what I'm getting for my money at times, but I've always been confused by people who make the 'student-as-consumer' argument. It always appears to drastically misunderstand what a degree is and what puzzles me most is that many who make the argument don't just want the degree as a means-to-an-end. In other respects they still appear (at least) to value a degree for what it is.

      Delete
  3. 'They also drive me up the wall sometimes...' very much guilty I think.

    'One or two, I admit, I want to punch in the face...' again very much a possibilty following on from the first point.


    But that aside its a fantastic and well written piece and I cannot fault the arguments within it. I have always thought its a great shame that while us students get the ability to comment anonymously in feedback, it seems with little fear of the repercussions, that lecturers cannot enjoy the same freedom. I do tend to wonder when receiving feedback if there was a box underneath the official feedback entitled 'what I really think of you?' what it would actually contain.

    My only concern is that by writing about this you may be countering your own argument asking us all to ignore it by actually alerting us to its existence. I initially wanted to forward it to a friend but then thought better in case they forwarded it again and it resulted in increased and unwanted attention to the site.

    ReplyDelete
    Replies
    1. Thanks for that, um, feedback. I think that thankfully (and maybe unsurprisingly) most lecturers would be too mature and professional to fill in a 'what I really think' box. As to the other point, the site will get the traffic anyway, I imagine. I just feel that some critique and information also needs to be available. I didn't think that Bob Cooke made all the right points in his piece and drew a lot of flack for that and his response to critics. That said, his piece and the replies to it were helpful in framing my post.

      Delete
    2. I was not suggesting that the 'what I really think box' be used in a childish manner perhaps I should have changed the title.
      I merely meant that lecturers have to be careful about wording their responses to our work and contributions in seminars. An example could be a student who does not contribute to the seminars, has perhaps insulted people and produced bad work. This is unlikely to appear as such in feedback it is more likely to be disguised as there is room for improvement or something like that.
      My point is that I'm guessing that departments have rules to regulate lecturer feeback yet students do not have rules when we give feedback which is a little unfair on the lecturers part.

      Delete
    3. p.s also realized the first part of my initial comment made little sense I was referring to myself as guilty of being one of those students who drove you up the wall with the second quote following on from that, NOT that I was suggesting at all that you ever would punch one of us in the face. We all know you wouldn't.

      Delete
  4. I'm intrigued by the analogies used to justify the site... Another reason the 'everything gets reviewed' claim is flawed is because there are no restrictions on who can buy a book or a kettle on Amazon - publicly visible reviews are meaningful because any member of the public might buy the item in question. For a university course, only registered students at the university are eligible to take the course, so having a review available to the entire internet has no meaning (I don't have to enter my A level results or GPA in order to be accepted into a pool of people eligible to buy the kettle). I guess another way of saying this is that we don't generally expect people's work to be reviewed publicly UNLESS their work is a public commodity (a TV program, a book, a performance etc). A lecturer's work is not publicly available in this way. I'm sure anyone else who worked in a private institution would resent their work being reviewed for all to see, say their annual appraisal being published online.

    Also, I think the Rate My Student site is a great idea - was discussing the very same idea with a friend after reading the Bill Cooke and THES articles. Actually, there is a sort of (analogous) precedent in the US... Say restaurant reviews are like RMP/RYL, where consumers rate the product they've purchased, perhaps even mentioning individuals by name (e.g. the waitress who did a very good or bad job). Now, tipping is pretty compulsory in the US and actually makes up some portion of a server's wages. So, when a customer fails to tip adequately, the server might get ripped off financially, or at least feel affronted that their hard work was unappreciated by the customer... and there ARE 'bad tipper' websites out there where waitstaff name and shame people who failed to tip. Whether or not we might feel this is ethical, my point is simply that maybe we do live in a world where everything gets reviewed online, but it's not only the purchaser/consumer who can do that - isn't it (arguably) just as legitimate for the customer/student to be reviewed online when their actions and behavior impact the everyday working experience of the server/lecturer.

    ReplyDelete
    Replies
    1. Thanks. Good points. I should say, though, for anyone looking in, that I was only being flippant about the Rate Your Student thing... Then again...

      Delete

Note: only a member of this blog may post a comment.