For Reference Purposes

One of my pet peeves in letters is when the letter is too much about the research, not enough about the person. This is often worse in the case of a graduate school application than in the case of a faculty search, since the details of the research are much more relevant in the latter case, but even there I see it too often.

I've read a lot of applications to grad school where the letter-writer goes on for 2/3 of the letter about the research, and it's difficult to tell if the letter writer is trying to support the applicant, or the letter writer himself.

With a faculty job, yeah, it helps to have somebody who will tell you the research, and why it's significant and important, as many on the comimttee won't know that. But, even there, too many details just bog down the letter.

-Rob

This candidate amply fills a gap that would best remain empty.

The key here is your advice to faculty members to just say no. I don't write reference letters unless I can be strongly supportive with plenty of evidence to back up my recommendation.

But it's tough to say no, especially if it's someone you like. I usually make the excuse that I haven't been close enough to the research to make a valid judgement. If that's not true then I point out that there are others who are closer to the candidate and know their personal strengths. When neither of those excuses work I resort to the excuse of being too busy.

I think letters of recommendation say more about the person writing them than the person applying for a job/school. Some take their time to write a thoughtful letter. Others put together generic stuff because they have better things to do. Yet others will write a letter full of "exceptionally good" epithets for everyone they know, regardless of their qualities.

Letters are archaic and very strange way of evaluating candidates. If A says B is just great, but C says D is the best, does it mean D is better than B? There's no renormalization. Older people tend to write more honest and modest letters even for exceptional people, while younger people tend to exaggerate (similar to grade inflation problem).

Nobody will tell you frankly that a candidate is a jerk who is manipulating his way to get on every paper. So what's the point?

The letters may have some weight with evaluating entering PhD students, but even then - how well does anyone know undergraduate working for them for half-a-semester or a summer?

Faculty search committees should do the first round based solely on research credentials. Once the search is narrowed to 10 or 20 candidates they should contact references in person - via phone, for example. Even then I wouldn't weigh this opinion very highly. It's not a very objective measure, it says very little about candidate, and there's no way to renormalize to compare letters written by different people.

It's a waste of time, effort and paper, if you ask me. If someone is producing good science and is active and prolific, you don't need letters to tell you that. If someone is difficult to get along with - you will get a sniff of that during on-campus interview.

We might as well as the candidates mothers if they think their son or daughter is a nice enough person to deserve a tenure-track position.

Reminds me of Seinfeld episode where Jerry mother says "How can anyone not like you?"

It's a waste of time, effort and paper, if you ask me. If someone is producing good science and is active and prolific, you don't need letters to tell you that. If someone is difficult to get along with - you will get a sniff of that during on-campus interview.

It's a nice thought, but a terrible idea in practice, as we don't have the resources to invite more than a handful of candidates to campus. We really can't afford to waste campus visits on people who turn out to be difficult to work with.

And I do find the reference letters very useful in the hiring process, though it wouldn't be appropriate to give examples of why and how. Suffice to say I find the references more useful than the CV, in general.

Nobody will tell you frankly that a candidate is a jerk who is manipulating his way to get on every paper. So what's the point?

It's an art to read letters of recommendation. If somebody is merely "good", then that's a terrible letter. Sometimes you find carefully phrased setences where it's clear that the letter writer was fishing to find something that the applicant was "best" at. Sometimes you may see a hint or two that a person who has an otherwise excellent record might have some strings attached; when you catch the hint, you should call the letter writer and ask what the deal is. It is frusturating the level of hyperbole that happens in letters -- everybody is great -- but it's just what we deal with.

Some letter writers also will say right up front how they rank the person compared to other similar candidates they know about who are applying for jobs this year.

I've found that the letters of recommendation are the best and most important piece of information for students applying to grad school. Yes, grades and GREs and all of that matter. But GREs are standardized tests, and we all know all about that. Even grades aren't really the right thing for grad school... a letter stating the student's performance from the student's research advisor is far more useful.

When it comes to faculty searches, again, letters are of the utmost importance. Sure, you could develop statistics and do it all based on citations... and you'd be making a huge mistake. There are all kinds of differences in what should be expected between subfields, and indeed even between individuals. Already there's too much gaming and padding of publication lists and tricks to pump up citation counts. Aren't you more interested in who the person is, and what he or she is capable of? Judgement is needed. Letters of recommendation are crucial for that.

-Rob

for graduate students letters may prove to be valuable because there's very little else to judge them.
Similarly, Small Liberal Arts Colleges which emphasize teaching over research may have to rely on letters more heavily simply because it is difficult to evaluate teaching abilities, and personal skills have a lot to do with that.

Finally, don't get me wrong - I do agree that personal skills are as important if not more important than research skills.

Having said that, letters have a lot of problems. The original post over at Female Science professor reveals several. Inflation (similar to grade inflation) is one. Being "merely exceptional" is not enough anymore. God forbid your advisor is one of those old-fashioned folks who doesn't want to use words like "extraordinary", and instead favors more simple and less flashy language. People trying to read between the lines will think you are a moron and difficult to work with, simply because of inflation. It's like getting an A- from a fair grader when everyone else is getting A+ and A++.

Another point is that very often people are too busy to sit down and write a thoughtful letter for someone. There's no reward for writing a good letter vs. writing a crappy cookie-cutting one. This is why calling people to talk to them about the applicant is a thousand times more valuable than reading something on the paper.

I am curious as to what happens to applicants with one really good letter and two lukewarm ones, or two great ones and one so-so. Do they all count, so the applicant who happens to work with three people who are thoughtful and inflated letter writers always wins? Or do people occasionally dismiss a short lukewarm one, or the "cookie-cutting" one in favor of more detailed ones? Are all students working for an advisor who writes un-inflated letters doomed? Or if the advisor is too busy or thinks the whole letter writing is a sham anyways. In some cases there are clear conflicts of interest where if accepted, the applicant becomes a direct competitor of the letter-writer.

Sure, one could *try* to pad the CV and publication list. But for anyone with a few minutes of free time it doesn't take much effort to figure out who the applicant actually is and what they have done. There's clear paper trail of research, and 10 proceedings papers don't equal a solid PRL or Nature or Science or whatever. Finally, one could actually try to read at least some of the papers for at least top applicants (a novel concept, I know).
And chances are - people doing good productive research are also good communicators and have good personal skills, simply because it's what it takes to do good science in collaborations nowadays.

Physics committees should follow those in Materials Sciences and other disciplines where an applicant is asked only to provide a list of references, and those are contacted once the first or second round has narrowed the field to managable numbers of a dozen or so candidates. In some cases it could be done over the phone, where you can actually get a somewhat more honest response.
Otherwise, like in Chad's example, someone has to compare 200 applicants times 3-5 letters per applicant = 600-1000 letters, which are not normalizeable due to large variance among letter providers, and are highly subjective anyhow.

Again, it wouldn't be appropriate to go into much detail here, but I think the talk of rampant recommendation inflation is a little exaggerated. It may be that people wrote much more damning things in the past (my experience reading recommendation letters goes back a whopping five years), but I haven't really noticed a ridiculous degree of inflation, or felt that I had any difficulty in normalizing between letters, at least for those letters that are worth reading in the first place.

As for what happens to people who have one bad letter out of three, it's impossible to say anything definitive, as there are many other factors involved.

There's no reward for writing a good letter vs. writing a crappy cookie-cutting one.

Isn't there? Placing your students in the best jobs you can get them is one of the primo ways of expanding your own prestige.

some advisors realize a long-term benefit of placing students well. But for some there's a short-term threat of competition within their field too.

Chad, do not mean to pile on, but your post and responses is somewhat contradictory. You mention in original post that if your advisor(s) will tend to write letters anything like the ones described in Female professor blog, you better look for new people that you could ask for them. Then you seem to go on saying that letters system works just fine, no inflation, and it's more useful than CV.

First of all, it's impossible to tell what your letter looks like, or if your advisor/collaborator is one of those "uninflated/honest" or "lazy" or "standard" people, since letters are confidential. So one may not know if they are being unfairly penalized.

Second, at the level of grad student, or even postdoc, it's not always possible to choose your collaborators. Typically it's up to people who get the grant money. So even if you do know that your letters are unfairly bad, you are basically stuck with these people.

It may seem like there may be nothing wrong with the honesty/inflation of the letters (even though Female Professor's blog provides numerous examples of this not being the case), but only if one starts with underlying assumption that the letters are honest, detailed and not inflated. Unless you get multiple letters from the same person about different applicants, normalization is virtually impossible, so the statement becomes a self-fulfilling prophecy (system works because we assume it does).

I think in the end it comes down to risks the committee is willing/unwilling to take. With 200 candidates per spot, does it really matter if the absolutely best candidate is chosen? Maybe candidate A is indeed the best, but has a liability in a sense of a so-so letter from his advisor. Even if it's because the advisor is a jerk, rather than the candidate, does it make sense to look into this issue further? The easiest course of action is to dismiss A and go with candidate B who has perfect letters, even though somewhat less stellar research record. It's quite possible A is a perfectly nice person, even nicer than B, but was merely less lucky in choosing collaborators/advisors. But with 200 candidates to choose from, who cares? Committees play it safe, hedging their bets this way and saving themselves a lot of time.

In the end it's all about free market - departments that tend to value letters less than other indicators will benefit over the long run, by hiring candidates like A over B. Similarly, bean-counting (instead of going into details of what the candidate actually does), or discriminatory/cronyism/nepotism practices will benefit departments which are honest and fair. So maybe it's not such a bad thing overall.

Chad, do not mean to pile on, but your post and responses is somewhat contradictory. You mention in original post that if your advisor(s) will tend to write letters anything like the ones described in Female professor blog, you better look for new people that you could ask for them. Then you seem to go on saying that letters system works just fine, no inflation, and it's more useful than CV.

I never said anything about inflation, one way or another, in the original post. You're the one who brought that up.

I do find a good letter to be more useful than the CV, because a good recommender will spend a paragraph or so putting the research in context in a way that the list of publications doesn't. The job candidate should also do this in the statements, but it's important to get as many perspectives on that as possible, especially when the candidate is working outside my area of expertise. With the CV, all I get is a list of articles, with no real way to compare across candidates-- how many Astrophysical Journal articles equal one Physical Review Letter? Beats me, unless the research is put in its proper context, and recommendation writers have the ability to do this-- and can do it better than the candidates themselves, in most cases.

Will one tepid and uniformative letter completely sink a candidate? No. Does it help that candidate? Not one bit. And when you've got 235 applicants for a single tenure-track opening, you need every bit of help.

As for the whole inflation thing, that's your hang-up. I don't drop a candidate down a notch because an advisor describes them as "excellent" instead of "outstanding" or "superamazinglyterrific." I'm not giving points for adjectives, I'm looking for some context to the research (how important is the work within the field, how active was the candidate in producing the work, that sort of thing), some statements that go to teaching ability, and some sense of the candidate as a person.

The other point in favor of getting letters up front is that it streamlines the process a bit. Yeah, if we only asked for references after we got down to 20-ish candidates, we'd do less paper-shuffling, but that would also add a bunch of time between the initial reading and the campus invitations, to collect the recommendations. With the letters up front, not only do they help in getting down from 200 to 20, but we can go directly to the next step without having to track down somebody's advisor who's on vacation in Bermuda.

Have you considered that the Generic Q. Postdoc letter is written about a candidate that the author considers mediocre? I spend a lot of time writing letters about candidates I consider outstanding but tend to just dash something off for candidates for whom that seems appropriate. I would never agree to write a letter for someone I think is an idiot but there are lots of in-between students whom I can't see turning down.

I agree with Alison that lukewarm letters are an implicit warning that the candidate is mediocre at best.
Letters of recommendation often damn with very faint and fuzzy praise. Don't you think?