Inverse Functions are certainly a minefield for students, and the situation is not helped by teachers' use of sloppy language to describe the concept and the prescription of a mindless ritual for answering assigned questions.
This article points in the right direction although it's not quite perfect in my opinion1. But what got my friend Bruce to comment was one of the authors taking the objection to explanation by procedural prescription into another area where it might be less apt - namely the concept of average value.
Expressions like "the inverse of y=f(x)" are problematic because the relation defined by y=f(x) is the same as that defined by x=f^-1(y) and does have inverse relation defined by y=f^-1(x). So, contrary to the article, it is in some sense correct to say that the "the inverse of y=f(x) is y=f^-1(x)", and the formal definition of functions as sets of ordered pairs does justify "switching x and y" if this is interpreted and explained properly.