Dude, please... "little people."

I never got that. How is "little people" less offensive than "midget" or "dwarf"? I mean, dwarfs are cool, in that Tolkien retro way. When I think of "little people" I think "less than a real person." Consistency, that's all I'm asking for.