All of the "affairs" occurred before he was married, so they shouldn't be labeled as such.
I think they are called "relationships."
Certainly, the employer/employee aspect could be troubling, but this whole mess wasn't employee-driven, but rather, the machinations of a douchebag who got some dirt, and tried to profit from it via extortion, which, unlike "relationships", is a crime.
Certain Wingers could obtain momentary glee from having Letterman on the spit due to his prior jokes about Bush, Palin, etc. and his spot-on, face-to-face interviews with such lamentable personages as Limbaugh, O'Reilly, etc., but at least he wasn't adopting a "wide stance" in an airport bathroom, sexting with twinkish interns and/or serving divorce papers in a hospital ward, ala "Repug-style."