I think you are right about the conflict of interest, but the other models probably don't know him well enough to know he does this. In the reverse, he could very well lead his girlfriend to believe she's getting first consideration, when in fact she's not. Contacts Tom Ely once maintained in the northeast, don't have good things to say. I suspect that his new girlfriend in Florida has credit cards and other resources he cannot get himself, but that he needs to operate some aspects of operations. This was the situation in Massachusetts with his previous girlfriend.
Some people in the northeast have a lot to say about him, and it's not good. People were made promises and invested professional time on alleged projects that they now believe may have been more sales fluff than reality. He's considered a salesman who tricks people to give him what he wants based on inflated presentations.
Male models involved with that agency have played second fiddle to Ely's desires to have sex with certain models. As a result, they get less work. Certainly there is a conflict of interest there as well. His 'agency' is alleged to be used more as a vehicle for self-gratification than real management services, and it takes too much time for new models to realize this before they recognize they have wasted their time.