You can try and let him down easily.. just be honest and open with him.. tell him that you are very flattered but that you arent interested in playing around on your wife at all with anyone. Hopefully he will understand and you can remain friends.