I understand the general halachic issues with serious crossdressing, people with gender dysphoria etc. because of what it may lead to, using it to be secluded with the opposite sex etc. Is there a serious problem however, for a regular man with no gender challenges, to wear clothing of the opposite sex in private for recreational purposes and fun with no intention to mislead anyone?

Although there is a discussion in the talmud as to whether crossdressing (see Nazir 59a), when it will not lead to intermingling, is permitted, the conclusion, as well as the psak of the Rambam and Shulchan Aruch, is that it is forbidden in all curcumstances. There is a position that wearing clothes of the other gender is permitted when dressing in a costume for OPurim, but this is not generally practiced, and Rav Ovadiah Yosef, among others says it is not permitted to rely on this leniency.