There may not be specific laws that deal with going into opposite sex rest rooms. Janitors do it all the time. However, I am extremely confident that if a guy tries to walk in to a ladies room, on purpose, they'll find something.
I don't know exactly how "peeping tom" laws work, but if they can be applied, they will be. If he actually uses the facilities, then there's indecent exposure to be considered.
Basically, guys who try to get into ladies rooms are really creepy, and they'll find something.
On the other hand, women who use the men's room are in a hurry, practical, and not overly shy. No one wants to throw such a lady in jail.