I will challenge your experience here, and suggest that what you're missing is context.
In isolation, with good lighting, a lot of prep-work, undergarment scaffolding, very heavy makeup, and in the presences of similarly size males... a skilled transvestite might pass. But put one of those very convincing crossdressers in the presence of a few females, and it won't generally hold.
When you're looking at a handful of highly costumed males in the presence of other males, the size difference is masked. Put them in the presence of females, and the shoulders, hands, and feet almost always stand out. It's good mimicry, sure, but only effective in isolation.
Think about this - almost all of the transgender identified male athletes that you ever see pictures of are either alone or taken from an angle that de-emphasizes the size difference. Same with actors, or pretty much any other photo you've ever been shown. Careful lighting, heavy make-up, and no females nearby to compare too. Very, very, very few are small-statured and delicate enough to pass when they're among females.
By the same token, transgender identified females (transmen) don't pass as well when they're among males. The beards and lower voices go a really long way toward being convincing at a glance, as does the larger standard deviation in height among males - they're more likely to pass as a short male at a cursory look. But put them among actual males, and the difference in build, hands, and feet can very quickly become obvious.
Not always - some will pass effectively on both sides of this. But in reality, the vast majority do NOT pass as the opposite sex, especially when actually compared to the opposite sex.