Your line of reasoning suffers from one fundamental flaw: If people can simply self-identify, why would anyone want to self-identify as a race that they know would be discriminated against?
Well, one reason is because there's strong social pressure within the group to conform to the in-group social expectations; John McWhorter, for example, has spent quite some time documenting anti-intellectualism among American blacks, and -- interestingly -- black children who act intellectual are criticized specifically for "acting white." To attempt to self-identify as "white" will expose you to as much or more discrimination among blacks than self-identifying as "black" among whites. And if most of your social contacts are black, that's a much bigger disincentive.....
So your question itself is based on a false premise :
iWhy would a black man self-identify as black, if he knew it would get him in trouble more than it would benefit him?
For many, self-identification as "black" is actually beneficial if they're in a strongly "black" community, to the point that they will attempt to become
more prototypically black -- essentially, trying to make themselves more black instead of make themselves more white. Again, this can produce strongly counterintuitive results, such as Lichelle Laws' rather infamous "Trying to get to Watts, but I'm stuck in Baldwin Hills" lyrics.
Beyond that, of course, self-identification only works as long as other people tend to agree with the results of the self-identification. As you point out, correctly, a white
could self-identify as a black to try to get money out of a college-tuition-for-blacks fund. As a single isolated incident, this isn't necessarily a problem; in order to get the money, the white not only has to self-identify, but the fund officer has to
agree, otherwise, he doesn't get the money. The funds officer, of course, has his own prototype (again, there's no "bright line" involved). The system is robust enough to stand up to a few mistakes. If the funds officer's decisions vary too radically from society's as a whole, he will either be removed from his job as incompetent, or else donations to the fund will drop to the point that the fund is no longer viable.
So the system is robust to a few errors, but widespread attempting to work the system as you describe would produce too many errors, and people would stop relying on self-identification. For example, some schools/foundations used to require applicants to submit photographs with the application. This substitutes the prototype-based judgement of the foundation's officer with the self-identification of the applicant.
You still haven't argued why you need a single bright line to define people's race when every other categorization decision that people make can demonstrably be handled well by prototype-based decision mechanisms. If your point is simply that racial classification is not objective -- so what? If you argue that people can disagree about a given person's race -- again, so what? The fact that two people can disagree about a borderline case in a prototype-based classification system does not invalidate either the system or the classifications.
Or if you feel it does, then take it up with Dr. Rosch.