I've been reading "Atheism: The Case Against God" by George H. Smith and it talks about the Christian arguing that atheists have "faith" by trusting in authority. Anyway, I think it's applicable in this case:
"We acquire knowledge in a variety of different ways, but all knowledge must eventually meet the requirements of reason. All appeals to authority must be subsumed within the guidelines of reason. The appeal to authority is not a different means of acquiring knowledge; it is one aspect of rational inquiry, not an ultimate ground of truth.
A rational appeal to authority is fundamentally nonauthoritarian. For example, if we accept the testimony of physicists concerning the truth of scientific theories, we do so not because of their authority (i.e., not because they say so), but because we believe that they are able to provide strong evidence to support their positions. Again quoting Blanshard:
"If we ask why they {the Physicists} do accept certain results, the answer is very simple; given the conditions, they have seen these results to be necessary; and they are ready to supply the data and the reasoning to anyone who can follow. In short, they do not take these things to be true because they are authorities; they are authorities because they can see these things to be true.... The court to which in the end we shall take our appeal is not authority, but those reasons through seeing which an authority becomes an authority, namely, those that condition or determine the truth itself.... If this higher warrant is there, authority is superseded; if it is not there, authority fails. In neither case is authority itself the final court of appeal.*"
*Blanshard, Nature of Thought, Vol II, p.217
In other words, when Phil Plait claims something about astronomy I accept his testimony because I believe he can provide strong evidence to support it. When Sylvia Brown claims something about astrology (or pretty much anything else for that matter) I dismiss it unless she can site specific evidence that I can verify. I think that I'm willing to accept what scientists say they have tested to a comfortable degree of certainty.