Fantasy is wonderful stuff. Any set of rules is arbitrary; it's coherence within that ruleset that makes something believable. There's no reason we can't enjoy a story set in a universe using a different ruleset.
You've just expressed why I'm so dissatisfied with a mystery series I've been reading.
Rant coming your way -- duck!
Jacqueline Winspear's "Maisie Dobbs" books are presented as straightforward cozy-mystery stories, set in the 1930s: they
purport to be operating in our universe, to be using the ruleset we function under every day. Except that the heroine is, although it's never acknowledged on the covers or in the promotional material, magic. It's
cheating to say "This is a mystery story and nothing else", and then to have your detective character know what people are going to say before they say it. Or to have her -- flawlessly and inerrantly -- perceive someone else's emotions and even physical sensations by mimicking that other's body language. Or to have her know where a violent crime happened, because the place feels cold. Or to have her figure out a missing woman's thoughts and feelings by
meditating in the woman's bedroom:
Maisie concentrated on her breathing, stilling both her body and her mind, and she began to feel the strength of emotion that resided in the room. This was Charlotte's refuge while in the house and had become a receptacle for her every thought, feeling, inspiration, reflection, and wish. And as she sat in meditation, Maisie felt that Charlotte had been deeply troubled and that her departure had little to do with a broken engagement. Charlotte Waite had run away, but what was she running from? Or to? What had caused such an intense ache in her heart that even now in her room, Maisie felt Charlotte's lingering sorrow?
It IS possible to set mystery stories in a world of magic. Think of Randall Garrett! He set his "Lord Darcy" stories in an alternate twentieth century where magic and sorcery have been subject to legitimate "scientific" study and exploration for eight hundred years. Garrett did put a very strong and coherent ruleset -- thank you for those words -- around the functioning of magic in his universe, around who could use it and what they could and could not do with it.
Winspear doesn't do this. She pulls things out of the general vicinity of her seat cushion. The book I most recently read (not the latest in the series -- I have three more to go, if I can stomach them) abruptly retconned the heroine's family history to give her a Roma ("Gypsy") grandmother -- after introducing as guest characters a tribe of "Gypsies". The scene that had me jumping up and down and screaming and throwing the book across the room was the bit where the tribe's requisite Wise Old Woman character taught Maisie to
dowse. Or, more precisely, showed her how to use her
inborn talent for dowsing. And it gets worse: the first thing she finds by dowsing is her treasured watch. Her
silver watch. And one element of the plot involves a search for some stolen silver goods.
This is cheating. It's cheating to suddenly announce that your character has a magical ability that enables her to solve the mystery, and to give it no more basis or grounding than "Well she just CANNNNNNNN."