I've read some short stories based in that scenario, but don't think I've read any of the novels. Might have to look for a few
And what I stated was only as we currently understand. There's always a possibility that we're wrong

It's similar to considering the Hidden Variables hypothesis for quantum entanglement...if it's true it'd have to be because of something even wierder than the wierdness we already have. And, or coursee I may be way off (again, I am not na expert, just an interested and well-read layman).
And that possibly unknowable hits it on the head. It's like the HUP: You can't determine momentum and location precisely, and the more precise you measure one, the less precise the other. This is due to physical limitations: we can only measure it by bouncing something off it. Likewise, it can only interact by contacting something. So, we can bounce a particle off it (such as a photon or graviton). But, the more precise we want to measure the position, the smaller-wavelength particle we have to use, which means higher energy which means momentum gets whonked out. And vice-verse, we use lower-energy to measure momentum of the particle, but that smears out it's location. And this holds true for natural processes, as well, as something has to interact with a particle for anything to happen.
It's the same concept with the time idea. Since there is a minimum "distance" across which we can measure with any meaning--the universe's "maximum resolution", for shorthand--and a maximum speed--c--we end up with a minimum time.