If someone goes back in time, even if he does nothing there, he changes the whole world with the butterfly effect. Because it affects the people around it first, they affect others and spreads like dominoes to the world.
I mean, there is no traveling back in time. If there were, each visit would change the whole world in the long run.
Peace
You need to read more SF. There are a number of ways around this.
1 - "If there were, each visit would change the whole world in the long run."
So what? The timeline is in continual flux. Deal with it. You and I and our world are an ephemeral timeline which will eventually produce a time traveller who will snuff us out of existence. We won't know, of course, since we will never have existed.
Easy come, easy go.
2 - Timelines are self-healing. They permit details to change, but the larger issues will be preserved.
3 - While time travel is "possible", the universe acts to prevent it and thereby preserve its current existence. A civilization which is about to produce a working time machine will inevitably experience a supernova in its immediate stellar neighborhood, wiping out the civilization before the time machine is created. Or an undetected asteroid will impact in the vicinity of the inventor's lab, destroying it.
4 - The appearance of a time machine produces a split timeline. The original timeline produces a present/future in which an inventor fires up his invention and disappears without a trace. The new timeline differs from the original in that a time machine appears in the past, and produces differences from the original. Both timelines "exist", so far as the term has meaning. And, yes, this potentially implies the "existence" of a multitude of timelines. So what?
5 - The present exists because a time machine appeared. And has, from our perspective, "always" appeared. This is different from 4) because it says nothing about the fate of the "original" timeline. This obviously plays hob with conservation laws, but so what?
6 - If a time machine is operated, it goes back in time and creates a new history. If the new history produces a time machine, that in turn produces yet "another" history, and so on. See #1. However, if a time traveller produces changes, they may result in time travel never being invented. And that is how it has always been. Just because something is possible does not mean it will happen.