Yes, but if the big bang is right, the universe would have to expand faster than the speed of light if there are parts of it that are invisible for us. Or am I completely off base here?
The Big Bang is compatible with an infinite universe. As Schneibster said, only the observable universe was the size of a pea, the whole would have always been infinite. Even if the universe is finite, with all of it the size of a pea a long time ago, its radius is bigger than its age times the speed of light. This is not in contradiction with SR. You can read this tutorial or start with these two FAQ:
How can the Universe be infinite if it was all concentrated into a point at the Big Bang?
If the Universe is only 10 billion years old, how can we see objects that are now 30 billion light years away?