Regardless of whether you're a theist or an atheist, we all agree that the regress of explanation has to terminate somewhere. At a certain point, you get to some fundamental facts for which there is nothing to explain them – they simply are.
The difference between a believer and a non-believer, then, is that non-believers don't see any reason, scientific or otherwise, why the universe itself cannot simply be. Theists, though, are not satisfied with that and want to push the problem back a step, where "God" is the explanation for all physical reality. But what's the explanation for God being the way he is? Why does God even exist at all? Well, so the theist would opine, God doesn't need an explanation. He just is.