The problems of interstellar travel can be overcome a few ways, and I think eventually will.  Human life expectancy may be greatly lengthened, or generational ships set out, or hibernation, and so on.  I don't think the light-speed barrier can be fudged, though.

The thing that makes their presence most unlikely is the great number of hurdles life would have to cross in order for "them" to be there -- some of which are almost certainly very unlikely.  Not that "they" don't exist, but they are almost certainly excruciatingly rare -- as rare as only one example in millions or billions of galaxies.

The way intelligent life can become common is for it to spread, and so far in our galaxy that hasn't happened or they would be here.