I think if it's possible, we wouldn't see any time travelers because they would try to stay anonymous.
And, who would believe a time traveler's story? People would think that person's insane, probably, and put them into a mental institution if they can.
Also, it is not known by which means time travel would be possible and if we would be able to notice anything.
I think it's pretty ignorant of some scientists to claim that just because it isn't possible with today's technology that it will never be possible. In the 1980ies, some scientists claimed that our sun was the only one in the whole universe with planets, and now that instruments and methods have been refined, we're seeing planets everywhere.