"If people have theories that associate me with that, it’s a reasonable thing to think," Colbert said of discussion about why ...