Neighbours of the north Toronto property, where the Shermans’ house once stood, are worried about what it might become.