I guess I'm in the minority here, but we (hub or I) answer the door to solicitors. I want anyone possibly casing the house to know that someone is home, and that we are happy to engage with them.
I've never been scared of anyone who has come by, but it's not in my nature to be scared anyway. I'm firm with them, but polite...and I have never had any problems...