Know your rights, as a photographer.
Yes, you still have some. I know, from the way people talk about the harassment they get as an amateur photographer in public, you may not think you have any rights left, but you do. Not sure what they are? Head over to the ACLU's website and see their fantastic article Know Your Rights : Photographers.
In a nutshell, you can photograph pretty much anything you can see from a public space as long as you're not breaking any other laws or interfering with a police officer in their official duties. The only exceptions may be military bases. But, from everyone I've talked to, military police and security are actually quite polite and friendly and reasonable. And, to be honest, most of the police officers I've dealt with are, too. I've generally had more issues with private security guards than anyone else. Video with an audio component gets a little trickier.
Also, an important point to note, police officers cannot legally force you to show them your photos or force you to delete any photos without a search warrant or a court order.
Of course, on private property, whoever owns the property makes the rules about this stuff.
And, I think it's important to note that, while you may be in the legal right, you may also be harassed anyway. And, if things escalate, you may find yourself going to jail and being booked, not to mention seeing a judge, before your rights are recognized. There are sometimes prices to be paid for exercising your freedom, which you may or may not be willing to do on any given day.
Or, as your Uncle Jim always says: "Safety first, kids!"