You're exactly right. "Deinstitutionalization" is the source of many of our problems today. The inmates were released, but the local centers were not funded or built. So now we have a population of people with severe mental health problems who have nowhere to go. Many of them are harmless, but...
I'm a firm believer that most people go through life barely aware that there's a person in front of them, so they would take no notice of how that person is dressed. You can recognize the few who are aware of their surroundings, and they are probably carrying too ("hello, brother"). And LEOs...