If you consider yourself a rational person, and people are telling you you're hurting them, someone else, or the community, then please. Interrogate your motivations.
It is so amazingly easy to draw a chain of implications that ends with something like "they deserve it" or "they're weird" or "they hurt me first." But you not only need to ask why, but you need to ask why in the many different ways. Don't just ask "why do they deserve it?" that's a loaded question. Ask "why should I punish them if they do?"