I'll share my perspective. Instead of instructing you on what to do, I'll explain what I did.
The first chickens I got were straight run, which meant some were roosters. Out of the 13 chicks, I was lucky that only 3 turned out to be roosters. When they started fighting, I decided to keep Oreo, a beautiful BCM. Things were going well until a few hens got out and Oreo couldn't fit through the hole because he was too big. One was taken by a hawk and he couldn't help her. After this incident, he became very aggressive towards humans, so I rehomed him as I did with the other two roosters. I hope they are doing well, but I'm not sure.
This left my flock without a rooster for a while. Surprisingly, my hens seemed fine and showed no signs of distress.
One downside was that without a rooster, the hens had no one to protect them while free-ranging. I had to limit their time out of the run to when I could be with them. Luckily, I had a large coop and run, so it wasn't a major issue for me.
I understand the fear as I was scared of Oreo too. I was worried he might harm my kids or visitors. That's why I decided to get rid of aggressive roosters. I now have two roosters raised by mature hens who get along well and are good with my kids. There were a couple of incidents where one rooster chased my son, but it never escalated and has stopped completely.
My approach is to remove aggressive roosters, especially if they are past the hormone surge phase, and focus on caring for the good ones. If your rooster is taking away the joy of having chickens, then maybe you should consider rehoming him.
I keep telling him how I’m not sure if I can cull or rehome him, but that if he died of natural causes, I wouldn’t be sad. 
