No I wouldn't consider all male/ male relationships yaoi. Many fans of the so called genre constantly disagree with that topic. It's just easier to shorthand it that way because that is the way it's labeled by the media. It's just basically used as a red flag for people who don't want to be exposed to the material rather than what we choose to call it, hence why it would never become mainstream.
All I need in anime are these things- Samurai, Swords, Sports, Mecha, and Shounen Ai <3