Everyone has haters: no matter how virtuous or cruel a person is. Everyone is adored and despised by someone in some way whether we like it or not. None of it should hold you back from living your life.
Stop being afraid because you believe you will get hurt. Nobody has lived a life without getting a scratch; may it be physical or mental.