Learned this lesson the hard way too: always follow your own path, even when the world seems against you. People will judge regardless of what they know about your situation. Do what's right for you in the long run.