Risk of Continuing Finasteride After Side Effects?

I have been taking finasteride for the past 3 years and have found it good for maintaining my hair. I have, however, experienced side effects. I have a greatly reduced sex drive and no longer get erections unless stimulated physically.

My curiosity lies in whether or not I am doing damage to my body by continuing use after side effects. I am too afraid to stop taking the drug because I don't want to lose my hair, but I also don't want to cause excessive damage to my body.

Is there any difference in regards to hormonal profiles/receptor activity from taking the drug for 10 days vs. 10 months vs. 10 years or is it all the same as soon as you cease the inhibition of DHT?

Thanks!