>Why should the government limit my freedoms?

Government consist of people, because to this group of people, its in their best interest, its ethical for them to not allow rape.

Does the government care about this rapist freedom ? No, in their perspective its wrong.

So whichever side that can force other to comply (persuasively or physically) get to decide the ethics.

In the case of rape, the government get to decide whether you can rape.