If there’s one aspect of life where prejudice is sort of OK, it’s dating/choosing sex partners. You don’t owe anyone a detailed explanation of why you don’t want to spend leisure or social time with them; you don’t owe anyone anything more than simple courtesy.
A regular claim from anti-trans wingnuts, however, is that they are being punished/erased/discriminated against – or labelled
transphobic – for refusing to date or have sex with trans people. It’s not the fact that say you don’t fancy trans people that makes you a bigot, wingnuts. It’s the fact that you keep screaming about it when no trans person has actually asked you for a date. No one cares who you don’t want to sleep with; most people don’t care who you do want to sleep with unless they are the object of your desire, or a person who desires you. OK, some might raise their eyebrows or mutter a bit about someone who only seems to date a specific ‘type’, perhaps a little more if that type is notably different from the dater: always younger/older, always of a specific ethnicity that is not their own or whatever, but they are mostly not losing sleep over it.
Some monogamous people find their perfect partner at a young age, and never really consider their orientation afterwards: why would they, when they are happy with their circumstances? Most of us, though, are at least potentially pansexual and at least could 
be surprised by a sudden, intense desire for someone outside of our usual dating categories.
It’s not that pre-emptively shutting yourself off from a wider pool of potential partners is inherently bad, or wrong. Denying yourself something you might like only really affects you. Someone you reject might experience some hurt feelings, but they’ll soon move on. If you are incapable of shutting up about how many people you wouldn’t ever go near, you just come across to others – including people you might actually like – as rude, childish and ultimately boring.