Sex is complicated. Some plants require the cooperation of more than two different gametes to produce a viable seed, and some animals live parts of their lives as male and part as female while others switch sex according to chemical signals in their environment and others are hermaphroditic (and in some cases even capable of self-fertilization) while yet others are capable of parthogenesis (by means of forgoing meiosis). Among humans true hermaphrodism is rare but not unknown (though not sfaik with evidence of fertility in both directions), and various other intersex body forms can arise from genetic abnormality or fetal environment. Even more common than intersex bodies are the cases in which behavior is different from the norm for the apparent body type and/or genome. This includes both cases of gender dysphoria and exclusive homosexuality (with the latter being so common as to apply to on the order of 10% of all humans). The range of secondary sex-related characteristics is of course huge – with bearded women and breasted men being far more common than is visible after the adjustments of depilation, makeup, and clothing. Not to mention the range of other physical attributes, sensitivities, and attitudes, which, despite perhaps being distributed bi-modally with high correlation to apparent sex, are nonetheless of such variance that sex is a poor (and in any case inappropriate) predictor.
Getting back to the two most highly correlated variables (namely chromosome pattern and externally apparent genital structure), they do indeed define two non-overlapping clusters in the multi-dimensional spectrum of human sex types which almost encompass the entire species.
Because of the dominance of these two particular clusters and because of the correlation of these clusters with variation in physical size strength and aggression (and maybe other factors as well) those who identify with the “weaker” sex may feel at risk or disadvantage when exposed to the other group in vulnerable situations or in athletic competition. As a result, certain facilities and competitions are segregated, but then the question arises of how to deal with those who, for physical or psychological reasons, don’t fit into one of those two clusters.
Some people feel genuine terror at the prospect of entering an almost exclusively male environment (especially one involving public nudity) and others feel equally genuine terror at the prospect of not having certain spaces free from people who have penises (or in some extreme cases even from people who have some secondary male characteristics despite being biologically clearly female).
Some people who have changed their social presentation want to engage in sports in the category that they identify with, but many women feel that it is unfair to have to compete athletically with people who have the advantage of having grown through adolescence with typically male bodies and hormones.
These are not easy issues to resolve, but calling the latter groups above a derogatory term like TERF (or “bigoted fuckface”) is offensive – and so too is insisting on calling the former men and referring to them with male pronouns when that is easily avoided. (My personal preference is to refer to a person by whatever pronoun she or he prefers, but if someone feels unable to do that I don’t object so long as some other way is found of avoiding the pronouns that are known to be hurtful.)
Unfortunately both sides in the “debate” are most wrong when they claim that “science” supports their view.
YES it is true that, in humans, sex is almost always binary and immutable, BUT it is not always that way and it is certainly not a universal pattern in biology.
YES there is a (multi-dimensional) spectrum of sexual categories, BUT if anything, this contradicts the idea that one not in one of the two dominant categories can be appropriately assigned to one of them (either just by declaration or by some other means).
Science does not tell us how to behave, and in particular it does not tell us how to respond to people’s desire to appropriate an identity in a way that others feel is inappropriate. What we need for that is empathy, caring, and maybe a bit of wisdom.