people consider it bad because they fear it will be forced upon them by racist/ableist powers

Yes, because this was normal thinking 150 years ago when eugenics was gaining prominence:

At some future period, not very distant as measured by centuries, the civilised races of man will almost certainly exterminate, and replace, the savage races throughout the world.

Charles Darwin * The Descent of Man, and Selection in Relation to Sex*

Many people took this as, oh ya we should do that.