It’s mostly a game of chance.

Grossly oversimplified: Say you have black eyes, but carry a blue-eye gene (which doesn’t show because black is dominant). There’s a very good chance your sibling carries the same gene. If the two of you have a child together, there’s a higher chance that the kid will carry double blue genes and has blue eyes.

Had you had the kid with another black eyed person that’s not a sibling, there’s a higher chance that they may carry different genres (black, grey, brown, green eyes). So much lower chance of having a blue-eyed kid.

Imagine the same but with all sorts of physical and mental defects. They may not show in you, but if you carry the genes, there’s a good chance your sibling does too. Hence the higher chance of it showing up in your kids.

Even if the trait doesn’t show in your kids, over generations of inbreeding practice (common in old royal families) the bad genes would be so concentrated in the pool that so many defects will start showing up.