Depends on your system specs, but… yes, generally speaking. There is a reason most people and most distros use binaries. Even Gentoo can use binaries for some stuff.

Are you going to suffer significant damage if your updates take forever though? What’s the hurry? The number of times I have literally needed the absolute latest version of something installed right now are pretty damn minimal. The major exception is widespread, exploited zero-day remote-access vulnerabilities, but those are rare, and especially rare are ones that affect the exact versions and configurations of software that I am currently using and cannot reasonably just opt to “stop” using. Even so, there are usually other ways to block the network traffic, disable the offending part of the configuration, or otherwise mitigate the risk.

Patience is a virtue, and it’s generally good for you. You don’t have to be addicted to constant updates, but you do need to be thoughtful and understand how to build defense-in-depth.