That’s a decision you and the business need to decide. Turning off SSL decryption will prevent the ability to inspect the traffic, look for malware, perform DLP (if casb was purchased) and a whole bunch of other protections and options.
For us we go the other way. We SSL decrypt nearly everything and use exceptions in rare cases, like CPA’s. CPA’s really allow us to get granular in what we exempt from steering.
Other than speed test websites (again, not reliable with a SWG) are you experiencing observable issues? Like is a file taking 5x longer to download/upload? That shouldn’t be the case. I think for a business it’s accepted that an employees browsing experience might be a little slower (at the benefit of protecting all your company data).