OpenSSL is continuously scrutinised and has resources and pressure to fix issues.
A less ubiquitous library is not as scrutinised (so who knows what vulnerabilities lie within?) and has probably not the same resources/pressure to fix.
I’m not suggesting you fork it yourself, write your own, or use Joe Schmoe’s SSL library. There are other major implementations such as Amazon’s s2n that have many eyeballs on them daily.
Diversity of infrastructure components confers similar resistance as DNA diversity in the wild.