Is there any performance penalty using --with-ssl?

Hi!

I am preparing the update of the FreeBSD port of PostgreSQL with the 
upcoming 7.2, and I'm just wondering: is there any performance penalty 
intoduced by including --with-ssl in the default configure args? Of course, 
if SSL is actually *used*, I know what'll happen ;-)  Just wondering 
whether there is any reason not to include it by default if it exists on 
the system; will it decrease performance for those who don't use it?

Regards,
Palle