Some plugins, like BuddyPress and Events Manager, use their own email delivery solutions. By default, this option does not block their emails, as those plugins do not use the default wp_mail() function to send emails. You will need to consult the documentation of any such plugins to switch them to use default WordPress email delivery for this setting to have an effect. 