This is indeed the trade-off when using the 'only show followed content' toggle of the Recent Activity widget. Setting the notification frequency to 'Never' essentially sets that piece of content as unfollowed, so it won't show up.
To keep notifications turned off and still see recent activity, the advice would be to uncheck the 'Only show followed content' box of the Recent Activity widget. Alternatively, one could set a conservative notification frequency such as 'Monthly'.
Hope this helps,