Well, unless they check their watch list often and know if you're missing from it, the site won't exactly alert them to being unwatched. So if you feel it's come to a point where you have to unwatch them to avoid it, there shouldn't be a consequence. But otherwise I'd just ignore the journals entirely, like delete it from your inbox right away.