For me, it seems to work if I view the page with the reply, regardless of whether I get there by actually clicking the alert or whether I get there some other way (click the thread in a list of recent threads, refresh the page I had open, get there with a bookmark, etc.)
I do have to be logged in while I do it. Clicking the link in an email, and going to the page without being logged in to BYC, will not do it.
But yes, it seems I have to actually load the page with the reply before the system will send me any other alerts about replies in that thread.