In my opinion you are free to do as you please - i.e. enduring his attacks every time he sees you - as long as you are the only person attacked and nobody else is involved.
As soon as other people ( your child, husband, neighbours, postman etc.) are endangered by your overaggressive animal it...