To delete all custom avoidances, select Select All > Delete.

To delete one custom avoidance, select the avoidance and select Delete.