The more information you manage to store into a dot product between input vectors and a weight vector the greater the average angle to the weight vector. Then an attack is to input a zero angle vector to get an extreme response. That suggests you should either limit the minimum angle to minimum seen at the last stage of training or in a similar way limit the output of the dot product output to the maximum magnitude seen.
The more information you manage to store into a dot product between input vectors and a weight vector the greater the average angle to the weight vector. Then an attack is to input a zero angle vector to get an extreme response. That suggests you should either limit the minimum angle to minimum seen at the last stage of training or in a similar way limit the output of the dot product output to the maximum magnitude seen.