Yeah, the MNIST model seems to have issues. I can get it to make mistakes for all line widths and most digits. Maybe the issue with small digits is that there is no cropping happening and the model was trained on MNIST where all the digits are uniform size.

Actually this is probably a good demo for a ML visualization tool as we were pretty easily able to find these issues. If cropping is the issue, this is also something that a test set might not catch.