Consider a category of item you care about. Would you accept it if it were sold to you as being in new condition, but arrived with substantial cosmetic damage? A dining table with a gash down the middle of it still holds food just fine. A white door spattered with red paint still opens and closes and keeps the wind out. Carpeting stained with wine still keeps your feet warm.
Often, in any case, the damage is more than cosmetic. A paperback book that has been badly creased is more difficult to read than one that's in good condition.
Caring about the condition of books is not a strange idiosyncrasy. There's a clear stratification in the price of the book market as you descend from new to very good to acceptable condition - because buyers generally care.
For people who buy physical books, there's a lot of pride of ownership and enjoying the look and feel of the thing, as much as actually reading it.
I'll never forget lending one of the few books I bothered to keep physically to my wife who immediately trying to crack the spine on it to have it open the way she was used to.
The majority of the books I read are digital, though, and have been since PDAs were invented.
How do you lose a page from inside of a box/envelope?