Its quite possible that Robert LeFevre first said it. I first came upon the phrase in one of L. Neil Smiths books. I think it was the second book in the Probability Broach series, the one where he starts each chapter with a quotation, and this was one of them (attributed to a fictional character, according to memory.)
Apparently, this one is usually attributed to Robert LeFevre. Do you have any sources attributing it to L. Neil Smith?