I think the "diffuse harm" in the case of smoking weed in private is not talking about the act of smoking and who is or is not affected by the chemicals involved, etc.
It is rather looking at the overall picture of "What are the consequences of allowing weed to be bought/sold/grown. Is there harm in it?" The whole infrastructure, not just the end user.
Sure, some people look at it that way, some people look at it other ways. I just see no link between people thinking that a crime causes diffuse harm, and people thinking that a crime is victimless.
It is rather looking at the overall picture of "What are the consequences of allowing weed to be bought/sold/grown. Is there harm in it?" The whole infrastructure, not just the end user.