If one take the position tha minds are not part of the "real world" then you end up defining "reality" by some random model.
It is just as absurd as saying that the only thing that exists is my own perception and you are "just" a ghost my mind is "thinking into existence".
It sounds less crazy only because we are used to reductionism being generally more useful, but what is the usefullness of concepts like reality and existence when defined to mean that we and our minds (the only thing we perceive) are not "real" or do not really "exist"?
Even before the appearance of life, the things group in sets spontaneously, due to the interplay between attractive and repulsive forces and the positive feedbacks that appear in certain conditions.
The world is not made of a homogeneous substance, but there are various kinds of groupings at various levels, nucleons and electrons group in atoms, atoms group in molecules, molecules group in pebbles, stars group in galaxies and so on.
By who?