It is always listening just like your dog is always listening. If you are not talking to it and you are not saying its name you are being (mostly) ignored.
Most of the time, it is not paying attention - the chip that is processing the sound is looking for the ONE word that will activate it. That passive audio processing is happening locally on a chip that is dedicated to the task. Once activated - the expensive processing happens and the sound gets processed, converted to text, sent to the cloud.
I understand this, and I don't happen to agree with the people who feel this type of technology should not be embraced, but, to be fair, the chip is controlled by software that is constantly connected to the cloud and updating over the air. It would take very little to update the software to disable the on-chip keyword detection and just record everything. That update could easily be done without your knowledge and in a way that would be almost undetectable since the software stack doesn't appear to be open and the server-stack is in the cloud and out of your control.
Most of the time, it is not paying attention - the chip that is processing the sound is looking for the ONE word that will activate it. That passive audio processing is happening locally on a chip that is dedicated to the task. Once activated - the expensive processing happens and the sound gets processed, converted to text, sent to the cloud.