If you send a message to a node, the operations it perform is not atomic, so it can either
* Perform the action, then acknowledge the message.
or
* Acknowledge the message and then perform the action.
(assuming delivery of messages is reliable)
In either case, if they crash after the first part, it will either be carried out twice, or not at all. When/if communication is established again, you need to ensure that the node is A. able to gurantee that it knows if an action was carried out or not, B. able to undo an action if the coordinator decided to send it elsewhere.
This is not always possible, and you cannot have a message system that gurantee exactly-once, without the recieving system supporting it.
* Perform the action, then acknowledge the message.
or
* Acknowledge the message and then perform the action.
(assuming delivery of messages is reliable)
In either case, if they crash after the first part, it will either be carried out twice, or not at all. When/if communication is established again, you need to ensure that the node is A. able to gurantee that it knows if an action was carried out or not, B. able to undo an action if the coordinator decided to send it elsewhere.
This is not always possible, and you cannot have a message system that gurantee exactly-once, without the recieving system supporting it.