The state machine is a powerful/clear metaphor to align your design. IMO hierarchical state machines (aka Harel statecharts) are a superior variant that makes behavior yet clearer and easier to design.
FWIW, I wrote one for a game engine, and I'm aware of several others. I can safely say they map cleanly enough that the names are more or less synonymous, at least as far as the underlying concept goes.
A behaviour tree is sort of a hierarchical state machine but one in which the states are not explicit. They fall out of how you arrange the nodes in the tree.
Actual hierarchical state machines are pretty common for driving a variety of things. Probably most commonly animation but also driving game state.
