On recent x86 processors, you would just use doubles in most cases. It turns out that they're on par with integers for speed, although I believe there were some differences on latency.
Integer operations have 2-3x the throughput and 4-5x better latency than floating-point operations on recent Intel CPUs.
According to the Intel optimization manual, there's a 5 cycle latency for doubles (4 for floats), but both fp multiply/add and alu ops have a one operation per cycle throughput.
I guess it comes down to how effectively you can schedule stuff.
Integer operations have 2-3x the throughput and 4-5x better latency than floating-point operations on recent Intel CPUs.