They made it in-house. The engine is called COBRA. Their other games (Planet Coaster, Jurassic Park) use it too. The engine has gone through several generations now and started in 1988 (by name at least).
Frontier is probably my favourite game studio right now. RCT3, Zoo Tycoon 1 and Thrillville on the ps2 where probably my favourite games as a kid and they absolutely smashed it out of the park with their Planet series getting everything that made the originals good right while making it so much more advanced.
Simcity 2013 and Zoo Tycoon 2013 were such disappointments so I'm glad there is a trend along with cities skylines to revive classics in a way that doesn't stomp on what made the originals great.