The primary downside to a small TTL is cost. For many DNS providers you are allotted a number of requests per month that DNS will resolve. On particularly popular websites having a ton of DNS requests can cause the cost of DNS services to inflate significantly.