Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Ok, so question (because I really like the DAG approach in principle but don't have enough experience to have had my fingers burned yet):

The way you use Airflow, what advantage does it have over crontab? Or to put it another way, once you remove the pipeline logic, what's left?



Airflow provides straightforward parallelism and error handling of dependent subtasks. Cron really doesn’t.

With cron you have to be more thoughtful about failover especially when convincing others to write failure safe cron in invoked code. With airflow you shouldn’t be running code locally so you can have a mini framework for failure handling.

Cron doesn’t natively provide singleton locking so if the system bogs down you can end up running N of the same jobs at the same time which slows things down further. Airflow isn’t immune to this by default but it’s easier to setup centralized libraries that everything uses so more junior people avoid this when writing quick one off jobs.


Observability is a huge upside.


Backfilling is also very useful


Thanks to both comments.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: