This is exactly what we do, but with Spark instead. We develop the functions loc... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		csunbird on June 19, 2024 \| parent \| context \| favorite \| on: Ask HN: Why do message queue-based architectures s... This is exactly what we do, but with Spark instead. We develop the functions locally in a package and call necessary functions for the job notebooks, and the job notebooks are very minimalistic because of this

appplication on June 19, 2024 [–]

Spark-via-Airflow is also the context we use this, glad of see the pattern also works for you.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact