We believe that Julia is uniquely well-positioned to pioneer new approaches to dataflow orchestration that are currently dominated by monolithic frameworks. In this BoF, Julia's nascent Data Engineering community will swap experiences and identify opportunities to collaborate on open-source next-generation data engineering tools.
Join the discussion on the bof-voice channel in discord.
Julia has already succeeded by empowering many scientists and engineers to author their own high-performance compute kernels without the usual ergonomics/composability sacrifices that high-performance code often entails. However, actually leveraging these kernels within production contexts often requires packaging them into an automated service, usually within the context of wider automated pipelines. It is not surprising that in the past few years, many new capabilities and packages have emerged that facilitate this by enabling Julia to be executed atop Kubernetes, interop with tabular data sources/sinks via Apache Arrow, and integrate with other popular cloud-native technologies. This blossoming ecosystem within the wider Julia community demonstrates both the desire and opportunity for Julia's usage in production data engineering contexts.
Topics of discussion for this BoF include:
Our goal is two-fold: