![]() ![]() ![]() So I’m going to use that operator, and then I want to send the data somewhere else. And basically, the community develops and contributes to these operators so that the users, in the end, are basically saying the task I want to do is pull data from here. And then there’s an operator that will send the data to an SQL server or something like that. “So, for example, there’s an operator to write data to. “You just chain them together in different ways,” he said. Each operator does one specific task, Ferruzzi said. Operators are like generic building blocks. So companies like AWS, and Google and Databricks, are all contributing these operators, which really wrap their underlying SDK.” ‘That Blueprint Exists for Everyone’ And two, we have this operator ecosystem. ![]() “The beautiful thing about Airflow, that has made it so popular is that it’s so easy,” Oliveira said. Raphaël Vandon, a senior software engineer at AWS, is an Apache Airflow contributor working on performance improvements for Airflow and leveraging async capabilities in AWS Operators, the part of Airflow that allows for seamless interactions with AWS. It allows Airflow to be a more pluggable architecture, which makes it easier for users to build and write their own Airflow Executors. A recent project included writing and implementing AIP-51 (Airflow Improvement Proposal), which modifies and updates the Executor interface in Airflow. He spends much time reviewing, approving and merging pull requests. Niko Oliveira, a senior software development engineer at AWS, is a committer/maintainer for Apache Airflow. The API will allow for more granular metrics and better visibility into Airflow environments. In an On the Road episode of Makers recorded at the Linux Foundation’s Open Source Summit North America, our guests, who all work with the AWS Managed Service for Airflow team, reflected on the work on Apache Airflow to improve the overall experience:ĭennis Ferruzzi, a software developer at AWS, is an Airflow contributor working on project API-49, which will update Airflow’s logging and metrics backend to the OpenTelemetry standard. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |