L. D. Nicolas May

Published in

·Pinned

Break Up a Big Airflow DAG into Multiple Files

Modularize Chunks of Your Large Airflow DAG for Easy Reüse and Maintainability — I was working on a Airflow DAG file that was growing into the hundreds of lines. Making changes required bouncing back-and-forth around the file, taking notes on a scratchpad to get everything right. Once I got to the point of opening multiple views of the DAG file in an IDE…

Airflow

4 min read

Airflow

4 min read

Feb 16

Dagster + Airbyte + dbt: How Software-Defined Assets Change the Way We Orchestrate

154

Tamas Foldi

Thanks for this! It helped me align asset keys between Fivetran and dbt.

1 min read

Published in

Towards Data Science

·Apr 13, 2021

Introspecting Databases with Airflow and SQLAlchemy

A Tutorial For Iterating Over Automatically Introspected Database Objects — For a recent data migration project that utilized Airflow, I needed to connect to a database and automatically introspect its schemas and tables. One database that needed to be introspected had hundreds of schemas. …

Airflow

6 min read

Airflow

6 min read

Dec 4, 2020

Getting SSL certificate behind a proxy working with Python requests library from Docker

Context At work we use a REDCap instance supported by one of our university research institutes. We often write code against the REDCap API to ETL and/or analyze clinical research data for various purposes. …

Python

4 min read

Python

4 min read

Jul 20, 2020

Deploying a free Python Dash Open Source app from a Docker container with Gunicorn

Do you need to deploy a Dash app without plunking down the cash for Dash Enterprise? Here’s the straightforward steps I used to deploy a simple test app in a Docker container. The container runs on a RedHat virtual machine provisioned by my employer’s sysadmins. 1. Create App Use https://dash.plotly.com/layout…

Dash

3 min read

Dash

3 min read