I made a discovery at work this week. We have five separate instances of Airflow. One for each Development environment (of which we have four) and one for our Datalake environment.
Our IT department set them up for us in the middle of last year. Due to the way life and work has gone, we have only just now got around to finally utilizing our last environment, the one for the Datalake.
Imagine my surprise when the changes I made to the git repository aren’t being automatically picked up by the Datalake instance of Airflow.
Now, you would think that I could get this resolved by simply contacting IT and asking them to take a look into this. However, like any good IT department they are understaffed and overworked. Not only that but the person who set it all up for us is now working on different projects and has no capacity to assist us.
What I should have done is when is test EVERY environment. Even though I didn’t have anything ready for the Datalake, I should have at least done a simple test that would have proven to me it worked.
Now, I don’t know when it will be fixed.