#AIRFLOW DOCKER FILE INSTALL#
I am actually quite confused about this specific warning message referring to google_cloud_os_login because the build step that is hanging is the line I shared starting with RUN set -ex, which doesn't look to have any google cloud installations? We install some google cloud stuff via requirements.txt (-core, -datastore), but the lines to COPY and RUN pip install on requirements.txt are much lower in our dockerfile (as indicated by the. & apt-get purge -auto-remove -yqq $buildDeps \Īnd here is our requirements.txt google-cloud-core=1.4.1 So, open your terminal and run: mkdir airflow-docker cd airflow-docker I created a folder called airflow-docker. & update-locale LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 \ It is convenient to create an Airflow directory where you will have your folders like dags etc. & apt-get install -yqq -no-install-recommends \ Here is the line in our Dockerfile that is taking the hour+ RUN set -ex \ You might need to provide the dependency resolver with stricter constraints => => # INFO: This is taking longer than usual. We're seeing a lot of warnings that look like this during build: => => # Downloading google_cloud_os_login-2.3.1-py2.p圓-none-any.whl (42 kB) With that said we haven't had to rebuild our image in a few months, so not sure when the issue came to be. Each step in your pipeline is a standalone file containing modular code thats reusable and testable with data validations.
It used to take ~5 minutes for our Airflow deployment's docker image to build, and all of a sudden it is taking over an hour.
0 kommentar(er)
