alnoda-workspaces/workspaces/notebook-old-workspace/README.md

Data Workstation

docker build -t data-workstation-base:3.8 --build-arg docker_registry=rg.fr-par.scw.cloud/dgym .
docker run -p 3000:3000 -p 8001:8000 -p 3012:3012 -p 8092:8092 -p 8448:8448 -p 9992:9992 -p 8085:8085 -p 8086:8086 -p 8082:8082 -p 8084:8084  data-workstation-base:3.8
docker run -p 3000:3000 -p 8001:8000 -p 3012:3012 -p 8092:8092 -p 8448:8448 -p 9992:9992 -p 8085:8085 -p 8086:8086 -p 8082:8082 -p 8084:8084 rg.fr-par.scw.cloud/dgym/python-workstation:3.8

Luigi

Useful links:

• Luigi Github Repo
• A Tutorial on Luigi, the Spotify’s Pipeline
• Create your first ETL in Luigi
• Luigi on PyPi

DBT

Useful links:

• DBT main page
• dbt(Data Build Tool) Tutorial
• DBT on PyPi
• Analytics Engineering with dbt and PostgreSQL

dbt init simple_dbt_project --adapter postgres

Great expectations

Useful links:

• Great Expectations main page
• Great Expectations documentation
• Great Expectations on PyPi
• Understanding Great Expectations and How to Use It
• Know Your Data Pipelines with Great Expectations

https://www.startdataengineering.com/post/ensuring-data-quality-with-great-expectations/ https://medium.com/hashmapinc/understanding-great-expectations-and-how-to-use-it-7754c78962f4 https://docs.greatexpectations.io/en/stable/guides/tutorials/how_to_create_expectations.html

Papermill

• Papermill Report GitHub
• Automated Report Generation with Papermill: Part 1
• [Automated Report Generation with Papermill: Part 2]https://pbpython.com/papermil-rclone-report-2.html)

Prefect

https://docs.prefect.io/core/getting_started/installation.html

ADVANCED DATA

https://www.datacouncil.ai/blog/25-hot-new-data-tools-and-what-they-dont-do

PREFECT

RUN pip install prefect==0.14.20

[program:prefect]
directory=/home/
command=/bin/sh -c " prefect backend server; prefect server start --ui-port 8095; prefect agent local start "
stderr_logfile = /var/log/prefect-stderr.log
stdout_logfile = /var/log/prefect-stdout.log
logfile_maxbytes = 1024

-p 8095:8095