diff --git a/README.md b/README.md index 2995243..368b68e 100644 --- a/README.md +++ b/README.md @@ -1,44 +1,11 @@ # Workspaces in docker - -### Contents - -* [What is workspace In Docker](#what-is-workspace-in-docker) -* [Available workspaces](#available-workspaces) -* [Why workspace In Docker](#why-workspace-in-docker) -* [Use-cases](#use-cases) -* [How to make your own workspace](#how-to-make-your-own-workspace) -* [Publications](#publications) -* [Feedback and contribution](#feedback-an-contribution) - -## What is workspace In Docker - -***TLDR: Workspaces are toolsets packed together for specific tasks.*** - -Workspaces are a single docker image, running multiple processes. Workspaces include only browser-based -applications, making them suitable for both local and cloud deployment. - -Workspaces have browser-based VS-code, terminal, scheduler, and can be helpful to interact with solutions, deployed on K8s -that do not expose entrypoints beyound K8s cluster, such as databases, kafka. Workspace can be used as a convenient -terminal and IDE to develop annd schedule maintenance tasks. - -When running locally, workspaces help to isolate tasks, easily switch between projects. -They be moved between laptops, PCs, and cloud servers; can run in the cloud and be used from any device; have versions and -backups; can be used by several users and make collaboration easier. - -
- -
+ + -Essentially, workspace - is a toolset for a specific kind of work, packed in a docker image. It is a self-contained environment -that is ready to use as soon as you execute `docker run`. Dockerized workspaces do not require other tools to be set up on the host -machine, apart from docker itself. You can use such workspace locally or launch it securely on any cloud server. - -Workspace would typically include file browser, code editor, IDE, terminal, job scheduler, resource and process monitor, documentation -solution. Designed for specific IT projects, workspaces could also include frameworks, linters, profilers, testing, and auto-documentation tools, -tools to visualize and study data, report generators, task orchestrators, infrastructure visualization tools, interactive notebooks, -simulation UIs, custom dashboards, etc. +Workspace is a single docker image with a number of applications and tools for specific task, +such as Python development or Postgres management.+ +
+ +## Features: +1. [Browser-based VS-code](https://theia-ide.org/) with extensions: - [PostgreSQL Client for vscode](https://open-vsx.org/extension/cweijan/vscode-postgresql-client2) - [SQLTools PostgreSQL](https://open-vsx.org/extension/mtxr/sqltools-driver-pg) - - and additional color & icon extensions -2. Browser-based terminal (finally no need to ssh into a running docker container) -3. [Cronicle](https://github.com/jhuckaby/Cronicle). Browser-based jobs scheduler. Schedule backups, data extracts, maintenance, etc. -4. Ubuntu with many apps and tools installed and configured: +2. Browser-based terminal. Finally no need to ssh into a running docker container. +3. [Cronicle](https://github.com/jhuckaby/Cronicle). Browser-based jobs scheduler. Schedule backups, data extracts, maintenance scripts. +4. Ubuntu with many apps installed: - text editors - [Zsh](https://ohmyz.sh/) - cron - mc - and many more -5. [Rclone](https://rclone.org/) to save backups to S3, and pull then back to restore a database. -6. Up and running Postgres 14 database (to try things locally first) +5. [Rclone](https://rclone.org/). Save backups to S3. +6. Up and running Postgres 14 database. To try things locally first. 7. Postgres CLI tools: - [psql](https://www.postgresql.org/docs/current/app-psql.html). Standard PostgreSQL interactive terminal. - [pgcli](https://www.pgcli.com/). Command line interface for Postgres with auto-completion and syntax highlighting. - - [pspg](https://iboxshare.com/okbob/pspg). Visualisation of Postgres SQL queries data output. + - [pspg](https://iboxshare.com/okbob/pspg). Data visualizer. 8. Performance and load testing tools: - - [pgmetrics](https://pgmetrics.io/). A tool that can collect 350+ metrics from a running PostgreSQL server. - - [pgCenter](https://github.com/lesovsky/pgcenter). A command-line admin tool for observing and troubleshooting Postgres. - - [pgbench](https://www.postgresql.org/docs/devel/pgbench.html). Run a benchmark test on PostgreSQL. + - [pgmetrics](https://pgmetrics.io/). Collect 350+ metrics from a running PostgreSQL server. + - [pgCenter](https://github.com/lesovsky/pgcenter). Admin tool for observing and troubleshooting Postgres. + - [pgbench](https://www.postgresql.org/docs/devel/pgbench.html). Run benchmark tests on PostgreSQL. 9. Backups and restore tools: - [pg_dump](https://www.postgresql.org/docs/current/backup-dump.html) - [pg_dumpall](https://www.postgresql.org/docs/current/backup-dump.html) @@ -41,48 +39,62 @@ __Features:__ - [synth](https://github.com/getsynth/synth). A tool for generating realistic data. - [mock-data](https://github.com/faisaltheparttimecoder/mock-data). Generate mock data, tables and databases easily. 12. Database change management toolset: - - [sqitch](https://github.com/sqitchers/sqitch). A database change management application. + - [sqitch](https://github.com/sqitchers/sqitch). Database change management application. - [yuniql](https://github.com/rdagumampan/yuniql). Schema versioning and database migration engine. - [pgmigrate](https://github.com/yandex/pgmigrate). Database migration tool developed by Yandex. - - [migra](https://databaseci.com/docs/migra). Find differences in database schemas as easily as running a diff on two text files. + - [migra](https://databaseci.com/docs/migra). Find differences in database schemas easily. 13. [pg_sample](https://github.com/mla/pg_sample). Create database sample while maintaining referential integrity. 14. [Postgres explain visualizer](https://github.com/AlexTatiyants/pev). Understand Postgres execution plans. 15. [DBdesigner](https://github.com/akreienbring/dbdesigner). Draw ERD diagrams and use it to generate DDL code. -16. [Schemaspy](https://github.com/schemaspy/schemaspy). Generate extensive HTML report about Postgres database. +16. [Schemaspy](https://github.com/schemaspy/schemaspy). Generate extensive HTML database documentation that looks great. 17. [tbls](https://github.com/k1LoW/tbls). Generate documentation (essentially data catalog) from the database -18. [pg_flame](https://github.com/mgartner/pg_flame). A flamegraph generator for Postgres EXPLAIN ANALYZE output. +18. [pg_flame](https://github.com/mgartner/pg_flame). A flamegraph generator for Postgres explain output. 19. Programming languages. Develop/execute directly inside the workspace using browser-based IDE. - python - nodejs - java -20. [Filebrowser](https://github.com/filebrowser/filebrowser). Browse files, push and pull files to/from workspace (finally we don't need to mount folders for that). +20. [Filebrowser](https://github.com/filebrowser/filebrowser). Browse and download files (finally no need to mount folders for that). 21. [Static file server](https://github.com/vercel/serve). View generated HTML reports. 22. [Ungit](https://github.com/FredrikNoren/ungit). The easiest way to use git. 23. Browser-based process and resource utilization monitor 24. [Mkdocs](https://www.mkdocs.org/). Create documentation with Markdown. 25. Workspace own UI, from which you can open all applications and documentation. -__Try it:__ +__*Below you'll find examples and short tutorials how to use most of the tools included in the workspace.*__ + +> If you like this project, please support it by simply putting a Github star. +Share this library with friends on Twitter and everywhere else you can. + +### Try it out ``` docker run --name space-1 -d -p 8020-8035:8020-8035 alnoda/postgres-workspace ``` -Open [http://localhost:8020](http://localhost:8020) +Open [localhost:8020](http://localhost:8020) ### Why this image -So that you don't need to set up all those tools yourself, could have an isolated environment specifically configured +So that you could have an isolated environment specifically configured to manage and develop Postgres, get started fast, try things out on the internal Postgres first, import and export data, create backup dumps, create sample databases for development and testing, schedule all these tasks, enforce database change management in your team, analyse metrics annd uderstand performance, conduct load tests on the databases you use, look on visual execution plans, create complete documentations from the Postgres databases, develop scripts and apps with Python or Javascript. -_Below you'll find examples how to use many of the applications included in the workspace._ ++ +
+ +## Workspace UI +Workspace UI can be accessed on [localhost:8020](http://localhost:8020). It provides quick access to the IDE, terrminal and other apps. + ++ +
## Postgres The workspace includes Postgres 14 up and running. You can use it to experiment and try things before trying on the application database. Connection parameters: host - _localhost_, user - _abc_ (no password), database - _abc_. -You can load sample database to check out the workspace features and applications. +You can load sample database to check out the workspace features and applications. Load "usda" database ``` git clone https://github.com/morenoh149/postgresDBSamples /home/pg_dbs cd /home/pg_dbs/usda-r18-1.0 @@ -92,7 +104,7 @@ psql -f usda.sql usda pgcli usda SELECT * FROM data_src ``` -Load db "Chinook" +load Chinook database ``` cd /home/pg_dbs/chinook-1.4 createdb -E UTF8 chinook @@ -101,7 +113,7 @@ psql -f Chinook_PostgreSql_utf8.sql -d chinook pgcli chinook SELECT * FROM "Track" ``` -Load db Adventureworks: +load Adventureworks database ``` cd /home/pg_dbs/adventureworks/ unzip data.zip # this will unzip all csv to the current directory @@ -113,33 +125,28 @@ SELECT * FROM sales.salesorderdetail ``` ## VS-code & plugins -[Apache Theia](https://theia-ide.org/) is an browser-based version of the VS-code. It is part of the workspace, and -it is easy and convenient to develop directly in the workspace. Moreover, 2 database extensions are already installed, -and can be used to connect to local or remote PostgreSQL, explore data and create python/node/shell scripts to manage Postgres. +[Apache Theia](https://theia-ide.org/), a browser-based version of the VS-code, is part of the workspace. It has +extensions to explore Postgres- +
If you loaded sample databases to the local postgres, connect & explore data directly in the IDE- +
## Postgres CLI tools -Set-up and configured CLI tools that enable very convenient working with Postgres from the command line, -with auto-completion and data visualisation -- [psql](https://www.postgresql.org/docs/current/app-psql.html) -- [pgcli](https://www.pgcli.com/) -- [pspg](https://iboxshare.com/okbob/pspg) +Querying from command line should be awesome- +
- +
### psql @@ -174,10 +181,12 @@ You can disable pspg by deleting the (last) line _PAGER='pspg -s 4 --interactive ## Performance and load testing -Monitor live activity, collect and analyse metrics, perform load tests with -- [pgmetrics](https://pgmetrics.io/) -- [pgCenter](https://github.com/lesovsky/pgcenter) -- [pgbench](https://www.postgresql.org/docs/devel/pgbench.html) +Monitor live activity, collect and analyse metrics, perform load tests. + ++ +
+ ### pgmetrics [pgmetrics](https://pgmetrics.io/) is an open-source, zero-dependency, single-binary tool that can collect 350+ metrics from a running PostgreSQL server and display it in easy-to-read text format or export it as JSON and CSV for scripting. Example use: @@ -219,9 +228,8 @@ __Learn more:__ ## Backups and restore -- [pg_dump](https://www.postgresql.org/docs/current/backup-dump.html) -- [rclone](https://rclone.org/) -- backup scheduling +Create backup dumps with [pg_dump](https://www.postgresql.org/docs/current/backup-dump.html) or [pg_dumpall](https://www.postgresql.org/docs/14/app-pg-dumpall.html), +push to S3 with [rclone](https://rclone.org/) and schedule with [Cronicle](https://github.com/jhuckaby/Cronicle). ### pgdump The purpose of [pg_dump](https://www.postgresql.org/docs/current/backup-dump.html) is to generate a file with SQL commands that, when fed back to the server, will recreate the database in the same state as it was at the time of the dump. Example of use @@ -290,6 +298,10 @@ __Schedule with cron__: open workspace terminal, and execute `crontab -e`. Creat __Schedule with Cronicle__: open Cronicle WEB UI from the main Workspace page, or directly on [http://localhost:8023/](http://localhost:8023/) (user/pass - admin/admin) and create scheduled event (plugin - shell script). Provide commands to be executed. ++ +
+ ## Database sample Who hasn't faced a need to make a sample database for testing/development purposes? :-) ``` @@ -302,9 +314,8 @@ psql usda_sample < /home/project/sample_db.sql ``` ## Data import/export -- [pgclimb](https://github.com/lukasmartinelli/pgclimb) -- [pgfutter](https://github.com/lukasmartinelli/pgfutter) -- [pgloader](https://pgloader.readthedocs.io/en/latest/index.html) +Import data from various files, export to csv, json, html. Download with [Filebrowser](https://github.com/filebrowser/filebrowser), +upload to S3 with [Rclone](https://rclone.org/), schedule with [Cronicle](https://github.com/jhuckaby/Cronicle). ### pgclimb [pgclimb](https://github.com/lukasmartinelli/pgclimb) is a PostgreSQL utility to export data into different data formats with support for templates. Example of use: @@ -390,8 +401,7 @@ pgloader /home/project/test.load ## Data generators/mocking -- [synth](https://github.com/getsynth/synth) -- [mock-data](https://github.com/faisaltheparttimecoder/mock-data) +Generate realistic data for existing tables, create new mock tables or databases. ### [synth](https://github.com/getsynth/synth) Synth is a tool for generating realistic data using a declarative data model. Synth is database agnostic and can scale to millions of rows of data. @@ -444,10 +454,7 @@ __Learn more:__ ## Database change management -- [sqitch](https://github.com/sqitchers/sqitch) -- [yuniql](https://github.com/rdagumampan/yuniql) -- [pgmigrate](https://github.com/yandex/pgmigrate) -- [migra](https://databaseci.com/docs/migra) +Implement database change management in your team and drive higher software delivery and organizational performance. ### sqitch [sqitch](https://github.com/sqitchers/sqitch) - is a database change management application. To demonstrate (very briefly) how Sqitch works, create a new database called 'flipr_test' in the local Postgres: @@ -530,28 +537,40 @@ Migra will produce a set of SQL commands to make database "abc" have the same DD ## Visualization tools -- [pev](https://github.com/AlexTatiyants/pev) -- [pg_flame](https://github.com/mgartner/pg_flame) -- [schemaspy](https://github.com/schemaspy/schemaspy) -- [tbls](https://github.com/k1LoW/tbls) -- [dbdesigner](https://github.com/akreienbring/dbdesigner) -### [schemaspy](https://github.com/schemaspy/schemaspy) -Generate complete database documentation that has absolutely amazing look and feel generated content. +### dbdesigner +[dbdesigner](https://github.com/akreienbring/dbdesigner) a visual tool to create entity relationship diagrams and +generate Postgres DDL code. + ++ +
+ +### schemaspy +[schemaspy](https://github.com/schemaspy/schemaspy) generates complete database documentation that looks great. ``` schemaspy -h localhost -p 5432 -d usda -u abc -o /home/static-server/usda-schemaspy ``` And open static file server on [http://localhost:8022/](http://localhost:8022/) folder usda-schemaspy +- +
+ +### Postgres explain visualizer +[Pev](https://github.com/AlexTatiyants/pev) helps to understand Postgres execution plans + ++ +
+ ### tbls [tbls](https://github.com/k1LoW/tbls) - a tool for document a database, written in Go. -For example, generate Markdown docs as +For example, generate Markdown docs ``` tbls doc "postgres://abc@localhost:5432/usda?sslmode=disable" /home/static-server/usda-tbls ``` -and push to Github. +Now you can push these docks push to Github. ### pg_flame [pg_flame](https://github.com/mgartner/pg_flame) - a flamegraph generator for Postgres EXPLAIN ANALYZE output. ``` @@ -565,6 +584,10 @@ psql usda -qAtc 'SELECT * | pg_flame > /home/static-server/flamegraph.html ``` ++ +
+ ## Other useful tools ### pg_insights Convenient SQL for monitoring Postgres database health. Clone repo and use it to analyse Postgres databases diff --git a/workspaces/postgres-workspace/img/Cronicle2.png b/workspaces/postgres-workspace/img/Cronicle2.png new file mode 100644 index 0000000..818de6a Binary files /dev/null and b/workspaces/postgres-workspace/img/Cronicle2.png differ diff --git a/workspaces/postgres-workspace/img/Postgres-wid-UI.png b/workspaces/postgres-workspace/img/Postgres-wid-UI.png new file mode 100644 index 0000000..851f533 Binary files /dev/null and b/workspaces/postgres-workspace/img/Postgres-wid-UI.png differ diff --git a/workspaces/postgres-workspace/img/dbdesigner.png b/workspaces/postgres-workspace/img/dbdesigner.png new file mode 100644 index 0000000..d250a73 Binary files /dev/null and b/workspaces/postgres-workspace/img/dbdesigner.png differ diff --git a/workspaces/postgres-workspace/img/pev.gif b/workspaces/postgres-workspace/img/pev.gif new file mode 100644 index 0000000..4cb2228 Binary files /dev/null and b/workspaces/postgres-workspace/img/pev.gif differ diff --git a/workspaces/postgres-workspace/img/pgflame.png b/workspaces/postgres-workspace/img/pgflame.png new file mode 100644 index 0000000..e63f6fc Binary files /dev/null and b/workspaces/postgres-workspace/img/pgflame.png differ diff --git a/workspaces/postgres-workspace/img/pgmetrics.png b/workspaces/postgres-workspace/img/pgmetrics.png new file mode 100644 index 0000000..69ed49b Binary files /dev/null and b/workspaces/postgres-workspace/img/pgmetrics.png differ diff --git a/workspaces/postgres-workspace/img/postgres-wid-collage.png b/workspaces/postgres-workspace/img/postgres-wid-collage.png new file mode 100644 index 0000000..5a16d90 Binary files /dev/null and b/workspaces/postgres-workspace/img/postgres-wid-collage.png differ diff --git a/workspaces/postgres-workspace/img/postgres-wid-demo.gif b/workspaces/postgres-workspace/img/postgres-wid-demo.gif new file mode 100644 index 0000000..f6f3ebe Binary files /dev/null and b/workspaces/postgres-workspace/img/postgres-wid-demo.gif differ diff --git a/workspaces/postgres-workspace/img/schemaspy.gif b/workspaces/postgres-workspace/img/schemaspy.gif new file mode 100644 index 0000000..38222dd Binary files /dev/null and b/workspaces/postgres-workspace/img/schemaspy.gif differ