CodeMirror/alnoda-workspaces
1
0
Fork 0
mirror of https://github.com/bluxmit/alnoda-workspaces.git synced 2024-05-06 23:22:19 +12:00
Code Issues Projects Releases Wiki Activity

Postgres and main README updated

Browse source
This commit is contained in:
bluxmit 2022-02-01 14:52:18 +00:00
parent bd1d8499a2
commit 6bfc14d196
12 changed files with 203 additions and 237 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

211
README.md
View file

@ -1,44 +1,11 @@
# Workspaces in docker
### Contents
* [What is workspace In Docker](#what-is-workspace-in-docker)
* [Available workspaces](#available-workspaces)
* [Why workspace In Docker](#why-workspace-in-docker)
* [Use-cases](#use-cases)
* [How to make your own workspace](#how-to-make-your-own-workspace)
* [Publications](#publications)
* [Feedback and contribution](#feedback-an-contribution)
## What is workspace In Docker
***TLDR: Workspaces are toolsets packed together for specific tasks.***
Workspaces are a single docker image, running multiple processes. Workspaces include only browser-based
applications, making them suitable for both local and cloud deployment.
Workspaces have browser-based VS-code, terminal, scheduler, and can be helpful to interact with solutions, deployed on K8s
that do not expose entrypoints beyound K8s cluster, such as databases, kafka. Workspace can be used as a convenient
terminal and IDE to develop annd schedule maintenance tasks.
When running locally, workspaces help to isolate tasks, easily switch between projects.
They be moved between laptops, PCs, and cloud servers; can run in the cloud and be used from any device; have versions and
backups; can be used by several users and make collaboration easier.
<p align="center">
<img src="./img/wid-advantages.svg" alt="Htop">
</p>
<img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/img/Alnoda-white.svg" alt="Alnoda logo" width="250">
</p>
Essentially, workspace - is a toolset for a specific kind of work, packed in a docker image. It is a self-contained environment
that is ready to use as soon as you execute `docker run`. Dockerized workspaces do not require other tools to be set up on the host
machine, apart from docker itself. You can use such workspace locally or launch it securely on any cloud server.
Workspace would typically include file browser, code editor, IDE, terminal, job scheduler, resource and process monitor, documentation
solution. Designed for specific IT projects, workspaces could also include frameworks, linters, profilers, testing, and auto-documentation tools,
tools to visualize and study data, report generators, task orchestrators, infrastructure visualization tools, interactive notebooks,
simulation UIs, custom dashboards, etc.
Workspace is a single docker image with a number of applications and tools for specific task,
such as Python development or Postgres management.
<div align="center" style="font-style: italic;">
Demo: Ansible-Terraform workspace
@ -48,18 +15,16 @@ simulation UIs, custom dashboards, etc.
<img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/ansible-terraform-workspace/img/ansible-terraform-wid.gif" alt="Htop" width="900">
</p>
## Available workspaces
> If you like this project, please support it by simply putting a Github star.
Share this library with friends on Twitter and everywhere else you can.
### Contents
### [`Ubuntu-workspace`](./workspaces/ubuntu-workspace/README.md)
Docker as a light-weight Virtual Machine. It
provides isolation of environments, but uses less resources than VMs. Allows to start multiple processes inside the
same docker container, has docker-in-docker, Python and Node.js, and a collection of common applications such as text editors,
git, supervisord, z-shell etc. Ubuntu-workspace with browser-based terminal can be used on both local and remote server with the same experience.
When it runs on the remote server, the access can be restricted with a password, and secured with TLS encryption.
* [List of workspaces](#list-of-workspaces)
* [Why workspace In Docker](#why-workspace-in-docker)
* [Contribution](#contribution)
```
docker run --name space-1 -d -p 8020-8030:8020-8030 alnoda/ubuntu-workspace
```
## List of workspaces
### [`Workspace-in-docker`](./workspaces/workspace-in-docker/README.md)
General-purpose dockerized workspace - an environment fully isolated inside a docker container.
@ -102,6 +67,14 @@ the Base-workspace and Ubuntu-workspace.
docker run --name space-1 -d -p 8020-8035:8020-8035 -p 9000:9000 alnoda/ansible-terraform-workspace
```
### [`Postgres-Workspace`](./workspaces/postgres-workspace/README.md)
Great number of tools to manage PostgreSQL database, develop and schedule database scripts and applications.
All in a single docker image.
```
docker run --name space-1 -d -p 8020-8035:8020-8035 alnoda/postgres-workspace
```
### [`Redis-Workspace`](./workspaces/redis-workspace/README.md)
Docker image with Redis, and many other tools that make development with Redis much more convenient. Substitute
in your development docker-compose.yaml file standard redis image with this one, add port-mapping -p 8020-8035:8020-8035.
@ -134,133 +107,33 @@ of documentation, tutorials, and training. Has all the features of the Base-work
docker run --name space-1 -d -p 8020-8035:8020-8035 alnoda/mkdocs-magicspace
```
### [`Ubuntu-workspace`](./workspaces/ubuntu-workspace/README.md)
Docker as a "light-weight Virtual Machine". It
provides isolation of environments, but uses less resources than VMs. Allows to start multiple processes inside the
same docker container, has docker-in-docker, Python and Node.js, and a collection of common applications such as text editors,
git, supervisord, z-shell etc. Ubuntu-workspace with browser-based terminal can be used on both local and remote server with the same experience.
When it runs on the remote server, the access can be restricted with a password, and secured with TLS encryption.
```
docker run --name space-1 -d -p 8020-8030:8020-8030 alnoda/ubuntu-workspace
```
## Why workspace In Docker
Compared to the local environment, where you can install any application with a nice native interface, dockerized workspaces are much
less flexible. They contain only terminal-based or browser-based tools. Also, workspaces in docker have an overhead of running inside
the docker container, which can result in some performance implications.
When used locally, workspaces help to isolate development environments and easily switch between projects. Workspaces can be moved between laptops, PCs,
backed up and shared with friends.
At the same time, dockerized workspaces have several very strong advantages, that local workspaces don't.
Some workspace benefits come from docker itself:
When launched on remote server, workspaces can be used to schedule scripts and tasks, such as data ingestion.
- **Containerization.** Everything in the workspace is packed in a docker container.
- **Isolation.** Changes in one environment cannot affect another environment.
- **Easy-to-switch.** Start and stop workspaces with just one action.
- **Shareability.** Share workspace as a whole (code, files, secrets, configuration, data, etc.) with your peers.
- **Platform-agnostic.** Run on any system where docker can run.
- **Movability.** You can move the entire workspaces between laptops, PCs, and servers.
- **Collaboration.** Launch workspace in cloud, and use together with your peers.
- **Versions and backups.** Commit workspaces to images, or save to files. Back up the entire workspace before making changes, keep versions.
Other features are coming from the concept of the workspace itself, and only satisfied if the workspace is done properly:
- **Cloud-ready.** You work with workspace through browser-based or terminal-based interfaces. Hence workspace is ready to be launched on the remote server,
container orchestration platform or Kubernetes, and be used from any of your devices.
- **Feature-reach.** Workspace is not merely an IDE inside a docker container. It includes the whole toolkit necessary to solve the problem.
- **Readiness.** Tooling that is needed to tackle a specific problem is pre-selected, set up, and configured with sane defaults.
- **Easy to use.** Workspace is convenient, it has UI, and tooling is well-documented.
- **Independence.** Workspace is fully open-source. It does not use any proprietary technology, it does not depend on any cloud.
- **Opinionated.** Workspace is an opinionated system of chosen tools and practices, that according to its creator, are best suited
for the specific purpose.
## Use-cases
There are many problems that workspaces in docker can solve. Dockerized workspaces can be a good choice in situations when:
**You need to get started fast.** The creation of a new workspace is a tedious task. You need to install a programming language,
set up and configure multiple tools (this can take days). You'd have to research, chose, and learn how to do such things as linting,
testing, pre-commit hooks, debugging, and profiling (can take months). This workspace has those tools selected and set up.
Documentation on how to use them is provided, with an example.
**You are working on multiple projects.** If you are working on a serious IT project, the chances are that you are dealing with multiple
Git branches, various configuration files, and environmental variables. There is a good chance Git branches have different dependencies
and versions, so you also use virtual environments. It is likely that you also have different development and production environments,
with the need to change the profile of a cloud provider, kubeconfig, use another ssh-key. Switching between such environments even of
a single project requires several actions, which is terrible. Things become even less convenient when we are working with multiple IT projects.
Managing such projects, and switching between them becomes a real overhead. When you are working in a dockerized workspace, the switch
is just one action of stopping one container and starting another!
**You need to move workspaces.** Let's assume you mostly work on a Linux/macOS laptop, but you have a powerful gaming PC at home,
and might want to use it for builds, simulations, training ML models, etc. You might realize the algorithm you are using needs extra
high resources or needs to run for many hours to finish, and a very powerful cloud server should be used. Dockerized workspace
can be moved as-is from one machine to another within minutes, no matter if it is a laptop, PC, or virtual cloud server.
**You need to put your work to production really fast.** You have developed a piece of code that must be running all the time. Or jobs that should run on schedule.
You cannot keep your laptop running all the time. You need to quickly move to the cloud. If you don't have time or resources to build production-ready
docker images, create CI/CD, configure servers, you can simply move your workspace to the cloud as-is. Depending on the situation, such "rapid" deployment might
be well-justified for proofs-of-concept, experiments, scheduling supporting tasks, etc.
**Yo need an environment to isolate an experiment.** You want to try a new framework, library, package, or simply update your dependencies to the latest version. Try to do it in
a self-contained isolated environment first.
**You require backups and versions for the entire workspace.** Your work may go well beyond the development-debugging-testing-deployment cycle.
For example, if you are a data analyst, data scientist, researcher you make experiments, simulations, train ML models. Such work often involves
frequent changes to your environment (updating packages, install new applications, change datasets, etc.). To reproduce analytics,
experiment, or ML model training you need to have an exact version of your environment at a specific point in time. Dockerized workspaces in docker
solve this problem very well! At any moment you can backup and save your entire workspace as an image, and launch it
as a new workspace whenever and wherever you want.
**Collaboration made simple.** You need a workspace that can be shared with your peers or colleagues. Launch it on the cloud server
and use together.
**You need to deliver results.** If you are a freelancer or contractor, you might want to send your client the whole workspace
you worked in. This will help your client to reproduce results without bothering you with questions after the contract is closed.
**You want to improve team's productivity.** The workspace for a serious project includes a lot of tooling, standards and conventions,
secrets and configurations, ssh keys, environmental variables, VPN, and much more. Your team probably would like to do it once.
When everyone needs to make this work, it is such a waste of time! An how much easier it is to onboard new team members if they can use
an environment where everything is ready to get started right away.
**You need to onboard freelancer/contractor asap.** The time freelancer spends to set up the environment - is the time you are paying for.
Use a workspace, customized for the tasks you typically hire an external workforce for. Let freelancers be able to start real work
as soon as possible. Save time and money.
**You are buildding internal development platform.** An Internal Developer Platform, or IDP, is a self-service layer that allows developers
to interact independently with their organization's delivery setup, enabling them to self-serve environments, deployments, databases,
logs, and anything else they need to run their applications. Customize workspaces for the specifics of your IDP. Support and maintain
workspaces for your company, and let engineers do the real work rather than bothering how to configure and use your company's IDP.
**You want to help others.** You know how to tackle a complex project. You know how to use different tooling, practices, and standards,
to be successful and efficient in the specific area or tasks. You spent months setting it up, you have tried different versions and package combinations.
You have made scripts to automate work. Everything seems easy for you now, and you are ready to write a blog post to help others.
You can go one step further! Fellow engineers, analysts, and scientists are months behind you. Create a workspace with everything set up and ready.
Together with your publication, it will greatly help people who struggle with the same problem.
## How to make your own workspace
This repo was created with the idea in mind - to set up basics for building custom workspaces. The workspaces, shared in this project
are build based on each other. Some of the workspaces are highly customized, have a narrow scope, and focus on a specific task.
Other workspaces are more general and intended to be used for further customization.
The most general workspace - is [`Ubuntu-workspace`](./workspaces/ubuntu-workspace/README.md). It sets the basics for using docker
in order to run multiple processes, adds cron, zsh, and other applications that will be used in most workspaces.
Other general workspaces are the [`Base-Workspace`](./workspaces/base-workspace/README.md), which adds several browser-based applications,
like task scheduler, file browser, documentation framework and workspace own UI. [`Workspace-in-docker`](./workspaces/workspace-in-docker/README.md)
adds [Eclipse Theia](https://theia-ide.org/) to the Base-Workspace. Theia - is an open-source browser-based VS-Code version, making
Workspace-in-docker to be a general base workspace, to be used in order to create workspaces for specific tasks (like Python workspace, or
Ansible workspace).
[`Codeserver-Workspace`](./workspaces/codeserver-workspace/README.md) - is an alternative to the Workspace-in-docker. It is also a general
base workspace to build upon, but it has a different VS-Code version - [Codeserver](https://github.com/cdr/code-server).
## Publications
- [The way I use Docker as interactive environment for tries and experiments](https://medium.com/@bluxmit/the-way-i-use-docker-as-interactive-environment-for-tries-and-experiments-52ac06c0ec69)
- [Docker as a lightweight VM - docker image that you can use as VM substitute](https://medium.com/@bluxmit/docker-as-a-lightweight-vm-docker-image-that-you-can-use-as-vm-substitute-164032e4ed0b)
- [The better development experience with Kafka](https://medium.com/@bluxmit/the-better-development-experience-with-kafka-54f15705a8ee)
- [Improve development experience with Redis](https://medium.com/@bluxmit/improve-development-experience-with-redis-96d14ef473ae)
## Feedback and contribution
Workspaces in docker helped me to manage multiple projects in different clouds and k8s clusters. They made my work so much easier,
and I hope they can be useful for other people too.
I will highly appreciate any suggestions, ideas for improvements, contribution or any other form of collaboration.
Send me an email bluxmit@gmail.com
When deployed on K8S, workspaces can be used to manage databases, create backups.
## Contribution
Simply create a pull request. It will be welcome.
If you feel you need to discuss first, create an issue.

70
img/Alnoda-white.svg Normal file
View file

@ -0,0 +1,70 @@
<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="1280" height="1024" viewBox="0 0 1280 1024" xml:space="preserve">
<desc>Created with Fabric.js 3.6.3</desc>
<defs>
</defs>
<g transform="matrix(2,0,0,2,640,512)" id="background-logo" >
</g>
<g transform="matrix(2,0,0,2,640,416.3)" id="logo-logo" >
<g transform="matrix(1.8,0,0,2.3,-1.2,-40.1)" style="" paint-order="stroke" >
<g transform="matrix(0.2,0,0,-0.2,0,-61.6)" >
<path style="fill: rgb(64, 50, 44); fill-rule: nonzero; stroke: none; stroke-width: 1; stroke-linecap: butt; stroke-linejoin: miter; stroke-dasharray: none; stroke-dashoffset: 0; stroke-miterlimit: 4; opacity: 1" paint-order="stroke" transform="translate(-1625,-1745)" d="M 2326.7 1581.5 c -466.3 107.3 -950.8 107.3 -1417.1 0 c -16.2 -3.7 -26.6 -18.2 -23.3 -32.1 c 3.4 -13.9 19 -22.1 34.9 -18.4 c 458.6 105.5 935.2 105.5 1393.9 0 c 15.9 -3.7 31.5 4.5 34.9 18.4 c 3.4 13.9 -7 28.4 -23.3 32.1" stroke-linecap="round" />
</g>
<g transform="matrix(0.2,0,0,-0.2,-95.7,12.6)" >
<path style="fill: rgb(64, 50, 44); fill-rule: nonzero; stroke: none; stroke-width: 1; stroke-linecap: butt; stroke-linejoin: miter; stroke-dasharray: none; stroke-dashoffset: 0; stroke-miterlimit: 4; opacity: 1" paint-order="stroke" transform="translate(-1210,-1423.3)" d="M 1112.4 1410.1 c 0 52.1 44.1 94.1 97 90.6 c 48.2 -3.2 84.6 -45.5 84.6 -93.8 V 1021.9 c 0 -14.3 11.6 -25.9 25.9 -25.9 c 14.3 0 25.9 11.6 25.9 25.9 v 384 c 0 76.7 -58.8 142.8 -135.5 146.6 c -81.9 4 -149.8 -61.4 -149.8 -142.5 V 1021.9 c 0 -14.3 11.6 -25.9 25.9 -25.9 c 14.3 0 25.9 11.6 25.9 25.9 v 388.2" stroke-linecap="round" />
</g>
<g transform="matrix(0.2,0,0,-0.2,-111.1,21.9)" >
<path style="fill: rgb(81, 173, 229); fill-rule: nonzero; stroke: none; stroke-width: 1; stroke-linecap: butt; stroke-linejoin: miter; stroke-dasharray: none; stroke-dashoffset: 0; stroke-miterlimit: 4; opacity: 1" paint-order="stroke" transform="translate(-1143.2,-1383.2)" d="M 1126 1443.8 v -419.2 c 0 -5.8 4.7 -10.4 10.4 -10.4 c 5.7 0 10.4 4.7 10.4 10.4 v 419.2 c 0 5.8 -4.6 10.4 -10.4 10.4 c -5.7 0 -10.4 -4.7 -10.4 -10.4" stroke-linecap="round" />
</g>
<g transform="matrix(0.2,0,0,-0.2,-103.4,18.6)" >
<path style="fill: rgb(81, 173, 229); fill-rule: nonzero; stroke: none; stroke-width: 1; stroke-linecap: butt; stroke-linejoin: miter; stroke-dasharray: none; stroke-dashoffset: 0; stroke-miterlimit: 4; opacity: 1" paint-order="stroke" transform="translate(-1176.6,-1397.2)" d="M 1159.4 1471.8 v -447.2 c 0 -5.8 4.7 -10.5 10.4 -10.5 c 5.7 0 10.4 4.7 10.4 10.5 v 447.2 c 0 5.8 -4.6 10.4 -10.4 10.4 c -5.7 0 -10.4 -4.7 -10.4 -10.4" stroke-linecap="round" />
</g>
<g transform="matrix(0.2,0,0,-0.2,-95.7,17.5)" >
<path style="fill: rgb(81, 173, 229); fill-rule: nonzero; stroke: none; stroke-width: 1; stroke-linecap: butt; stroke-linejoin: miter; stroke-dasharray: none; stroke-dashoffset: 0; stroke-miterlimit: 4; opacity: 1" paint-order="stroke" transform="translate(-1210,-1401.8)" d="M 1192.8 1480.9 v -456.1 c 0 -5.9 4.6 -10.6 10.4 -10.6 c 5.7 0 10.4 4.8 10.4 10.6 v 456.1 c 0 5.9 -4.6 10.7 -10.4 10.7 c -5.7 0 -10.4 -4.8 -10.4 -10.7" stroke-linecap="round" />
</g>
<g transform="matrix(0.2,0,0,-0.2,-88,18.6)" >
<path style="fill: rgb(81, 173, 229); fill-rule: nonzero; stroke: none; stroke-width: 1; stroke-linecap: butt; stroke-linejoin: miter; stroke-dasharray: none; stroke-dashoffset: 0; stroke-miterlimit: 4; opacity: 1" paint-order="stroke" transform="translate(-1243.4,-1397.2)" d="M 1226.2 1471.2 v -445.9 c 0 -6.1 4.6 -11.1 10.4 -11.1 c 5.7 0 10.4 5 10.4 11.1 v 445.9 c 0 6.1 -4.7 11.1 -10.4 11.1 c -5.7 0 -10.4 -5 -10.4 -11.1" stroke-linecap="round" />
</g>
<g transform="matrix(0.2,0,0,-0.2,-80.3,21.9)" >
<path style="fill: rgb(81, 173, 229); fill-rule: nonzero; stroke: none; stroke-width: 1; stroke-linecap: butt; stroke-linejoin: miter; stroke-dasharray: none; stroke-dashoffset: 0; stroke-miterlimit: 4; opacity: 1" paint-order="stroke" transform="translate(-1276.8,-1383.2)" d="M 1259.6 1443.8 v -419.2 c 0 -5.8 4.6 -10.4 10.4 -10.4 c 5.7 0 10.4 4.7 10.4 10.4 v 419.2 c 0 5.8 -4.7 10.4 -10.4 10.4 c -5.7 0 -10.4 -4.7 -10.4 -10.4" stroke-linecap="round" />
</g>
<g transform="matrix(0.2,0,0,-0.2,0,12.6)" >
<path style="fill: rgb(64, 50, 44); fill-rule: nonzero; stroke: none; stroke-width: 1; stroke-linecap: butt; stroke-linejoin: miter; stroke-dasharray: none; stroke-dashoffset: 0; stroke-miterlimit: 4; opacity: 1" paint-order="stroke" transform="translate(-1625,-1423.3)" d="M 1527.4 1410.1 c 0 52.1 44.1 94.1 97 90.6 c 48.2 -3.2 84.6 -45.5 84.6 -93.8 V 1021.9 c 0 -14.3 11.6 -25.9 25.9 -25.9 c 14.3 0 25.9 11.6 25.9 25.9 v 384 c 0 76.7 -58.8 142.8 -135.5 146.6 c -81.9 4 -149.8 -61.4 -149.8 -142.5 V 1021.9 c 0 -14.3 11.6 -25.9 25.9 -25.9 c 14.3 0 25.9 11.6 25.9 25.9 v 388.2" stroke-linecap="round" />
</g>
<g transform="matrix(0.2,0,0,-0.2,-15.4,21.9)" >
<path style="fill: rgb(191, 27, 44); fill-rule: nonzero; stroke: none; stroke-width: 1; stroke-linecap: butt; stroke-linejoin: miter; stroke-dasharray: none; stroke-dashoffset: 0; stroke-miterlimit: 4; opacity: 1" paint-order="stroke" transform="translate(-1558.2,-1383.2)" d="M 1541 1443.8 v -419.2 c 0 -5.8 4.7 -10.4 10.4 -10.4 c 5.7 0 10.4 4.7 10.4 10.4 v 419.2 c 0 5.8 -4.6 10.4 -10.4 10.4 c -5.7 0 -10.4 -4.7 -10.4 -10.4" stroke-linecap="round" />
</g>
<g transform="matrix(0.2,0,0,-0.2,-7.7,18.6)" >
<path style="fill: rgb(191, 27, 44); fill-rule: nonzero; stroke: none; stroke-width: 1; stroke-linecap: butt; stroke-linejoin: miter; stroke-dasharray: none; stroke-dashoffset: 0; stroke-miterlimit: 4; opacity: 1" paint-order="stroke" transform="translate(-1591.6,-1397.2)" d="M 1574.4 1471.8 v -447.2 c 0 -5.8 4.7 -10.5 10.4 -10.5 c 5.7 0 10.4 4.7 10.4 10.5 v 447.2 c 0 5.8 -4.6 10.4 -10.4 10.4 c -5.7 0 -10.4 -4.7 -10.4 -10.4" stroke-linecap="round" />
</g>
<g transform="matrix(0.2,0,0,-0.2,0,17.5)" >
<path style="fill: rgb(191, 27, 44); fill-rule: nonzero; stroke: none; stroke-width: 1; stroke-linecap: butt; stroke-linejoin: miter; stroke-dasharray: none; stroke-dashoffset: 0; stroke-miterlimit: 4; opacity: 1" paint-order="stroke" transform="translate(-1625,-1401.8)" d="M 1607.8 1480.9 v -456.1 c 0 -5.9 4.6 -10.6 10.4 -10.6 c 5.7 0 10.4 4.8 10.4 10.6 v 456.1 c 0 5.9 -4.6 10.7 -10.4 10.7 c -5.7 0 -10.4 -4.8 -10.4 -10.7" stroke-linecap="round" />
</g>
<g transform="matrix(0.2,0,0,-0.2,7.7,18.6)" >
<path style="fill: rgb(191, 27, 44); fill-rule: nonzero; stroke: none; stroke-width: 1; stroke-linecap: butt; stroke-linejoin: miter; stroke-dasharray: none; stroke-dashoffset: 0; stroke-miterlimit: 4; opacity: 1" paint-order="stroke" transform="translate(-1658.4,-1397.2)" d="M 1641.2 1471.2 v -445.9 c 0 -6.1 4.6 -11.1 10.4 -11.1 c 5.7 0 10.4 5 10.4 11.1 v 445.9 c 0 6.1 -4.7 11.1 -10.4 11.1 c -5.7 0 -10.4 -5 -10.4 -11.1" stroke-linecap="round" />
</g>
<g transform="matrix(0.2,0,0,-0.2,15.4,21.9)" >
<path style="fill: rgb(191, 27, 44); fill-rule: nonzero; stroke: none; stroke-width: 1; stroke-linecap: butt; stroke-linejoin: miter; stroke-dasharray: none; stroke-dashoffset: 0; stroke-miterlimit: 4; opacity: 1" paint-order="stroke" transform="translate(-1691.8,-1383.2)" d="M 1674.6 1443.8 v -419.2 c 0 -5.8 4.6 -10.4 10.4 -10.4 c 5.7 0 10.4 4.7 10.4 10.4 v 419.2 c 0 5.8 -4.7 10.4 -10.4 10.4 c -5.7 0 -10.4 -4.7 -10.4 -10.4" stroke-linecap="round" />
</g>
<g transform="matrix(0.2,0,0,-0.2,95.7,12.6)" >
<path style="fill: rgb(64, 50, 44); fill-rule: nonzero; stroke: none; stroke-width: 1; stroke-linecap: butt; stroke-linejoin: miter; stroke-dasharray: none; stroke-dashoffset: 0; stroke-miterlimit: 4; opacity: 1" paint-order="stroke" transform="translate(-2040,-1423.3)" d="M 1942.4 1410.1 c 0 52.1 44.1 94.1 97 90.6 c 48.2 -3.2 84.6 -45.5 84.6 -93.8 V 1021.9 c 0 -14.3 11.6 -25.9 25.9 -25.9 c 14.3 0 25.9 11.6 25.9 25.9 v 384 c 0 76.7 -58.8 142.8 -135.5 146.6 c -81.9 4 -149.8 -61.4 -149.8 -142.5 V 1021.9 c 0 -14.3 11.6 -25.9 25.9 -25.9 c 14.3 0 25.9 11.6 25.9 25.9 v 388.2" stroke-linecap="round" />
</g>
<g transform="matrix(0.2,0,0,-0.2,80.3,21.9)" >
<path style="fill: rgb(128, 204, 40); fill-rule: nonzero; stroke: none; stroke-width: 1; stroke-linecap: butt; stroke-linejoin: miter; stroke-dasharray: none; stroke-dashoffset: 0; stroke-miterlimit: 4; opacity: 1" paint-order="stroke" transform="translate(-1973.2,-1383.2)" d="M 1956 1443.8 v -419.2 c 0 -5.8 4.7 -10.4 10.4 -10.4 c 5.7 0 10.4 4.7 10.4 10.4 v 419.2 c 0 5.8 -4.6 10.4 -10.4 10.4 c -5.7 0 -10.4 -4.7 -10.4 -10.4" stroke-linecap="round" />
</g>
<g transform="matrix(0.2,0,0,-0.2,88,18.6)" >
<path style="fill: rgb(128, 204, 40); fill-rule: nonzero; stroke: none; stroke-width: 1; stroke-linecap: butt; stroke-linejoin: miter; stroke-dasharray: none; stroke-dashoffset: 0; stroke-miterlimit: 4; opacity: 1" paint-order="stroke" transform="translate(-2006.6,-1397.2)" d="M 1989.4 1471.8 v -447.2 c 0 -5.8 4.7 -10.5 10.4 -10.5 c 5.7 0 10.4 4.7 10.4 10.5 v 447.2 c 0 5.8 -4.6 10.4 -10.4 10.4 c -5.7 0 -10.4 -4.7 -10.4 -10.4" stroke-linecap="round" />
</g>
<g transform="matrix(0.2,0,0,-0.2,95.7,17.5)" >
<path style="fill: rgb(128, 204, 40); fill-rule: nonzero; stroke: none; stroke-width: 1; stroke-linecap: butt; stroke-linejoin: miter; stroke-dasharray: none; stroke-dashoffset: 0; stroke-miterlimit: 4; opacity: 1" paint-order="stroke" transform="translate(-2040,-1401.8)" d="M 2022.8 1480.9 v -456.1 c 0 -5.9 4.6 -10.6 10.4 -10.6 c 5.7 0 10.4 4.8 10.4 10.6 v 456.1 c 0 5.9 -4.6 10.7 -10.4 10.7 c -5.7 0 -10.4 -4.8 -10.4 -10.7" stroke-linecap="round" />
</g>
<g transform="matrix(0.2,0,0,-0.2,103.4,18.6)" >
<path style="fill: rgb(128, 204, 40); fill-rule: nonzero; stroke: none; stroke-width: 1; stroke-linecap: butt; stroke-linejoin: miter; stroke-dasharray: none; stroke-dashoffset: 0; stroke-miterlimit: 4; opacity: 1" paint-order="stroke" transform="translate(-2073.4,-1397.2)" d="M 2056.2 1471.2 v -445.9 c 0 -6.1 4.6 -11.1 10.4 -11.1 c 5.7 0 10.4 5 10.4 11.1 v 445.9 c 0 6.1 -4.7 11.1 -10.4 11.1 c -5.7 0 -10.4 -5 -10.4 -11.1" stroke-linecap="round" />
</g>
<g transform="matrix(0.2,0,0,-0.2,111.1,21.9)" >
<path style="fill: rgb(128, 204, 40); fill-rule: nonzero; stroke: none; stroke-width: 1; stroke-linecap: butt; stroke-linejoin: miter; stroke-dasharray: none; stroke-dashoffset: 0; stroke-miterlimit: 4; opacity: 1" paint-order="stroke" transform="translate(-2106.8,-1383.2)" d="M 2089.5 1443.8 v -419.2 c 0 -5.8 4.6 -10.4 10.4 -10.4 c 5.7 0 10.4 4.7 10.4 10.4 v 419.2 c 0 5.8 -4.7 10.4 -10.4 10.4 c -5.7 0 -10.4 -4.7 -10.4 -10.4" stroke-linecap="round" />
</g>
</g>
</g>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 11 KiB

159
workspaces/postgres-workspace/README.md
View file

@ -1,34 +1,32 @@
> Please support this project by simply putting a Github star.
Share this library with friends on Twitter and everywhere else you can.
> If you notice a bug or have something not working, please report an issue, we'll try to fix it as soon as possible.
Great number of tools to manage PostgreSQL database, develop and schedule database scripts and applications.
In a single docker image.
All in a single docker image.
__Features:__
1. [Browser-based VS-code](https://theia-ide.org/) version with the following extensions installed:
<p align="center">
<img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/postgres-wid-collage.png" alt="postgres-wid-collage" width="750">
</p>
## Features:
1. [Browser-based VS-code](https://theia-ide.org/) with extensions:
- [PostgreSQL Client for vscode](https://open-vsx.org/extension/cweijan/vscode-postgresql-client2)
- [SQLTools PostgreSQL](https://open-vsx.org/extension/mtxr/sqltools-driver-pg)
- and additional color & icon extensions
2. Browser-based terminal (finally no need to ssh into a running docker container)
3. [Cronicle](https://github.com/jhuckaby/Cronicle). Browser-based jobs scheduler. Schedule backups, data extracts, maintenance, etc.
4. Ubuntu with many apps and tools installed and configured:
2. Browser-based terminal. Finally no need to ssh into a running docker container.
3. [Cronicle](https://github.com/jhuckaby/Cronicle). Browser-based jobs scheduler. Schedule backups, data extracts, maintenance scripts.
4. Ubuntu with many apps installed:
- text editors
- [Zsh](https://ohmyz.sh/)
- cron
- mc
- and many more
5. [Rclone](https://rclone.org/) to save backups to S3, and pull then back to restore a database.
6. Up and running Postgres 14 database (to try things locally first)
5. [Rclone](https://rclone.org/). Save backups to S3.
6. Up and running Postgres 14 database. To try things locally first.
7. Postgres CLI tools:
- [psql](https://www.postgresql.org/docs/current/app-psql.html). Standard PostgreSQL interactive terminal.
- [pgcli](https://www.pgcli.com/). Command line interface for Postgres with auto-completion and syntax highlighting.
- [pspg](https://iboxshare.com/okbob/pspg). Visualisation of Postgres SQL queries data output.
- [pspg](https://iboxshare.com/okbob/pspg). Data visualizer.
8. Performance and load testing tools:
- [pgmetrics](https://pgmetrics.io/). A tool that can collect 350+ metrics from a running PostgreSQL server.
- [pgCenter](https://github.com/lesovsky/pgcenter). A command-line admin tool for observing and troubleshooting Postgres.
- [pgbench](https://www.postgresql.org/docs/devel/pgbench.html). Run a benchmark test on PostgreSQL.
- [pgmetrics](https://pgmetrics.io/). Collect 350+ metrics from a running PostgreSQL server.
- [pgCenter](https://github.com/lesovsky/pgcenter). Admin tool for observing and troubleshooting Postgres.
- [pgbench](https://www.postgresql.org/docs/devel/pgbench.html). Run benchmark tests on PostgreSQL.
9. Backups and restore tools:
- [pg_dump](https://www.postgresql.org/docs/current/backup-dump.html)
- [pg_dumpall](https://www.postgresql.org/docs/current/backup-dump.html)
@ -41,48 +39,62 @@ __Features:__
- [synth](https://github.com/getsynth/synth). A tool for generating realistic data.
- [mock-data](https://github.com/faisaltheparttimecoder/mock-data). Generate mock data, tables and databases easily.
12. Database change management toolset:
- [sqitch](https://github.com/sqitchers/sqitch). A database change management application.
- [sqitch](https://github.com/sqitchers/sqitch). Database change management application.
- [yuniql](https://github.com/rdagumampan/yuniql). Schema versioning and database migration engine.
- [pgmigrate](https://github.com/yandex/pgmigrate). Database migration tool developed by Yandex.
- [migra](https://databaseci.com/docs/migra). Find differences in database schemas as easily as running a diff on two text files.
- [migra](https://databaseci.com/docs/migra). Find differences in database schemas easily.
13. [pg_sample](https://github.com/mla/pg_sample). Create database sample while maintaining referential integrity.
14. [Postgres explain visualizer](https://github.com/AlexTatiyants/pev). Understand Postgres execution plans.
15. [DBdesigner](https://github.com/akreienbring/dbdesigner). Draw ERD diagrams and use it to generate DDL code.
16. [Schemaspy](https://github.com/schemaspy/schemaspy). Generate extensive HTML report about Postgres database.
16. [Schemaspy](https://github.com/schemaspy/schemaspy). Generate extensive HTML database documentation that looks great.
17. [tbls](https://github.com/k1LoW/tbls). Generate documentation (essentially data catalog) from the database
18. [pg_flame](https://github.com/mgartner/pg_flame). A flamegraph generator for Postgres EXPLAIN ANALYZE output.
18. [pg_flame](https://github.com/mgartner/pg_flame). A flamegraph generator for Postgres explain output.
19. Programming languages. Develop/execute directly inside the workspace using browser-based IDE.
- python
- nodejs
- java
20. [Filebrowser](https://github.com/filebrowser/filebrowser). Browse files, push and pull files to/from workspace (finally we don't need to mount folders for that).
20. [Filebrowser](https://github.com/filebrowser/filebrowser). Browse and download files (finally no need to mount folders for that).
21. [Static file server](https://github.com/vercel/serve). View generated HTML reports.
22. [Ungit](https://github.com/FredrikNoren/ungit). The easiest way to use git.
23. Browser-based process and resource utilization monitor
24. [Mkdocs](https://www.mkdocs.org/). Create documentation with Markdown.
25. Workspace own UI, from which you can open all applications and documentation.
__Try it:__
__*Below you'll find examples and short tutorials how to use most of the tools included in the workspace.*__
> If you like this project, please support it by simply putting a Github star.
Share this library with friends on Twitter and everywhere else you can.
### Try it out
```
docker run --name space-1 -d -p 8020-8035:8020-8035 alnoda/postgres-workspace
```
Open [http://localhost:8020](http://localhost:8020)
Open [localhost:8020](http://localhost:8020)
### Why this image
So that you don't need to set up all those tools yourself, could have an isolated environment specifically configured
So that you could have an isolated environment specifically configured
to manage and develop Postgres, get started fast, try things out on the internal Postgres first, import and export data,
create backup dumps, create sample databases for development and testing,
schedule all these tasks, enforce database change management in your team, analyse metrics annd uderstand performance,
conduct load tests on the databases you use, look on visual execution
plans, create complete documentations from the Postgres databases, develop scripts and apps with Python or Javascript.
_Below you'll find examples how to use many of the applications included in the workspace._
<p align="center">
<img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/postgres-wid-demo.gif" alt="postgres-wid-demo" width="750">
</p>
## Workspace UI
Workspace UI can be accessed on [localhost:8020](http://localhost:8020). It provides quick access to the IDE, terrminal and other apps.
<p align="center">
<img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/Postgres-wid-UI.png" alt="Postgres-wid UI" width="750">
</p>
## Postgres
The workspace includes Postgres 14 up and running. You can use it to experiment and try things before trying on the application database.
Connection parameters: host - _localhost_, user - _abc_ (no password), database - _abc_.
You can load sample database to check out the workspace features and applications.
You can load sample database to check out the workspace features and applications. Load "usda" database
```
git clone https://github.com/morenoh149/postgresDBSamples /home/pg_dbs
cd /home/pg_dbs/usda-r18-1.0
@ -92,7 +104,7 @@ psql -f usda.sql usda
pgcli usda
SELECT * FROM data_src
```
Load db "Chinook"
load Chinook database
```
cd /home/pg_dbs/chinook-1.4
createdb -E UTF8 chinook
@ -101,7 +113,7 @@ psql -f Chinook_PostgreSql_utf8.sql -d chinook
pgcli chinook
SELECT * FROM "Track"
```
Load db Adventureworks:
load Adventureworks database
```
cd /home/pg_dbs/adventureworks/
unzip data.zip # this will unzip all csv to the current directory
@ -113,33 +125,28 @@ SELECT * FROM sales.salesorderdetail
```
## VS-code & plugins
[Apache Theia](https://theia-ide.org/) is an browser-based version of the VS-code. It is part of the workspace, and
it is easy and convenient to develop directly in the workspace. Moreover, 2 database extensions are already installed,
and can be used to connect to local or remote PostgreSQL, explore data and create python/node/shell scripts to manage Postgres.
[Apache Theia](https://theia-ide.org/), a browser-based version of the VS-code, is part of the workspace. It has
extensions to explore Postgres
<p align="center">
<img src="./img/Theia connect.png" alt="Theia connect" width="750">
<img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/Theia connect.png" alt="Theia connect" width="750">
</p>
If you loaded sample databases to the local postgres, connect & explore data directly in the IDE
<p align="center">
<img src="./img/Theia explore data.png" alt="Theia explore" width="750">
<img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/Theia explore data.png" alt="Theia explore" width="750">
</p>
## Postgres CLI tools
Set-up and configured CLI tools that enable very convenient working with Postgres from the command line,
with auto-completion and data visualisation
- [psql](https://www.postgresql.org/docs/current/app-psql.html)
- [pgcli](https://www.pgcli.com/)
- [pspg](https://iboxshare.com/okbob/pspg)
Querying from command line should be awesome
<p align="center">
<img src="./img/pgcli.png" alt="pgcli" width="450">
<img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/pgcli.png" alt="pgcli" width="750">
</p>
<p align="center">
<img src="./img/pspg.png" alt="pspg" width="450">
<img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/pspg.png" alt="pspg" width="750">
</p>
### psql
@ -174,10 +181,12 @@ You can disable pspg by deleting the (last) line _PAGER='pspg -s 4 --interactive
## Performance and load testing
Monitor live activity, collect and analyse metrics, perform load tests with
- [pgmetrics](https://pgmetrics.io/)
- [pgCenter](https://github.com/lesovsky/pgcenter)
- [pgbench](https://www.postgresql.org/docs/devel/pgbench.html)
Monitor live activity, collect and analyse metrics, perform load tests.
<p align="center">
<img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/pgmetrics.png" alt="pgmetrics" width="750">
</p>
### pgmetrics
[pgmetrics](https://pgmetrics.io/) is an open-source, zero-dependency, single-binary tool that can collect 350+ metrics from a running PostgreSQL
server and display it in easy-to-read text format or export it as JSON and CSV for scripting. Example use:
@ -219,9 +228,8 @@ __Learn more:__
## Backups and restore
- [pg_dump](https://www.postgresql.org/docs/current/backup-dump.html)
- [rclone](https://rclone.org/)
- backup scheduling
Create backup dumps with [pg_dump](https://www.postgresql.org/docs/current/backup-dump.html) or [pg_dumpall](https://www.postgresql.org/docs/14/app-pg-dumpall.html),
push to S3 with [rclone](https://rclone.org/) and schedule with [Cronicle](https://github.com/jhuckaby/Cronicle).
### pgdump
The purpose of [pg_dump](https://www.postgresql.org/docs/current/backup-dump.html) is to generate a file with SQL commands that,
when fed back to the server, will recreate the database in the same state as it was at the time of the dump. Example of use
@ -290,6 +298,10 @@ __Schedule with cron__: open workspace terminal, and execute `crontab -e`. Creat
__Schedule with Cronicle__: open Cronicle WEB UI from the main Workspace page, or directly on [http://localhost:8023/](http://localhost:8023/)
(user/pass - admin/admin) and create scheduled event (plugin - shell script). Provide commands to be executed.
<p align="center">
<img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/Cronicle2.png" alt="Cronicle" width="750">
</p>
## Database sample
Who hasn't faced a need to make a sample database for testing/development purposes? :-)
```
@ -302,9 +314,8 @@ psql usda_sample < /home/project/sample_db.sql
```
## Data import/export
- [pgclimb](https://github.com/lukasmartinelli/pgclimb)
- [pgfutter](https://github.com/lukasmartinelli/pgfutter)
- [pgloader](https://pgloader.readthedocs.io/en/latest/index.html)
Import data from various files, export to csv, json, html. Download with [Filebrowser](https://github.com/filebrowser/filebrowser),
upload to S3 with [Rclone](https://rclone.org/), schedule with [Cronicle](https://github.com/jhuckaby/Cronicle).
### pgclimb
[pgclimb](https://github.com/lukasmartinelli/pgclimb) is a PostgreSQL utility to export data into different data formats with support for templates.
Example of use:
@ -390,8 +401,7 @@ pgloader /home/project/test.load
## Data generators/mocking
- [synth](https://github.com/getsynth/synth)
- [mock-data](https://github.com/faisaltheparttimecoder/mock-data)
Generate realistic data for existing tables, create new mock tables or databases.
### [synth](https://github.com/getsynth/synth)
Synth is a tool for generating realistic data using a declarative data model. Synth is database agnostic and can scale to millions of rows of data.
@ -444,10 +454,7 @@ __Learn more:__
## Database change management
- [sqitch](https://github.com/sqitchers/sqitch)
- [yuniql](https://github.com/rdagumampan/yuniql)
- [pgmigrate](https://github.com/yandex/pgmigrate)
- [migra](https://databaseci.com/docs/migra)
Implement database change management in your team and drive higher software delivery and organizational performance.
### sqitch
[sqitch](https://github.com/sqitchers/sqitch) - is a database change management application.
To demonstrate (very briefly) how Sqitch works, create a new database called 'flipr_test' in the local Postgres:
@ -530,28 +537,40 @@ Migra will produce a set of SQL commands to make database "abc" have the same DD
## Visualization tools
- [pev](https://github.com/AlexTatiyants/pev)
- [pg_flame](https://github.com/mgartner/pg_flame)
- [schemaspy](https://github.com/schemaspy/schemaspy)
- [tbls](https://github.com/k1LoW/tbls)
- [dbdesigner](https://github.com/akreienbring/dbdesigner)
### [schemaspy](https://github.com/schemaspy/schemaspy)
Generate complete database documentation that has absolutely amazing look and feel generated content.
### dbdesigner
[dbdesigner](https://github.com/akreienbring/dbdesigner) a visual tool to create entity relationship diagrams and
generate Postgres DDL code.
<p align="center">
<img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/dbdesigner.png" alt="dbdesigner" width="750">
</p>
### schemaspy
[schemaspy](https://github.com/schemaspy/schemaspy) generates complete database documentation that looks great.
```
schemaspy -h localhost -p 5432 -d usda -u abc -o /home/static-server/usda-schemaspy
```
And open static file server on [http://localhost:8022/](http://localhost:8022/) folder usda-schemaspy
<p align="center">
<img src="./img/schemaspy.png" alt="schemaspy" width="750">
<img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/schemaspy.gif" alt="schemaspy" width="750">
</p>
### Postgres explain visualizer
[Pev](https://github.com/AlexTatiyants/pev) helps to understand Postgres execution plans
<p align="center">
<img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/pev.gif" alt="pev" width="750">
</p>
### tbls
[tbls](https://github.com/k1LoW/tbls) - a tool for document a database, written in Go.
For example, generate Markdown docs as
For example, generate Markdown docs
```
tbls doc "postgres://abc@localhost:5432/usda?sslmode=disable" /home/static-server/usda-tbls
```
and push to Github.
Now you can push these docks push to Github.
### pg_flame
[pg_flame](https://github.com/mgartner/pg_flame) - a flamegraph generator for Postgres EXPLAIN ANALYZE output.
```
@ -565,6 +584,10 @@ psql usda -qAtc 'SELECT *
| pg_flame > /home/static-server/flamegraph.html
```
<p align="center">
<img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/pgflame.png" alt="pgflame" width="750">
</p>
## Other useful tools
### pg_insights
Convenient SQL for monitoring Postgres database health. Clone repo and use it to analyse Postgres databases

BIN
workspaces/postgres-workspace/img/Cronicle2.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 566 KiB

BIN
workspaces/postgres-workspace/img/Postgres-wid-UI.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 1.1 MiB

BIN
workspaces/postgres-workspace/img/dbdesigner.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 452 KiB

BIN
workspaces/postgres-workspace/img/pev.gif Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 1.2 MiB

BIN
workspaces/postgres-workspace/img/pgflame.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 156 KiB

BIN
workspaces/postgres-workspace/img/pgmetrics.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 266 KiB

BIN
workspaces/postgres-workspace/img/postgres-wid-collage.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 777 KiB

BIN
workspaces/postgres-workspace/img/postgres-wid-demo.gif Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 16 MiB

BIN
workspaces/postgres-workspace/img/schemaspy.gif Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 2.4 MiB