alnoda-workspaces/workspaces/postgres-workspace/removeme_docs/getting-started.md

# Getting started

## Intro

These tutorials demonstrates how to use Postgres workspace to explore and interact with Postgres.  

To start, open Quickstart page [http://localhost:8020/](http://localhost:8020/) for quick access to all the tools. 
Use terminal to execute commands from this tutorial.

## Postgres

The workspace includes Postgres 14 up and running. You can use it to experiment and try things before trying on the application database. 
Connection parameters: host - _localhost_, user - _abc_ (no password), database - _abc_.  

You can load sample database to check out the workspace features and applications. For example, load "usda" database

```
git clone https://github.com/morenoh149/postgresDBSamples /home/pg_dbs
cd /home/pg_dbs/usda-r18-1.0
createdb usda
psql -f usda.sql usda
#Explore data
pgcli usda
SELECT * FROM data_src
```

load Chinook database

```
cd /home/pg_dbs/chinook-1.4
createdb -E UTF8 chinook
psql -f Chinook_PostgreSql_utf8.sql -d chinook   
# Explore data
pgcli chinook
SELECT * FROM "Track"
```

load Adventureworks database

```
cd /home/pg_dbs/adventureworks/
unzip data.zip # this will unzip all csv to the current directory
psql -c "CREATE DATABASE \"adventureworks\";"
psql -d adventureworks < install.sql
# Explore data
pgcli adventureworks
SELECT * FROM sales.salesorderdetail
```

Further examples will use this local Postgres instance, but can easily applied to another Postgres database.

# VS-code & plugins

[Apache Theia](https://theia-ide.org/), a browser-based version of the VS-code, is part of the workspace. It has 
extensions to query Postgres  

<p align="center">
  <img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/Theia connect.png" alt="Theia connect" width="750">
</p> 

If you loaded sample databases to the local Postgres, connect & explore data directly in the IDE

<p align="center">
  <img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/Theia explore data.png" alt="Theia explore" width="750">
</p> 

## Postgres CLI tools

Querying from command line can be awesome! 

### psql

[psql](https://www.postgresql.org/docs/14/app-psql.html) is a PostgreSQL interactive terminal

```
# connect to local Postgres
psql
# connect to database chinook (local Postgres)
psql chinook
# general psql connection pattern
psql postgresql://dbmaster:5432/mydb
# with password prompt
psql -U postgres -W postgresql://my-postgres-host:5432/mydb
$ without password prompt
PGPASSWORD=xxxxxxxxxxx psql -U postgres postgresql://my-postgres-host:5432/mydb
```

### pgcli
[pgcli](https://www.pgcli.com/) - is a postgres client that does auto-completion and syntax highlighting. 

```
# pgcli to the local Postgres 
pgcli 
# connect to database usda (local Postgres)
pgcli usda
# pgcli to the remote Postgres 
pgcli -h my-postgres-host -p 5432 -U postgres -W -d mydb
```

<p align="center">
  <img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/pgcli.png" alt="pgcli" width="750">
</p> 

<p align="center">
  <img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/pspg.png" alt="pspg" width="750">
</p> 

### pspg

[Pspg](https://github.com/okbob/pspg) is a unix pager (with very rich functionality) designed for work with tables. 
Pspg is enabled by default, and will automatically applied to psql and pgcli if the result of SELECT statement is large. 
You can disable pspg by deleting the (last) line _PAGER='pspg -s 4 --interactive --no-mouse'_  from file /home/abc/.zshrc. 
(or you can modify configuration).


## Performance and load testing
Monitor live activity, collect and analyse metrics, perform load tests.

<p align="center">
  <img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/pgmetrics.png" alt="pgmetrics" width="500">
</p> 

### pgmetrics

[pgmetrics](https://pgmetrics.io/) is an open-source, zero-dependency, single-binary tool that can collect 350+ metrics from a running PostgreSQL 
server and display it in easy-to-read text format or export it as JSON and CSV for scripting.

```
# Try on local Postgres database "usda" (upon password prompt just hit Enter) 
pgmetrics --no-pager -h localhost -U abc usda 
# Remote Postgres 
pgmetrics --no-pager -h my-postgres-host -p 5432 -U postgres mydb
```

_Hint:_ Output to txt file `pgmetrics -h localhost -U abc usda >> /home/static-server/pgmetrics-usda.txt` and open static server [http://localhost:8022](http://localhost:8022)

### pgCenter

[pgCenter](https://github.com/lesovsky/pgcenter) is a command-line admin tool for observing and troubleshooting Postgres. 

```
# for local Postgres 
pgcenter top -h localhost -p 5432 -U abc -d usda
# for remote Postgres (password prompt)
pgcenter top -h my-postgres-host -p 5432 -U postgres -d mydb
# for remote Postgres (no password prompt)
PGPASSWORD=xxxxxxxxxxx pgcenter top -h my-postgres-host -p 5432 -U postgres -d mydb
```

### pgbench
[pgbench](https://www.postgresql.org/docs/devel/pgbench.html) is a simple program for running benchmark tests on PostgreSQL. 

```
# initialize 
pgbench -h localhost -U abc -i -s 50 abc
# test
pgbench -h localhost -U abc -c 10 -j 2 -t 10000 abc
```
Example use on remote Postgres:
```
# initialize 
pgbench -h my-postgres-host -U postgres -i -s 50 mydb
# test
pgbench -h my-postgres-host -U postgres -c 10 -j 2 -t 10000 mydb
```
__Learn more:__
- [Pgbench tutorial](https://www.cloudbees.com/blog/tuning-postgresql-with-pgbench). 


## Backups and restore
Create backup dumps with [pg_dump](https://www.postgresql.org/docs/current/backup-dump.html) or [pg_dumpall](https://www.postgresql.org/docs/14/app-pg-dumpall.html), 
push to S3 with [rclone](https://rclone.org/) and schedule with [Cronicle](https://github.com/jhuckaby/Cronicle).

### pgdump

The purpose of [pg_dump](https://www.postgresql.org/docs/current/backup-dump.html) is to generate a file with SQL commands that, 
when fed back to the server, will recreate the database in the same state as it was at the time of the dump.

```
mkdir /home/dumps/
# Dump from local Postgres
pg_dump -h localhost -p 5432 -U abc usda > /home/dumps/local_postgres_dump
# Dump from remote Postgres
PGPASSWORD=xxxxxxxxxxx pg_dump -h my-postgres-host -p 5432 -U postgres mydb > /home/dumps/remote_postgres_dump
# Restore from dump to local Postgres
psql dbname < /home/dumps/remote_postgres_dump
```

__Other versions of pg_dump__
The default pg_dump version is for Postgres 14. If you see error _error: aborting because of server version mismatch_, try 
pg_dump for your postgres version (currently Postgres versions 12, 13 and 14 are supported)

```
pg_dump_12 -h localhost -p 5432 -U abc abc > /home/dumps/local_postgres_dump_12
pg_dump_13 -h localhost -p 5432 -U abc abc > /home/dumps/local_postgres_dump_13
```

### pg_dumpall
[pg_dumpall](https://www.postgresql.org/docs/14/app-pg-dumpall.html) is a utility for writing out ("dumping") 
all PostgreSQL databases of a cluster into one script file. 

```
pg_dumpall -h localhost -p 5432 -U abc > /home/dumps/local_postgres_dumpall
# Dump from remote Postgres
PGPASSWORD=xxxxxxxxxxx pg_dump -h my-postgres-host -p 5432 -U postgres mydb > /home/dumps/remote_postgres_dump
# Restore from dump to local Postgres
psql dbname < /home/dumps/remote_postgres_dump
```

The default is pg_dumpall for Postgres 14, if you need earlier version, use `pg_dumpall_12`, `pg_dumpall_13`. 

### rclone

[rclone](https://rclone.org/) is a command line program to manage files on cloud storage. 
It is a feature rich alternative to cloud vendors' web storage interfaces.  
__Copy dump to S3:__  
Create file `~/.config/rclone/rclone.conf` with the following content

```
[remote]
type = s3
provider = AWS
access_key_id = XXXXXXXXXXXXXXXXXXXXXX
secret_access_key = XXxxXXxXXXxxxXXxXXXxxXXXxxxXXXxXXXX
region = xx-xxxx-x
```

Use Rclone to copy to S3 and delete from local

```
rclone move /home/dumps/ remote:my-s3-bucket/dumps/
```

__Restore from S3 to local:__
When there is a need to restore database from a dump, copy dupm from S3 or mount S3 bucket to local folder. 
In case of mounting, file will not be physically copied. Database restoration will happen directly from dump on S3.

```
# copy & restore
rclone copy remote:dwh-1/dumps/remote_postgres_dump /home/s3-dumps/
psql dbname < /home/s3-dumps/remote_postgres_dump
# mount & restore
rclone sync remote:dwh-1/dumps /home/s3-dumps
psql dbname < /home/s3-dumps/remote_postgres_dump
```

### scheduling backups

__Schedule with Cronicle__: open Cronicle WEB UI from the main Workspace page, or directly on [http://localhost:8023/](http://localhost:8023/) 
(user/pass - admin/admin) and create scheduled event (plugin - shell script). Provide commands to be executed.

<p align="center">
  <img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/Cronicle2.png" alt="Cronicle" width="750">
</p> 

## Database sample

Who hasn't faced a need to make a sample database for testing/development purposes? :-)

```
# Create sample from database "usda" in local Postgres
pg_sample -h localhost -U abc --limit=500 --file=/home/project/sample_db.sql usda
# create new database (local Postgres)
createdb usda_sample
# restore sample to the new database
psql usda_sample < /home/project/sample_db.sql 
```

## Data import/export

Import data from various files, export to csv, json, html. Download with [Filebrowser](https://github.com/filebrowser/filebrowser), 
upload to S3 with [Rclone](https://rclone.org/), schedule with [Cronicle](https://github.com/jhuckaby/Cronicle).

### pgclimb
[pgclimb](https://github.com/lukasmartinelli/pgclimb) is a PostgreSQL utility to export data into different data formats with support for templates. 

```
# From local Postgres instance
pgclimb --host localhost --port 5432 --dbname abc --username abc --query "SELECT * FROM distributors" -o /home/project/distributors.csv csv
# From remote Postgres instance
pgclimb --host my-postgres-host --port 5432 --dbname mydb --username postgres --password xxxxxxxxxxx --query "SELECT * FROM phonebook" -o /home/project/phone.csv csv
pgclimb --host my-postgres-host --port 5432 --dbname mydb --username postgres --password xxxxxxxxxxx --query "SELECT * FROM phonebook" -o /home/static-server/phone.html html
```

### pgfutter
[pgfutter](https://github.com/lukasmartinelli/pgfutter) helps to import CSV and line delimited JSON into PostgreSQL the easy way.  
For example, create file `/home/project/friends.json` with the following content

```json
{"name": "Jacob", "age": 26, "friends": ["Anthony"]}
{"name": "Anthony", "age": 25, "friends": []}
{"name": "Emma", "age": 28, "friends": ["Jacob", "Anthony"]}
```

Import data to Postgres

```
# to the local Postgres
pgfutter --host localhost --port 5432 --dbname abc --username abc --table friends json /home/project/friends.json
# to the remote Postgres
pgfutter --host my-postgres-host --port 5432 --dbname mydb --username postgres --pass admin --schema imports --table friends json /home/project/friends.json
```

Another example: create file `/home/project/friends2.csv` with the following content

```
name,age,friends
Jacob,26,"Anthony"
Anthony,25,""
Emma,28,"Jacob,Anthony"
```

Import data to Postgres

```
pgfutter --host my-postgres-host --port 5432 --dbname mydb --username postgres --pass admin --schema public --table friends2 csv /home/project/friends2.csv
```

### PGLoader

[PGLoader](https://pgloader.readthedocs.io/en/latest/index.html) - a very versatile data loading tool for PostgreSQL. 
Load data from files, or migrate entire databases to Postgres.   

Example of use: create test load file `/home/project/test.csv` with the following content

```
Header, with a © sign
"2.6.190.56","2.6.190.63","33996344","33996351","GB","United Kingdom"
"3.0.0.0","4.17.135.31","50331648","68257567","US","United States"
"4.17.135.32","4.17.135.63","68257568","68257599","CA","Canada"
"4.17.135.64","4.17.142.255","68257600","68259583","US","United States"
"4.17.143.0","4.17.143.15","68259584","68259599","CA","Canada"
"4.17.143.16","4.18.32.71","68259600","68296775","US","United States"
```

Create PGLoader file `/home/project/test.load` with the following content (use your postgres connection arguments)

```
LOAD CSV
     FROM '/home/project/test.csv' (x, y, a, b, c, d)
     -- local Postgres
     INTO postgresql://abc@localhost:5432/abc?csv (a, b, d, c)
     -- remote Postgres
     -- INTO postgresql://user:password@my-postgres-host:5432/mydb?csv (a, b, d, c)

     WITH truncate,
          skip header = 1,
          fields optionally enclosed by '"',
          fields escaped by double-quote,
          fields terminated by ','

      SET client_encoding to 'latin1',
          work_mem to '12MB',
          standard_conforming_strings to 'on'

   BEFORE LOAD DO
    $$ drop table if exists csv; $$,
    $$ create table csv (
        a bigint,
        b bigint,
        c char(2),
        d text
       );
  $$;
```

Load data into new table called "csv"

```
pgloader /home/project/test.load
```


## Data generators/mocking
Generate realistic data for existing tables, create new mock tables or databases.

### Synth

[Synth](https://github.com/getsynth/synth) is a tool for generating realistic data using a declarative data model. Synth is database agnostic and can scale to millions of rows of data. 

Create dir `mkdir /home/project/synth` and create file `/home/project/synth/companies.json` with the following contents

```json
{
    "type": "array",
    "length": {
        "type": "number",
        "constant": 1
    },
    "content": {
        "type": "object",
        "company_id": {
            "type": "number",
            "id": {}
        },
        "company_name": {
            "type": "string",
            "faker": {
                "generator": "company_name"
            }
        }
    }
}
```

Generate data (2 entries only) as json - `synth generate /home/project/synth/ --size 2 | jq`. 
Generate data into (local) Postgres

```
# Create table users in local Postgres database
echo "CREATE TABLE companies(company_id SERIAL PRIMARY KEY, company_name VARCHAR(255) NOT NULL);" | psql
# generate random data to table users
synth generate /home/project/synth/ --size 200 --to postgres://abc@localhost:5432/abc
# check 10 records
echo "SELECT * FROM companies LIMIT 10" | psql
```

__Learn more:__
- [Synth docs](https://www.getsynth.com/docs/getting_started/synth)
- [How to Create PostgreSQL Test Data](https://www.getsynth.com/docs/blog/2021/03/09/postgres-data-gen)

### mock-data

The idea behind [mock-data](https://github.com/faisaltheparttimecoder/mock-data)is to allow users to test database queries with sets of fake data in any pre-defined table. 

```
# Create table in (local) Postgres
echo "CREATE TABLE distributors (did integer, name varchar(40));" | psql
# Populate (local) Postgres with fake data (2000 rows)
mock tables -t "distributors" -r 2000 --uri="postgres://abc@localhost:5432/abc?sslmode=disable"
```

__Learn more:__
- [Mock-data documentation](https://github.com/faisaltheparttimecoder/mock-data/wiki)


## Database change management
Implement database change management in your team and drive higher software delivery and organizational performance.

### sqitch

[sqitch](https://github.com/sqitchers/sqitch) - is a database change management application. 
To demonstrate (very briefly) how Sqitch works, create a new database called 'flipr_test' in the local Postgres:

```
createdb flipr_test
```

Initialize and configure a Sqitch project, called 'flipr'

```
mkdir /home/project/flipr && cd /home/project/flipr
sqitch init flipr --engine pg
```

Open file `/home/project/flipr/sqitch.conf` and update configuration for the "pg" engine to conect to local Postgres as following

```
[core]
	engine = pg
[engine "pg"]
	target = db:pg:
	client = psql
``` 

Create database and changes to be managed by Sqitch

```
# create new Postgres database 
createdb flipr_test
# create sqitch change
sqitch add appschema -n 'Add schema for all flipr objects.'
# add SQL command to change file
echo "CREATE SCHEMA flipr;" >> /home/project/flipr/deploy/appschema.sql
# add SQL command to change-revert file
echo "DROP SCHEMA flipr;" >> /home/project/flipr/revert/appschema.sql
```

Apply db change with Squitch

```
sqitch deploy db:pg:flipr_test
```

Connet to local Postgres, database 'flipr_test' and check schema 'flipr' was created

```
pgcli flipr_test
SELECT schema_name FROM information_schema.schemata
```

Now you can open terminal back, and revert this change with the command `sqitch revert db:pg:flipr_test`.  

__Learn more__:
- [Sqitch Postgres tutorial](https://sqitch.org/docs/manual/sqitchtutorial/)

### yuniql

[Yuniql](https://github.com/rdagumampan/yuniql) is a data platform devops tool using migration-based and database-first 
delivery model. Migration-based as each changeset to the schema and seed data is a set of carefully prepared scripts 
controlled with a version number.  
To briefly demonstrate how Yuniql works, clone example Yuniql project

```
git clone https://github.com/rdagumampan/yuniql.git /home/project/yuniql
```

Create new database in the local Postgres instance

```
createdb helloyuniql
```

Apply to the local Postgres a Yuniql project in `/home/project/yuniql/samples/basic-postgresql-sample`

```
export YUNIQL_PLATFORM="postgresql"
export YUNIQL_WORKSPACE="/home/project/yuniql/samples/basic-postgresql-sample"
export YUNIQL_CONNECTION_STRING="Host=localhost;Port=5432;Username=abc;Password=;Database=helloyuniql"
```

Finally, run migrations

```
cd /home/project/yuniql/samples/basic-postgresql-sample
yuniql run -a --platform postgresql
```

Check the results

```
pgcli helloyuniql
SELECT * FROM regions
```

### migra

[migra](https://databaseci.com/docs/migra) is a schema comparison tool for PostgreSQL. 
Find differences in database schemas as easily as running a diff on two text files.
Migra makes schema changes almost automatic. Management of database migration deployments becomes much easier, 
faster, and more reliable. For example, compare two local Postgres databases

```
migra postgresql://abc:@localhost:5432/abc postgresql://abc:@localhost:5432/helloyuniql --unsafe
```

Migra will produce a set of SQL commands to make database "abc" have the same DDL as databas "helloyuniql". 


## Visualization tools

### dbdesigner
[dbdesigner](https://github.com/akreienbring/dbdesigner) a visual tool to create entity relationship diagrams and 
generate Postgres DDL code. Open dbdesigner from the Quickstart page.

<p align="center">
  <img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/dbdesigner.png" alt="dbdesigner" width="750">
</p> 

### schemaspy

[schemaspy](https://github.com/schemaspy/schemaspy) generates complete database documentation that looks great.

```
schemaspy -h localhost -p 5432 -d usda -u abc -o /home/static-server/usda-schemaspy
```

And open static file server on [http://localhost:8022/](http://localhost:8022/) folder usda-schemaspy

<p align="center">
  <img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/schemaspy.gif" alt="schemaspy" width="750">
</p> 

### Postgres explain visualizer

[Pev](https://github.com/AlexTatiyants/pev) helps to understand Postgres execution plans

<p align="center">
  <img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/pev.gif" alt="pev" width="750">
</p> 

### tbls

[tbls](https://github.com/k1LoW/tbls) - a tool for document a database, written in Go.
For example, generate Markdown docs  

```
tbls doc "postgres://abc@localhost:5432/usda?sslmode=disable" /home/static-server/usda-tbls
```

Now you can push these docks push to Github.

### pg_flame 

[pg_flame](https://github.com/mgartner/pg_flame) - a flamegraph generator for Postgres EXPLAIN ANALYZE output.

```
psql usda -qAtc 'EXPLAIN (ANALYZE, FORMAT JSON) SELECT *
   FROM nut_data N
   JOIN food_des F ON F.ndb_no = N.ndb_no
   JOIN datsrcln D ON D.nutr_no = N.nutr_no
   LEFT JOIN data_src ON data_src.datasrc_id = D.datasrc_id
   WHERE N.nutr_val > 20
       AND data_src.year > 1975 AND data_src.year < 1986;' \
    | pg_flame > /home/static-server/flamegraph.html
```

<p align="center">
  <img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/postgres-workspace/img/pgflame.png" alt="pgflame" width="750">
</p> 

## Other useful tools

### pg_insights

Convenient SQL for monitoring Postgres database health. Clone repo and use it to analyse Postgres databases

```
git clone https://github.com/lob/pg_insights /home/pg_insights
```

### sqlfluff

[Sqlfluff](https://www.sqlfluff.com/) is a moduler sql linter for humans.

```
echo "select * from t where id=1" >/tmp/test.sql
sqlfluff lint /tmp/test.sql
```

Results in:

```
== [/tmp/test.sql] FAIL
L:   1 | P:   1 | L044 | Query produces an unknown number of result columns.
L:   1 | P:  27 | L006 | Missing whitespace before =
L:   1 | P:  27 | L006 | Missing whitespace after =
```