CodeMirror/budibase
1
0
Fork 0
mirror of synced 2024-06-26 18:10:51 +12:00
Code Issues Projects Releases Wiki Activity

Merge branch 'develop' of github.com:Budibase/budibase into feature/automation-logs

Browse source
This commit is contained in:
mike12345567 2022-06-24 17:43:59 +01:00
parent a2dc3dc3b1 e3196adc18
commit dd6617465c
65 changed files with 842 additions and 262 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

75
.github/workflows/README.md vendored
View file

@ -6,7 +6,7 @@ Welcome to the budibase CI pipelines directory. This document details what each
## All CI Pipelines
### Note
- When running workflow dispatch jobs, ensure you always run them off the `master` branch. It defaults to `develop`, so double check before running any jobs.
- When running workflow dispatch jobs, ensure you always run them off the `master` branch. It defaults to `develop`, so double check before running any jobs. The exception to this case is the `deploy-release` job which requires the develop branch.
### Standard CI Build Job (budibase_ci.yml)
Triggers:
@ -115,4 +115,75 @@ This job is responsible for deploying to our production, cloud kubernetes enviro
### Rollback A Bad Cloud Deployment
- Kick off cloud deploy job
- Ensure you are running off master
- Enter the version number of the last known good version of budibase. For example `1.0.0`
- Enter the version number of the last known good version of budibase. For example `1.0.0`
## Pro
### Installing Pro
The pro package is always installed from source in our CI jobs.
This is done to prevent pro needing to be published prior to CI runs in budiabse. This is required for two reasons:
- To reduce developer need to manually bump versions, i.e:
- release pro, bump pro dep in budibase, now ci can run successfully
- The cyclic dependency on backend-core, i.e:
- pro depends on backend-core
- server depends on pro
- backend-core lives in the monorepo, so it can't be released independently to be used in pro
- therefore the only option is to pull pro from source and release it as a part of the monorepo release, as if it were a mono package
The install is performed using the same steps as local development, via the `yarn bootstrap` command, see the [Contributing Guide#Pro](../CONTRIBUTING.md#pro)
The branch to install pro from can vary depending on ref of the commit that triggered the budibase CI job. This is done to enable branches which have changes in both the monorepo and the pro repo to have their CI pass successfully.
This is done using the [pro/install.sh](../../scripts/pro/install.sh) script. The script will:
- Clone pro to it's default branch (`develop`)
- Check if the clone worked, on forked versions of budibase this will fail due to no access
- This is fine as the `yarn` command will install the version from NPM
- Community PRs should never touch pro so this will always work
- Checkout the `BRANCH` argument, if this fails fallback to `BASE_BRANCH`
- This enables the more complex case of a feature branch being merged to another feature branch, e.g.
- I am working on a branch `epic/stonks` which exists on budibase and pro.
- I want to merge a change to this branch in budibase from `feature/stonks-ui`, which only exists in budibase
- The base branch ensures that `epic/stonks` in pro will still be checked out for the CI run, rather than falling back to `develop`
- Run `yarn setup` to build and install dependencies
- `yarn`
- `yarn bootstrap`
- `yarn build`
- The will build .ts files, and also update the `main` and `types` of `package.json` to point to `dist` rather than src
- The build command will only ever work in CI, it is prevented in local dev
#### `BRANCH` and `BASE_BRANCH` arguments
These arguments are supplied by the various budibase build and release pipelines
- `budibase_ci`
- `BRANCH: ${{ github.event.pull_request.head.ref }}` -> The branch being merged
- `BASE_BRANCH: ${{ github.event.pull_request.base.ref}}` -> The base branch
- `release-develop`
- `BRANCH: develop` -> always use the `develop` branch in pro
- `release`
- `BRANCH: master` -> always use the `master` branch in pro
### Releasing Pro
After budibase dependencies have been released we will release the new version of pro to match the release version of budibase dependencies. This is to ensure that we are always keeping the version of `backend-core` in sync in the pro package and in budibase packages. Without this we could run into scenarios where different versions are being used when installed via `yarn` inside the docker images, creating very difficult to debug cases.
Pro is released using the [pro/release.sh](../../scripts/pro/release.sh) script. The script will:
- Inspect the `VERSION` from the `lerna.json` file in budibase
- Determine whether to use the `latest` or `develop` tag based on the command argument
- Go to pro directory
- install npm creds
- update the version of `backend-core` to be `VERSION`, the version just released by lerna
- publish to npm. Uses a `lerna publish` command, pro itself is a mono repo.
- force the version to be the same as `VERSION` to keep pro and budibase in sync
- reverts the changes to `main` and `types` in `package.json` that were made by the build step, to point back to source
- commit & push: `Prep next development iteration`
- Go to budibase
- Update to the new version of pro in `server` and `worker` so the latest pro version is used in the docker builds
- commit & push: `Update pro version to $VERSION`
#### `COMMAND` argument
This argument is supplied by the existing `release` and `release:develop` budibase commands, which invoke the pro release
- `release` will supply no command and default to use `latest`
- `release:develop` will supply `develop`

15
docs/CONTRIBUTING.md
View file

@ -190,6 +190,21 @@ yarn mode:account
```
### CI
An overview of the CI pipelines can be found [here](./workflows/README.md)
### Pro
@budibase/pro is the closed source package that supports licensed features in budibase. By default the package will be pulled from NPM and will not normally need to be touched in local development. If you require to update code inside the pro package it can be cloned to the same root level as budibase, e.g.
```
.
|_ budibase
|_ budibase-pro
```
Note that only budibase maintainers will be able to access the pro repo.
The `yarn bootstrap` command can be used to replace the NPM supplied dependency with the local source aware version. This is achieved using the `yarn link` command. To see specifically how dependencies are linked see [scripts/link-dependencies.sh](../scripts/link-dependencies.sh). The same link script is used to link dependencies to account-portal in local dev.
### Troubleshooting
Sometimes, things go wrong. This can be due to incompatible updates on the budibase platform. To clear down your development environment and start again follow **Step 6. Cleanup**, then proceed from **Step 3. Install and Build** in the setup guide above to create a fresh Budibase installation.

13
hosting/letsencrypt/certificate-renew.sh Normal file
View file

@ -0,0 +1,13 @@
#!/bin/bash
CUSTOM_DOMAIN="$1"
if [[ ! -z "${CUSTOM_DOMAIN}" ]]; then
certbot certonly --webroot --webroot-path="/var/www/html" \
--register-unsafely-without-email \
--domains $CUSTOM_DOMAIN \
--rsa-key-size 4096 \
--agree-tos \
--force-renewal
nginx -s reload
fi

24
hosting/letsencrypt/certificate-request.sh Normal file
View file

@ -0,0 +1,24 @@
#!/bin/bash
CUSTOM_DOMAIN="$1"
# Request from Lets Encrypt
certbot certonly --webroot --webroot-path="/var/www/html" \
--register-unsafely-without-email \
--domains $CUSTOM_DOMAIN \
--rsa-key-size 4096 \
--agree-tos \
--force-renewal
if (($? != 0)); then
echo "ERROR: certbot request failed for $CUSTOM_DOMAIN use http on port 80 - exiting"
nginx -s stop
exit 1
else
cp /app/letsencrypt/options-ssl-nginx.conf /etc/letsencrypt/options-ssl-nginx.conf
cp /app/letsencrypt/ssl-dhparams.pem /etc/letsencrypt/ssl-dhparams.pem
cp /app/letsencrypt/nginx-ssl.conf /etc/nginx/sites-available/nginx-ssl.conf
sed -i 's/CUSTOM_DOMAIN/$CUSTOM_DOMAIN/g' /etc/nginx/sites-available/nginx-ssl.conf
ln -s /etc/nginx/sites-available/nginx-ssl.conf /etc/nginx/sites-enabled/nginx-ssl.conf
echo "INFO: restart nginx after certbot request"
nginx -s reload
fi

94
hosting/letsencrypt/nginx-ssl.conf Normal file
View file

@ -0,0 +1,94 @@
server {
listen 443 ssl default_server;
listen [::]:443 ssl default_server;
server_name _;
ssl_certificate /etc/letsencrypt/live/CUSTOM_DOMAIN/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/CUSTOM_DOMAIN/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
client_max_body_size 1000m;
ignore_invalid_headers off;
proxy_buffering off;
# port_in_redirect off;
location ^~ /.well-known/acme-challenge/ {
default_type "text/plain";
root /var/www/html;
break;
}
location = /.well-known/acme-challenge/ {
return 404;
}
location /app {
proxy_pass http://127.0.0.1:4001;
}
location = / {
proxy_pass http://127.0.0.1:4001;
}
location ~ ^/(builder|app_) {
proxy_http_version 1.1;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:4001;
}
location ~ ^/api/(system|admin|global)/ {
proxy_pass http://127.0.0.1:4002;
}
location /worker/ {
proxy_pass http://127.0.0.1:4002;
rewrite ^/worker/(.*)$ /$1 break;
}
location /api/ {
# calls to the API are rate limited with bursting
limit_req zone=ratelimit burst=20 nodelay;
# 120s timeout on API requests
proxy_read_timeout 120s;
proxy_connect_timeout 120s;
proxy_send_timeout 120s;
proxy_http_version 1.1;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:4001;
}
location /db/ {
proxy_pass http://127.0.0.1:5984;
rewrite ^/db/(.*)$ /$1 break;
}
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass http://127.0.0.1:9000;
}
client_header_timeout 60;
client_body_timeout 60;
keepalive_timeout 60;
# gzip
gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_types text/plain text/css text/xml application/json application/javascript application/rss+xml application/atom+xml image/svg+xml;
}

13
hosting/letsencrypt/options-ssl-nginx.conf Normal file
View file

@ -0,0 +1,13 @@
# This file contains important security parameters. If you modify this file
# manually, Certbot will be unable to automatically provide future security
# updates. Instead, Certbot will print and log an error message with a path to
# the up-to-date file that you will need to refer to when manually updating
# this file.
ssl_session_cache shared:le_nginx_SSL:10m;
ssl_session_timeout 1440m;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers off;
ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384";

8
hosting/letsencrypt/ssl-dhparams.pem Normal file
View file

@ -0,0 +1,8 @@
-----BEGIN DH PARAMETERS-----
MIIBCAKCAQEA//////////+t+FRYortKmq/cViAnPTzx2LnFg84tNpWp4TZBFGQz
+8yTnc4kmz75fS/jY2MMddj2gbICrsRhetPfHtXV/WVhJDP1H18GbtCFY2VVPe0a
87VXE15/V8k1mE8McODmi3fipona8+/och3xWKE2rec1MKzKT0g6eXq8CrGCsyT7
YdEIqUuyyOP7uWrat2DX9GgdT0Kj3jlN9K5W7edjcrsZCwenyO4KbXCeAvzhzffi
7MA0BM0oNC9hkXL+nOmFg/+OTxIy7vKBg8P+OxtMb61zO7X8vC7CIAXFjvGDfRaD
ssbzSibBsu/6iGtCOGEoXJf//////////wIBAg==
-----END DH PARAMETERS-----

33
hosting/scripts/healthcheck.sh Normal file
View file

@ -0,0 +1,33 @@
#!/usr/bin/env bash
healthy=true
if [[ $(curl -Lfk -s -w "%{http_code}\n" http://localhost/ -o /dev/null) -ne 200 ]]; then
echo 'ERROR: Budibase is not running';
healthy=false
fi
if [[ $(curl -s -w "%{http_code}\n" http://localhost:4001/health -o /dev/null) -ne 200 ]]; then
echo 'ERROR: Budibase backend is not running';
healthy=false
fi
if [[ $(curl -s -w "%{http_code}\n" http://localhost:4002/health -o /dev/null) -ne 200 ]]; then
echo 'ERROR: Budibase worker is not running';
healthy=false
fi
if [[ $(curl -s -w "%{http_code}\n" http://localhost:5984/ -o /dev/null) -ne 200 ]]; then
echo 'ERROR: CouchDB is not running';
healthy=false
fi
if [[ $(redis-cli -a $REDIS_PASSWORD --no-auth-warning ping) != 'PONG' ]]; then
echo 'ERROR: Redis is down';
healthy=false
fi
# mino, clouseau,
if [ $healthy == true ]; then
exit 0
else
exit 1
fi

66
hosting/single/Dockerfile
View file

@ -1,7 +1,7 @@
FROM node:14-slim as build
# install node-gyp dependencies
RUN apt-get update && apt-get install -y --no-install-recommends g++ make python
RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends apt-utils cron g++ make python
# add pin script
WORKDIR /
@ -20,32 +20,36 @@ RUN node /pinVersions.js && yarn && yarn build && /cleanup.sh
FROM couchdb:3.2.1
ARG TARGETARCH amd64
COPY --from=build /app /app
COPY --from=build /worker /worker
ENV DEPLOYMENT_ENVIRONMENT=docker \
POSTHOG_TOKEN=phc_fg5I3nDOf6oJVMHSaycEhpPdlgS8rzXG2r6F2IpxCHS \
ENV \
APP_PORT=4001 \
ARCHITECTURE=amd \
BUDIBASE_ENVIRONMENT=PRODUCTION \
CLUSTER_PORT=80 \
COUCHDB_PASSWORD=budibase \
COUCHDB_USER=budibase \
COUCH_DB_URL=http://budibase:budibase@localhost:5984 \
BUDIBASE_ENVIRONMENT=PRODUCTION \
MINIO_URL=http://localhost:9000 \
REDIS_URL=localhost:6379 \
WORKER_URL=http://localhost:4002 \
CUSTOM_DOMAIN=budi001.custom.com \
DEPLOYMENT_ENVIRONMENT=docker \
INTERNAL_API_KEY=budibase \
JWT_SECRET=testsecret \
MINIO_ACCESS_KEY=budibase \
MINIO_SECRET_KEY=budibase \
SELF_HOSTED=1 \
CLUSTER_PORT=10000 \
MINIO_URL=http://localhost:9000 \
POSTHOG_TOKEN=phc_fg5I3nDOf6oJVMHSaycEhpPdlgS8rzXG2r6F2IpxCHS \
REDIS_PASSWORD=budibase \
ARCHITECTURE=amd \
APP_PORT=4001 \
WORKER_PORT=4002
REDIS_URL=localhost:6379 \
SELF_HOSTED=1 \
WORKER_PORT=4002 \
WORKER_URL=http://localhost:4002
# install base dependencies
RUN apt-get update && \
apt-get install software-properties-common wget -y && \
apt-get install -y software-properties-common wget nginx && \
apt-add-repository 'deb http://security.debian.org/debian-security stretch/updates main' && \
apt-get update
@ -53,20 +57,19 @@ RUN apt-get update && \
WORKDIR /nodejs
RUN curl -sL https://deb.nodesource.com/setup_16.x -o /tmp/nodesource_setup.sh && \
bash /tmp/nodesource_setup.sh && \
apt-get install libaio1 nodejs nginx openjdk-8-jdk redis-server unzip -y && \
apt-get install -y libaio1 nodejs nginx openjdk-8-jdk redis-server unzip && \
npm install --global yarn pm2
# setup nginx
ADD hosting/single/nginx.conf /etc/nginx
RUN mkdir /etc/nginx/logs && \
useradd www && \
touch /etc/nginx/logs/error.log && \
touch /etc/nginx/logs/nginx.pid
RUN mkdir -p /var/log/nginx && \
touch /var/log/nginx/error.log && \
touch /var/run/nginx.pid
WORKDIR /
RUN mkdir -p scripts/integrations/oracle
ADD packages/server/scripts/integrations/oracle scripts/integrations/oracle
RUN /bin/bash -e ./scripts/integrations/oracle/instantclient/linux/x86-64/install.sh
RUN /bin/bash -e ./scripts/integrations/oracle/instantclient/linux/install.sh
# setup clouseau
WORKDIR /
@ -87,20 +90,41 @@ ADD hosting/single/vm.args ./etc/
# setup minio
WORKDIR /minio
RUN wget https://dl.min.io/server/minio/release/linux-${ARCHITECTURE}64/minio && chmod +x minio
ADD scripts/install-minio.sh ./install.sh
RUN chmod +x install.sh && ./install.sh
# setup runner file
WORKDIR /
ADD hosting/single/runner.sh .
RUN chmod +x ./runner.sh
ADD hosting/scripts/healthcheck.sh .
RUN chmod +x ./healthcheck.sh
# cleanup cache
RUN yarn cache clean -f
EXPOSE 10000
EXPOSE 80
EXPOSE 443
VOLUME /opt/couchdb/data
VOLUME /minio
# setup letsencrypt certificate
RUN apt-get install -y certbot python3-certbot-nginx
ADD hosting/letsencrypt /app/letsencrypt
RUN chmod +x /app/letsencrypt/certificate-request.sh /app/letsencrypt/certificate-renew.sh
# Remove cached files
RUN rm -rf \
/root/.cache \
/root/.npm \
/root/.pip \
/usr/local/share/doc \
/usr/share/doc \
/usr/share/man \
/var/lib/apt/lists/* \
/tmp/*
HEALTHCHECK --interval=15s --timeout=15s --start-period=45s CMD "/healthcheck.sh"
# must set this just before running
ENV NODE_ENV=production
WORKDIR /

105
hosting/single/README.md Normal file
View file

@ -0,0 +1,105 @@
# Docker Single Image for Budibase
## Overview
As an alternative to running several docker containers via docker-compose, the files under ./hosting/single can be used to build a docker image containing all of the Budibase components (minio, couch, clouseau etc).
We call this the 'single image' container as the Dockerfile adds all the components to a single docker image.
## Usage
- Amend Environment Variables
- Build Requirements
- Build the Image
- Run the Container
### Amend Environment Variables
Edit the Dockerfile in this directory amending the environment variables to suit your usage. Pay particular attention to changing passwords.
The CUSTOM_DOMAIN variable will be used to request a certificate from LetsEncrypt and if successful you can point traffic to port 443. If you choose to use the CUSTOM_DOMAIN variable ensure that the DNS for your custom domain points to the public IP address where you are running Budibase - otherwise the certificate issuance will fail.
If you have other arrangements for a proxy in front of the single image container you can omit the CUSTOM_DOMAIN environment variable and the request to LetsEncrypt will be skipped. You can then point traffic to port 80.
### Build Requirements
We would suggest building the image with 6GB of RAM and 20GB of free disk space for build artifacts. The resulting image size will use approx 2GB of disk space.
### Build the Image
The guidance below is based on building the Budibase single image on Debian 11. If you use another distro or OS you will need to amend the commands to suit.
Install Node
Budibase requires a recent version of node (14+) than is in the base Debian repos so:
```
curl -sL https://deb.nodesource.com/setup_16.x | sudo bash -
apt install -y nodejs
node -v
```
Install yarn and lerna:
```
npm install -g yarn jest lerna
```
Install Docker
```
apt install -y docker.io
apt install -y python3-pip
pip3 install docker-compose
```
Check the versions of each installed version. This process was tested with the version numbers below so YMMV using anything else:
- Docker: 20.10.5
- docker-compose: 1.29.2
- node: 16.15.1
- yarn: 1.22.19
- lerna: 5.1.4
Clone the Budibase repo
```
git clone https://github.com/Budibase/budibase.git
cd budibase
```
Node setup:
```
node ./hosting/scripts/setup.js
yarn
yarn bootstrap
yarn build
```
Build the image from the Dockerfile:
```
yarn build:docker:single
```
If the docker build step fails run that step again manually with:
```
docker build --no-cache -t budibase:latest -f ./hosting/single/Dockerfile .
```
### Run the Container
```
docker run -d -p 80:80 -p 443:443 --name budibase budibase:latest
```
Where:
- -d runs the container in detached mode
- -p forwards ports from your host to the ports inside the container. If you are already using port 80 on your host for something else you can try running with an alternative port e.g. `-p 8080:80`
- --name is the name for the container as shown in `docker ps` and can be used with other docker commands e.g. `docker restart budibase`
When the container runs you should be able to access the container over http at your host address e.g. http://1.2.3.4/ or using your custom domain e.g. https://my.custom.domain/
When the Budibase UI appears you will be prompted to create an account to get started.
### Check
There are many things that could go wrong so if your container is not building or running as expected please check the following before opening a support issue.
Verify the healthcheck status of the container:
```
docker ps
```
Check the container logs:
```
docker logs budibase
```
### Support
This single image build is still a work-in-progress so if you open an issue please provide the following information:
- The OS and OS version you are building on
- The versions you are using of docker, docker-compose, yarn, node, lerna
- For build errors please provide zipped output
- For container errors please provide zipped container logs

19
hosting/single/nginx.conf
View file

@ -1,6 +1,6 @@
user www www;
error_log /etc/nginx/logs/error.log;
pid /etc/nginx/logs/nginx.pid;
user www-data www-data;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
worker_processes auto;
worker_rlimit_nofile 8192;
@ -33,14 +33,23 @@ http {
}
server {
listen 10000 default_server;
listen [::]:10000 default_server;
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
client_max_body_size 1000m;
ignore_invalid_headers off;
proxy_buffering off;
# port_in_redirect off;
location ^~ /.well-known/acme-challenge/ {
default_type "text/plain";
root /var/www/html;
break;
}
location = /.well-known/acme-challenge/ {
return 404;
}
location /app {
proxy_pass http://127.0.0.1:4001;
}

16
hosting/single/runner.sh
View file

@ -2,6 +2,15 @@ redis-server --requirepass $REDIS_PASSWORD &
/opt/clouseau/bin/clouseau &
/minio/minio server /minio &
/docker-entrypoint.sh /opt/couchdb/bin/couchdb &
/etc/init.d/nginx restart
if [[ ! -z "${CUSTOM_DOMAIN}" ]]; then
# Add monthly cron job to renew certbot certificate
echo -n "* * 2 * * root exec /app/letsencrypt/certificate-renew.sh ${CUSTOM_DOMAIN}" >> /etc/cron.d/certificate-renew
chmod +x /etc/cron.d/certificate-renew
# Request the certbot certificate
/app/letsencrypt/certificate-request.sh ${CUSTOM_DOMAIN}
fi
/etc/init.d/nginx restart
pushd app
pm2 start --name app "yarn run:docker"
@ -10,7 +19,6 @@ pushd worker
pm2 start --name worker "yarn run:docker"
popd
sleep 10
URL=http://${COUCHDB_USER}:${COUCHDB_PASSWORD}@localhost:5984
curl -X PUT ${URL}/_users
curl -X PUT ${URL}/_replicator
sleep infinity
curl -X PUT ${COUCH_DB_URL}/_users
curl -X PUT ${COUCH_DB_URL}/_replicator
sleep infinity

2
lerna.json
View file

@ -1,5 +1,5 @@
{
"version": "1.0.207-alpha.6",
"version": "1.0.207-alpha.10",
"npmClient": "yarn",
"packages": [
"packages/*"

1
package.json
View file

@ -62,6 +62,7 @@
"build:docker:develop": "node scripts/pinVersions && lerna run build:docker && npm run build:docker:proxy:compose && cd hosting/scripts/linux/ && ./release-to-docker-hub.sh develop && cd -",
"build:docker:airgap": "node hosting/scripts/airgapped/airgappedDockerBuild",
"build:digitalocean": "cd hosting/digitalocean && ./build.sh && cd -",
"build:docker:single:multiarch": "docker buildx build --platform linux/arm64,linux/amd64 -f hosting/single/Dockerfile -t budibase:latest .",
"build:docker:single:image": "docker build -f hosting/single/Dockerfile -t budibase:latest .",
"build:docker:single": "lerna run build && lerna run predocker && npm run build:docker:single:image",
"build:docs": "lerna run build:docs",

4
packages/backend-core/package.json
View file

@ -1,6 +1,6 @@
{
"name": "@budibase/backend-core",
"version": "1.0.207-alpha.6",
"version": "1.0.207-alpha.10",
"description": "Budibase backend core libraries used in server and worker",
"main": "dist/src/index.js",
"types": "dist/src/index.d.ts",
@ -20,7 +20,7 @@
"test:watch": "jest --watchAll"
},
"dependencies": {
"@budibase/types": "^1.0.207-alpha.6",
"@budibase/types": "^1.0.207-alpha.10",
"@techpass/passport-openidconnect": "0.3.2",
"aws-sdk": "2.1030.0",
"bcrypt": "5.0.1",

49
packages/backend-core/src/db/pouch.js
View file

@ -1,21 +1,42 @@
const PouchDB = require("pouchdb")
const env = require("../environment")
function getUrlInfo() {
let url = env.COUCH_DB_URL
let username, password, host
const [protocol, rest] = url.split("://")
if (url.includes("@")) {
const hostParts = rest.split("@")
host = hostParts[1]
const authParts = hostParts[0].split(":")
username = authParts[0]
password = authParts[1]
} else {
host = rest
exports.getUrlInfo = (url = env.COUCH_DB_URL) => {
let cleanUrl, username, password, host
if (url) {
// Ensure the URL starts with a protocol
const protoRegex = /^https?:\/\//i
if (!protoRegex.test(url)) {
url = `http://${url}`
}
// Split into protocol and remainder
const split = url.split("://")
const protocol = split[0]
const rest = split.slice(1).join("://")
// Extract auth if specified
if (url.includes("@")) {
// Split into host and remainder
let parts = rest.split("@")
host = parts[parts.length - 1]
let auth = parts.slice(0, -1).join("@")
// Split auth into username and password
if (auth.includes(":")) {
const authParts = auth.split(":")
username = authParts[0]
password = authParts.slice(1).join(":")
} else {
username = auth
}
} else {
host = rest
}
cleanUrl = `${protocol}://${host}`
}
return {
url: `${protocol}://${host}`,
url: cleanUrl,
auth: {
username,
password,
@ -24,7 +45,7 @@ function getUrlInfo() {
}
exports.getCouchInfo = () => {
const urlInfo = getUrlInfo()
const urlInfo = exports.getUrlInfo()
let username
let password
if (env.COUCH_DB_USERNAME) {

62
packages/backend-core/src/db/tests/pouch.spec.js Normal file
View file

@ -0,0 +1,62 @@
require("../../../tests/utilities/TestConfiguration")
const getUrlInfo = require("../pouch").getUrlInfo
describe("pouch", () => {
describe("Couch DB URL parsing", () => {
it("should handle a null Couch DB URL", () => {
const info = getUrlInfo(null)
expect(info.url).toBeUndefined()
expect(info.auth.username).toBeUndefined()
})
it("should be able to parse a basic Couch DB URL", () => {
const info = getUrlInfo("http://host.com")
expect(info.url).toBe("http://host.com")
expect(info.auth.username).toBeUndefined()
})
it("should be able to parse a Couch DB basic URL with HTTPS", () => {
const info = getUrlInfo("https://host.com")
expect(info.url).toBe("https://host.com")
expect(info.auth.username).toBeUndefined()
})
it("should be able to parse a basic Couch DB URL with a custom port", () => {
const info = getUrlInfo("https://host.com:1234")
expect(info.url).toBe("https://host.com:1234")
expect(info.auth.username).toBeUndefined()
})
it("should be able to parse a Couch DB URL with auth", () => {
const info = getUrlInfo("https://user:pass@host.com:1234")
expect(info.url).toBe("https://host.com:1234")
expect(info.auth.username).toBe("user")
expect(info.auth.password).toBe("pass")
})
it("should be able to parse a Couch DB URL with auth and special chars", () => {
const info = getUrlInfo("https://user:s:p@s://@://:d@;][~s@host.com:1234")
expect(info.url).toBe("https://host.com:1234")
expect(info.auth.username).toBe("user")
expect(info.auth.password).toBe("s:p@s://@://:d@;][~s")
})
it("should be able to parse a Couch DB URL without a protocol", () => {
const info = getUrlInfo("host.com:1234")
expect(info.url).toBe("http://host.com:1234")
expect(info.auth.username).toBeUndefined()
})
it("should be able to parse a Couch DB URL with auth and without a protocol", () => {
const info = getUrlInfo("user:s:p@s://@://:d@;][~s@host.com:1234")
expect(info.url).toBe("http://host.com:1234")
expect(info.auth.username).toBe("user")
expect(info.auth.password).toBe("s:p@s://@://:d@;][~s")
})
it("should be able to parse a Couch DB URL with only username auth", () => {
const info = getUrlInfo("https://user@host.com:1234")
expect(info.url).toBe("https://host.com:1234")
expect(info.auth.username).toBe("user")
expect(info.auth.password).toBeUndefined()
})
it("should be able to parse a Couch DB URL with only username auth and without a protocol", () => {
const info = getUrlInfo("user@host.com:1234")
expect(info.url).toBe("http://host.com:1234")
expect(info.auth.username).toBe("user")
expect(info.auth.password).toBeUndefined()
})
})
})

4
packages/bbui/package.json
View file

@ -1,7 +1,7 @@
{
"name": "@budibase/bbui",
"description": "A UI solution used in the different Budibase projects.",
"version": "1.0.207-alpha.6",
"version": "1.0.207-alpha.10",
"license": "MPL-2.0",
"svelte": "src/index.js",
"module": "dist/bbui.es.js",
@ -38,7 +38,7 @@
],
"dependencies": {
"@adobe/spectrum-css-workflow-icons": "^1.2.1",
"@budibase/string-templates": "^1.0.207-alpha.6",
"@budibase/string-templates": "^1.0.207-alpha.10",
"@spectrum-css/actionbutton": "^1.0.1",
"@spectrum-css/actiongroup": "^1.0.1",
"@spectrum-css/avatar": "^3.0.2",

2
packages/bbui/src/Typography/Detail.svelte
View file

@ -1,9 +1,7 @@
<script>
import "@spectrum-css/typography/dist/index-vars.css"
// Sizes
export let size = "M"
export let serif = false
</script>

2
packages/builder/cypress/support/commands.js
View file

@ -361,7 +361,7 @@ Cypress.Commands.add("createTable", (tableName, initialTable) => {
cy.get(`[data-cy="new-table"]`).click()
}
cy.wait(5000)
cy.get(".spectrum-Dialog-grid")
cy.get(".item")
.contains("Budibase DB")
.click({ force: true })
.then(() => {

10
packages/builder/package.json
View file

@ -1,6 +1,6 @@
{
"name": "@budibase/builder",
"version": "1.0.207-alpha.6",
"version": "1.0.207-alpha.10",
"license": "GPL-3.0",
"private": true,
"scripts": {
@ -69,10 +69,10 @@
}
},
"dependencies": {
"@budibase/bbui": "^1.0.207-alpha.6",
"@budibase/client": "^1.0.207-alpha.6",
"@budibase/frontend-core": "^1.0.207-alpha.6",
"@budibase/string-templates": "^1.0.207-alpha.6",
"@budibase/bbui": "^1.0.207-alpha.10",
"@budibase/client": "^1.0.207-alpha.10",
"@budibase/frontend-core": "^1.0.207-alpha.10",
"@budibase/string-templates": "^1.0.207-alpha.10",
"@sentry/browser": "5.19.1",
"@spectrum-css/page": "^3.0.1",
"@spectrum-css/vars": "^3.0.1",

116
packages/builder/src/builderStore/store/frontend.js
View file

@ -190,6 +190,7 @@ export const getFrontendStore = () => {
// Build array of promises to speed up bulk deletions
const promises = []
let deleteUrls = []
screensToDelete.forEach(screen => {
// Delete the screen
promises.push(
@ -199,14 +200,10 @@ export const getFrontendStore = () => {
})
)
// Remove links to this screen
promises.push(
store.actions.components.links.delete(
screen.routing.route,
screen.props._instanceName
)
)
deleteUrls.push(screen.routing.route)
})
promises.push(store.actions.links.delete(deleteUrls))
await Promise.all(promises)
const deletedIds = screensToDelete.map(screen => screen._id)
store.update(state => {
@ -578,89 +575,38 @@ export const getFrontendStore = () => {
})
await store.actions.preview.saveSelected()
},
links: {
save: async (url, title) => {
const layout = get(mainLayout)
if (!layout) {
return
}
},
links: {
save: async (url, title) => {
const layout = get(mainLayout)
if (!layout) {
return
}
// Add link setting to main layout
if (layout.props._component.endsWith("layout")) {
// If using a new SDK, add to the layout component settings
if (!layout.props.links) {
layout.props.links = []
}
layout.props.links.push({
text: title,
url,
})
} else {
// If using an old SDK, add to the navigation component
// TODO: remove this when we can assume everyone has updated
const nav = findComponentType(
layout.props,
"@budibase/standard-components/navigation"
)
if (!nav) {
return
}
// Add link setting to main layout
if (!layout.props.links) {
layout.props.links = []
}
layout.props.links.push({
text: title,
url,
})
let newLink
if (nav._children && nav._children.length) {
// Clone an existing link if one exists
newLink = cloneDeep(nav._children[0])
await store.actions.layouts.save(layout)
},
delete: async urls => {
const layout = get(mainLayout)
if (!layout?.props.links?.length) {
return
}
// Set our new props
newLink._id = Helpers.uuid()
newLink._instanceName = `${title} Link`
newLink.url = url
newLink.text = title
} else {
// Otherwise create vanilla new link
newLink = {
...store.actions.components.createInstance("link"),
url,
text: title,
_instanceName: `${title} Link`,
}
nav._children = [...nav._children, newLink]
}
}
// Filter out the URLs to delete
urls = Array.isArray(urls) ? urls : [urls]
layout.props.links = layout.props.links.filter(
link => !urls.includes(link.url)
)
// Save layout
await store.actions.layouts.save(layout)
},
delete: async (url, title) => {
const layout = get(mainLayout)
if (!layout) {
return
}
// Add link setting to main layout
if (layout.props._component.endsWith("layout")) {
// If using a new SDK, add to the layout component settings
layout.props.links = layout.props.links.filter(
link => !(link.text === title && link.url === url)
)
} else {
// If using an old SDK, add to the navigation component
// TODO: remove this when we can assume everyone has updated
const nav = findComponentType(
layout.props,
"@budibase/standard-components/navigation"
)
if (!nav) {
return
}
nav._children = nav._children.filter(
child => !(child.url === url && child.text === title)
)
}
// Save layout
await store.actions.layouts.save(layout)
},
await store.actions.layouts.save(layout)
},
},
settings: {

2
packages/builder/src/builderStore/store/screenTemplates/newRowScreen.js
View file

@ -15,7 +15,7 @@ export default function (tables) {
name: `${table.name} - New`,
create: () => createScreen(table),
id: NEW_ROW_TEMPLATE,
table: table.name,
table: table._id,
}
})
}

2
packages/builder/src/builderStore/store/screenTemplates/rowDetailScreen.js
View file

@ -17,7 +17,7 @@ export default function (tables) {
name: `${table.name} - Detail`,
create: () => createScreen(table),
id: ROW_DETAIL_TEMPLATE,
table: table.name,
table: table._id,
}
})
}

2
packages/builder/src/builderStore/store/screenTemplates/rowListScreen.js
View file

@ -10,7 +10,7 @@ export default function (tables) {
name: `${table.name} - List`,
create: () => createScreen(table),
id: ROW_LIST_TEMPLATE,
table: table.name,
table: table._id,
}
})
}

80
packages/builder/src/components/backend/DatasourceNavigator/modals/CreateDatasourceModal.svelte
View file

@ -5,12 +5,13 @@
Body,
Layout,
Detail,
Heading,
notifications,
} from "@budibase/bbui"
import { onMount } from "svelte"
import ICONS from "../icons"
import { API } from "api"
import { IntegrationNames, IntegrationTypes } from "constants/backend"
import { IntegrationTypes } from "constants/backend"
import CreateTableModal from "components/backend/TableNavigator/modals/CreateTableModal.svelte"
import DatasourceConfigModal from "components/backend/DatasourceNavigator/modals/DatasourceConfigModal.svelte"
import GoogleDatasourceConfigModal from "components/backend/DatasourceNavigator/modals/GoogleDatasourceConfigModal.svelte"
@ -118,7 +119,7 @@
<Modal bind:this={modal}>
<ModalContent
disabled={!Object.keys(integration).length}
title="Data"
title="Add data source"
confirmText="Continue"
showSecondaryButton={showImportButton}
secondaryButtonText="Import"
@ -129,27 +130,25 @@
chooseNextModal()
}}
>
<Layout noPadding>
<Body size="S"
>All apps need data. You can connect to a data source below, or add data
to your app using Budibase's built-in database.
</Body>
<Layout noPadding gap="XS">
<Body size="S">Get started with Budibase DB</Body>
<div
class:selected={integration.type === IntegrationTypes.INTERNAL}
on:click={() => selectIntegration(IntegrationTypes.INTERNAL)}
class="item hoverable"
>
<div class="item-body">
<svelte:component this={ICONS.BUDIBASE} height="18" width="18" />
<span class="icon-spacing"> <Body size="S">Budibase DB</Body></span>
<div class="item-body with-type">
<svelte:component this={ICONS.BUDIBASE} height="20" width="20" />
<div class="text">
<Heading size="XXS">Budibase DB</Heading>
<Detail size="S" class="type">Non-relational</Detail>
</div>
</div>
</div>
</Layout>
<Layout gap="XS" noPadding>
<div class="title-spacing">
<Detail size="S">Connect to data source</Detail>
</div>
<Layout noPadding gap="XS">
<Body size="S">Connect to an external data source</Body>
<div class="item-list">
{#each Object.entries(integrations).filter(([key]) => key !== IntegrationTypes.INTERNAL) as [integrationType, schema]}
<div
@ -157,18 +156,18 @@
on:click={() => selectIntegration(integrationType)}
class="item hoverable"
>
<div class="item-body">
<div class="item-body" class:with-type={!!schema.type}>
<svelte:component
this={ICONS[integrationType]}
height="18"
width="18"
height="20"
width="20"
/>
<span class="icon-spacing">
<Body size="S"
>{schema.name || IntegrationNames[integrationType]}</Body
></span
>
<div class="text">
<Heading size="XXS">{schema.friendlyName}</Heading>
{#if schema.type}
<Detail size="S">{schema.type || ""}</Detail>
{/if}
</div>
</div>
</div>
{/each}
@ -178,13 +177,6 @@
</Modal>
<style>
.icon-spacing {
margin-left: var(--spacing-m);
}
.item-body {
display: flex;
margin-left: var(--spacing-m);
}
.item-list {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));
@ -195,21 +187,35 @@
cursor: pointer;
display: grid;
grid-gap: var(--spectrum-alias-grid-margin-xsmall);
padding: var(--spectrum-alias-item-padding-s);
padding: var(--spectrum-alias-item-padding-s)
var(--spectrum-alias-item-padding-m);
background: var(--spectrum-alias-background-color-secondary);
transition: 0.3s all;
transition: background 0.13s ease-out;
border: solid var(--spectrum-alias-border-color);
border-radius: 5px;
box-sizing: border-box;
border-width: 2px;
}
.selected {
.item:hover,
.item.selected {
background: var(--spectrum-alias-background-color-tertiary);
}
.item:hover,
.selected {
background: var(--spectrum-alias-background-color-tertiary);
.item-body {
display: flex;
flex-direction: row;
justify-content: flex-start;
align-items: center;
gap: var(--spacing-m);
}
.item-body.with-type {
align-items: flex-start;
}
.item-body.with-type :global(svg) {
margin-top: 4px;
}
.text :global(.spectrum-Detail) {
color: var(--spectrum-global-color-gray-700);
}
</style>

16
packages/builder/src/components/design/NavigationPanel/DatasourceModal.svelte
View file

@ -14,14 +14,14 @@
let selectedScreens = [...initalScreens]
const toggleScreenSelection = (table, datasource) => {
if (selectedScreens.find(s => s.table === table.name)) {
if (selectedScreens.find(s => s.table === table._id)) {
selectedScreens = selectedScreens.filter(
screen => screen.table !== table.name
screen => screen.table !== table._id
)
} else {
let partialTemplates = getTemplates($store, $tables.list).reduce(
(acc, template) => {
if (template.table === table.name) {
if (template.table === table._id) {
template.datasource = datasource.name
acc.push(template)
}
@ -88,7 +88,7 @@
<div
class="data-source-entry"
class:selected={selectedScreens.find(
x => x.table === table.name
x => x.table === table._id
)}
on:click={() => toggleScreenSelection(table, datasource)}
>
@ -102,8 +102,7 @@
<use xlink:href="#spectrum-icon-18-Table" />
</svg>
{table.name}
{#if selectedScreens.find(x => x.table === table.name)}
{#if selectedScreens.find(x => x.table === table._id)}
<span class="data-source-check">
<Icon size="S" name="CheckmarkCircle" />
</span>
@ -116,7 +115,7 @@
<div
class="data-source-entry"
class:selected={selectedScreens.find(
x => x.table === datasource.entities[table_key].name
x => x.table === datasource.entities[table_key]._id
)}
on:click={() =>
toggleScreenSelection(
@ -134,8 +133,7 @@
<use xlink:href="#spectrum-icon-18-Table" />
</svg>
{datasource.entities[table_key].name}
{#if selectedScreens.find(x => x.table === datasource.entities[table_key].name)}
{#if selectedScreens.find(x => x.table === datasource.entities[table_key]._id)}
<span class="data-source-check">
<Icon size="S" name="CheckmarkCircle" />
</span>

3
packages/builder/src/components/design/NavigationPanel/ScreenWizard.svelte
View file

@ -66,7 +66,7 @@
// Add link in layout for list screens
if (screen.props._instanceName.endsWith("List")) {
await store.actions.components.links.save(
await store.actions.links.save(
screen.routing.route,
screen.routing.route.split("/")[1]
)
@ -131,6 +131,7 @@
const screens = selectedTemplates.map(template => {
let screenTemplate = template.create()
screenTemplate.datasource = template.datasource
screenTemplate.autoTableId = template.table
return screenTemplate
})
await createScreens({ screens, screenAccessRole })

63
packages/builder/src/components/design/PropertiesPanel/PropertyControls/ButtonActionEditor/actions/ExportData.svelte
View file

@ -1,27 +1,18 @@
<script>
import { Label, Select, Body } from "@budibase/bbui"
import { findAllMatchingComponents } from "builderStore/componentUtils"
import { Label, Select, Body, Multiselect } from "@budibase/bbui"
import {
findAllMatchingComponents,
findComponent,
} from "builderStore/componentUtils"
import { currentAsset } from "builderStore"
import { onMount } from "svelte"
import {
getDatasourceForProvider,
getSchemaForDatasource,
} from "builderStore/dataBinding"
export let parameters
$: tables = findAllMatchingComponents($currentAsset?.props, component =>
component._component.endsWith("table")
).map(table => ({
label: table._instanceName,
value: table._id,
}))
$: tableBlocks = findAllMatchingComponents($currentAsset?.props, component =>
component._component.endsWith("tableblock")
).map(block => ({
label: block._instanceName,
value: `${block._id}-table`,
}))
$: componentOptions = tables.concat(tableBlocks)
const FORMATS = [
{
label: "CSV",
@ -33,6 +24,32 @@
},
]
$: tables = findAllMatchingComponents($currentAsset?.props, component =>
component._component.endsWith("table")
).map(table => ({
label: table._instanceName,
value: table._id,
}))
$: tableBlocks = findAllMatchingComponents($currentAsset?.props, component =>
component._component.endsWith("tableblock")
).map(block => ({
label: block._instanceName,
value: `${block._id}-table`,
}))
$: componentOptions = tables.concat(tableBlocks)
$: columnOptions = getColumnOptions(parameters.tableComponentId)
const getColumnOptions = tableId => {
// Strip block suffix if block component
if (tableId?.includes("-")) {
tableId = tableId.split("-")[0]
}
const selectedTable = findComponent($currentAsset?.props, tableId)
const datasource = getDatasourceForProvider($currentAsset, selectedTable)
const { schema } = getSchemaForDatasource($currentAsset, datasource)
return Object.keys(schema || {})
}
onMount(() => {
if (!parameters.type) {
parameters.type = "csv"
@ -53,10 +70,16 @@
<Select
bind:value={parameters.tableComponentId}
options={componentOptions}
on:change={() => (parameters.columns = [])}
/>
<Label small>Export as</Label>
<Select bind:value={parameters.type} options={FORMATS} />
<Label small>Export columns</Label>
<Multiselect
placeholder="All columns"
bind:value={parameters.columns}
options={columnOptions}
/>
</div>
</div>
@ -80,7 +103,7 @@
display: grid;
column-gap: var(--spacing-xs);
row-gap: var(--spacing-s);
grid-template-columns: 70px 1fr;
grid-template-columns: 90px 1fr;
align-items: center;
}
</style>

2
packages/cli/package.json
View file

@ -1,6 +1,6 @@
{
"name": "@budibase/cli",
"version": "1.0.207-alpha.6",
"version": "1.0.207-alpha.10",
"description": "Budibase CLI, for developers, self hosting and migrations.",
"main": "src/index.js",
"bin": {

8
packages/client/package.json
View file

@ -1,6 +1,6 @@
{
"name": "@budibase/client",
"version": "1.0.207-alpha.6",
"version": "1.0.207-alpha.10",
"license": "MPL-2.0",
"module": "dist/budibase-client.js",
"main": "dist/budibase-client.js",
@ -19,9 +19,9 @@
"dev:builder": "rollup -cw"
},
"dependencies": {
"@budibase/bbui": "^1.0.207-alpha.6",
"@budibase/frontend-core": "^1.0.207-alpha.6",
"@budibase/string-templates": "^1.0.207-alpha.6",
"@budibase/bbui": "^1.0.207-alpha.10",
"@budibase/frontend-core": "^1.0.207-alpha.10",
"@budibase/string-templates": "^1.0.207-alpha.10",
"@spectrum-css/button": "^3.0.3",
"@spectrum-css/card": "^3.0.3",
"@spectrum-css/divider": "^1.0.3",

1
packages/client/src/utils/buttonActions.js
View file

@ -270,6 +270,7 @@ const exportDataHandler = async action => {
tableId: selection.tableId,
rows: selection.selectedRows,
format: action.parameters.type,
columns: action.parameters.columns,
})
download(data, `${selection.tableId}.${action.parameters.type}`)
} catch (error) {

4
packages/frontend-core/package.json
View file

@ -1,12 +1,12 @@
{
"name": "@budibase/frontend-core",
"version": "1.0.207-alpha.6",
"version": "1.0.207-alpha.10",
"description": "Budibase frontend core libraries used in builder and client",
"author": "Budibase",
"license": "MPL-2.0",
"svelte": "src/index.js",
"dependencies": {
"@budibase/bbui": "^1.0.207-alpha.6",
"@budibase/bbui": "^1.0.207-alpha.10",
"lodash": "^4.17.21",
"svelte": "^3.46.2"
}

5
packages/frontend-core/src/api/rows.js
View file

@ -65,12 +65,15 @@ export const buildRowEndpoints = API => ({
* Exports rows.
* @param tableId the table ID to export the rows from
* @param rows the array of rows to export
* @param format the format to export (csv or json)
* @param columns which columns to export (all if undefined)
*/
exportRows: async ({ tableId, rows, format }) => {
exportRows: async ({ tableId, rows, format, columns }) => {
return await API.post({
url: `/api/${tableId}/rows/exportRows?format=${format}`,
body: {
rows,
columns,
},
parseResponse: async response => {
return await response.text()

12
packages/server/package.json
View file

@ -1,7 +1,7 @@
{
"name": "@budibase/server",
"email": "hi@budibase.com",
"version": "1.0.207-alpha.6",
"version": "1.0.207-alpha.10",
"description": "Budibase Web Server",
"main": "src/index.ts",
"repository": {
@ -77,11 +77,11 @@
"license": "GPL-3.0",
"dependencies": {
"@apidevtools/swagger-parser": "10.0.3",
"@budibase/backend-core": "^1.0.207-alpha.6",
"@budibase/client": "^1.0.207-alpha.6",
"@budibase/pro": "1.0.207-alpha.6",
"@budibase/string-templates": "^1.0.207-alpha.6",
"@budibase/types": "^1.0.207-alpha.6",
"@budibase/backend-core": "^1.0.207-alpha.10",
"@budibase/client": "^1.0.207-alpha.10",
"@budibase/pro": "1.0.207-alpha.10",
"@budibase/string-templates": "^1.0.207-alpha.10",
"@budibase/types": "^1.0.207-alpha.10",
"@bull-board/api": "3.7.0",
"@bull-board/koa": "3.9.4",
"@elastic/elasticsearch": "7.10.0",

BIN
packages/server/scripts/integrations/oracle/instantclient/linux/arm64/basiclite-19.10.zip Normal file
View file

Binary file not shown.

23
packages/server/scripts/integrations/oracle/instantclient/linux/arm64/install.sh Executable file
View file

@ -0,0 +1,23 @@
#!/bin/bash
# Must be root to continue
if [[ $(id -u) -ne 0 ]] ; then echo "Please run as root" ; exit 1 ; fi
# Allow for re-runs
rm -rf /opt/oracle
echo "Installing oracle instant client"
# copy and unzip package
mkdir -p /opt/oracle
cp scripts/integrations/oracle/instantclient/linux/arm64/basiclite-19.10.zip /opt/oracle
cd /opt/oracle
unzip -qq basiclite-19.10.zip -d .
rm *.zip
mv instantclient* instantclient
# update runtime link path
sh -c "echo /opt/oracle/instantclient > /etc/ld.so.conf.d/oracle-instantclient.conf"
ldconfig /etc/ld.so.conf.d
echo "Installation complete"

10
packages/server/scripts/integrations/oracle/instantclient/linux/install.sh Executable file
View file

@ -0,0 +1,10 @@
#!/bin/bash
SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]:-$0}"; )" &> /dev/null && pwd 2> /dev/null; )"
if [[ $TARGETARCH == arm* ]] ;
then
echo "Installing ARM Oracle instant client..."
$SCRIPT_DIR/arm64/install.sh
else
echo "Installing x86-64 Oracle instant client..."
$SCRIPT_DIR/x86-64/install.sh
fi

21
packages/server/src/api/controllers/row/external.js
View file

@ -157,7 +157,8 @@ exports.validate = async () => {
exports.exportRows = async ctx => {
const { datasourceId } = breakExternalTableId(ctx.params.tableId)
const db = getAppDB()
let format = ctx.query.format
const format = ctx.query.format
const { columns } = ctx.request.body
const datasource = await db.get(datasourceId)
if (!datasource || !datasource.entities) {
ctx.throw(400, "Datasource has not been configured for plus API.")
@ -171,13 +172,27 @@ exports.exportRows = async ctx => {
}
let result = await exports.search(ctx)
let headers = Object.keys(result.rows[0])
let rows = []
// Filter data to only specified columns if required
if (columns && columns.length) {
for (let i = 0; i < result.rows.length; i++) {
rows[i] = {}
for (let column of columns) {
rows[i][column] = result.rows[i][column]
}
}
} else {
rows = result.rows
}
let headers = Object.keys(rows[0])
const exporter = exporters[format]
const filename = `export.${format}`
// send down the file
ctx.attachment(filename)
return apiFileReturn(exporter(headers, result.rows))
return apiFileReturn(exporter(headers, rows))
}
exports.fetchEnrichedRow = async ctx => {

1
packages/server/src/definitions/common.ts
View file

@ -14,6 +14,7 @@ export interface Application extends Base {
export interface FieldSchema {
// TODO: replace with field types enum when done
type: string
externalType?: string
fieldName?: string
name: string
tableId?: string

1
packages/server/src/definitions/datasource.ts
View file

@ -94,6 +94,7 @@ export interface Integration {
relationships?: boolean
description: string
friendlyName: string
type?: string
datasource: {}
query: {
[key: string]: QueryDefinition

1
packages/server/src/integrations/airtable.ts
View file

@ -18,6 +18,7 @@ module AirtableModule {
description:
"Airtable is a spreadsheet-database hybrid, with the features of a database but applied to a spreadsheet.",
friendlyName: "Airtable",
type: "Spreadsheet",
datasource: {
apiKey: {
type: DatasourceFieldTypes.PASSWORD,

1
packages/server/src/integrations/arangodb.ts
View file

@ -19,6 +19,7 @@ module ArangoModule {
const SCHEMA: Integration = {
docs: "https://github.com/arangodb/arangojs",
friendlyName: "ArangoDB",
type: "Non-relational",
description:
"ArangoDB is a scalable open-source multi-model database natively supporting graph, document and search. All supported data models & access patterns can be combined in queries allowing for maximal flexibility. ",
datasource: {

36
packages/server/src/integrations/base/sql.ts
View file

@ -89,6 +89,28 @@ function parseFilters(filters: SearchFilters | undefined): SearchFilters {
return filters
}
function generateSelectStatement(
json: QueryJson,
knex: Knex
): (string | Knex.Raw)[] {
const { resource, meta } = json
const schema = meta?.table?.schema
return resource.fields.map(field => {
const fieldNames = field.split(/\./g)
const tableName = fieldNames[0]
const columnName = fieldNames[1]
if (columnName && knex.client.config.client === SqlClients.POSTGRES) {
const externalType = schema?.[columnName].externalType
if (externalType?.includes("money")) {
return knex.raw(
`"${tableName}"."${columnName}"::money::numeric as "${field}"`
)
}
}
return `${field} as ${field}`
})
}
class InternalBuilder {
private readonly client: string
@ -216,9 +238,7 @@ class InternalBuilder {
}
addRelationships(
knex: Knex,
query: KnexQuery,
fields: string | string[],
fromTable: string,
relationships: RelationshipsJson[] | undefined
): KnexQuery {
@ -323,12 +343,12 @@ class InternalBuilder {
if (!resource) {
resource = { fields: [] }
}
let selectStatement: string | string[] = "*"
let selectStatement: string | (string | Knex.Raw)[] = "*"
// handle select
if (resource.fields && resource.fields.length > 0) {
// select the resources as the format "table.columnName" - this is what is provided
// by the resource builder further up
selectStatement = resource.fields.map(field => `${field} as ${field}`)
selectStatement = generateSelectStatement(json, knex)
}
let foundLimit = limit || BASE_LIMIT
// handle pagination
@ -363,13 +383,7 @@ class InternalBuilder {
preQuery = this.addSorting(preQuery, json)
}
// handle joins
query = this.addRelationships(
knex,
preQuery,
selectStatement,
tableName,
relationships
)
query = this.addRelationships(preQuery, tableName, relationships)
return this.addFilters(query, filters, { relationship: true })
}

1
packages/server/src/integrations/couchdb.ts
View file

@ -16,6 +16,7 @@ module CouchDBModule {
const SCHEMA: Integration = {
docs: "https://docs.couchdb.org/en/stable/",
friendlyName: "CouchDB",
type: "Non-relational",
description:
"Apache CouchDB is an open-source document-oriented NoSQL database, implemented in Erlang.",
datasource: {

1
packages/server/src/integrations/dynamodb.ts
View file

@ -21,6 +21,7 @@ module DynamoModule {
description:
"Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale.",
friendlyName: "DynamoDB",
type: "Non-relational",
datasource: {
region: {
type: DatasourceFieldTypes.STRING,

1
packages/server/src/integrations/elasticsearch.ts
View file

@ -17,6 +17,7 @@ module ElasticsearchModule {
description:
"Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.",
friendlyName: "ElasticSearch",
type: "Non-relational",
datasource: {
url: {
type: DatasourceFieldTypes.STRING,

1
packages/server/src/integrations/firebase.ts
View file

@ -16,6 +16,7 @@ module Firebase {
const SCHEMA: Integration = {
docs: "https://firebase.google.com/docs/firestore/quickstart",
friendlyName: "Firestore",
type: "Non-relational",
description:
"Cloud Firestore is a flexible, scalable database for mobile, web, and server development from Firebase and Google Cloud.",
datasource: {

1
packages/server/src/integrations/googlesheets.ts
View file

@ -49,6 +49,7 @@ module GoogleSheetsModule {
description:
"Create and collaborate on online spreadsheets in real-time and from any device. ",
friendlyName: "Google Sheets",
type: "Spreadsheet",
datasource: {
spreadsheetId: {
display: "Google Sheet URL",

1
packages/server/src/integrations/microsoftSqlServer.ts
View file

@ -44,6 +44,7 @@ module MSSQLModule {
description:
"Microsoft SQL Server is a relational database management system developed by Microsoft. ",
friendlyName: "MS SQL Server",
type: "Relational",
datasource: {
user: {
type: DatasourceFieldTypes.STRING,

1
packages/server/src/integrations/mongodb.ts
View file

@ -24,6 +24,7 @@ module MongoDBModule {
const SCHEMA: Integration = {
docs: "https://github.com/mongodb/node-mongodb-native",
friendlyName: "MongoDB",
type: "Non-relational",
description:
"MongoDB is a general purpose, document-based, distributed database built for modern application developers and for the cloud era.",
datasource: {

1
packages/server/src/integrations/mysql.ts
View file

@ -36,6 +36,7 @@ module MySQLModule {
docs: "https://github.com/sidorares/node-mysql2",
plus: true,
friendlyName: "MySQL",
type: "Relational",
description:
"MySQL Database Service is a fully managed database service to deploy cloud-native applications. ",
datasource: {

1
packages/server/src/integrations/oracle.ts
View file

@ -40,6 +40,7 @@ module OracleModule {
docs: "https://github.com/oracle/node-oracledb",
plus: true,
friendlyName: "Oracle",
type: "Relational",
description:
"Oracle Database is an object-relational database management system developed by Oracle Corporation",
datasource: {

1
packages/server/src/integrations/postgres.ts
View file

@ -47,6 +47,7 @@ module PostgresModule {
docs: "https://node-postgres.com",
plus: true,
friendlyName: "PostgreSQL",
type: "Relational",
description:
"PostgreSQL, also known as Postgres, is a free and open-source relational database management system emphasizing extensibility and SQL compliance.",
datasource: {

1
packages/server/src/integrations/redis.ts
View file

@ -17,6 +17,7 @@ module RedisModule {
docs: "https://redis.io/docs/",
description: "",
friendlyName: "Redis",
type: "Non-relational",
datasource: {
host: {
type: "string",

1
packages/server/src/integrations/rest.ts
View file

@ -64,6 +64,7 @@ module RestModule {
description:
"With the REST API datasource, you can connect, query and pull data from multiple REST APIs. You can then use the retrieved data to build apps.",
friendlyName: "REST API",
type: "API",
datasource: {
url: {
type: DatasourceFieldTypes.STRING,

1
packages/server/src/integrations/s3.ts
View file

@ -17,6 +17,7 @@ module S3Module {
description:
"Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance.",
friendlyName: "Amazon S3",
type: "Object store",
datasource: {
region: {
type: "string",

1
packages/server/src/integrations/snowflake.ts
View file

@ -16,6 +16,7 @@ module SnowflakeModule {
description:
"Snowflake is a solution for data warehousing, data lakes, data engineering, data science, data application development, and securely sharing and consuming shared data.",
friendlyName: "Snowflake",
type: "Relational",
datasource: {
account: {
type: "string",

24
packages/server/yarn.lock
View file

@ -1080,11 +1080,12 @@
resolved "https://registry.yarnpkg.com/@bcoe/v8-coverage/-/v8-coverage-0.2.3.tgz#75a2e8b51cb758a7553d6804a5932d7aace75c39"
integrity sha512-0hYQ8SB4Db5zvZB4axdMHGwEaQjkZzFjQiN9LVYvIFB2nSUHW9tYpxWriPrWDASIxiaXax83REcLxuSdnGPZtw==
"@budibase/backend-core@1.0.207-alpha.3":
version "1.0.207-alpha.3"
resolved "https://registry.yarnpkg.com/@budibase/backend-core/-/backend-core-1.0.207-alpha.3.tgz#98bced0575ec4e2b158239a8c73b39ca2d816719"
integrity sha512-DU4X6jJ+DfhzOv4TTa1w4Dk5ZEdlK/z1joCTruT+SGM5qI75bXrGeol5OX2OaEbNKtXFKJ1zeVTmBCYcu7OFUg==
"@budibase/backend-core@1.0.207-alpha.6":
version "1.0.207-alpha.6"
resolved "https://registry.yarnpkg.com/@budibase/backend-core/-/backend-core-1.0.207-alpha.6.tgz#47fed5cc78686e23951a050479c777673f725c17"
integrity sha512-mB3i9TyNbdlE8TmsAWGXhphwLRlrBd2bDfvOYTz3CP7xzql1FXGoWfOqA87vNaGBDrtOyQQnmbYeTc3Tn2OHcg==
dependencies:
"@budibase/types" "^1.0.207-alpha.6"
"@techpass/passport-openidconnect" "0.3.2"
aws-sdk "2.1030.0"
bcrypt "5.0.1"
@ -1160,12 +1161,12 @@
svelte-flatpickr "^3.2.3"
svelte-portal "^1.0.0"
"@budibase/pro@1.0.207-alpha.3":
version "1.0.207-alpha.3"
resolved "https://registry.yarnpkg.com/@budibase/pro/-/pro-1.0.207-alpha.3.tgz#9bde845ceb685f1b43286a124620c21fdf891a01"
integrity sha512-WFEMujpKTVAMvAgLBnMdw8ou9PxsbM4Oa9Dq+DAUsWpPACsMWOProyHLsdRxJyvHlgGfwVjo5MEusvStjI4j6g==
"@budibase/pro@1.0.207-alpha.6":
version "1.0.207-alpha.6"
resolved "https://registry.yarnpkg.com/@budibase/pro/-/pro-1.0.207-alpha.6.tgz#04a81281beeb230c0c1a1f48237a94e1150a7851"
integrity sha512-IDQdKHaojfGlL8xLSQ1gYrLyipgUYPJ6Mjrrp8TcWnpwTOA2Wtzen31E5HG6YxZU8g8rN6k9S0Nsp88JKOGSrg==
dependencies:
"@budibase/backend-core" "1.0.207-alpha.3"
"@budibase/backend-core" "1.0.207-alpha.6"
node-fetch "^2.6.1"
"@budibase/standard-components@^0.9.139":
@ -1186,6 +1187,11 @@
svelte-apexcharts "^1.0.2"
svelte-flatpickr "^3.1.0"
"@budibase/types@^1.0.207-alpha.6":
version "1.0.208"
resolved "https://registry.yarnpkg.com/@budibase/types/-/types-1.0.208.tgz#c45cb494fb5b85229e15a34c6ac1805bae5be867"
integrity sha512-zKIHg6TGK+soVxMNZNrGypP3DCrd3jhlUQEFeQ+rZR6/tCue1G74bjzydY5FjnLEsXeLH1a0hkS5HulTmvQ2bA==
"@bull-board/api@3.7.0":
version "3.7.0"
resolved "https://registry.yarnpkg.com/@bull-board/api/-/api-3.7.0.tgz#231f687187c0cb34e0b97f463917b6aaeb4ef6af"

2
packages/string-templates/package.json
View file

@ -1,6 +1,6 @@
{
"name": "@budibase/string-templates",
"version": "1.0.207-alpha.6",
"version": "1.0.207-alpha.10",
"description": "Handlebars wrapper for Budibase templating.",
"main": "src/index.cjs",
"module": "dist/bundle.mjs",

2
packages/types/package.json
View file

@ -1,6 +1,6 @@
{
"name": "@budibase/types",
"version": "1.0.207-alpha.6",
"version": "1.0.207-alpha.10",
"description": "Budibase types",
"main": "dist/index.js",
"types": "dist/index.d.ts",

10
packages/worker/package.json
View file

@ -1,7 +1,7 @@
{
"name": "@budibase/worker",
"email": "hi@budibase.com",
"version": "1.0.207-alpha.6",
"version": "1.0.207-alpha.10",
"description": "Budibase background service",
"main": "src/index.ts",
"repository": {
@ -34,10 +34,10 @@
"author": "Budibase",
"license": "GPL-3.0",
"dependencies": {
"@budibase/backend-core": "^1.0.207-alpha.6",
"@budibase/pro": "1.0.207-alpha.6",
"@budibase/string-templates": "^1.0.207-alpha.6",
"@budibase/types": "^1.0.207-alpha.6",
"@budibase/backend-core": "^1.0.207-alpha.10",
"@budibase/pro": "1.0.207-alpha.10",
"@budibase/string-templates": "^1.0.207-alpha.10",
"@budibase/types": "^1.0.207-alpha.10",
"@koa/router": "8.0.8",
"@sentry/node": "6.17.7",
"@techpass/passport-openidconnect": "0.3.2",

4
scripts/buildx-multiarch.sh Executable file
View file

@ -0,0 +1,4 @@
#!/bin/bash
sudo apt-get install -y qemu qemu-user-static
docker buildx create --name budibase
docker buildx use budibase

13
scripts/cleanup.sh
View file

@ -1,11 +1,16 @@
#!/bin/bash
dir=$(pwd)
mv dist /
mv package.json /
declare -a keep=("dist" "package.json" "yarn.lock" "client" "builder" "build" "pm2.config.js" "docker_run.sh")
for moveDir in "${keep[@]}"
do
mv $moveDir / 2>/dev/null
done
cd /
rm -r $dir
mkdir $dir
mv /dist $dir
mv /package.json $dir
for keepDir in "${keep[@]}"
do
mv /$keepDir $dir/ 2>/dev/null
done
cd $dir
NODE_ENV=production yarn

8
scripts/install-minio.sh Executable file
View file

@ -0,0 +1,8 @@
#!/bin/bash
if [[ $TARGETARCH == arm* ]] ;
then
wget https://dl.min.io/server/minio/release/linux-arm64/minio
else
wget https://dl.min.io/server/minio/release/linux-amd64/minio
fi
chmod +x minio