CodeMirror/budibase
1
0
Fork 0
mirror of synced 2024-06-02 18:44:54 +12:00
Code Issues Projects Releases Wiki Activity

Merge branch 'develop' into cypress-testing

Browse source
This commit is contained in:
Mitch-Budibase 2022-06-27 16:15:00 +01:00
parent 9c37fd1aa6 00888d4572
commit 3f597ba6e0
65 changed files with 1514 additions and 719 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

75
.github/workflows/README.md vendored
View file

@ -6,7 +6,7 @@ Welcome to the budibase CI pipelines directory. This document details what each
## All CI Pipelines
### Note
- When running workflow dispatch jobs, ensure you always run them off the `master` branch. It defaults to `develop`, so double check before running any jobs.
- When running workflow dispatch jobs, ensure you always run them off the `master` branch. It defaults to `develop`, so double check before running any jobs. The exception to this case is the `deploy-release` job which requires the develop branch.
### Standard CI Build Job (budibase_ci.yml)
Triggers:
@ -115,4 +115,75 @@ This job is responsible for deploying to our production, cloud kubernetes enviro
### Rollback A Bad Cloud Deployment
- Kick off cloud deploy job
- Ensure you are running off master
- Enter the version number of the last known good version of budibase. For example `1.0.0`
- Enter the version number of the last known good version of budibase. For example `1.0.0`
## Pro
### Installing Pro
The pro package is always installed from source in our CI jobs.
This is done to prevent pro needing to be published prior to CI runs in budiabse. This is required for two reasons:
- To reduce developer need to manually bump versions, i.e:
- release pro, bump pro dep in budibase, now ci can run successfully
- The cyclic dependency on backend-core, i.e:
- pro depends on backend-core
- server depends on pro
- backend-core lives in the monorepo, so it can't be released independently to be used in pro
- therefore the only option is to pull pro from source and release it as a part of the monorepo release, as if it were a mono package
The install is performed using the same steps as local development, via the `yarn bootstrap` command, see the [Contributing Guide#Pro](../CONTRIBUTING.md#pro)
The branch to install pro from can vary depending on ref of the commit that triggered the budibase CI job. This is done to enable branches which have changes in both the monorepo and the pro repo to have their CI pass successfully.
This is done using the [pro/install.sh](../../scripts/pro/install.sh) script. The script will:
- Clone pro to it's default branch (`develop`)
- Check if the clone worked, on forked versions of budibase this will fail due to no access
- This is fine as the `yarn` command will install the version from NPM
- Community PRs should never touch pro so this will always work
- Checkout the `BRANCH` argument, if this fails fallback to `BASE_BRANCH`
- This enables the more complex case of a feature branch being merged to another feature branch, e.g.
- I am working on a branch `epic/stonks` which exists on budibase and pro.
- I want to merge a change to this branch in budibase from `feature/stonks-ui`, which only exists in budibase
- The base branch ensures that `epic/stonks` in pro will still be checked out for the CI run, rather than falling back to `develop`
- Run `yarn setup` to build and install dependencies
- `yarn`
- `yarn bootstrap`
- `yarn build`
- The will build .ts files, and also update the `main` and `types` of `package.json` to point to `dist` rather than src
- The build command will only ever work in CI, it is prevented in local dev
#### `BRANCH` and `BASE_BRANCH` arguments
These arguments are supplied by the various budibase build and release pipelines
- `budibase_ci`
- `BRANCH: ${{ github.event.pull_request.head.ref }}` -> The branch being merged
- `BASE_BRANCH: ${{ github.event.pull_request.base.ref}}` -> The base branch
- `release-develop`
- `BRANCH: develop` -> always use the `develop` branch in pro
- `release`
- `BRANCH: master` -> always use the `master` branch in pro
### Releasing Pro
After budibase dependencies have been released we will release the new version of pro to match the release version of budibase dependencies. This is to ensure that we are always keeping the version of `backend-core` in sync in the pro package and in budibase packages. Without this we could run into scenarios where different versions are being used when installed via `yarn` inside the docker images, creating very difficult to debug cases.
Pro is released using the [pro/release.sh](../../scripts/pro/release.sh) script. The script will:
- Inspect the `VERSION` from the `lerna.json` file in budibase
- Determine whether to use the `latest` or `develop` tag based on the command argument
- Go to pro directory
- install npm creds
- update the version of `backend-core` to be `VERSION`, the version just released by lerna
- publish to npm. Uses a `lerna publish` command, pro itself is a mono repo.
- force the version to be the same as `VERSION` to keep pro and budibase in sync
- reverts the changes to `main` and `types` in `package.json` that were made by the build step, to point back to source
- commit & push: `Prep next development iteration`
- Go to budibase
- Update to the new version of pro in `server` and `worker` so the latest pro version is used in the docker builds
- commit & push: `Update pro version to $VERSION`
#### `COMMAND` argument
This argument is supplied by the existing `release` and `release:develop` budibase commands, which invoke the pro release
- `release` will supply no command and default to use `latest`
- `release:develop` will supply `develop`

15
docs/CONTRIBUTING.md
View file

@ -190,6 +190,21 @@ yarn mode:account
```
### CI
An overview of the CI pipelines can be found [here](./workflows/README.md)
### Pro
@budibase/pro is the closed source package that supports licensed features in budibase. By default the package will be pulled from NPM and will not normally need to be touched in local development. If you require to update code inside the pro package it can be cloned to the same root level as budibase, e.g.
```
.
|_ budibase
|_ budibase-pro
```
Note that only budibase maintainers will be able to access the pro repo.
The `yarn bootstrap` command can be used to replace the NPM supplied dependency with the local source aware version. This is achieved using the `yarn link` command. To see specifically how dependencies are linked see [scripts/link-dependencies.sh](../scripts/link-dependencies.sh). The same link script is used to link dependencies to account-portal in local dev.
### Troubleshooting
Sometimes, things go wrong. This can be due to incompatible updates on the budibase platform. To clear down your development environment and start again follow **Step 6. Cleanup**, then proceed from **Step 3. Install and Build** in the setup guide above to create a fresh Budibase installation.

5
hosting/letsencrypt/certificate-request.sh
View file

@ -10,15 +10,14 @@ certbot certonly --webroot --webroot-path="/var/www/html" \
if (($? != 0)); then
echo "ERROR: certbot request failed for $CUSTOM_DOMAIN use http on port 80 - exiting"
nginx -s stop
exit 1
else
cp /app/letsencrypt/options-ssl-nginx.conf /etc/letsencrypt/options-ssl-nginx.conf
cp /app/letsencrypt/ssl-dhparams.pem /etc/letsencrypt/ssl-dhparams.pem
cp /app/letsencrypt/nginx-ssl.conf /etc/nginx/sites-available/nginx-ssl.conf
sed -i 's/CUSTOM_DOMAIN/$CUSTOM_DOMAIN/g' /etc/nginx/sites-available/nginx-ssl.conf
sed -i "s/CUSTOM_DOMAIN/$CUSTOM_DOMAIN/g" /etc/nginx/sites-available/nginx-ssl.conf
ln -s /etc/nginx/sites-available/nginx-ssl.conf /etc/nginx/sites-enabled/nginx-ssl.conf
echo "INFO: restart nginx after certbot request"
nginx -s reload
/etc/init.d/nginx restart
fi

2
hosting/letsencrypt/nginx-ssl.conf
View file

@ -6,6 +6,7 @@ server {
ssl_certificate_key /etc/letsencrypt/live/CUSTOM_DOMAIN/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
client_max_body_size 1000m;
ignore_invalid_headers off;
proxy_buffering off;
@ -91,4 +92,5 @@ server {
gzip_proxied any;
gzip_comp_level 6;
gzip_types text/plain text/css text/xml application/json application/javascript application/rss+xml application/atom+xml image/svg+xml;
}

17
hosting/scripts/build-target-paths.sh Normal file
View file

@ -0,0 +1,17 @@
#!/bin/bash
echo ${TARGETBUILD} > /buildtarget.txt
if [[ "${TARGETBUILD}" = "aas" ]]; then
# Azure AppService uses /home for persisent data & SSH on port 2222
mkdir -p /home/budibase/{minio,couchdb}
mkdir -p /home/budibase/couchdb/data
chown -R couchdb:couchdb /home/budibase/couchdb/
apt update
apt-get install -y openssh-server
sed -i 's#dir=/opt/couchdb/data/search#dir=/home/budibase/couchdb/data/search#' /opt/clouseau/clouseau.ini
sed -i 's#/minio/minio server /minio &#/minio/minio server /home/budibase/minio &#' /runner.sh
sed -i 's#database_dir = ./data#database_dir = /home/budibase/couchdb/data#' /opt/couchdb/etc/default.ini
sed -i 's#view_index_dir = ./data#view_index_dir = /home/budibase/couchdb/data#' /opt/couchdb/etc/default.ini
sed -i "s/#Port 22/Port 2222/" /etc/ssh/sshd_config
/etc/init.d/ssh restart
fi

7
hosting/scripts/healthcheck.sh
View file

@ -25,6 +25,13 @@ if [[ $(redis-cli -a $REDIS_PASSWORD --no-auth-warning ping) != 'PONG' ]]; then
healthy=false
fi
# mino, clouseau,
nginx -t -q
NGINX_STATUS=$?
if [[ $NGINX_STATUS -gt 0 ]]; then
echo 'ERROR: Nginx config problem';
healthy=false
fi
if [ $healthy == true ]; then
exit 0

16
hosting/single/Dockerfile
View file

@ -19,8 +19,12 @@ ADD packages/worker .
RUN node /pinVersions.js && yarn && yarn build && /cleanup.sh
FROM couchdb:3.2.1
# TARGETARCH can be amd64 or arm e.g. docker build --build-arg TARGETARCH=amd64
ARG TARGETARCH amd64
#TARGETBUILD can be set to single (for single docker image) or aas (for azure app service)
# e.g. docker build --build-arg TARGETBUILD=aas ....
ARG TARGETBUILD single
ENV TARGETBUILD $TARGETBUILD
COPY --from=build /app /app
COPY --from=build /worker /worker
@ -33,7 +37,7 @@ ENV \
COUCHDB_PASSWORD=budibase \
COUCHDB_USER=budibase \
COUCH_DB_URL=http://budibase:budibase@localhost:5984 \
CUSTOM_DOMAIN=budi001.custom.com \
# CUSTOM_DOMAIN=budi001.custom.com \
DEPLOYMENT_ENVIRONMENT=docker \
INTERNAL_API_KEY=budibase \
JWT_SECRET=testsecret \
@ -44,6 +48,7 @@ ENV \
REDIS_PASSWORD=budibase \
REDIS_URL=localhost:6379 \
SELF_HOSTED=1 \
TARGETBUILD=$TARGETBUILD \
WORKER_PORT=4002 \
WORKER_URL=http://localhost:4002
@ -62,6 +67,7 @@ RUN curl -sL https://deb.nodesource.com/setup_16.x -o /tmp/nodesource_setup.sh &
# setup nginx
ADD hosting/single/nginx.conf /etc/nginx
ADD hosting/single/nginx-default-site.conf /etc/nginx/sites-enabled/default
RUN mkdir -p /var/log/nginx && \
touch /var/log/nginx/error.log && \
touch /var/run/nginx.pid
@ -100,6 +106,12 @@ RUN chmod +x ./runner.sh
ADD hosting/scripts/healthcheck.sh .
RUN chmod +x ./healthcheck.sh
ADD hosting/scripts/build-target-paths.sh .
RUN chmod +x ./build-target-paths.sh
# For Azure App Service install SSH & point data locations to /home
RUN /build-target-paths.sh
# cleanup cache
RUN yarn cache clean -f

35
hosting/single/README.md
View file

@ -4,7 +4,6 @@
As an alternative to running several docker containers via docker-compose, the files under ./hosting/single can be used to build a docker image containing all of the Budibase components (minio, couch, clouseau etc).
We call this the 'single image' container as the Dockerfile adds all the components to a single docker image.
## Usage
- Amend Environment Variables
@ -22,9 +21,9 @@ If you have other arrangements for a proxy in front of the single image containe
We would suggest building the image with 6GB of RAM and 20GB of free disk space for build artifacts. The resulting image size will use approx 2GB of disk space.
### Build the Image
The guidance below is based on building the Budibase single image on Debian 11. If you use another distro or OS you will need to amend the commands to suit.
Install Node
Budibase requires a recent version of node (14+) than is in the base Debian repos so:
The guidance below is based on building the Budibase single image on Debian 11 and AlmaLinux 8. If you use another distro or OS you will need to amend the commands to suit.
#### Install Node
Budibase requires a more recent version of node (14+) than is available in the base Debian repos so:
```
curl -sL https://deb.nodesource.com/setup_16.x | sudo bash -
@ -35,25 +34,26 @@ Install yarn and lerna:
```
npm install -g yarn jest lerna
```
Install Docker
#### Install Docker
```
apt install -y docker.io
apt install -y python3-pip
pip3 install docker-compose
```
Check the versions of each installed version. This process was tested with the version numbers below so YMMV using anything else:
- Docker: 20.10.5
- docker-compose: 1.29.2
- node: 16.15.1
- yarn: 1.22.19
- lerna: 5.1.4
#### Get the Code
Clone the Budibase repo
```
git clone https://github.com/Budibase/budibase.git
cd budibase
```
#### Setup Node
Node setup:
```
node ./hosting/scripts/setup.js
@ -61,15 +61,20 @@ yarn
yarn bootstrap
yarn build
```
Build the image from the Dockerfile:
#### Build Image
The following yarn command does some prep and then runs the docker build command:
```
yarn build:docker:single
```
If the docker build step fails run that step again manually with:
If the docker build step fails try running that step again manually with:
```
docker build --no-cache -t budibase:latest -f ./hosting/single/Dockerfile .
docker build --build-arg TARGETARCH=amd --no-cache -t budibase:latest -f ./hosting/single/Dockerfile .
```
#### Azure App Services
Azure have some specific requirements for running a container in their App Service. Specifically, installation of SSH to port 2222 and data storage under /home. If you would like to build a budibase container for Azure App Service add the build argument shown below setting it to 'aas'. You can remove the CUSTOM_DOMAIN env variable from the Dockerfile too as Azure terminate SSL before requests reach the container.
```
docker build --build-arg TARGETARCH=amd --build-arg TARGETBUILD=aas -t budibase:latest -f ./hosting/single/Dockerfile .
```
### Run the Container
@ -85,6 +90,9 @@ When the container runs you should be able to access the container over http at
When the Budibase UI appears you will be prompted to create an account to get started.
### Podman
The single image container builds fine when using podman in place of docker. You may be prompted for the registry to use for the CouchDB image and the HEALTHCHECK parameter is not OCI compliant so is ignored.
### Check
There are many things that could go wrong so if your container is not building or running as expected please check the following before opening a support issue.
Verify the healthcheck status of the container:
@ -96,7 +104,6 @@ Check the container logs:
docker logs budibase
```
### Support
This single image build is still a work-in-progress so if you open an issue please provide the following information:
- The OS and OS version you are building on

94
hosting/single/nginx-default-site.conf Normal file
View file

@ -0,0 +1,94 @@
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
client_max_body_size 1000m;
ignore_invalid_headers off;
proxy_buffering off;
# port_in_redirect off;
location ^~ /.well-known/acme-challenge/ {
default_type "text/plain";
root /var/www/html;
break;
}
location = /.well-known/acme-challenge/ {
return 404;
}
location /app {
proxy_pass http://127.0.0.1:4001;
}
location = / {
proxy_pass http://127.0.0.1:4001;
}
location ~ ^/(builder|app_) {
proxy_http_version 1.1;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:4001;
}
location ~ ^/api/(system|admin|global)/ {
proxy_pass http://127.0.0.1:4002;
}
location /worker/ {
proxy_pass http://127.0.0.1:4002;
rewrite ^/worker/(.*)$ /$1 break;
}
location /api/ {
# calls to the API are rate limited with bursting
limit_req zone=ratelimit burst=20 nodelay;
# 120s timeout on API requests
proxy_read_timeout 120s;
proxy_connect_timeout 120s;
proxy_send_timeout 120s;
proxy_http_version 1.1;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:4001;
}
location /db/ {
proxy_pass http://127.0.0.1:5984;
rewrite ^/db/(.*)$ /$1 break;
}
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass http://127.0.0.1:9000;
}
client_header_timeout 60;
client_body_timeout 60;
keepalive_timeout 60;
# gzip
gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_types text/plain text/css text/xml application/json application/javascript application/rss+xml application/atom+xml image/svg+xml;
}

90
hosting/single/nginx.conf
View file

@ -32,94 +32,6 @@ http {
default "upgrade";
}
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
client_max_body_size 1000m;
ignore_invalid_headers off;
proxy_buffering off;
# port_in_redirect off;
include /etc/nginx/sites-enabled/*;
location ^~ /.well-known/acme-challenge/ {
default_type "text/plain";
root /var/www/html;
break;
}
location = /.well-known/acme-challenge/ {
return 404;
}
location /app {
proxy_pass http://127.0.0.1:4001;
}
location = / {
proxy_pass http://127.0.0.1:4001;
}
location ~ ^/(builder|app_) {
proxy_http_version 1.1;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:4001;
}
location ~ ^/api/(system|admin|global)/ {
proxy_pass http://127.0.0.1:4002;
}
location /worker/ {
proxy_pass http://127.0.0.1:4002;
rewrite ^/worker/(.*)$ /$1 break;
}
location /api/ {
# calls to the API are rate limited with bursting
limit_req zone=ratelimit burst=20 nodelay;
# 120s timeout on API requests
proxy_read_timeout 120s;
proxy_connect_timeout 120s;
proxy_send_timeout 120s;
proxy_http_version 1.1;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:4001;
}
location /db/ {
proxy_pass http://127.0.0.1:5984;
rewrite ^/db/(.*)$ /$1 break;
}
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass http://127.0.0.1:9000;
}
client_header_timeout 60;
client_body_timeout 60;
keepalive_timeout 60;
# gzip
gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_types text/plain text/css text/xml application/json application/javascript application/rss+xml application/atom+xml image/svg+xml;
}
}

2
hosting/single/test.sh
View file

@ -1,4 +1,4 @@
#!/bin/bash
id=$(docker run -t -d -p 10000:10000 budibase:latest)
id=$(docker run -t -d -p 80:80 budibase:latest)
docker exec -it $id bash
docker kill $id

2
lerna.json
View file

@ -1,5 +1,5 @@
{
"version": "1.0.207-alpha.6",
"version": "1.0.212-alpha.0",
"npmClient": "yarn",
"packages": [
"packages/*"

1
packages/backend-core/cache.js
View file

@ -3,5 +3,6 @@ const generic = require("./src/cache/generic")
module.exports = {
user: require("./src/cache/user"),
app: require("./src/cache/appMetadata"),
writethrough: require("./src/cache/writethrough"),
...generic,
}

5
packages/backend-core/package.json
View file

@ -1,6 +1,6 @@
{
"name": "@budibase/backend-core",
"version": "1.0.207-alpha.6",
"version": "1.0.212-alpha.0",
"description": "Budibase backend core libraries used in server and worker",
"main": "dist/src/index.js",
"types": "dist/src/index.d.ts",
@ -20,7 +20,7 @@
"test:watch": "jest --watchAll"
},
"dependencies": {
"@budibase/types": "^1.0.207-alpha.6",
"@budibase/types": "^1.0.212-alpha.0",
"@techpass/passport-openidconnect": "0.3.2",
"aws-sdk": "2.1030.0",
"bcrypt": "5.0.1",
@ -63,6 +63,7 @@
"@types/koa": "2.0.52",
"@types/node": "14.18.20",
"@types/node-fetch": "2.6.1",
"@types/pouchdb": "6.4.0",
"@types/redlock": "4.0.3",
"@types/semver": "7.3.7",
"@types/tar-fs": "2.0.1",

2
packages/backend-core/redis.js
View file

@ -1,5 +1,5 @@
module.exports = {
Client: require("./src/redis"),
utils: require("./src/redis/utils"),
clients: require("./src/redis/authRedis"),
clients: require("./src/redis/init"),
}

2
packages/backend-core/src/cache/appMetadata.js vendored
View file

@ -1,4 +1,4 @@
const redis = require("../redis/authRedis")
const redis = require("../redis/init")
const { doWithDB } = require("../db")
const { DocumentTypes } = require("../db/constants")

92
packages/backend-core/src/cache/base/index.ts vendored Normal file
View file

@ -0,0 +1,92 @@
import { getTenantId } from "../../context"
import redis from "../../redis/init"
import RedisWrapper from "../../redis"
function generateTenantKey(key: string) {
const tenantId = getTenantId()
return `${key}:${tenantId}`
}
export = class BaseCache {
client: RedisWrapper | undefined
constructor(client: RedisWrapper | undefined = undefined) {
this.client = client
}
async getClient() {
return !this.client ? await redis.getCacheClient() : this.client
}
async keys(pattern: string) {
const client = await this.getClient()
return client.keys(pattern)
}
/**
* Read only from the cache.
*/
async get(key: string, opts = { useTenancy: true }) {
key = opts.useTenancy ? generateTenantKey(key) : key
const client = await this.getClient()
return client.get(key)
}
/**
* Write to the cache.
*/
async store(
key: string,
value: any,
ttl: number | null = null,
opts = { useTenancy: true }
) {
key = opts.useTenancy ? generateTenantKey(key) : key
const client = await this.getClient()
await client.store(key, value, ttl)
}
/**
* Remove from cache.
*/
async delete(key: string, opts = { useTenancy: true }) {
key = opts.useTenancy ? generateTenantKey(key) : key
const client = await this.getClient()
return client.delete(key)
}
/**
* Read from the cache. Write to the cache if not exists.
*/
async withCache(
key: string,
ttl: number,
fetchFn: any,
opts = { useTenancy: true }
) {
const cachedValue = await this.get(key, opts)
if (cachedValue) {
return cachedValue
}
try {
const fetchedValue = await fetchFn()
await this.store(key, fetchedValue, ttl, opts)
return fetchedValue
} catch (err) {
console.error("Error fetching before cache - ", err)
throw err
}
}
async bustCache(key: string, opts = { client: null }) {
const client = await this.getClient()
try {
await client.delete(generateTenantKey(key))
} catch (err) {
console.error("Error busting cache - ", err)
throw err
}
}
}

75
packages/backend-core/src/cache/generic.js vendored
View file

@ -1,5 +1,6 @@
const redis = require("../redis/authRedis")
const { getTenantId } = require("../context")
const BaseCache = require("./base")
const GENERIC = new BaseCache()
exports.CacheKeys = {
CHECKLIST: "checklist",
@ -16,67 +17,13 @@ exports.TTL = {
ONE_DAY: 86400,
}
function generateTenantKey(key) {
const tenantId = getTenantId()
return `${key}:${tenantId}`
function performExport(funcName) {
return (...args) => GENERIC[funcName](...args)
}
exports.keys = async pattern => {
const client = await redis.getCacheClient()
return client.keys(pattern)
}
/**
* Read only from the cache.
*/
exports.get = async (key, opts = { useTenancy: true }) => {
key = opts.useTenancy ? generateTenantKey(key) : key
const client = await redis.getCacheClient()
const value = await client.get(key)
return value
}
/**
* Write to the cache.
*/
exports.store = async (key, value, ttl, opts = { useTenancy: true }) => {
key = opts.useTenancy ? generateTenantKey(key) : key
const client = await redis.getCacheClient()
await client.store(key, value, ttl)
}
exports.delete = async (key, opts = { useTenancy: true }) => {
key = opts.useTenancy ? generateTenantKey(key) : key
const client = await redis.getCacheClient()
return client.delete(key)
}
/**
* Read from the cache. Write to the cache if not exists.
*/
exports.withCache = async (key, ttl, fetchFn, opts = { useTenancy: true }) => {
const cachedValue = await exports.get(key, opts)
if (cachedValue) {
return cachedValue
}
try {
const fetchedValue = await fetchFn()
await exports.store(key, fetchedValue, ttl, opts)
return fetchedValue
} catch (err) {
console.error("Error fetching before cache - ", err)
throw err
}
}
exports.bustCache = async key => {
const client = await redis.getCacheClient()
try {
await client.delete(generateTenantKey(key))
} catch (err) {
console.error("Error busting cache - ", err)
throw err
}
}
exports.keys = performExport("keys")
exports.get = performExport("get")
exports.store = performExport("store")
exports.delete = performExport("delete")
exports.withCache = performExport("withCache")
exports.bustCache = performExport("bustCache")

59
packages/backend-core/src/cache/tests/writethrough.spec.js vendored Normal file
View file

@ -0,0 +1,59 @@
require("../../../tests/utilities/TestConfiguration")
const { Writethrough } = require("../writethrough")
const { dangerousGetDB } = require("../../db")
const tk = require("timekeeper")
const START_DATE = Date.now()
tk.freeze(START_DATE)
const DELAY = 5000
const db = dangerousGetDB("test")
const db2 = dangerousGetDB("test2")
const writethrough = new Writethrough(db, DELAY), writethrough2 = new Writethrough(db2, DELAY)
describe("writethrough", () => {
describe("put", () => {
let first
it("should be able to store, will go to DB", async () => {
const response = await writethrough.put({ _id: "test", value: 1 })
const output = await db.get(response.id)
first = output
expect(output.value).toBe(1)
})
it("second put shouldn't update DB", async () => {
const response = await writethrough.put({ ...first, value: 2 })
const output = await db.get(response.id)
expect(first._rev).toBe(output._rev)
expect(output.value).toBe(1)
})
it("should put it again after delay period", async () => {
tk.freeze(START_DATE + DELAY + 1)
const response = await writethrough.put({ ...first, value: 3 })
const output = await db.get(response.id)
expect(response.rev).not.toBe(first._rev)
expect(output.value).toBe(3)
})
})
describe("get", () => {
it("should be able to retrieve", async () => {
const response = await writethrough.get("test")
expect(response.value).toBe(3)
})
})
describe("same doc, different databases (tenancy)", () => {
it("should be able to two different databases", async () => {
const resp1 = await writethrough.put({ _id: "db1", value: "first" })
const resp2 = await writethrough2.put({ _id: "db1", value: "second" })
expect(resp1.rev).toBeDefined()
expect(resp2.rev).toBeDefined()
expect((await db.get("db1")).value).toBe("first")
expect((await db2.get("db1")).value).toBe("second")
})
})
})

2
packages/backend-core/src/cache/user.js vendored
View file

@ -1,4 +1,4 @@
const redis = require("../redis/authRedis")
const redis = require("../redis/init")
const { getTenantId, lookupTenantId, doWithGlobalDB } = require("../tenancy")
const env = require("../environment")
const accounts = require("../cloud/accounts")

120
packages/backend-core/src/cache/writethrough.ts vendored Normal file
View file

@ -0,0 +1,120 @@
import BaseCache from "./base"
import { getWritethroughClient } from "../redis/init"
const DEFAULT_WRITE_RATE_MS = 10000
let CACHE: BaseCache | null = null
interface CacheItem {
doc: any
lastWrite: number
}
async function getCache() {
if (!CACHE) {
const client = await getWritethroughClient()
CACHE = new BaseCache(client)
}
return CACHE
}
function makeCacheKey(db: PouchDB.Database, key: string) {
return db.name + key
}
function makeCacheItem(doc: any, lastWrite: number | null = null): CacheItem {
return { doc, lastWrite: lastWrite || Date.now() }
}
export async function put(
db: PouchDB.Database,
doc: any,
writeRateMs: number = DEFAULT_WRITE_RATE_MS
) {
const cache = await getCache()
const key = doc._id
let cacheItem: CacheItem | undefined = await cache.get(makeCacheKey(db, key))
const updateDb = !cacheItem || cacheItem.lastWrite < Date.now() - writeRateMs
let output = doc
if (updateDb) {
const writeDb = async (toWrite: any) => {
// doc should contain the _id and _rev
const response = await db.put(toWrite)
output = {
...doc,
_id: response.id,
_rev: response.rev,
}
}
try {
await writeDb(doc)
} catch (err: any) {
if (err.status !== 409) {
throw err
} else {
// get the rev, update over it - this is risky, may change in future
const readDoc = await db.get(doc._id)
doc._rev = readDoc._rev
await writeDb(doc)
}
}
}
// if we are updating the DB then need to set the lastWrite to now
cacheItem = makeCacheItem(output, updateDb ? null : cacheItem?.lastWrite)
await cache.store(makeCacheKey(db, key), cacheItem)
return { ok: true, id: output._id, rev: output._rev }
}
export async function get(db: PouchDB.Database, id: string): Promise<any> {
const cache = await getCache()
const cacheKey = makeCacheKey(db, id)
let cacheItem: CacheItem = await cache.get(cacheKey)
if (!cacheItem) {
const doc = await db.get(id)
cacheItem = makeCacheItem(doc)
await cache.store(cacheKey, cacheItem)
}
return cacheItem.doc
}
export async function remove(
db: PouchDB.Database,
docOrId: any,
rev?: any
): Promise<void> {
const cache = await getCache()
if (!docOrId) {
throw new Error("No ID/Rev provided.")
}
const id = typeof docOrId === "string" ? docOrId : docOrId._id
rev = typeof docOrId === "string" ? rev : docOrId._rev
try {
await cache.delete(makeCacheKey(db, id))
} finally {
await db.remove(id, rev)
}
}
export class Writethrough {
db: PouchDB.Database
writeRateMs: number
constructor(
db: PouchDB.Database,
writeRateMs: number = DEFAULT_WRITE_RATE_MS
) {
this.db = db
this.writeRateMs = writeRateMs
}
async put(doc: any) {
return put(this.db, doc, this.writeRateMs)
}
async get(id: string) {
return get(this.db, id)
}
async remove(docOrId: any, rev?: any) {
return remove(this.db, docOrId, rev)
}
}

49
packages/backend-core/src/db/pouch.js
View file

@ -1,21 +1,42 @@
const PouchDB = require("pouchdb")
const env = require("../environment")
function getUrlInfo() {
let url = env.COUCH_DB_URL
let username, password, host
const [protocol, rest] = url.split("://")
if (url.includes("@")) {
const hostParts = rest.split("@")
host = hostParts[1]
const authParts = hostParts[0].split(":")
username = authParts[0]
password = authParts[1]
} else {
host = rest
exports.getUrlInfo = (url = env.COUCH_DB_URL) => {
let cleanUrl, username, password, host
if (url) {
// Ensure the URL starts with a protocol
const protoRegex = /^https?:\/\//i
if (!protoRegex.test(url)) {
url = `http://${url}`
}
// Split into protocol and remainder
const split = url.split("://")
const protocol = split[0]
const rest = split.slice(1).join("://")
// Extract auth if specified
if (url.includes("@")) {
// Split into host and remainder
let parts = rest.split("@")
host = parts[parts.length - 1]
let auth = parts.slice(0, -1).join("@")
// Split auth into username and password
if (auth.includes(":")) {
const authParts = auth.split(":")
username = authParts[0]
password = authParts.slice(1).join(":")
} else {
username = auth
}
} else {
host = rest
}
cleanUrl = `${protocol}://${host}`
}
return {
url: `${protocol}://${host}`,
url: cleanUrl,
auth: {
username,
password,
@ -24,7 +45,7 @@ function getUrlInfo() {
}
exports.getCouchInfo = () => {
const urlInfo = getUrlInfo()
const urlInfo = exports.getUrlInfo()
let username
let password
if (env.COUCH_DB_USERNAME) {

62
packages/backend-core/src/db/tests/pouch.spec.js Normal file
View file

@ -0,0 +1,62 @@
require("../../../tests/utilities/TestConfiguration")
const getUrlInfo = require("../pouch").getUrlInfo
describe("pouch", () => {
describe("Couch DB URL parsing", () => {
it("should handle a null Couch DB URL", () => {
const info = getUrlInfo(null)
expect(info.url).toBeUndefined()
expect(info.auth.username).toBeUndefined()
})
it("should be able to parse a basic Couch DB URL", () => {
const info = getUrlInfo("http://host.com")
expect(info.url).toBe("http://host.com")
expect(info.auth.username).toBeUndefined()
})
it("should be able to parse a Couch DB basic URL with HTTPS", () => {
const info = getUrlInfo("https://host.com")
expect(info.url).toBe("https://host.com")
expect(info.auth.username).toBeUndefined()
})
it("should be able to parse a basic Couch DB URL with a custom port", () => {
const info = getUrlInfo("https://host.com:1234")
expect(info.url).toBe("https://host.com:1234")
expect(info.auth.username).toBeUndefined()
})
it("should be able to parse a Couch DB URL with auth", () => {
const info = getUrlInfo("https://user:pass@host.com:1234")
expect(info.url).toBe("https://host.com:1234")
expect(info.auth.username).toBe("user")
expect(info.auth.password).toBe("pass")
})
it("should be able to parse a Couch DB URL with auth and special chars", () => {
const info = getUrlInfo("https://user:s:p@s://@://:d@;][~s@host.com:1234")
expect(info.url).toBe("https://host.com:1234")
expect(info.auth.username).toBe("user")
expect(info.auth.password).toBe("s:p@s://@://:d@;][~s")
})
it("should be able to parse a Couch DB URL without a protocol", () => {
const info = getUrlInfo("host.com:1234")
expect(info.url).toBe("http://host.com:1234")
expect(info.auth.username).toBeUndefined()
})
it("should be able to parse a Couch DB URL with auth and without a protocol", () => {
const info = getUrlInfo("user:s:p@s://@://:d@;][~s@host.com:1234")
expect(info.url).toBe("http://host.com:1234")
expect(info.auth.username).toBe("user")
expect(info.auth.password).toBe("s:p@s://@://:d@;][~s")
})
it("should be able to parse a Couch DB URL with only username auth", () => {
const info = getUrlInfo("https://user@host.com:1234")
expect(info.url).toBe("https://host.com:1234")
expect(info.auth.username).toBe("user")
expect(info.auth.password).toBeUndefined()
})
it("should be able to parse a Couch DB URL with only username auth and without a protocol", () => {
const info = getUrlInfo("user@host.com:1234")
expect(info.url).toBe("http://host.com:1234")
expect(info.auth.username).toBe("user")
expect(info.auth.password).toBeUndefined()
})
})
})

3
packages/backend-core/src/environment.ts
View file

@ -16,7 +16,7 @@ if (!LOADED && isDev() && !isTest()) {
LOADED = true
}
const env: any = {
const env = {
isTest,
isDev,
JWT_SECRET: process.env.JWT_SECRET,
@ -66,6 +66,7 @@ const env: any = {
for (let [key, value] of Object.entries(env)) {
// handle the edge case of "0" to disable an environment variable
if (value === "0") {
// @ts-ignore
env[key] = 0
}
}

2
packages/backend-core/src/pkg/redis.ts
View file

@ -2,7 +2,7 @@
// The outer exports can't be used as they now reference dist directly
import Client from "../redis"
import utils from "../redis/utils"
import clients from "../redis/authRedis"
import clients from "../redis/init"
export = {
Client,

132
packages/backend-core/src/redis/index.js → packages/backend-core/src/redis/index.ts
View file

@ -1,3 +1,4 @@
import RedisWrapper from "../redis"
const env = require("../environment")
// ioredis mock is all in memory
const Redis = env.isTest() ? require("ioredis-mock") : require("ioredis")
@ -6,24 +7,34 @@ const {
removeDbPrefix,
getRedisOptions,
SEPARATOR,
SelectableDatabases,
} = require("./utils")
const RETRY_PERIOD_MS = 2000
const STARTUP_TIMEOUT_MS = 5000
const CLUSTERED = false
const DEFAULT_SELECT_DB = SelectableDatabases.DEFAULT
// for testing just generate the client once
let CLOSED = false
let CLIENT = env.isTest() ? new Redis(getRedisOptions()) : null
let CLIENTS: { [key: number]: any } = {}
// if in test always connected
let CONNECTED = !!env.isTest()
let CONNECTED = env.isTest()
function connectionError(timeout, err) {
function pickClient(selectDb: number): any {
return CLIENTS[selectDb]
}
function connectionError(
selectDb: number,
timeout: NodeJS.Timeout,
err: Error | string
) {
// manually shut down, ignore errors
if (CLOSED) {
return
}
CLIENT.disconnect()
pickClient(selectDb).disconnect()
CLOSED = true
// always clear this on error
clearTimeout(timeout)
@ -38,59 +49,69 @@ function connectionError(timeout, err) {
* Inits the system, will error if unable to connect to redis cluster (may take up to 10 seconds) otherwise
* will return the ioredis client which will be ready to use.
*/
function init() {
let timeout
function init(selectDb = DEFAULT_SELECT_DB) {
let timeout: NodeJS.Timeout
CLOSED = false
// testing uses a single in memory client
if (env.isTest() || (CLIENT && CONNECTED)) {
let client = pickClient(selectDb)
// already connected, ignore
if (client && CONNECTED) {
return
}
// testing uses a single in memory client
if (env.isTest()) {
CLIENTS[selectDb] = new Redis(getRedisOptions())
}
// start the timer - only allowed 5 seconds to connect
timeout = setTimeout(() => {
if (!CONNECTED) {
connectionError(timeout, "Did not successfully connect in timeout")
connectionError(
selectDb,
timeout,
"Did not successfully connect in timeout"
)
}
}, STARTUP_TIMEOUT_MS)
// disconnect any lingering client
if (CLIENT) {
CLIENT.disconnect()
if (client) {
client.disconnect()
}
const { redisProtocolUrl, opts, host, port } = getRedisOptions(CLUSTERED)
if (CLUSTERED) {
CLIENT = new Redis.Cluster([{ host, port }], opts)
client = new Redis.Cluster([{ host, port }], opts)
} else if (redisProtocolUrl) {
CLIENT = new Redis(redisProtocolUrl)
client = new Redis(redisProtocolUrl)
} else {
CLIENT = new Redis(opts)
client = new Redis(opts)
}
// attach handlers
CLIENT.on("end", err => {
connectionError(timeout, err)
client.on("end", (err: Error) => {
connectionError(selectDb, timeout, err)
})
CLIENT.on("error", err => {
connectionError(timeout, err)
client.on("error", (err: Error) => {
connectionError(selectDb, timeout, err)
})
CLIENT.on("connect", () => {
client.on("connect", () => {
clearTimeout(timeout)
CONNECTED = true
})
CLIENTS[selectDb] = client
}
function waitForConnection() {
function waitForConnection(selectDb: number = DEFAULT_SELECT_DB) {
return new Promise(resolve => {
if (CLIENT == null) {
if (pickClient(selectDb) == null) {
init()
} else if (CONNECTED) {
resolve()
resolve("")
return
}
// check if the connection is ready
const interval = setInterval(() => {
if (CONNECTED) {
clearInterval(interval)
resolve()
resolve("")
}
}, 500)
})
@ -100,25 +121,26 @@ function waitForConnection() {
* Utility function, takes a redis stream and converts it to a promisified response -
* this can only be done with redis streams because they will have an end.
* @param stream A redis stream, specifically as this type of stream will have an end.
* @param client The client to use for further lookups.
* @return {Promise<object>} The final output of the stream
*/
function promisifyStream(stream) {
function promisifyStream(stream: any, client: RedisWrapper) {
return new Promise((resolve, reject) => {
const outputKeys = new Set()
stream.on("data", keys => {
stream.on("data", (keys: string[]) => {
keys.forEach(key => {
outputKeys.add(key)
})
})
stream.on("error", err => {
stream.on("error", (err: Error) => {
reject(err)
})
stream.on("end", async () => {
const keysArray = Array.from(outputKeys)
const keysArray: string[] = Array.from(outputKeys) as string[]
try {
let getPromises = []
for (let key of keysArray) {
getPromises.push(CLIENT.get(key))
getPromises.push(client.get(key))
}
const jsonArray = await Promise.all(getPromises)
resolve(
@ -134,48 +156,52 @@ function promisifyStream(stream) {
})
}
class RedisWrapper {
constructor(db) {
export = class RedisWrapper {
_db: string
_select: number
constructor(db: string, selectDb: number | null = null) {
this._db = db
this._select = selectDb || DEFAULT_SELECT_DB
}
getClient() {
return CLIENT
return pickClient(this._select)
}
async init() {
CLOSED = false
init()
await waitForConnection()
init(this._select)
await waitForConnection(this._select)
return this
}
async finish() {
CLOSED = true
CLIENT.disconnect()
this.getClient().disconnect()
}
async scan(key = "") {
async scan(key = ""): Promise<any> {
const db = this._db
key = `${db}${SEPARATOR}${key}`
let stream
if (CLUSTERED) {
let node = CLIENT.nodes("master")
let node = this.getClient().nodes("master")
stream = node[0].scanStream({ match: key + "*", count: 100 })
} else {
stream = CLIENT.scanStream({ match: key + "*", count: 100 })
stream = this.getClient().scanStream({ match: key + "*", count: 100 })
}
return promisifyStream(stream)
return promisifyStream(stream, this.getClient())
}
async keys(pattern) {
async keys(pattern: string) {
const db = this._db
return CLIENT.keys(addDbPrefix(db, pattern))
return this.getClient().keys(addDbPrefix(db, pattern))
}
async get(key) {
async get(key: string) {
const db = this._db
let response = await CLIENT.get(addDbPrefix(db, key))
let response = await this.getClient().get(addDbPrefix(db, key))
// overwrite the prefixed key
if (response != null && response.key) {
response.key = key
@ -188,39 +214,37 @@ class RedisWrapper {
}
}
async store(key, value, expirySeconds = null) {
async store(key: string, value: any, expirySeconds: number | null = null) {
const db = this._db
if (typeof value === "object") {
value = JSON.stringify(value)
}
const prefixedKey = addDbPrefix(db, key)
await CLIENT.set(prefixedKey, value)
await this.getClient().set(prefixedKey, value)
if (expirySeconds) {
await CLIENT.expire(prefixedKey, expirySeconds)
await this.getClient().expire(prefixedKey, expirySeconds)
}
}
async getTTL(key) {
async getTTL(key: string) {
const db = this._db
const prefixedKey = addDbPrefix(db, key)
return CLIENT.ttl(prefixedKey)
return this.getClient().ttl(prefixedKey)
}
async setExpiry(key, expirySeconds) {
async setExpiry(key: string, expirySeconds: number | null) {
const db = this._db
const prefixedKey = addDbPrefix(db, key)
await CLIENT.expire(prefixedKey, expirySeconds)
await this.getClient().expire(prefixedKey, expirySeconds)
}
async delete(key) {
async delete(key: string) {
const db = this._db
await CLIENT.del(addDbPrefix(db, key))
await this.getClient().del(addDbPrefix(db, key))
}
async clear() {
let items = await this.scan()
await Promise.all(items.map(obj => this.delete(obj.key)))
await Promise.all(items.map((obj: any) => this.delete(obj.key)))
}
}
module.exports = RedisWrapper

13
packages/backend-core/src/redis/authRedis.js → packages/backend-core/src/redis/init.js
View file

@ -2,7 +2,7 @@ const Client = require("./index")
const utils = require("./utils")
const { getRedlock } = require("./redlock")
let userClient, sessionClient, appClient, cacheClient
let userClient, sessionClient, appClient, cacheClient, writethroughClient
let migrationsRedlock
// turn retry off so that only one instance can ever hold the lock
@ -13,6 +13,10 @@ async function init() {
sessionClient = await new Client(utils.Databases.SESSIONS).init()
appClient = await new Client(utils.Databases.APP_METADATA).init()
cacheClient = await new Client(utils.Databases.GENERIC_CACHE).init()
writethroughClient = await new Client(
utils.Databases.WRITE_THROUGH,
utils.SelectableDatabases.WRITE_THROUGH
).init()
// pass the underlying ioredis client to redlock
migrationsRedlock = getRedlock(
cacheClient.getClient(),
@ -25,6 +29,7 @@ process.on("exit", async () => {
if (sessionClient) await sessionClient.finish()
if (appClient) await appClient.finish()
if (cacheClient) await cacheClient.finish()
if (writethroughClient) await writethroughClient.finish()
})
module.exports = {
@ -52,6 +57,12 @@ module.exports = {
}
return cacheClient
},
getWritethroughClient: async () => {
if (!writethroughClient) {
await init()
}
return writethroughClient
},
getMigrationsRedlock: async () => {
if (!migrationsRedlock) {
await init()

37
packages/backend-core/src/redis/utils.js
View file

@ -6,6 +6,14 @@ const SEPARATOR = "-"
const REDIS_URL = !env.REDIS_URL ? "localhost:6379" : env.REDIS_URL
const REDIS_PASSWORD = !env.REDIS_PASSWORD ? "budibase" : env.REDIS_PASSWORD
/**
* These Redis databases help us to segment up a Redis keyspace by prepending the
* specified database name onto the cache key. This means that a single real Redis database
* can be split up a bit; allowing us to use scans on small databases to find some particular
* keys within.
* If writing a very large volume of keys is expected (say 10K+) then it is better to keep these out
* of the default keyspace and use a separate one - the SelectableDatabases can be used for this.
*/
exports.Databases = {
PW_RESETS: "pwReset",
VERIFICATIONS: "verification",
@ -19,6 +27,35 @@ exports.Databases = {
QUERY_VARS: "queryVars",
LICENSES: "license",
GENERIC_CACHE: "data_cache",
WRITE_THROUGH: "writeThrough",
}
/**
* These define the numeric Redis databases that can be access with the SELECT command -
* (https://redis.io/commands/select/). By default a Redis server/cluster will have 16 selectable
* databases, increasing this count increases the amount of CPU/memory required to run the server.
* Ideally new Redis keyspaces should be used sparingly, only when absolutely necessary for performance
* to be maintained. Generally a keyspace can grow to be very large is scans are not needed or desired,
* but if you need to walk through all values in a database periodically then a separate selectable
* keyspace should be used.
*/
exports.SelectableDatabases = {
DEFAULT: 0,
WRITE_THROUGH: 1,
UNUSED_1: 2,
UNUSED_2: 3,
UNUSED_3: 4,
UNUSED_4: 5,
UNUSED_5: 6,
UNUSED_6: 7,
UNUSED_7: 8,
UNUSED_8: 9,
UNUSED_9: 10,
UNUSED_10: 11,
UNUSED_11: 12,
UNUSED_12: 13,
UNUSED_13: 14,
UNUSED_14: 15,
}
exports.SEPARATOR = SEPARATOR

2
packages/backend-core/src/security/sessions.js
View file

@ -1,4 +1,4 @@
const redis = require("../redis/authRedis")
const redis = require("../redis/init")
const { v4: uuidv4 } = require("uuid")
// a week in seconds

4
packages/backend-core/src/utils.js
View file

@ -197,3 +197,7 @@ exports.platformLogout = async ({ ctx, userId, keepActiveSession }) => {
await events.auth.logout()
await userCache.invalidateUser(userId)
}
exports.timeout = timeMs => {
return new Promise(resolve => setTimeout(resolve, timeMs))
}

158
packages/backend-core/yarn.lock
View file

@ -656,6 +656,13 @@
"@types/keygrip" "*"
"@types/node" "*"
"@types/debug@*":
version "4.1.7"
resolved "https://registry.yarnpkg.com/@types/debug/-/debug-4.1.7.tgz#7cc0ea761509124709b8b2d1090d8f6c17aadb82"
integrity sha512-9AonUzyTjXXhEOa0DnqpzZi6VHlqKMswga9EXjpXnnqxwLtdvPPtlO8evrI5D9S6asFRCQ6v+wpiUKbw+vKqyg==
dependencies:
"@types/ms" "*"
"@types/express-serve-static-core@^4.17.18":
version "4.17.28"
resolved "https://registry.yarnpkg.com/@types/express-serve-static-core/-/express-serve-static-core-4.17.28.tgz#c47def9f34ec81dc6328d0b1b5303d1ec98d86b8"
@ -762,6 +769,11 @@
resolved "https://registry.yarnpkg.com/@types/mime/-/mime-1.3.2.tgz#93e25bf9ee75fe0fd80b594bc4feb0e862111b5a"
integrity sha512-YATxVxgRqNH6nHEIsvg6k2Boc1JHI9ZbH5iWFFv/MTkchz3b1ieGDa5T0a9RznNdI0KhVbdbWSN+KWWrQZRxTw==
"@types/ms@*":
version "0.7.31"
resolved "https://registry.yarnpkg.com/@types/ms/-/ms-0.7.31.tgz#31b7ca6407128a3d2bbc27fe2d21b345397f6197"
integrity sha512-iiUgKzV9AuaEkZqkOLDIvlQiL6ltuZd9tGcW3gwpnX8JbuiuhFlEGmmFXEXkN50Cvq7Os88IY2v0dkDqXYWVgA==
"@types/node-fetch@2.6.1":
version "2.6.1"
resolved "https://registry.yarnpkg.com/@types/node-fetch/-/node-fetch-2.6.1.tgz#8f127c50481db65886800ef496f20bbf15518975"
@ -780,6 +792,152 @@
resolved "https://registry.yarnpkg.com/@types/node/-/node-14.18.20.tgz#268f028b36eaf51181c3300252f605488c4f0650"
integrity sha512-Q8KKwm9YqEmUBRsqJ2GWJDtXltBDxTdC4m5vTdXBolu2PeQh8LX+f6BTwU+OuXPu37fLxoN6gidqBmnky36FXA==
"@types/pouchdb-adapter-cordova-sqlite@*":
version "1.0.1"
resolved "https://registry.yarnpkg.com/@types/pouchdb-adapter-cordova-sqlite/-/pouchdb-adapter-cordova-sqlite-1.0.1.tgz#49e5ee6df7cc0c23196fcb340f43a560e74eb1d6"
integrity sha512-nqlXpW1ho3KBg1mUQvZgH2755y3z/rw4UA7ZJCPMRTHofxGMY8izRVw5rHBL4/7P615or0J2udpRYxgkT3D02g==
dependencies:
"@types/pouchdb-core" "*"
"@types/pouchdb-adapter-fruitdown@*":
version "6.1.3"
resolved "https://registry.yarnpkg.com/@types/pouchdb-adapter-fruitdown/-/pouchdb-adapter-fruitdown-6.1.3.tgz#9b140ad9645cc56068728acf08ec19ac0046658e"
integrity sha512-Wz1Z1JLOW1hgmFQjqnSkmyyfH7by/iWb4abKn684WMvQfmxx6BxKJpJ4+eulkVPQzzgMMSgU1MpnQOm9FgRkbw==
dependencies:
"@types/pouchdb-core" "*"
"@types/pouchdb-adapter-http@*":
version "6.1.3"
resolved "https://registry.yarnpkg.com/@types/pouchdb-adapter-http/-/pouchdb-adapter-http-6.1.3.tgz#6e592d5f48deb6274a21ddac1498dd308096bcf3"
integrity sha512-9Z4TLbF/KJWy/D2sWRPBA+RNU0odQimfdvlDX+EY7rGcd3aVoH8qjD/X0Xcd/0dfBH5pKrNIMFFQgW/TylRCmA==
dependencies:
"@types/pouchdb-core" "*"
"@types/pouchdb-adapter-idb@*":
version "6.1.4"
resolved "https://registry.yarnpkg.com/@types/pouchdb-adapter-idb/-/pouchdb-adapter-idb-6.1.4.tgz#cb9a18864585d600820cd325f007614c5c3989cd"
integrity sha512-KIAXbkF4uYUz0ZwfNEFLtEkK44mEWopAsD76UhucH92XnJloBysav+TjI4FFfYQyTjoW3S1s6V+Z14CUJZ0F6w==
dependencies:
"@types/pouchdb-core" "*"
"@types/pouchdb-adapter-leveldb@*":
version "6.1.3"
resolved "https://registry.yarnpkg.com/@types/pouchdb-adapter-leveldb/-/pouchdb-adapter-leveldb-6.1.3.tgz#17c7e75d75b992050bca15991e97fba575c61bb3"
integrity sha512-ex8NFqQGFwEpFi7AaZ5YofmuemfZNsL3nTFZBUCAKYMBkazQij1pe2ILLStSvJr0XS0qxgXjCEW19T5Wqiiskg==
dependencies:
"@types/pouchdb-core" "*"
"@types/pouchdb-adapter-localstorage@*":
version "6.1.3"
resolved "https://registry.yarnpkg.com/@types/pouchdb-adapter-localstorage/-/pouchdb-adapter-localstorage-6.1.3.tgz#0dde02ba6b9d6073a295a20196563942ba9a54bd"
integrity sha512-oor040tye1KKiGLWYtIy7rRT7C2yoyX3Tf6elEJRpjOA7Ja/H8lKc4LaSh9ATbptIcES6MRqZDxtp7ly9hsW3Q==
dependencies:
"@types/pouchdb-core" "*"
"@types/pouchdb-adapter-memory@*":
version "6.1.3"
resolved "https://registry.yarnpkg.com/@types/pouchdb-adapter-memory/-/pouchdb-adapter-memory-6.1.3.tgz#9eabdbc890fcf58960ee8b68b8685f837e75c844"
integrity sha512-gVbsIMzDzgZYThFVT4eVNsmuZwVm/4jDxP1sjlgc3qtDIxbtBhGgyNfcskwwz9Zu5Lv1avkDsIWvcxQhnvRlHg==
dependencies:
"@types/pouchdb-core" "*"
"@types/pouchdb-adapter-node-websql@*":
version "6.1.3"
resolved "https://registry.yarnpkg.com/@types/pouchdb-adapter-node-websql/-/pouchdb-adapter-node-websql-6.1.3.tgz#aa18bc68af8cf509acd12c400010dcd5fab2243d"
integrity sha512-F/P+os6Jsa7CgHtH64+Z0HfwIcj0hIRB5z8gNhF7L7dxPWoAfkopK5H2gydrP3sQrlGyN4WInF+UJW/Zu1+FKg==
dependencies:
"@types/pouchdb-adapter-websql" "*"
"@types/pouchdb-core" "*"
"@types/pouchdb-adapter-websql@*":
version "6.1.4"
resolved "https://registry.yarnpkg.com/@types/pouchdb-adapter-websql/-/pouchdb-adapter-websql-6.1.4.tgz#359fbe42ccac0ac90b492ddb8c32fafd0aa96d79"
integrity sha512-zMJQCtXC40hBsIDRn0GhmpeGMK0f9l/OGWfLguvczROzxxcOD7REI+e6SEmX7gJKw5JuMvlfuHzkQwjmvSJbtg==
dependencies:
"@types/pouchdb-core" "*"
"@types/pouchdb-browser@*":
version "6.1.3"
resolved "https://registry.yarnpkg.com/@types/pouchdb-browser/-/pouchdb-browser-6.1.3.tgz#8f33d6ef58d6817d1f6d36979148a1c7f63244d8"
integrity sha512-EdYowrWxW9SWBMX/rux2eq7dbHi5Zeyzz+FF/IAsgQKnUxgeCO5VO2j4zTzos0SDyJvAQU+EYRc11r7xGn5tvA==
dependencies:
"@types/pouchdb-adapter-http" "*"
"@types/pouchdb-adapter-idb" "*"
"@types/pouchdb-adapter-websql" "*"
"@types/pouchdb-core" "*"
"@types/pouchdb-mapreduce" "*"
"@types/pouchdb-replication" "*"
"@types/pouchdb-core@*":
version "7.0.10"
resolved "https://registry.yarnpkg.com/@types/pouchdb-core/-/pouchdb-core-7.0.10.tgz#d1ea1549e7fad6cb579f71459b1bc27252e06a5a"
integrity sha512-mKhjLlWWXyV3PTTjDhzDV1kc2dolO7VYFa75IoKM/hr8Er9eo8RIbS7mJLfC8r/C3p6ihZu9yZs1PWC1LQ0SOA==
dependencies:
"@types/debug" "*"
"@types/pouchdb-find" "*"
"@types/pouchdb-find@*":
version "6.3.7"
resolved "https://registry.yarnpkg.com/@types/pouchdb-find/-/pouchdb-find-6.3.7.tgz#f713534a53c1a7f3fd8fbbfb74131a1b04711ddc"
integrity sha512-b2dr9xoZRK5Mwl8UiRA9l5j9mmCxNfqXuu63H1KZHwJLILjoIIz7BntCvM0hnlnl7Q8P8wORq0IskuaMq5Nnnw==
dependencies:
"@types/pouchdb-core" "*"
"@types/pouchdb-http@*":
version "6.1.3"
resolved "https://registry.yarnpkg.com/@types/pouchdb-http/-/pouchdb-http-6.1.3.tgz#09576c0d409da1f8dee34ec5b768415e2472ea52"
integrity sha512-0e9E5SqNOyPl/3FnEIbENssB4FlJsNYuOy131nxrZk36S+y1R/6qO7ZVRypWpGTqBWSuVd7gCsq2UDwO/285+w==
dependencies:
"@types/pouchdb-adapter-http" "*"
"@types/pouchdb-core" "*"
"@types/pouchdb-mapreduce@*":
version "6.1.7"
resolved "https://registry.yarnpkg.com/@types/pouchdb-mapreduce/-/pouchdb-mapreduce-6.1.7.tgz#9ab32d1e0f234f1bf6d1e4c5d7e216e9e23ac0a3"
integrity sha512-WzBwm7tmO9QhfRzVaWT4v6JQSS/fG2OoUDrWrhX87rPe2Pn6laPvdK5li6myNRxCoI/l5e8Jd+oYBAFnaiFucA==
dependencies:
"@types/pouchdb-core" "*"
"@types/pouchdb-node@*":
version "6.1.4"
resolved "https://registry.yarnpkg.com/@types/pouchdb-node/-/pouchdb-node-6.1.4.tgz#5214c0169fcfd2237d373380bbd65a934feb5dfb"
integrity sha512-wnTCH8X1JOPpNOfVhz8HW0AvmdHh6pt40MuRj0jQnK7QEHsHS79WujsKTKSOF8QXtPwpvCNSsI7ut7H7tfxxJQ==
dependencies:
"@types/pouchdb-adapter-http" "*"
"@types/pouchdb-adapter-leveldb" "*"
"@types/pouchdb-core" "*"
"@types/pouchdb-mapreduce" "*"
"@types/pouchdb-replication" "*"
"@types/pouchdb-replication@*":
version "6.4.4"
resolved "https://registry.yarnpkg.com/@types/pouchdb-replication/-/pouchdb-replication-6.4.4.tgz#743406c90f13a988fa3e346ea74ce40acd170d00"
integrity sha512-BsE5LKpjJK4iAf6Fx5kyrMw+33V+Ip7uWldUnU2BYrrvtR+MLD22dcImm7DZN1st2wPPb91i0XEnQzvP0w1C/Q==
dependencies:
"@types/pouchdb-core" "*"
"@types/pouchdb-find" "*"
"@types/pouchdb@6.4.0":
version "6.4.0"
resolved "https://registry.yarnpkg.com/@types/pouchdb/-/pouchdb-6.4.0.tgz#f9c41ca64b23029f9bf2eb4bf6956e6431cb79f8"
integrity sha512-eGCpX+NXhd5VLJuJMzwe3L79fa9+IDTrAG3CPaf4s/31PD56hOrhDJTSmRELSXuiqXr6+OHzzP0PldSaWsFt7w==
dependencies:
"@types/pouchdb-adapter-cordova-sqlite" "*"
"@types/pouchdb-adapter-fruitdown" "*"
"@types/pouchdb-adapter-http" "*"
"@types/pouchdb-adapter-idb" "*"
"@types/pouchdb-adapter-leveldb" "*"
"@types/pouchdb-adapter-localstorage" "*"
"@types/pouchdb-adapter-memory" "*"
"@types/pouchdb-adapter-node-websql" "*"
"@types/pouchdb-adapter-websql" "*"
"@types/pouchdb-browser" "*"
"@types/pouchdb-core" "*"
"@types/pouchdb-http" "*"
"@types/pouchdb-mapreduce" "*"
"@types/pouchdb-node" "*"
"@types/pouchdb-replication" "*"
"@types/prettier@^2.1.5":
version "2.6.3"
resolved "https://registry.yarnpkg.com/@types/prettier/-/prettier-2.6.3.tgz#68ada76827b0010d0db071f739314fa429943d0a"

4
packages/bbui/package.json
View file

@ -1,7 +1,7 @@
{
"name": "@budibase/bbui",
"description": "A UI solution used in the different Budibase projects.",
"version": "1.0.207-alpha.6",
"version": "1.0.212-alpha.0",
"license": "MPL-2.0",
"svelte": "src/index.js",
"module": "dist/bbui.es.js",
@ -38,7 +38,7 @@
],
"dependencies": {
"@adobe/spectrum-css-workflow-icons": "^1.2.1",
"@budibase/string-templates": "^1.0.207-alpha.6",
"@budibase/string-templates": "^1.0.212-alpha.0",
"@spectrum-css/actionbutton": "^1.0.1",
"@spectrum-css/actiongroup": "^1.0.1",
"@spectrum-css/avatar": "^3.0.2",

2
packages/bbui/src/Typography/Detail.svelte
View file

@ -1,9 +1,7 @@
<script>
import "@spectrum-css/typography/dist/index-vars.css"
// Sizes
export let size = "M"
export let serif = false
</script>

2
packages/builder/cypress/support/commands.js
View file

@ -400,7 +400,7 @@ Cypress.Commands.add("createTable", (tableName, initialTable) => {
cy.get(`[data-cy="new-table"]`).click()
}
cy.wait(500)
cy.get(".spectrum-Dialog-grid")
cy.get(".item")
.contains("Budibase DB")
.click({ force: true })
.then(() => {

10
packages/builder/package.json
View file

@ -1,6 +1,6 @@
{
"name": "@budibase/builder",
"version": "1.0.207-alpha.6",
"version": "1.0.212-alpha.0",
"license": "GPL-3.0",
"private": true,
"scripts": {
@ -69,10 +69,10 @@
}
},
"dependencies": {
"@budibase/bbui": "^1.0.207-alpha.6",
"@budibase/client": "^1.0.207-alpha.6",
"@budibase/frontend-core": "^1.0.207-alpha.6",
"@budibase/string-templates": "^1.0.207-alpha.6",
"@budibase/bbui": "^1.0.212-alpha.0",
"@budibase/client": "^1.0.212-alpha.0",
"@budibase/frontend-core": "^1.0.212-alpha.0",
"@budibase/string-templates": "^1.0.212-alpha.0",
"@sentry/browser": "5.19.1",
"@spectrum-css/page": "^3.0.1",
"@spectrum-css/vars": "^3.0.1",

80
packages/builder/src/components/backend/DatasourceNavigator/modals/CreateDatasourceModal.svelte
View file

@ -5,12 +5,13 @@
Body,
Layout,
Detail,
Heading,
notifications,
} from "@budibase/bbui"
import { onMount } from "svelte"
import ICONS from "../icons"
import { API } from "api"
import { IntegrationNames, IntegrationTypes } from "constants/backend"
import { IntegrationTypes } from "constants/backend"
import CreateTableModal from "components/backend/TableNavigator/modals/CreateTableModal.svelte"
import DatasourceConfigModal from "components/backend/DatasourceNavigator/modals/DatasourceConfigModal.svelte"
import GoogleDatasourceConfigModal from "components/backend/DatasourceNavigator/modals/GoogleDatasourceConfigModal.svelte"
@ -118,7 +119,7 @@
<Modal bind:this={modal}>
<ModalContent
disabled={!Object.keys(integration).length}
title="Data"
title="Add data source"
confirmText="Continue"
showSecondaryButton={showImportButton}
secondaryButtonText="Import"
@ -129,27 +130,25 @@
chooseNextModal()
}}
>
<Layout noPadding>
<Body size="S"
>All apps need data. You can connect to a data source below, or add data
to your app using Budibase's built-in database.
</Body>
<Layout noPadding gap="XS">
<Body size="S">Get started with Budibase DB</Body>
<div
class:selected={integration.type === IntegrationTypes.INTERNAL}
on:click={() => selectIntegration(IntegrationTypes.INTERNAL)}
class="item hoverable"
>
<div class="item-body">
<svelte:component this={ICONS.BUDIBASE} height="18" width="18" />
<span class="icon-spacing"> <Body size="S">Budibase DB</Body></span>
<div class="item-body with-type">
<svelte:component this={ICONS.BUDIBASE} height="20" width="20" />
<div class="text">
<Heading size="XXS">Budibase DB</Heading>
<Detail size="S" class="type">Non-relational</Detail>
</div>
</div>
</div>
</Layout>
<Layout gap="XS" noPadding>
<div class="title-spacing">
<Detail size="S">Connect to data source</Detail>
</div>
<Layout noPadding gap="XS">
<Body size="S">Connect to an external data source</Body>
<div class="item-list">
{#each Object.entries(integrations).filter(([key]) => key !== IntegrationTypes.INTERNAL) as [integrationType, schema]}
<div
@ -157,18 +156,18 @@
on:click={() => selectIntegration(integrationType)}
class="item hoverable"
>
<div class="item-body">
<div class="item-body" class:with-type={!!schema.type}>
<svelte:component
this={ICONS[integrationType]}
height="18"
width="18"
height="20"
width="20"
/>
<span class="icon-spacing">
<Body size="S"
>{schema.name || IntegrationNames[integrationType]}</Body
></span
>
<div class="text">
<Heading size="XXS">{schema.friendlyName}</Heading>
{#if schema.type}
<Detail size="S">{schema.type || ""}</Detail>
{/if}
</div>
</div>
</div>
{/each}
@ -178,13 +177,6 @@
</Modal>
<style>
.icon-spacing {
margin-left: var(--spacing-m);
}
.item-body {
display: flex;
margin-left: var(--spacing-m);
}
.item-list {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));
@ -195,21 +187,35 @@
cursor: pointer;
display: grid;
grid-gap: var(--spectrum-alias-grid-margin-xsmall);
padding: var(--spectrum-alias-item-padding-s);
padding: var(--spectrum-alias-item-padding-s)
var(--spectrum-alias-item-padding-m);
background: var(--spectrum-alias-background-color-secondary);
transition: 0.3s all;
transition: background 0.13s ease-out;
border: solid var(--spectrum-alias-border-color);
border-radius: 5px;
box-sizing: border-box;
border-width: 2px;
}
.selected {
.item:hover,
.item.selected {
background: var(--spectrum-alias-background-color-tertiary);
}
.item:hover,
.selected {
background: var(--spectrum-alias-background-color-tertiary);
.item-body {
display: flex;
flex-direction: row;
justify-content: flex-start;
align-items: center;
gap: var(--spacing-m);
}
.item-body.with-type {
align-items: flex-start;
}
.item-body.with-type :global(svg) {
margin-top: 4px;
}
.text :global(.spectrum-Detail) {
color: var(--spectrum-global-color-gray-700);
}
</style>

2
packages/cli/package.json
View file

@ -1,6 +1,6 @@
{
"name": "@budibase/cli",
"version": "1.0.207-alpha.6",
"version": "1.0.212-alpha.0",
"description": "Budibase CLI, for developers, self hosting and migrations.",
"main": "src/index.js",
"bin": {

8
packages/client/package.json
View file

@ -1,6 +1,6 @@
{
"name": "@budibase/client",
"version": "1.0.207-alpha.6",
"version": "1.0.212-alpha.0",
"license": "MPL-2.0",
"module": "dist/budibase-client.js",
"main": "dist/budibase-client.js",
@ -19,9 +19,9 @@
"dev:builder": "rollup -cw"
},
"dependencies": {
"@budibase/bbui": "^1.0.207-alpha.6",
"@budibase/frontend-core": "^1.0.207-alpha.6",
"@budibase/string-templates": "^1.0.207-alpha.6",
"@budibase/bbui": "^1.0.212-alpha.0",
"@budibase/frontend-core": "^1.0.212-alpha.0",
"@budibase/string-templates": "^1.0.212-alpha.0",
"@spectrum-css/button": "^3.0.3",
"@spectrum-css/card": "^3.0.3",
"@spectrum-css/divider": "^1.0.3",

4
packages/frontend-core/package.json
View file

@ -1,12 +1,12 @@
{
"name": "@budibase/frontend-core",
"version": "1.0.207-alpha.6",
"version": "1.0.212-alpha.0",
"description": "Budibase frontend core libraries used in builder and client",
"author": "Budibase",
"license": "MPL-2.0",
"svelte": "src/index.js",
"dependencies": {
"@budibase/bbui": "^1.0.207-alpha.6",
"@budibase/bbui": "^1.0.212-alpha.0",
"lodash": "^4.17.21",
"svelte": "^3.46.2"
}

13
packages/server/package.json
View file

@ -1,7 +1,7 @@
{
"name": "@budibase/server",
"email": "hi@budibase.com",
"version": "1.0.207-alpha.6",
"version": "1.0.212-alpha.0",
"description": "Budibase Web Server",
"main": "src/index.ts",
"repository": {
@ -77,11 +77,11 @@
"license": "GPL-3.0",
"dependencies": {
"@apidevtools/swagger-parser": "10.0.3",
"@budibase/backend-core": "^1.0.207-alpha.6",
"@budibase/client": "^1.0.207-alpha.6",
"@budibase/pro": "1.0.207-alpha.6",
"@budibase/string-templates": "^1.0.207-alpha.6",
"@budibase/types": "^1.0.207-alpha.6",
"@budibase/backend-core": "^1.0.212-alpha.0",
"@budibase/client": "^1.0.212-alpha.0",
"@budibase/pro": "1.0.212-alpha.0",
"@budibase/string-templates": "^1.0.212-alpha.0",
"@budibase/types": "^1.0.212-alpha.0",
"@bull-board/api": "3.7.0",
"@bull-board/koa": "3.9.4",
"@elastic/elasticsearch": "7.10.0",
@ -152,7 +152,6 @@
"@babel/core": "7.17.4",
"@babel/preset-env": "7.16.11",
"@budibase/standard-components": "^0.9.139",
"@budibase/types": "^1.0.207-alpha.3",
"@jest/test-sequencer": "24.9.0",
"@types/apidoc": "0.50.0",
"@types/bson": "4.2.0",

2
packages/server/src/api/controllers/row/internalSearch.js
View file

@ -145,7 +145,7 @@ class QueryBuilder {
function build(structure, queryFn) {
for (let [key, value] of Object.entries(structure)) {
key = builder.preprocess(key.replace(/ /, "_"), {
key = builder.preprocess(key.replace(/ /g, "_"), {
escape: true,
})
const expression = queryFn(key, value)

1
packages/server/src/definitions/common.ts
View file

@ -12,6 +12,7 @@ export interface Application extends Base {
export interface FieldSchema {
// TODO: replace with field types enum when done
type: string
externalType?: string
fieldName?: string
name: string
tableId?: string

1
packages/server/src/definitions/datasource.ts
View file

@ -94,6 +94,7 @@ export interface Integration {
relationships?: boolean
description: string
friendlyName: string
type?: string
datasource: {}
query: {
[key: string]: QueryDefinition

1
packages/server/src/integrations/airtable.ts
View file

@ -18,6 +18,7 @@ module AirtableModule {
description:
"Airtable is a spreadsheet-database hybrid, with the features of a database but applied to a spreadsheet.",
friendlyName: "Airtable",
type: "Spreadsheet",
datasource: {
apiKey: {
type: DatasourceFieldTypes.PASSWORD,

1
packages/server/src/integrations/arangodb.ts
View file

@ -19,6 +19,7 @@ module ArangoModule {
const SCHEMA: Integration = {
docs: "https://github.com/arangodb/arangojs",
friendlyName: "ArangoDB",
type: "Non-relational",
description:
"ArangoDB is a scalable open-source multi-model database natively supporting graph, document and search. All supported data models & access patterns can be combined in queries allowing for maximal flexibility. ",
datasource: {

36
packages/server/src/integrations/base/sql.ts
View file

@ -89,6 +89,28 @@ function parseFilters(filters: SearchFilters | undefined): SearchFilters {
return filters
}
function generateSelectStatement(
json: QueryJson,
knex: Knex
): (string | Knex.Raw)[] {
const { resource, meta } = json
const schema = meta?.table?.schema
return resource.fields.map(field => {
const fieldNames = field.split(/\./g)
const tableName = fieldNames[0]
const columnName = fieldNames[1]
if (columnName && knex.client.config.client === SqlClients.POSTGRES) {
const externalType = schema?.[columnName].externalType
if (externalType?.includes("money")) {
return knex.raw(
`"${tableName}"."${columnName}"::money::numeric as "${field}"`
)
}
}
return `${field} as ${field}`
})
}
class InternalBuilder {
private readonly client: string
@ -216,9 +238,7 @@ class InternalBuilder {
}
addRelationships(
knex: Knex,
query: KnexQuery,
fields: string | string[],
fromTable: string,
relationships: RelationshipsJson[] | undefined
): KnexQuery {
@ -323,12 +343,12 @@ class InternalBuilder {
if (!resource) {
resource = { fields: [] }
}
let selectStatement: string | string[] = "*"
let selectStatement: string | (string | Knex.Raw)[] = "*"
// handle select
if (resource.fields && resource.fields.length > 0) {
// select the resources as the format "table.columnName" - this is what is provided
// by the resource builder further up
selectStatement = resource.fields.map(field => `${field} as ${field}`)
selectStatement = generateSelectStatement(json, knex)
}
let foundLimit = limit || BASE_LIMIT
// handle pagination
@ -363,13 +383,7 @@ class InternalBuilder {
preQuery = this.addSorting(preQuery, json)
}
// handle joins
query = this.addRelationships(
knex,
preQuery,
selectStatement,
tableName,
relationships
)
query = this.addRelationships(preQuery, tableName, relationships)
return this.addFilters(query, filters, { relationship: true })
}

1
packages/server/src/integrations/couchdb.ts
View file

@ -16,6 +16,7 @@ module CouchDBModule {
const SCHEMA: Integration = {
docs: "https://docs.couchdb.org/en/stable/",
friendlyName: "CouchDB",
type: "Non-relational",
description:
"Apache CouchDB is an open-source document-oriented NoSQL database, implemented in Erlang.",
datasource: {

1
packages/server/src/integrations/dynamodb.ts
View file

@ -21,6 +21,7 @@ module DynamoModule {
description:
"Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale.",
friendlyName: "DynamoDB",
type: "Non-relational",
datasource: {
region: {
type: DatasourceFieldTypes.STRING,

1
packages/server/src/integrations/elasticsearch.ts
View file

@ -17,6 +17,7 @@ module ElasticsearchModule {
description:
"Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.",
friendlyName: "ElasticSearch",
type: "Non-relational",
datasource: {
url: {
type: DatasourceFieldTypes.STRING,

1
packages/server/src/integrations/firebase.ts
View file

@ -16,6 +16,7 @@ module Firebase {
const SCHEMA: Integration = {
docs: "https://firebase.google.com/docs/firestore/quickstart",
friendlyName: "Firestore",
type: "Non-relational",
description:
"Cloud Firestore is a flexible, scalable database for mobile, web, and server development from Firebase and Google Cloud.",
datasource: {

1
packages/server/src/integrations/googlesheets.ts
View file

@ -49,6 +49,7 @@ module GoogleSheetsModule {
description:
"Create and collaborate on online spreadsheets in real-time and from any device. ",
friendlyName: "Google Sheets",
type: "Spreadsheet",
datasource: {
spreadsheetId: {
display: "Google Sheet URL",

1
packages/server/src/integrations/microsoftSqlServer.ts
View file

@ -44,6 +44,7 @@ module MSSQLModule {
description:
"Microsoft SQL Server is a relational database management system developed by Microsoft. ",
friendlyName: "MS SQL Server",
type: "Relational",
datasource: {
user: {
type: DatasourceFieldTypes.STRING,

7
packages/server/src/integrations/mongodb.ts
View file

@ -24,6 +24,7 @@ module MongoDBModule {
const SCHEMA: Integration = {
docs: "https://github.com/mongodb/node-mongodb-native",
friendlyName: "MongoDB",
type: "Non-relational",
description:
"MongoDB is a general purpose, document-based, distributed database built for modern application developers and for the cloud era.",
datasource: {
@ -263,6 +264,12 @@ module MongoDBModule {
filter: FilterQuery<any>
options: CommonOptions
}
if (!json.options) {
json = {
filter: json,
options: {},
}
}
switch (query.extra.actionTypes) {
case "deleteOne": {

1
packages/server/src/integrations/mysql.ts
View file

@ -36,6 +36,7 @@ module MySQLModule {
docs: "https://github.com/sidorares/node-mysql2",
plus: true,
friendlyName: "MySQL",
type: "Relational",
description:
"MySQL Database Service is a fully managed database service to deploy cloud-native applications. ",
datasource: {

1
packages/server/src/integrations/oracle.ts
View file

@ -40,6 +40,7 @@ module OracleModule {
docs: "https://github.com/oracle/node-oracledb",
plus: true,
friendlyName: "Oracle",
type: "Relational",
description:
"Oracle Database is an object-relational database management system developed by Oracle Corporation",
datasource: {

1
packages/server/src/integrations/postgres.ts
View file

@ -47,6 +47,7 @@ module PostgresModule {
docs: "https://node-postgres.com",
plus: true,
friendlyName: "PostgreSQL",
type: "Relational",
description:
"PostgreSQL, also known as Postgres, is a free and open-source relational database management system emphasizing extensibility and SQL compliance.",
datasource: {

1
packages/server/src/integrations/redis.ts
View file

@ -17,6 +17,7 @@ module RedisModule {
docs: "https://redis.io/docs/",
description: "",
friendlyName: "Redis",
type: "Non-relational",
datasource: {
host: {
type: "string",

1
packages/server/src/integrations/rest.ts
View file

@ -64,6 +64,7 @@ module RestModule {
description:
"With the REST API datasource, you can connect, query and pull data from multiple REST APIs. You can then use the retrieved data to build apps.",
friendlyName: "REST API",
type: "API",
datasource: {
url: {
type: DatasourceFieldTypes.STRING,

1
packages/server/src/integrations/s3.ts
View file

@ -17,6 +17,7 @@ module S3Module {
description:
"Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance.",
friendlyName: "Amazon S3",
type: "Object store",
datasource: {
region: {
type: "string",

1
packages/server/src/integrations/snowflake.ts
View file

@ -16,6 +16,7 @@ module SnowflakeModule {
description:
"Snowflake is a solution for data warehousing, data lakes, data engineering, data science, data application development, and securely sharing and consuming shared data.",
friendlyName: "Snowflake",
type: "Relational",
datasource: {
account: {
type: "string",

18
packages/server/yarn.lock
View file

@ -1094,10 +1094,10 @@
resolved "https://registry.yarnpkg.com/@bcoe/v8-coverage/-/v8-coverage-0.2.3.tgz#75a2e8b51cb758a7553d6804a5932d7aace75c39"
integrity sha512-0hYQ8SB4Db5zvZB4axdMHGwEaQjkZzFjQiN9LVYvIFB2nSUHW9tYpxWriPrWDASIxiaXax83REcLxuSdnGPZtw==
"@budibase/backend-core@1.0.207-alpha.3":
version "1.0.207-alpha.3"
resolved "https://registry.yarnpkg.com/@budibase/backend-core/-/backend-core-1.0.207-alpha.3.tgz#98bced0575ec4e2b158239a8c73b39ca2d816719"
integrity sha512-DU4X6jJ+DfhzOv4TTa1w4Dk5ZEdlK/z1joCTruT+SGM5qI75bXrGeol5OX2OaEbNKtXFKJ1zeVTmBCYcu7OFUg==
"@budibase/backend-core@1.0.211":
version "1.0.211"
resolved "https://registry.yarnpkg.com/@budibase/backend-core/-/backend-core-1.0.211.tgz#579bcc97acf1df2510e302bb70b43245e5ab0e37"
integrity sha512-ham+Qk6WXQi37Lgnz1Gh/+ItcvCo+dnPZDcVvPESrVxqQkVVIejlxQinZSueIrsgOYHE1j3TUH5zER5M2t6RWw==
dependencies:
"@techpass/passport-openidconnect" "0.3.2"
aws-sdk "2.1030.0"
@ -1175,12 +1175,12 @@
svelte-flatpickr "^3.2.3"
svelte-portal "^1.0.0"
"@budibase/pro@1.0.207-alpha.3":
version "1.0.207-alpha.3"
resolved "https://registry.yarnpkg.com/@budibase/pro/-/pro-1.0.207-alpha.3.tgz#9bde845ceb685f1b43286a124620c21fdf891a01"
integrity sha512-WFEMujpKTVAMvAgLBnMdw8ou9PxsbM4Oa9Dq+DAUsWpPACsMWOProyHLsdRxJyvHlgGfwVjo5MEusvStjI4j6g==
"@budibase/pro@1.0.211":
version "1.0.211"
resolved "https://registry.yarnpkg.com/@budibase/pro/-/pro-1.0.211.tgz#d362e9af8c15f6ed386f27b7cca95cc096a91344"
integrity sha512-dfFByJhlTIURT3sXei5mVXq5rczFMM/ij2Scze0uqPZNpmIlWVqiYdGX7/HEcmIFSS0+UfcdBxjCJJlTGiK4/w==
dependencies:
"@budibase/backend-core" "1.0.207-alpha.3"
"@budibase/backend-core" "1.0.211"
node-fetch "^2.6.1"
"@budibase/standard-components@^0.9.139":

2
packages/string-templates/package.json
View file

@ -1,6 +1,6 @@
{
"name": "@budibase/string-templates",
"version": "1.0.207-alpha.6",
"version": "1.0.212-alpha.0",
"description": "Handlebars wrapper for Budibase templating.",
"main": "src/index.cjs",
"module": "dist/bundle.mjs",

2
packages/types/package.json
View file

@ -1,6 +1,6 @@
{
"name": "@budibase/types",
"version": "1.0.207-alpha.6",
"version": "1.0.212-alpha.0",
"description": "Budibase types",
"main": "dist/index.js",
"types": "dist/index.d.ts",

11
packages/worker/package.json
View file

@ -1,7 +1,7 @@
{
"name": "@budibase/worker",
"email": "hi@budibase.com",
"version": "1.0.207-alpha.6",
"version": "1.0.212-alpha.0",
"description": "Budibase background service",
"main": "src/index.ts",
"repository": {
@ -34,10 +34,10 @@
"author": "Budibase",
"license": "GPL-3.0",
"dependencies": {
"@budibase/backend-core": "^1.0.207-alpha.6",
"@budibase/pro": "1.0.207-alpha.6",
"@budibase/string-templates": "^1.0.207-alpha.6",
"@budibase/types": "^1.0.207-alpha.6",
"@budibase/backend-core": "^1.0.212-alpha.0",
"@budibase/pro": "1.0.212-alpha.0",
"@budibase/string-templates": "^1.0.212-alpha.0",
"@budibase/types": "^1.0.212-alpha.0",
"@koa/router": "8.0.8",
"@sentry/node": "6.17.7",
"@techpass/passport-openidconnect": "0.3.2",
@ -67,7 +67,6 @@
"server-destroy": "1.0.1"
},
"devDependencies": {
"@budibase/types": "^1.0.207-alpha.3",
"@types/jest": "26.0.23",
"@types/koa": "2.13.4",
"@types/koa-router": "7.4.4",

833
yarn.lock
View file

File diff suppressed because it is too large Load diff