1
0
Fork 0
mirror of synced 2024-09-21 20:01:32 +12:00
Commit graph

67 commits

Author SHA1 Message Date
Hector Valcarcel
65c2039d0c Chore: Allow using an AWS_SESSION_TOKEN for object storage with AWS S3 2024-06-10 11:48:54 +01:00
mike12345567
155de99b68 Streaming to disk before passing onto S3. 2024-06-03 11:46:20 +01:00
mike12345567
75501c2251 Updating object store stream upload to make sure the stream has finished being processed before trying to upload to AWS (and only uploading a partial stream). 2024-05-31 17:57:31 +01:00
mike12345567
39147b80b6 PR comments. 2024-05-14 11:23:50 +01:00
mike12345567
5b5e7e47a2 Fixing some issues with updating rows with attachments, there were some UI bugs that had been raised (undefined being displayed) and the uploading from a URL was not working as expected, due to the stream not matching the expected format of the S3 SDK - this has been fixed by converting the stream to a true readable stream with . 2024-05-13 18:11:40 +01:00
Peter Clement
db273bcd36
Support attachment columns in Automations (#13567)
* base work to support attachments in create / update row

* handle single attachment column

* fix tests

* pro

* fix some types

* handle case where file exists in storage

* improve attacment processing

* refactor slightly and ensure correct url is used for existing attachments

* add test

* Fixing a build issue.

* update tests

* some lint

* remove cursed backend-core test util

* addressing pr comments

* refactoring nasty automationUtils upload code

* remove uneeded check

* use basneeame for fallback filename

* add a test to ensure coverage of single attachment column type

* fail early when fetching object metadata

---------

Co-authored-by: mike12345567 <me@michaeldrury.co.uk>
2024-05-09 14:30:56 +01:00
Peter Clement
1d300c2577
Remove aws sdk global mock and update tests (#13637)
* Remove aws sdk global mock and update tests

* add awaits

* Minio healthcheck in tests.

* Bind to 127.0.0.1 instead of 0.0.0.0

* Fix port fetching for minio container.

* Actually fix port mapping this time.

* Pull minio container before running tests.

* Enable testcontainers debug logging.

* Promote minio container to always running in tests, like CouchDB.

* Remove testcontainers debug logging.

---------

Co-authored-by: Sam Rose <hello@samwho.dev>
2024-05-08 14:08:34 +01:00
Michael Drury
2187d25711
Revert "Changing client library to be retrieved via pre-signed URL for Cloud" 2024-04-24 17:12:36 +01:00
mike12345567
f76abb0ca9 Getting rid of linting error. 2024-04-24 15:59:11 +01:00
mike12345567
849253faba Bringing back the old mechanism of returning the client library through a pre-signed URL, rather than always serving through the service. 2024-04-24 15:37:47 +01:00
Peter Clement
a4c0328c53
REST file handling and SMTP automation block attachments (#13403)
* handle files in rest connector

* fetch presigned url and return

* further updates to handle files in rest connector

* remove unused important and fix extension bug

* wrong expiry param

* tests

* add const for temp bucket

* handle ttl on bucket

* more bucket ttl work

* split out fileresponse and xmlresponse into utils

* lint

* remove log

* fix tests

* some pr comments

* update function naming and lint

* adding back needed response for frontend

* use fsp

* handle different content-disposition and potential path traversal

* add test container for s3 / minio

* add test case for filename* and ascii filenames

* move tests into separate describe

* remove log

* up timeout

* switch to minio image instead of localstack

* use minio image instead of s3 for testing

* stream file upload instead

* use streamUpload and update signatures

* update bucketcreate return

* throw real error

* tidy up

* pro

* pro ref fix?

* pro fix

* pro fix?

* move minio test provider to backend-core

* update email builder to allow attachments

* testing for sending files via smtp

* use backend-core minio test container in server

* handle different types of url

* fix minio test provider

* test with container host

* lint

* try different hostname?

* Revert "try different hostname?"

This reverts commit cfefdb8ded.

* fix issue with fetching of signed url with test minio

* update autoamtion attachments to take filename and url

* fix tests

* pro ref

* fix parsing of url object

* pr comments and linting

* pro ref

* fix pro again

* fix pro

* account-portal

* fix null issue

* fix ref

* ref

* When sending a file attachment in email fetch it directly from our object store

* add more checks to ensure we're working with a signed url

* update test to account for direct object store read

* formatting

* fix time issues within test

* update bucket and path extraction to regex

* use const in regex

* pro

* Updating TTL handling in upload functions (#13539)

* Updating TTL handling in upload functions

* describe ttl type

* account for ttl creation in existing buckets and update types

* fix tests

* pro

* pro
2024-04-22 15:30:57 +00:00
Sam Rose
217b39c232
Reenable no-unused-vars 2024-03-20 11:46:39 +00:00
Sam Rose
2e8eda47f1
Respond to PR feedback. 2024-02-26 17:38:33 +00:00
Sam Rose
c15554547b
Respond to PR feedback. 2024-02-26 17:28:37 +00:00
Sam Rose
b9600d8330
More progress on modernising application tests. 2024-02-26 11:57:56 +00:00
mike12345567
7dc2c3551f Updating the listObjects functionality to correctly handle truncated responses, when not all objects can be returned at once we need to loop, but we weren't correctly picking up the token that should be passed. 2024-02-07 17:03:44 +00:00
mike12345567
85f7d66a99 Handling very large exports/backup downloads. 2024-01-31 17:16:51 +00:00
Martin McKeaveney
a23f76b8c8 update expiry time for attachments/images 2024-01-31 08:57:54 -03:00
melohagan
9d56039d03
Revert TTL on signed urls (#12556) 2023-12-12 10:52:14 +00:00
mike12345567
00a3c630ef Initial work to get file streaming working when dealing with a large amount of file exports, tested with up to 1.5GB of attachments. 2023-12-01 18:36:40 +00:00
Martin McKeaveney
5301c1edd9 update presigned url time 2023-11-27 19:03:37 +00:00
Martin McKeaveney
a65b29eb88 banner changes for new pricing, fix for onboarding to prevent flash of UI before onboarding tutorial 2023-11-27 18:50:44 +00:00
Adria Navarro
db2c20e3ae eslint import/newline-after-import 2023-11-20 21:52:29 +01:00
Sam Rose
a27a9dc2af
Eliminate TOCTOU problem in creating bbTmp. 2023-11-03 14:29:54 +00:00
Michael Drury
c6b377c565
Revert "Revert "Client library fetching through server"" 2023-10-24 18:13:36 +01:00
Michael Drury
e2e5c9044c
Revert "Client library fetching through server" 2023-10-24 18:12:20 +01:00
mike12345567
701d30abb4 Don't pass through default tenant ID. 2023-10-24 17:30:57 +01:00
mike12345567
ba5b1d7cfa Adding tenant ID to query string and fixing test cases. 2023-10-24 17:27:58 +01:00
mike12345567
b53158ac08 Moving the client library to be retrieved using the same mechanism as we do in development, through the server instance (same as the builder itself). This should help with the CDN issues that we have been having, although attachments are still currently affected. 2023-10-24 16:25:14 +01:00
Sam Rose
767900aec6
Remove type information from JSDoc in .ts files, it's redundant. 2023-10-17 16:46:32 +01:00
Adria Navarro
5ef90a3855 Fix import 2023-10-13 09:08:07 +02:00
Adria Navarro
18545b1c79 Fix vulnerabilities 2023-10-03 23:45:39 +02:00
adrinr
ac4e45a433 Replace usages 2023-04-17 10:27:37 +01:00
mike12345567
24fc72657d Fixing all broken test cases after breaking out context fully from jestSetup/backend-core jest loading. 2023-03-30 16:35:25 +01:00
Rory Powell
ae9979929a Enable higher concurrency and resiliency in worker tests 2023-02-13 11:57:30 +00:00
adrinr
7027abeaa2 Fix issues with parallel tests and self/cloud modes 2023-01-26 18:20:21 +00:00
Hong Phuc Nguyen
6825b2e3fc Doing sanitizeKey when doing deleteFile(s) (#9209)
* Doing sanitizeKey when doing deleteFile(s)

Doing sanitizeKey when doing deleteFile(s). 
In case of backup file name in backups bucket backup-2023-01-03T185512.686Z.tar.gz but in global-db store backup filename is backup-2023-01-03T18:55:12.686Z.tar.gz
sanitizeKey on deleteFile(s) method will prevent all other case related to this bug

* use .promise() instead callback for workaround aws-sdk bug
2023-01-05 08:54:10 +00:00
Rory Powell
f4a39080de Minio and local licensing fixes (#9071) 2022-12-16 11:14:31 +00:00
Rory Powell
786039055e File authentication using presigned URLs (#8883)
* file auth wip

* Private plugin bucket wip

* Add etag to logo request, fix circular dependency

* Resolve cyclic dependency

* Ensure browser doesn't cache custom component

* Global file fixes

* File fixes / remove permaUrl

* Add ctx local storage

* Enable presigned url to work from any host

* Test fix

* Self hosting w/ S3 + other fixes

* Fix for serving dev app in multi tenant

* Fix backup restore and import attachment url processing, never store attachment urls

* Test fixes

* Unit tests WIP

* Replace file path with base64 encoded private key

* Multi tenancy tests for files

* Environment files fixes

* Post-merge build and test fixes

* Add debug conditional to all jest logging and revert/fix prod minio proxy_pass

* Re-add default '/' minio route :(

* Rename iconFile -> iconFileName

* Add cloudfront vars to k8s templates

* Remove public bucket policy

* Remove duplicated test
2022-12-15 11:35:22 +00:00
mike12345567
d3c4c99e16 Complete conversion of backend-core to Typescript. 2022-11-24 18:48:51 +00:00
mike12345567
de82eca502 Some typescript conversions, moving a few imports around. 2022-11-21 18:33:34 +00:00
mike12345567
693d6b6242 Typescript re-write of the roles layer, this is the backbone of a lot of our security features, and I believe the issue was generally to do with a lack of handling of null-ish inputs. 2022-11-08 18:25:37 +00:00
mike12345567
2f01caa88d Adding more type handling around the upload object store function. 2022-11-08 11:49:07 +00:00
mike12345567
913eb96ab0 Fixing an issue which was alerting, x-amz-meta-fieldname contains an invalid value, this error isn't really documented, but comes from if a metadata tag is sent up with nullish, or a non-string type. Fixing this in the core library, removing an invalid inputs. 2022-11-08 11:34:16 +00:00
Rory Powell
887470d4b5 Ensure tmpDir/.budibase always exists 2022-10-26 09:24:28 +01:00
mike12345567
0bd2a18e46 Main types and work for the CRUD operations of app backup backend in pro + the listeners to handle exporting apps from the server. 2022-10-14 19:24:03 +01:00
mike12345567
19133f08e6 Adding import functionality - still need to update the attachments URL. 2022-10-11 20:25:22 +01:00
mike12345567
f237befbce Some fixes + cleanup of tmp directory. 2022-10-11 19:28:13 +01:00
mike12345567
1f36eec89a Some updates towards supporting attachments in app exports. 2022-10-10 20:08:59 +01:00
mike12345567
95136a54f2 Adding ability for datasource plugins to have a custom icon svg. 2022-09-29 19:30:53 +01:00