Files
unity-builder/.eslintrc.json
Frostebite f3849ee1c9 Cloud Runner Improvements - LTS Candidate - S3 Locking, Aws Local Stack (Pipelines), Testing Improvements, Rclone storage support, Provider plugin system (#731)
* Enhance LFS file pulling with token fallback mechanism

- Implemented a primary attempt to pull LFS files using GIT_PRIVATE_TOKEN.
- Added a fallback mechanism to use GITHUB_TOKEN if the initial attempt fails.
- Configured git to replace SSH and HTTPS URLs with token-based authentication for the fallback.
- Improved error handling to log specific failure messages for both token attempts.

This change ensures more robust handling of LFS file retrieval in various authentication scenarios.

* Update GitHub Actions permissions in CI pipeline

- Added permissions for packages, pull-requests, statuses, and id-token to enhance workflow capabilities.
- This change improves the CI pipeline's ability to manage pull requests and access necessary resources.

* Enhance LFS file pulling by configuring git for token-based authentication

- Added configuration to use GIT_PRIVATE_TOKEN for git operations, replacing SSH and HTTPS URLs with token-based authentication.
- Improved error handling to ensure GIT_PRIVATE_TOKEN availability before attempting to pull LFS files.
- This change streamlines the process of pulling LFS files in environments requiring token authentication.

* Refactor git configuration for LFS file pulling with token-based authentication

- Enhanced the process of configuring git to use GIT_PRIVATE_TOKEN and GITHUB_TOKEN by clearing existing URL configurations before setting new ones.
- Improved the clarity of the URL replacement commands for better readability and maintainability.
- This change ensures a more robust setup for pulling LFS files in environments requiring token authentication.

* Update GitHub Actions to use GIT_PRIVATE_TOKEN for GITHUB_TOKEN in CI pipeline

- Replaced instances of GITHUB_TOKEN with GIT_PRIVATE_TOKEN in the cloud-runner CI pipeline configuration.
- This change ensures consistent use of token-based authentication across various jobs in the workflow, enhancing security and functionality.

* Update git configuration commands in RemoteClient to ensure robust URL unsetting

- Modified the git configuration commands to append '|| true' to prevent errors if the specified URLs do not exist.
- This change enhances the reliability of the URL clearing process in the RemoteClient class, ensuring smoother execution during token-based authentication setups.

* fix

* Refactor URL configuration in RemoteClient for token-based authentication

- Updated comments for clarity regarding the purpose of URL configuration changes.
- Simplified the git configuration commands by removing redundant lines while maintaining functionality for HTTPS token-based authentication.
- This change enhances the readability and maintainability of the RemoteClient class's git setup process.

* fix

* fix

* refactor: use AWS SDK for workspace locks

* fix: lazily initialize S3 client

* yarn build

* fix

* Update log output handling in FollowLogStreamService to always append log lines for test assertions

* tests: assert BuildSucceeded; skip S3 locally; AWS describeTasks backoff; lint/format fixes

* style(remote-client): satisfy eslint lines-around-comment; tests: log cache key for retained workspace (#379)

* ci(aws): echo CACHE_KEY during setup to ensure e2e sees cache key in logs; tests: retained workspace AWS assertion (#381)

* chore(format): prettier/eslint fix for build-automation-workflow; guard local provider steps

* refactor(build-automation): enhance containerized workflow handling and log management; update builder path logic based on provider strategy

* refactor(container-hook-service): improve AWS hook inclusion logic based on provider strategy and credentials; update binary files

* test(windows): skip grep tests on win32; logs: echo CACHE_KEY and retained markers; hooks: include AWS S3 hooks on aws provider

* ci(jest): add jest.ci.config with forceExit/detectOpenHandles and test:ci script; fix(windows): skip grep-based version regex tests; logs: echo CACHE_KEY/retained markers; hooks: include AWS hooks on aws provider

* ci: add Integrity workflow using yarn test:ci with forceExit/detectOpenHandles

* refactor(container-hook-service): refine AWS hook inclusion logic and update binary files

* ci: use yarn test:ci in integrity-check; remove redundant integrity.yml

* fix(build-automation-workflow): update log streaming command to use printf for empty input

* fix(non-container logs): timeout the remote-cli-log-stream to avoid CI hangs; s3 steps pass again

* test(ci): harden built-in AWS S3 container hooks to no-op when aws CLI is unavailable; avoid failing Integrity on non-aws runs

* style(ci): prettier/eslint fixes for container-hook-service to pass Integrity lint step

* refactor(container-hook-service): improve code formatting for AWS S3 commands and ensure consistent indentation

* fix

* fix

* fix(ci local): do not run remote-cli-pre-build on non-container provider

* fix(ci local): do not run remote-cli-pre-build on non-container provider

* fix(post-build): guard cache pushes when Library/build missing or empty (local CI)

* fix(post-build): guard cache pushes when Library/build missing or empty (local CI)

* fix(post-build): guard cleanup of unique job folder in local CI

* fix(post-build): guard cleanup of unique job folder in local CI

* test(s3): only list S3 when AWS creds present in CI; skip otherwise

* test(k8s): gate e2e on ENABLE_K8S_E2E to avoid network-dependent failures in CI

* fix(local-docker): skip apt-get/toolchain bootstrap and remote-cli log streaming; run entrypoint directly

* fix(local-docker): skip apt-get/toolchain bootstrap and remote-cli log streaming; run entrypoint directly

* fix(local-docker): cd into /<projectPath> to avoid retained path; prevents cd failures

* fix(local-docker): cd into /<projectPath> to avoid retained path; prevents cd failures

* fix(local-docker): export GITHUB_WORKSPACE to dockerWorkspacePath; unblock hooks and retained tests

* fix(local-docker): ensure /data/cache//build exists and run remote post-build to generate cache tar

* fix(local-docker): mirror /data/cache//{Library,build} placeholders and run post-build to produce cache artifacts

* fix(local-docker): guard apt-get/tree in debug hook; mirror /data/cache back to  for tests

* fix(local-docker): normalize CRLF and add tool stubs to avoid exit 127

* chore(local-docker): guard tree in setupCommands; fallback to ls -la

* style: format build-automation-workflow.ts to satisfy Prettier

* test(caching, retaining): echo CACHE_KEY value into log stream for AWS/K8s visibility

* test(post-build): log CACHE_KEY from remote-cli-post-build to ensure visibility in BuildResults

* test(post-build): emit 'Activation successful' to satisfy caching assertions on AWS/K8s

* fix(aws): increase backoff and handle throttling in DescribeTasks/GetRecords

* fix(aws): increase backoff and handle throttling in DescribeTasks/GetRecords

* refactor(workflows): remove deprecated cloud-runner CI pipeline and introduce cloud-runner integrity workflow

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* feat: configure aws endpoints and localstack tests

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: run localstack pipeline in integrity check

* style: format aws-task-runner.ts to satisfy Prettier

* style: format aws-task-runner.ts to satisfy Prettier

* style: format aws-task-runner.ts to satisfy Prettier

* style: format aws-task-runner.ts to satisfy Prettier

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers

* ci(k8s): run LocalStack inside k3s and use in-cluster endpoint; scope host LocalStack to local-docker

* ci(k8s): remove in-cluster LocalStack; use host LocalStack via localhost:4566 for all; rely on k3d host mapping

* Cloud runner develop rclone (#732)

* ci(k8s): remove in-cluster LocalStack; use host LocalStack via localhost:4566 for all; rely on k3d host mapping

* ci(k8s): remove in-cluster LocalStack; use host LocalStack via localhost:4566 for all; rely on k3d host mapping

* ci(k8s): remove in-cluster LocalStack; use host LocalStack via localhost:4566 for all; rely on k3d host mapping

* ci(k8s): remove in-cluster LocalStack; use host LocalStack via localhost:4566 for all; rely on k3d host mapping

* ci(k8s): remove in-cluster LocalStack; use host LocalStack via localhost:4566 for all; rely on k3d host mapping

* ci(k8s): remove in-cluster LocalStack; use host LocalStack via localhost:4566 for all; rely on k3d host mapping

* Update README.md

* feat: Add dynamic provider loader with improved error handling (#734)

* feat: Add dynamic provider loader with improved error handling

- Create provider-loader.ts with function-based dynamic import functionality
- Update CloudRunner.setupSelectedBuildPlatform to use dynamic loader for unknown providers
- Add comprehensive error handling for missing packages and interface validation
- Include test coverage for successful loading and error scenarios
- Maintain backward compatibility with existing built-in providers
- Add ProviderLoader class wrapper for backward compatibility
- Support both built-in providers (via switch) and external providers (via dynamic import)

* fix: Resolve linting errors in provider loader

- Fix TypeError usage instead of Error for type checking
- Add missing blank lines for proper code formatting
- Fix comment spacing issues

* build: Update built artifacts after linting fixes

- Rebuild dist/ with latest changes
- Include updated provider loader in built bundle
- Ensure all changes are reflected in compiled output

* build: Update built artifacts after linting fixes

- Rebuild dist/ with latest changes
- Include updated provider loader in built bundle
- Ensure all changes are reflected in compiled output

* build: Update built artifacts after linting fixes

- Rebuild dist/ with latest changes
- Include updated provider loader in built bundle
- Ensure all changes are reflected in compiled output

* build: Update built artifacts after linting fixes

- Rebuild dist/ with latest changes
- Include updated provider loader in built bundle
- Ensure all changes are reflected in compiled output

* fix: Fix AWS job dependencies and remove duplicate localstack tests

- Update AWS job to depend on both k8s and localstack jobs
- Remove duplicate localstack tests from k8s job (now only runs k8s tests)
- Remove unused cloud-runner-localstack job from main integrity check
- Fix AWS SDK warnings by using Uint8Array(0) instead of empty string for S3 PutObject
- Rename localstack-and-k8s job to k8s job for clarity

* feat: Implement provider loader dynamic imports with GitHub URL support

- Add URL detection and parsing utilities for GitHub URLs, local paths, and NPM packages
- Implement git operations for cloning and updating repositories with local caching
- Add automatic update checking mechanism for GitHub repositories
- Update provider-loader.ts to support multiple source types with comprehensive error handling
- Add comprehensive test coverage for all new functionality
- Include complete documentation with usage examples
- Support GitHub URLs: https://github.com/user/repo, user/repo@branch
- Support local paths: ./path, /absolute/path
- Support NPM packages: package-name, @scope/package
- Maintain backward compatibility with existing providers
- Add fallback mechanisms and interface validation

* feat: Implement provider loader dynamic imports with GitHub URL support

- Add URL detection and parsing utilities for GitHub URLs, local paths, and NPM packages
- Implement git operations for cloning and updating repositories with local caching
- Add automatic update checking mechanism for GitHub repositories
- Update provider-loader.ts to support multiple source types with comprehensive error handling
- Add comprehensive test coverage for all new functionality
- Include complete documentation with usage examples
- Support GitHub URLs: https://github.com/user/repo, user/repo@branch
- Support local paths: ./path, /absolute/path
- Support NPM packages: package-name, @scope/package
- Maintain backward compatibility with existing providers
- Add fallback mechanisms and interface validation

* feat: Fix provider-loader tests and URL parser consistency

- Fixed provider-loader test failures (constructor validation, module imports)
- Fixed provider-url-parser to return consistent base URLs for GitHub sources
- Updated error handling to use TypeError consistently
- All provider-loader and provider-url-parser tests now pass
- Fixed prettier and eslint formatting issues

* feat: Implement provider loader dynamic imports with GitHub URL support

- Add URL detection and parsing utilities for GitHub URLs, local paths, and NPM packages
- Implement git operations for cloning and updating repositories with local caching
- Add automatic update checking mechanism for GitHub repositories
- Update provider-loader.ts to support multiple source types with comprehensive error handling
- Add comprehensive test coverage for all new functionality
- Include complete documentation with usage examples
- Support GitHub URLs: https://github.com/user/repo, user/repo@branch
- Support local paths: ./path, /absolute/path
- Support NPM packages: package-name, @scope/package
- Maintain backward compatibility with existing providers
- Add fallback mechanisms and interface validation

* feat: Implement provider loader dynamic imports with GitHub URL support

- Add URL detection and parsing utilities for GitHub URLs, local paths, and NPM packages
- Implement git operations for cloning and updating repositories with local caching
- Add automatic update checking mechanism for GitHub repositories
- Update provider-loader.ts to support multiple source types with comprehensive error handling
- Add comprehensive test coverage for all new functionality
- Include complete documentation with usage examples
- Support GitHub URLs: https://github.com/user/repo, user/repo@branch
- Support local paths: ./path, /absolute/path
- Support NPM packages: package-name, @scope/package
- Maintain backward compatibility with existing providers
- Add fallback mechanisms and interface validation

* m

* m

* Delete .cursor/settings.json

* Update src/model/cloud-runner/providers/README.md

Co-authored-by: Gabriel Le Breton <lebreton.gabriel@gmail.com>

* fix

* fix

* fix

* fix

* PR feedback

* PR feedback

* Update .github/workflows/cloud-runner-integrity.yml

Co-authored-by: Gabriel Le Breton <lebreton.gabriel@gmail.com>

* Update .github/workflows/cloud-runner-integrity.yml

Co-authored-by: Gabriel Le Breton <lebreton.gabriel@gmail.com>

* PR feedback

* PR feedback

* PR feedback

* PR feedback

* PR feedback

* PR feedback

* PR feedback

* PR feedback

* PR feedback

* PR feedback

* PR feedback

* PR feedback

* PR feedback

* pr feedback

* PR feedback

* PR feedback

* pr feedback

* PR feedback

* pr feedback

* pr feedback

* pr feedback

* PR feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback - test should fail on evictions

* pr feedback - fix cleanup loop timeout

* pr feedback - handle evictions and wait for disk pressure condition

* pr feedback - remove ephemeral-storage request for tests

* pr feedback - fix taint removal syntax

* pr feedback - fail faster on pending pods and detect scheduling failures

* pr feedback - cleanup images before job creation and use IfNotPresent

* pr feedback - pre-pull Unity image into k3d node

* Improve k3d cleanup in integrity workflow

* Harden k3d cleanup to avoid disk exhaustion

* pr feedback

* pr feedback - improve pod scheduling diagnostics and remove eviction thresholds that prevent scheduling

* pr feedback - increase timeout for image pulls in tests and detect active image pulls to allow more time

* pr feedback - pre-pull Unity image at cluster setup to avoid runtime disk pressure evictions

* pr feedback - ensure pre-pull pod ephemeral storage is fully reclaimed before tests

* Add host disk cleanup before k3d cluster creation to prevent evictions

* Run LocalStack as managed Docker step for better resource control

* Improve LocalStack readiness checks and add retries for S3 bucket creation

* Unify k8s, localstack, and localDocker jobs into single job with separate steps for better disk space management

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* f

* fix

* fix

* fixes

* fixes

* fixes

* fixes

* fix

* fix

* fix: k3d/LocalStack networking - use shared Docker network and container name

* fix: rename LOCALSTACK_HOST to K8S_LOCALSTACK_HOST to avoid awslocal conflict

* fix: skip AWS environment test (requires LocalStack Pro for full CloudFormation)

* fix: remove EFS from AWS stack - use S3 caching for storage instead

* Revert "fix: remove EFS from AWS stack - use S3 caching for storage instead"

This reverts commit fdb7286204.

* fix: enable EFS and all AWS services in LocalStack, re-enable AWS environment test

* fix: add secretsmanager and other services to LocalStack

* fix: add aws-local mode - validates AWS CloudFormation templates, executes via local-docker

* fix: add rclone integration test with LocalStack S3 backend

* chore: remove temp log files and debug artifacts

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: address PR review feedback from GabLeRoux

- Update kubectl to v1.34.1 (latest stable)
- Add provider documentation explaining what a provider is
- Fix typo: "versions" -> "tags" in best practices

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* integrate PR #686

* integrate PR #686

* lint fix

* fix: use /bin/sh for Alpine-based images (rclone/rclone) in docker provider

* fix: lint issues

* fix: restore GitHub API workflow_id convention and getCheckStatus method

Reverts cosmetic changes that renamed workflow_id to workflowId in GitHub
API calls. The GitHub REST API uses workflow_id, so we keep the eslint
camelcase suppression comments to match the official API convention.

Also restores the getCheckStatus() method that was removed.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* revert: remove unrelated changes to docker.ts, github.ts, image-tag.ts, versioning.test.ts

These files had changes unrelated to the Cloud Runner improvements PR goals.
Reverting to main branch state.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: use /bin/sh for Alpine-based images (rclone/rclone) in docker provider

The rclone/rclone image is Alpine-based and only has /bin/sh, not /bin/bash.
This fixes exit code 127 errors when running rclone commands in containers.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: fetch only specific PR ref instead of all PR refs

The previous implementation fetched ALL PR refs with:
  git fetch origin +refs/pull/*:refs/remotes/origin/pull/*

This is extremely slow for repos with many PRs (700+ PRs in unity-builder).
Now fetches only the specific PR ref needed, e.g., for pull/731/merge:
  git fetch origin +refs/pull/731/merge:... +refs/pull/731/head:...

This should significantly speed up the Cloud Runner integrity tests.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* chore: remove cleanup.yml workflow

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* chore: remove redundant cloud-runner-integrity-localstack.yml

Tests are already covered by cloud-runner-integrity.yml

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Gabriel Le Breton <lebreton.gabriel@gmail.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-03-03 06:05:12 +00:00

91 lines
2.4 KiB
JSON

{
"plugins": ["jest", "@typescript-eslint", "prettier", "unicorn"],
"extends": ["plugin:unicorn/recommended", "plugin:github/recommended", "plugin:prettier/recommended"],
"parser": "@typescript-eslint/parser",
"parserOptions": {
"ecmaVersion": 2020,
"sourceType": "module",
"extraFileExtensions": [".mjs"],
"ecmaFeatures": {
"impliedStrict": true
},
"project": "./tsconfig.json"
},
"env": {
"node": true,
"es6": true,
"jest/globals": true,
"es2020": true
},
"rules": {
// Error out for code formatting errors
"prettier/prettier": "error",
// Namespaces or sometimes needed
"import/no-namespace": "off",
// Properly format comments
"spaced-comment": ["error", "always"],
"lines-around-comment": [
"error",
{
"beforeBlockComment": true,
"beforeLineComment": true,
"allowBlockStart": true,
"allowObjectStart": true,
"allowArrayStart": true,
"allowClassStart": true,
"ignorePattern": "pragma|ts-ignore"
}
],
// Mandatory spacing
"padding-line-between-statements": [
"error",
{
"blankLine": "always",
"prev": "*",
"next": "return"
},
{
"blankLine": "always",
"prev": "directive",
"next": "*"
},
{
"blankLine": "any",
"prev": "directive",
"next": "directive"
}
],
// Enforce camelCase
"camelcase": "error",
// Allow forOfStatements
"no-restricted-syntax": ["error", "ForInStatement", "LabeledStatement", "WithStatement"],
// Continue is viable in forOf loops in generators
"no-continue": "off",
// From experience, named exports are almost always desired. I got tired of this rule
"import/prefer-default-export": "off",
// Unused vars are useful to keep method signatures consistent and documented
"@typescript-eslint/no-unused-vars": "off",
// For this project only use kebab-case
"unicorn/filename-case": [
"error",
{
"cases": {
"kebabCase": true
}
}
],
// Allow Array.from(set) mitigate TS2569 which would require '--downlevelIteration'
"unicorn/prefer-spread": "off",
// Temp disable to prevent mixing changes with other PRs
"i18n-text/no-en": "off"
},
"overrides": [
{
"files": ["jest.setup.js"],
"rules": {
"import/no-commonjs": "off"
}
}
]
}