Cloud runner v0.2 - continued quality of life improvements (#387)

* Update cloud-runner-aws-pipeline.yml * Update cloud-runner-k8s-pipeline.yml * yarn build * yarn build * correct branch ref * correct branch ref passed to target repo * Create k8s-tests.yml * Delete k8s-tests.yml * correct branch ref passed to target repo * correct branch ref passed to target repo * Always describe AWS tasks for now, because unstable error handling * Remove unused tree commands * Use lfs guid sum * Simple override cache push * Simple override cache push and pull override to allow pure cloud storage driven caching * Removal of early branch (breaks lfs caching) * Remove unused tree commands * Update action.yml * Update action.yml * Support cache and input override commands as input + full support custom hooks * Increase k8s timeout * replace filename being appended for unknclear reason * cache key should not contain whitespaces * Always try and deploy rook for k8s * Apply k8s files for rook * Update action.yml * Apply k8s files for rook * Apply k8s files for rook * cache test and action description for kuber storage class * Correct test and implement dependency health check and start * GCP-secret run, cache key * lfs smudge set explicit and undo explicit * Run using external secret provider to speed up input * Update cloud-runner-aws-pipeline.yml * Add nodejs as build step dependency * Add nodejs as build step dependency * Cloud Runner Tests must be specified to capture logs from cloud runner for tests * Cloud Runner Tests must be specified to capture logs from cloud runner for tests * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * better defaults for new inputs * better defaults * merge latest * force build update * use npm n to update node in unity builder * use npm n to update node in unity builder * use npm n to update node in unity builder * correct new line * quiet zipping * quiet zipping * default secrets for unity username and password * default secrets for unity username and password * ls active directory before lfs install * Get cloud runner secrets from * Get cloud runner secrets from * Cleanup setup of default secrets * Various fixes * Cleanup setup of default secrets * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * AWS secrets manager support * less caching logs * default k8s storage class to pd-standard * more readable build commands * Capture aws exit code 1 reliably * Always replace /head from branch * k8s default storage class to standard-rwo * cleanup * further cleanup input * further cleanup input * further cleanup input * further cleanup input * further cleanup input * folder sizes to inspect caching * dir command for local cloud runner test * k8s wait for pending because pvc will not create earlier * prefer k8s standard storage * handle empty string as cloud runner cluster input * local-system is now used for cloud runner test implementation AND correctly unset test CLI input * local-system is now used for cloud runner test implementation AND correctly unset test CLI input * fix unterminated quote * fix unterminated quote * do not share build parameters in tests - in cloud runner this will cause conflicts with resouces of the same name * remove head and heads from branch prefix * fix reversed caching direction of cache-push * fixes * fixes * fixes * cachePull cli * fixes * fixes * fixes * fixes * fixes * order cache test to be first * order cache test to be first * fixes * populate cache key instead of using branch * cleanup cli * garbage-collect-aws cli can iterate over aws resources and cli scans all ts files * import cli methods * import cli files explicitly * import cli files explicitly * import cli files explicitly * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * log parameters in cloud runner parameter test * log parameters in cloud runner parameter test * log parameters in cloud runner parameter test * Cloud runner param test before caching because we have a fast local cache test now * Using custom build path relative to repo root rather than project root * aws-garbage-collect at end of pipeline * aws-garbage-collect do not actually delete anything for now - just list * remove some legacy du commands * Update cloud-runner-aws-pipeline.yml * log contents after cache pull and fix some scenarios with duplicate secrets * log contents after cache pull and fix some scenarios with duplicate secrets * log contents after cache pull and fix some scenarios with duplicate secrets * PR comments * Replace guid with uuid package * use fileExists lambda instead of stat to check file exists in caching * build failed results in core error message * Delete sample.txt * cloud-runner-system prefix changed to cloud-runner * Update cloud-runner-aws-pipeline.yml * remove du from caching, should run manually if interested in size, adds too much runtime to job to include by default * github ephemeral pipeline support * github ephemeral pipeline support * Merge remote-tracking branch 'origin/main' into cloud-runner-develop # Conflicts: # dist/index.js.map # src/model/cloud-runner/providers/aws/aws-task-runner.ts # src/model/cloud-runner/providers/aws/index.ts * garbage collection * garbage collection * self hosted runner pipeline * self hosted runner pipeline * self hosted runner pipeline * self hosted runner pipeline * self hosted runner pipeline * self hosted runner pipeline * self hosted runner pipeline * self hosted runner pipeline * self hosted runner pipeline * self hosted runner pipeline * ephemeral runner pipeline * ephemeral runner pipeline * ephemeral runner pipeline * download runner each time * download runner each time * download runner each time * garbage collect all older than 1d as part of cleanup * download runner each time * number container cpu and memory for aws * per provider container defaults * per provider container defaults * per provider container defaults * per provider container defaults * Skip printing size unless cloudRunnerIntegrationTests is true * transition zip usage in cache to uncompressed tar for speed * transition zip usage in cache to uncompressed tar for speed * transition zip usage in cache to uncompressed tar for speed * transition zip usage in cache to uncompressed tar for speed * per provider container defaults * per provider container defaults * per provider container defaults * per provider container defaults * per provider container defaults * per provider container defaults * per provider container defaults * per provider container defaults * baked in cloud formation template * baked in cloud formation template * baked in cloud formation template * baked in cloud formation template * baked in cloud formation template * baked in cloud formation template * baked in cloud formation template * baked in cloud formation template * better aws commands * better aws commands * parse number for cloud formation template * remove container resource defaults from actions yaml * remove container resource defaults from actions yaml * skip all input readers when cloud runner is local * prefer fs/promises * actually set aws cloud runner step as failure if unity build fails * default to 3gb of ram - webgl fails on 2
2026-06-14 20:16:48 -07:00 · 2022-04-22 00:47:45 +01:00
parent 5ae03dfef6
commit 8abce48a48
26 changed files with 995 additions and 863 deletions
@@ -1,7 +1,6 @@
-AWSTemplateFormatVersion: '2010-09-09'
-Description: AWS Fargate cluster that can span public and private subnets. Supports
-  public facing load balancers, private internal load balancers, and
-  both internal and external service discovery namespaces.
+export class BaseStackFormation {
+  public static readonly formation: string = `AWSTemplateFormatVersion: '2010-09-09'
+Description: Game-CI base stack
 Parameters:
  EnvironmentName:
    Type: String
@@ -335,57 +334,58 @@ Outputs:
    Description: 'The connection endpoint for the database.'
    Value: !Ref EfsFileStorage
    Export:
-      Name: !Sub ${EnvironmentName}:EfsFileStorageId
+      Name: !Sub ${'${EnvironmentName}'}:EfsFileStorageId
  ClusterName:
    Description: The name of the ECS cluster
    Value: !Ref 'ECSCluster'
    Export:
-      Name: !Sub ${EnvironmentName}:ClusterName
+      Name: !Sub${' ${EnvironmentName}'}:ClusterName
  AutoscalingRole:
    Description: The ARN of the role used for autoscaling
    Value: !GetAtt 'AutoscalingRole.Arn'
    Export:
-      Name: !Sub ${EnvironmentName}:AutoscalingRole
+      Name: !Sub ${'${EnvironmentName}'}:AutoscalingRole
  ECSRole:
    Description: The ARN of the ECS role
    Value: !GetAtt 'ECSRole.Arn'
    Export:
-      Name: !Sub ${EnvironmentName}:ECSRole
+      Name: !Sub ${'${EnvironmentName}'}:ECSRole
  ECSTaskExecutionRole:
    Description: The ARN of the ECS role tsk execution role
    Value: !GetAtt 'ECSTaskExecutionRole.Arn'
    Export:
-      Name: !Sub ${EnvironmentName}:ECSTaskExecutionRole
+      Name: !Sub ${'${EnvironmentName}'}:ECSTaskExecutionRole

  DeleteCFNLambdaExecutionRole:
    Description: Lambda execution role for cleaning up cloud formations
    Value: !GetAtt 'DeleteCFNLambdaExecutionRole.Arn'
    Export:
-      Name: !Sub ${EnvironmentName}:DeleteCFNLambdaExecutionRole
+      Name: !Sub ${'${EnvironmentName}'}:DeleteCFNLambdaExecutionRole

  CloudWatchIAMRole:
    Description: The ARN of the CloudWatch role for subscription filter
    Value: !GetAtt 'CloudWatchIAMRole.Arn'
    Export:
-      Name: !Sub ${EnvironmentName}:CloudWatchIAMRole
+      Name: !Sub ${'${EnvironmentName}'}:CloudWatchIAMRole
  VpcId:
    Description: The ID of the VPC that this stack is deployed in
    Value: !Ref 'VPC'
    Export:
-      Name: !Sub ${EnvironmentName}:VpcId
+      Name: !Sub ${'${EnvironmentName}'}:VpcId
  PublicSubnetOne:
    Description: Public subnet one
    Value: !Ref 'PublicSubnetOne'
    Export:
-      Name: !Sub ${EnvironmentName}:PublicSubnetOne
+      Name: !Sub ${'${EnvironmentName}'}:PublicSubnetOne
  PublicSubnetTwo:
    Description: Public subnet two
    Value: !Ref 'PublicSubnetTwo'
    Export:
-      Name: !Sub ${EnvironmentName}:PublicSubnetTwo
-
+      Name: !Sub ${'${EnvironmentName}'}:PublicSubnetTwo
  ContainerSecurityGroup:
    Description: A security group used to allow Fargate containers to receive traffic
    Value: !Ref 'ContainerSecurityGroup'
    Export:
-      Name: !Sub ${EnvironmentName}:ContainerSecurityGroup
+      Name: !Sub ${'${EnvironmentName}'}:ContainerSecurityGroup
+`;
+}
@@ -1,4 +1,5 @@
-AWSTemplateFormatVersion: 2010-09-09
+export class TaskDefinitionFormation {
+  public static readonly formation: string = `AWSTemplateFormatVersion: 2010-09-09
 Description: >-
  AWS Fargate cluster that can span public and private subnets. Supports public
  facing load balancers, private internal load balancers, and both internal and
@@ -23,12 +24,12 @@ Parameters:
    Default: 80
    Description: What port number the application inside the docker container is binding to
  ContainerCpu:
-    Type: Number
    Default: 1024
+    Type: Number
    Description: How much CPU to give the container. 1024 is 1 CPU
  ContainerMemory:
-    Type: Number
    Default: 2048
+    Type: Number
    Description: How much memory in megabytes to give the container
  BUILDGUID:
    Type: String
@@ -78,7 +79,7 @@ Resources:
    Properties:
      FilterPattern: ''
      RoleArn:
-        'Fn::ImportValue': !Sub '${EnvironmentName}:CloudWatchIAMRole'
+        'Fn::ImportValue': !Sub '${'${EnvironmentName}'}:CloudWatchIAMRole'
      LogGroupName: !Ref ServiceName
      DestinationArn:
        'Fn::GetAtt':
@@ -98,9 +99,7 @@ Resources:
    Metadata:
      'AWS::CloudFormation::Designer':
        id: c6f18447-b879-4696-8873-f981b2cedd2b
-
  # template secrets p2 - secret
-
  TaskDefinition:
    Type: 'AWS::ECS::TaskDefinition'
    Properties:
@@ -112,12 +111,12 @@ Resources:
        - Name: efs-data
          EFSVolumeConfiguration:
            FilesystemId:
-              'Fn::ImportValue': !Sub '${EnvironmentName}:EfsFileStorageId'
+              'Fn::ImportValue': !Sub '${'${EnvironmentName}'}:EfsFileStorageId'
            TransitEncryption: ENABLED
      RequiresCompatibilities:
        - FARGATE
      ExecutionRoleArn:
-        'Fn::ImportValue': !Sub '${EnvironmentName}:ECSTaskExecutionRole'
+        'Fn::ImportValue': !Sub '${'${EnvironmentName}'}:ECSTaskExecutionRole'
      TaskRoleArn:
        'Fn::If':
          - HasCustomRole
@@ -153,69 +152,7 @@ Resources:
              awslogs-group: !Ref ServiceName
              awslogs-region: !Ref 'AWS::Region'
              awslogs-stream-prefix: !Ref ServiceName
-    Metadata:
-      'AWS::CloudFormation::Designer':
-        id: dabb0116-abe0-48a6-a8af-cf9111c879a5
    DependsOn:
      - LogGroup
-Metadata:
-  'AWS::CloudFormation::Designer':
-    dabb0116-abe0-48a6-a8af-cf9111c879a5:
-      size:
-        width: 60
-        height: 60
-      position:
-        x: 270
-        'y': 90
-      z: 1
-      embeds: []
-      dependson:
-        - aece53ae-b82d-4267-bc16-ed964b05db27
-    c6f18447-b879-4696-8873-f981b2cedd2b:
-      size:
-        width: 60
-        height: 60
-      position:
-        x: 270
-        'y': 210
-      z: 1
-      embeds: []
-    7f809e91-9e5d-4678-98c1-c5085956c480:
-      size:
-        width: 60
-        height: 60
-      position:
-        x: 60
-        'y': 300
-      z: 1
-      embeds: []
-      dependson:
-        - aece53ae-b82d-4267-bc16-ed964b05db27
-        - c6f18447-b879-4696-8873-f981b2cedd2b
-    aece53ae-b82d-4267-bc16-ed964b05db27:
-      size:
-        width: 150
-        height: 150
-      position:
-        x: 60
-        'y': 90
-      z: 1
-      embeds: []
-    4d2da56c-3643-46b8-aaee-e46e19f95fcc:
-      source:
-        id: 7f809e91-9e5d-4678-98c1-c5085956c480
-      target:
-        id: aece53ae-b82d-4267-bc16-ed964b05db27
-      z: 11
-    14eb957b-f094-4653-93c4-77b2f851953c:
-      source:
-        id: 7f809e91-9e5d-4678-98c1-c5085956c480
-      target:
-        id: c6f18447-b879-4696-8873-f981b2cedd2b
-      z: 12
-    85c57444-e5bb-4230-bc85-e545cd4558f6:
-      source:
-        id: dabb0116-abe0-48a6-a8af-cf9111c879a5
-      target:
-        id: aece53ae-b82d-4267-bc16-ed964b05db27
-      z: 13
+`;
+}