#Running CodeScoring in Kubernetes

#Installation with a Helm chart

Installation procedure:

1. Create a namespace.

   kubectl create namespace codescoring

2. Create a secret for accessing the private Docker image registry for CodeScoring. Use the registry address (REGISTRY_URL), login (USERNAME), and password (PASSWORD) received from the vendor.

   kubectl create secret docker-registry codescoring-regcred --docker-server=REGISTRY_URL --docker-username=USERNAME --docker-password=PASSWORD -n codescoring

   Alternatively, you can create a registry access secret through values.yaml by adding the following entry to .Values.imagePullSecrets:

   registry.org: |
     {"auths":{"registry.org":{"auth":"<base64>"}}}

3. Install Helm using your preferred method.

4. Run the following commands to add the current Helm repository to the local machine:

   helm repo add codescoring-org https://{REGISTRY_URL}/repository/helm/ --username USERNAME --password PASSWORD
   helm repo update

#Helm chart settings

To edit CodeScoring parameters conveniently, download and unpack the Helm chart source code:

helm pull codescoring-org/codescoring-helm --version CHART_VERSION --untar --untardir codescoring-src && cd codescoring-src

In values.yaml, edit the required variables and then run the installation command from the Helm chart source directory:

helm install codescoring . -f values.yaml -n codescoring --atomic --version CHART_VERSION

The chart stores all configuration, including Kubernetes resource definitions, in values.yaml, which makes the file fairly large and not always convenient to edit. If you need to override any parameters, it is acceptable and recommended to create an additional file with an arbitrary name, for example values-override.yaml. This document uses that name below. For convenience, the chart includes a values-override.yaml template.

In this case, the installation command looks as follows. The order of files matters:

helm install codescoring . -f values.yaml -f values-override.yaml -n codescoring --atomic --version CHART_VERSION

#Installation with built-in PostgreSQL and Redis

The chart can deploy PostgreSQL and Redis using StatefulSet resources. To enable PostgreSQL and Redis installation, define the following parameters in values-override.yaml:

statefulSets:
  codescoring-postgresql:
    enabled: true
    containers:
      postgresql:
        resources:
          limits:
            cpu: 2000m
            memory: 12Gi
          requests:
            cpu: 1m
            memory: 500Mi

  codescoring-redis:
    enabled: true
    containers:
      redis:
        resources:
          limits:
            cpu: 2000m
            memory: 4Gi

pvcs:
  codescoring-postgresql:
    accessModes:
      - ReadWriteOnce
    size: 20Gi
    storageClassName: "default"
  codescoring-redis:
    accessModes:
      - ReadWriteOnce
    size: 1Gi
    storageClassName: "default"

When using built-in databases, no additional connection configuration is required.

#Connecting external databases

#Connecting external Redis

To connect external Redis, specify the corresponding connection strings in the following fields and keep the database numbers consistent:

configMaps:
  ipcs-backend-env:
    data:
      DJANGO_CACHES_REDIS_URL: "redis://codescoring-redis:6379/1"
      HUEY_REDIS_URL: "redis://codescoring-redis:6379/0"
      CELERY_BROKER_URL: "redis://codescoring-redis:6379/2"
      CELERY_RESULT_BACKEND: "redis://codescoring-redis:6379/2"

#Connecting to PostgreSQL through PgBouncer

This option is suitable when PostgreSQL is already deployed in the existing infrastructure, but a connection pooler is not used. The Helm chart deploys PgBouncer and connects it to the existing PostgreSQL. Complete the following steps:

1. Configure the connection pooler by specifying the corresponding values:

   secrets:
     pgbouncer-secrets:
       data:
         # secrets for connecting pgbouncer to PostgreSQL
         POSTGRES_DB: "codescoring_db"
         POSTGRES_USER: "codescoring_user"
         POSTGRES_PASSWORD: "changeme"
         # secrets for internal pgbouncer pools
         TRANSACTION_POOL_PASSWORD: "changeme"
         SESSION_POOL_PASSWORD: "changeme"
         STATS_USER: "codescoring_user-stats"
         STATS_PASSWORD: "changeme"
         ADMIN_USER: "codescoring_user-admin"
         ADMIN_PASSWORD: "changeme"
   
   configMaps:
     pgbouncer-env:
       data:
         # parameters for connecting pgbouncer to PostgreSQL
         POSTGRES_HOST: "codescoring-postgresql"
         POSTGRES_PORT: "5432"
         # parameters for internal pools
         TRANSACTION_POOL_USER: "codescoring_user"
         SESSION_POOL_USER: "codescoring_user-session"
         TRANSACTION_POOL_SIZE: "50"
         TRANSACTION_POOL_MIN_SIZE: "1"
         SESSION_POOL_SIZE: "50"
         SESSION_POOL_MIN_SIZE: "1"
         # virtual pgbouncer databases that exist only inside pgbouncer
         TRANSACTION_POOL_DATABASE_NAME: "codescoring_generic"
         SESSION_POOL_DATABASE_NAME: "codescoring_generic-session"
         MAX_CLIENT_CONN: "500"

2. Pass the configuration parameters created in step 1 to CodeScoring components:

   secrets:
     ipcs-secrets:
       data:
         # secrets for connecting to pools from pgbouncer-secrets
         TRANSACTION_POOL_PASSWORD: "changeme"
         SESSION_POOL_PASSWORD: "changeme"
         DATABASE_PASSWORD: "changeme" # matches TRANSACTION_POOL_PASSWORD
   
   configMaps:
     ipcs-backend-env:
       data:
         # connection parameters for osa-api and judge
         DATABASE_HOST: "pgbouncer"
         DATABASE_PORT: "6432"
         DATABASE_USERNAME: "codescoring_user" # matches TRANSACTION_POOL_USER
         DATABASE_NAME: "codescoring_db" # matches TRANSACTION_POOL_DATABASE_NAME
   
         TRANSACTION_POOL_HOST: "pgbouncer"
         TRANSACTION_POOL_PORT: "6432"
         TRANSACTION_POOL_USER: "codescoring_user"
         TRANSACTION_POOL_DATABASE_NAME: "codescoring_db"
   
         SESSION_POOL_HOST: "pgbouncer"
         SESSION_POOL_PORT: "6432"
         SESSION_POOL_USER: "codescoring_user-session"
         SESSION_POOL_DATABASE_NAME: "codescoring_generic-session"

#Configuring volumes

#Dynamic Volume Provisioning with the required StorageClass

The chart creates the required volumes through Dynamic Volume Provisioning using an explicitly specified StorageClass.

To create PersistentVolumeClaim resources, fill in the following values section:

pvcs:
  # replace storageClassName with the StorageClass used in the cluster
  codescoring-ipcs-django-static:
    accessModes:
      - ReadWriteOnce
    size: 10Gi
    storageClassName: "default"
  codescoring-ipcs-analysis-root:
    accessModes:
      - ReadWriteOnce
    size: 10Gi
    storageClassName: "default"
  codescoring-ipcs-media-root:
    accessModes:
      - ReadWriteOnce
    size: 10Gi
    storageClassName: "default"

If the cluster supports volumes with the ReadWriteMany (RWX) access mode, it is recommended to use that mode because it allows installation components to be placed on different cluster nodes. If the cluster does not support RWX, or if RWO is required for another reason, configure podAffinity so that pods using the same volumes are scheduled on the same node. To do this, add the following block to values-override.yaml:

deploymentsGeneral:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: codescoring-component
                operator: In
                values:
                  - backend
                  - frontend
                  - huey
                  - celery
          topologyKey: kubernetes.io/hostname

#PersistentVolumeClaim for pre-created PersistentVolume resources

You can specify names of pre-created PersistentVolume resources in the volumeName field, for example:

pvcs:
  codescoring-ipcs-analysis-root:
    accessModes:
      - ReadWriteOnce
    size: 10Gi
    storageClassName: "default"
    volumeName: "codescoring-precreated-pv"

#Configuring resource limits

By default, requests and limits are set to demonstration values that are not intended for production use. This makes it possible to run CodeScoring in clusters with limited resources, for example in minikube, for testing purposes. When running in a production environment, you may need to configure resource limits. Edit the following fields:

#Resource limits for init containers

# limits applied to all init containers
# this field can only be edited in values.yaml; override is not possible because of YAML structure limitations
init-container-resources: &init-container-resources
  resources:
    limits:
      cpu: 1000m
      memory: 1Gi
    requests:
      cpu: 1m
      memory: 100Mi

# limits for backend init containers are configured separately
deployments:
  ipcs-backend:
    initContainers:
      ipcs-collectstatic:
        resources:
          limits:
            cpu: 1000m
            memory: 2Gi
          requests:
            cpu: 1m
            memory: 500Mi
      ipcs-migrate:
        resources:
          limits:
            cpu: 1000m
            memory: 2Gi
          requests:
            cpu: 1m
            memory: 500Mi

#Resource limits for main component containers

In general, resources are configured using the following template:

deployments:
  <deployment-name>:
    containers:
      <container-name>:
        resources:
          limits:
            cpu: 8000m
            memory: 12Gi
          requests:
            cpu: 1m
            memory: 500Mi

You can specify requests and limits together or separately. However, if only limits are specified, Kubernetes automatically sets requests to the same values, which may negatively affect pod scheduling.

#Adding a Certificate Authority (CA) certificate

To allow CodeScoring to access resources with TLS certificates signed by a corporate Certificate Authority (CA), add the root CA certificate (RootCA) to values.yaml in key: value format. The key is the certificate file name, including the .crt extension. The value is the certificate in PEM format. Add it to the following field:

secrets:
  ca-certificates:
    data:
      my-root-CA.crt: |-
        -----BEGIN CERTIFICATE-----
        MIIDTDCCAjSgAwIBAgIBATANBgkqhkiG9w0BAQUFADA3MQswCQYDVQQGEwJERTEP
        MA0GA1UEChMGZWR1UEtJMRcwFQYDVQQDEw5lZHVQS0kgVGVzdCBDQTAeFw0xMDAz
        MzExMjIwMjRaFw0zMDAzMjYxMjIwMjRaMDcxCzAJBgNVBAYTAkRFMQ8wDQYDVQQK
        EwZlZHVQS0kxFzAVBgNVBAMTDmVkdVBLSSBUZXN0IENBMIIBIjANBgkqhkiG9w0B
        AQEFAAOCAQ8AMIIBCgKCAQEAt5IxCk/NQPOLqeA1lGuB3pvqHGQPxRQ1udYGcXQY
        t7EuSMFymUR9m5TsifG1ktktJTtOWyaWFC4ac0vai49wGVeuDYptfZBoHLIUvCwN
        DOofLYHxk04WzfrtSiUTptn1o6QPOw8YR0XH30MEi1zgD8fLMZmVTJ+XwA5Eus6c
        XtTmI4XhNrHUtvWt4UsNgLmp5/djUgRMpNqxIdrpFQzl+XycRJRAaoAwUzHFl14t
        49qwBhGChxQ8AdDMQGA7kv6VR8o0ktCPv3a4GQbs8+z0cX0w5dC+XhJ1xpqW6TOg
        qAY9XBFIDe5j21hjKmNZ39rsODVGUS2wUtNEhSz+3YqxLwIDAQABo2MwYTAdBgNV
        HQ4EFgQUqHe3saMjZZLan8RlFJs+Xuz4yiAwHwYDVR0jBBgwFoAUqHe3saMjZZLa
        n8RlFJs+Xuz4yiAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwDQYJ
        KoZIhvcNAQEFBQADggEBAEjQGyHZQis47c2kf+zXJJoDDlRgFzr9xfcnrHFaJvYx
        nuqNE0T+xmujnwGm3VrgddeAQJuW3sD6y0Ox8NgL4z886VFeaDQ0GmFPI6HEVtg6
        mixMhi+YzdkC+PFrEdYUeVNNwVO+bvJb1Rc08BYU4v7VtTkssHjru76E2/ahn/Ct
        kaVTEojEWeRaxsw5/0VLkgyf8SwDaukM2aamqgEzfsw5GTdSAh7ERZKc+zF7Sr5s
        DY8c5lOmyCwuNh9ODuw4cAThICrn7G8bh8ZyxLyj4Znxh0X45SwMZKTmYLfy9ab8
        b/j7FK8uBNRL+pXl9HGBWAFA01uJw4HkYK+Uo+RcAzo=
        -----END CERTIFICATE-----

You can specify any number of files, but each certificate must be stored in a separate file.

#Environment variables

#Managing environment variables through values

Environment variables and configuration files used by installation components are described in the configMaps and secrets blocks in values.yaml. The most commonly overridden configuration parameters are included in values-override.yaml.

#Managing environment variables through External Secret

External secret storage can also be connected. To use it, the cluster must have External Secrets Operator (ESO) installed. ESO adds the required CRDs (Custom Resource Definitions) to the cluster and connects it to secret storage.

To create an ExternalSecret resource and connect it to an existing SecretStore, add the corresponding entry to the vaults block:

vaults:
  ipcs-backend-secret-external: # name of the secret created as a result
    apiVersion: external-secrets.io/v1 # external-secrets.io/v1beta1 by default
    enabled: true
    store:
      name: vault-backend # name of the pre-created SecretStore
      kind: ClusterSecretStore # type of the pre-created SecretStore
    path: secret/data/my-vault-secret # path to the secret in Vault

To pass the created secret to all Deployment resources, add the resource name from the vaults block to deploymentsGeneral.envSecrets, for example:

deploymentsGeneral:
  envConfigmaps:
    - ipcs-backend-env
  envSecrets:
    - ipcs-secrets
    - ipcs-backend-secret-external # the resource name from the vaults block

To pass the secret only to one Deployment resource, specify it in the deployments block along with all secrets already used by that resource, for example:

deployments:
  ipcs-backend:
    envSecrets:
      - ipcs-secrets
      - ipcs-backend-secret-external

#Configuring Ingress

An Ingress controller must be installed in the cluster to expose the web interface and API through Ingress. Enable the Ingress, specify its class, and replace the domain name:

ingresses:
  codescoring:
    enabled: true
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt
    serviceName: frontend
    servicePort: 8080
    hosts:
      - hostname: codescoring.example.com
        paths:
          - path: /
    ingressClassName: nginx
    tls:
      - hosts:
          - codescoring.example.com
        secretName: codescoring-tls

httpRoutes:
  codescoring:
    enabled: false

Set ingressClassName, hosts, and tls to match the Ingress controller, DNS, and TLS certificate configuration in your cluster. If cert-manager is not used, remove its annotation and create the Secret specified in tls.secretName by another method.

#Configuring Gateway API

If the cluster uses Kubernetes Gateway API, you can create an HTTPRoute resource instead of an Ingress. The referenced Gateway resource must exist before CodeScoring is installed.

Disable the Ingress, specify the Gateway settings in httpRoutesGeneral, and enable the httpRoutes.codescoring route:

ingresses:
  codescoring:
    enabled: false

httpRoutesGeneral:
  gatewayName: codescoring-gateway
  gatewayNamespace: gateway-system
  gatewaySectionName: https
  gatewayKind: Gateway
  gatewayGroup: gateway.networking.k8s.io

httpRoutes:
  codescoring:
    enabled: true
    hostnames:
      - "codescoring.example.com"
    rules:
      - matches:
          - path:
              type: PathPrefix
              value: /
        backendRefs:
          - name: codescoring-frontend
            port: 8080

Set gatewayName, gatewayNamespace, gatewaySectionName, and hostnames to match the Gateway and DNS name in your cluster.

#Migration from Helm Chart 2.x.x

The installation will be unavailable during migration, so it is recommended to allocate a maintenance window. The migration takes 5-10 minutes in total.

Migration steps:

1. Delete all Deployment resources: kubectl delete deployment --all -n codescoring.

2. Delete all Service resources: kubectl delete service --all -n codescoring.

3. Run helm upgrade after filling in values.yaml and values-override.yaml according to Helm chart settings:

   helm upgrade codescoring codescoring-org/codescoring-helm --version <CHART_VERSION> -f values.yaml -f values-override.yaml -n codescoring