The MEGA infrastructure project.
The MEGA Project is hosted in Openshift and split to the following 3 stages (Openshift Projects):
-
mega-dev
The development stage, where usually the currentdevelopstate is deployed on. -
mega-test
The test environment where the acceptance tests are performed. -
mega-prod
The production system of the MEGA Project.
The cited image illustrates the system of the MEGA Project, its involved services, interactions and zones.
Each zone is protected by OAuth whereby each zone uses its own clientId.
The mega databaseis backed up by a Jenkins Pipeline and stored in an Openshift PVC.
|
Important
|
This type of backup is not working because we cannot get our Pod-Networks of the different OCP Projects merged. |
|
Important
|
The CronJobs are working only within the project they are defined in. |
The backups are stored on a PVc within the hosting Project and can be synced locally via the mega-zep-db`service.
oc rsync <POD_ID>:/mega-db-backup/ .The backups are located on PVC within a project, so actually we can only restore backups to the project they were made from.
|
Important
|
First modify /apps/mega-zep-db/restore-job.yml env var DUMPFILE with the filename you want to restore which you retrieved from the mega-db-service.
|
# Then create the job, which is executed right away
./apps/ocp.sh createMegaDbRestore
# Check the pod state
oc get pods
# Check the logs of the COMPLETED job
oc logs mega-db-restore-*
# Delete the restore OCP Job
./apps/ocp.sh deleteMegaDbRestore# login to the OpenShift Console oc login https://manage.cloud.itandtel.at --token=<token> # create the cronjob with the current date time. Format: yyyy-MM-dd-HH-mm-ss kubectl create job --from=cronjob/mega-db-backup mega-db-backup-<date>-<time>
!!!does only work in prod!!! In dev and test there is no cronjob
The cited Dockerfiles are used to build the jenkins build agents for the MEGA Project.
-
agent-nodejs/Dockerfile
The Dockerfile for building the nodejs build agent. -
agent-jdk/Dockerfile
The Dockerfile for building the OpenJDK build agent.
We use a custom Jenkins which is based on the official Openshift-Jenkins-Image and which is build via the S2I mechanism.
-
s2i/jenkins
The directory holding the s2i sources. -
s2i/jenkins/plugins.txt
The plugins.txt which defines the plugins and plugins version the Mega Jenkins uses.
Jenkins is configured via the configureation-as-code plugin which allows to configure Jenkins via YAML config files.
-
/config/jenkins/jenkins-config.yaml
The YAML configuration file which configures the Jenkins instance and the MEGA Job Definitions.
|
Tip
|
The YAML configuration file references secrets via environment variables, which are injected into the container by Openshift. Because the configuration files doesn’t contain any secrets, this configuration file is part of the repository. |
The configuration for MEGA is provided via a properties file and a secret file. Each used stage needs its own configuration file to separate the configurations per stage.
-
application.<STAGE>.properties
The staged configuration file. (Part of the repository) -
application.<STAGE>.properties
The staged secret file. (In Cloud storage)
|
Tip
|
The secret file must be provided manually. See MEGA Repository for possible configuration values. |
For the Jenkins service we use multiple Openshift Templates which are used to define an part of the Jenkins service and its dependencies.
-
/apps/jenkins/jenkins.yaml
The Jenkins Openshift Template defining the Jenkins service. -
/apps/jenkins/jenkins-agent-bc.yaml
The Openshift Template holding the definitions of the by agents used Container Images and Openshift Build Configurations. -
/apps/jenkins/jenkins-bc.yaml
The Openshift Template holding the Openshift Build Configurations for the custom Jenkins build. -
/apps/jenkins/jenkins-pvc.yaml
The Openshift Template holding the definition for the persistence storage of Jenkins in Openshift. -
/apps/jenkins/maven-pvc.yaml
The Openshift Template holding the definition for the persistence storage for the caching of the maven repository used by the build agents. -
/apps/jenkins/jenkins.properties
The properties file holding the values for thejenkins.yamlOpenshift Template defined parameters.
The Jenkins Openshift Templates can be configured via the cited configuration properties.
-
JENKINS_SERVICE_NAME=jenkins
The name used for the Jenkins service and Openshift Resources -
JENKINS_S2I_IMAGE=docker.io/openshift/jenkins-2-centos7:v3.11The image the custom Jenkins Build is based on -
JENKINS_IMAGE_STREAM_TAG=jenkins-custom:latest
The name of the Openshift image Stream for the custom Jenkins Build Container Images -
GIT_URL=https://github.com/Gepardec/mega-infrastructure.git
The git repository url for the Jenkins related Openshift Build Configurations -
GIT_REF=master
The git reference used for the Jenkins related Openshift Build Configurations -
GIT_SECRET=github-http
The name of the Openshift secret providing the Openshift Secret of typekubernetes.io/basic-auth -
MAVEN_VERSION=3.8.1
The Maven version used by the JDK build agent Container Image -
MEMORY_LIMIT=2Gi
The Jenkins Service assgined RAM -
MAVEN_VERSION=3.8.1
The Maven Version to use for theagent-jdkJenkins Agent Pod -
MEGA_BACKUP_PVC
The name of the backup pvc of mega
The MEGA Project uses 2 Openshift Templates, one for the backend and one for the frontend to integrate the MEGA services in openshift.
The templates are parametrized and whereby the parameter values are provided via properties files.
-
mega-zep-backend.yaml
ThebackendOpenshift Template -
mega-zep-backend.<STAGE>.properties
The stagedbackendconfiguration file -
mega-zep-frontend.yaml
ThebackendOpenshift Template -
mega-zep-frontend.<STAGE>.properties
The stagedfrontendconfiguration file
The backend can be configured via the cited configuration properties.
-
APP=mega-zep
The value for the app label, which is add to each created Openshift Resource -
NAME=mega-zep-backend
The value for the service and created Openshift Resources -
SECRET=mega
The name of the secret holding the MEGA secrets for the specific stage -
CPU=500m
The assigned CPU resources to the service -
MEMORY=256Mi
The assigned RAM resources to the services -
TIMEZONE=Europe/Vienna
The timezone to set on the container
The frontend can be configured via the cited configuration properties.
-
APP=mega-zep
The value for the app label, which is add to each created Openshift Resource -
NAME=mega-zep-frontend
The value for the service and created Openshift Resources -
CPU=500m
The assigned CPU resources to the service -
MEMORY=256Mi
The assigned RAM resources to the services -
TIMEZONE=Europe/Vienna
The timezone to set on the container
The db can be configured via the cited configuration properties.
-
APP=mega-zep
The value for the app label, which is add to each created Openshift Resource -
NAME=mega-zep-db
The value for the service and created Openshift Resources -
VERSION=10
The version of the postgres to use -
SECRET=mega
The name of the secret holding the MEGA secrets for the specific stage -
PVC=mega-zep-db-10
The name of persistence volume for the database -
CPU=500m
The assigned CPU resources to the service -
MEMORY=256Mi
The assigned RAM resources to the services -
TIMEZONE=Europe/Vienna
The timezone to set on the container -
DATABASE_MAX_CONNECTIONS=100
The maximal allowed connections to the database -
DATABASE_SHARED_BUFFERS=64MB
The maximal shared buffer size -
PGCTLTIMEOUT=150
The pgctl timeout in seconds
The cited Jenkinsfiles are provided by this repository.
-
/jenkins/src/release/Jenkinsfile
The release pipeline which releases a MEGA version to a specific stage.
The cited scripts are provided by this repository.
-
/apps/ocp.sh
This scripts holds functions which can be used to managed Jenkins and Jenkins related resources.
The cited secret files have to be provided in the root of this repository.
emailUser=service@gepardec.com
emailPassword=****
megaBuildWebhookToken=***This secret is used to configure the mailing and the github webhook.
username=mega-dev
password=***This secret is used for checkout of the mega Github repositories.
|
Important
|
The password must be a generated API Token and not the account password. |
The MEGA Project configuration properties are described on the MEGA MEGA Repository.
-
mega-secrets.dev.properties
The DEV stage MEGA configuration -
mega-secrets.test.properties
The TEST stage MEGA configuration -
mega-secrets.prod.properties
The PROD stage MEGA configuration
Each stage uses its own OAuth clientId/clientSecret and therefore each stages has to be setup in Google. See here for a description how to setup a google oauth client.
# Create secrets
STAGE=dev apps/ocp.sh createMegaSecrets
apps/ocp.sh createJenkinsSecrets
# Delete secrets
apps/ocp.sh deleteMegaSecrets
apps/ocp.sh deleteJenkinsSecrets
# Delete/Create secrets
STAGE=dev apps/ocp.sh recreateMegaSecrets
apps/ocp.sh recreateJenkinsSecrets# Create all build configs
apps/ocp.sh createBuildConfigs
# Delete all build configs
apps/ocp.sh deleteBuildConfigs
# Delete/Create all build configs
apps/ocp.sh recreateBuildConfigs# Create Backup PVC
apps/ocp.sh createMegaBackupPvc
# Delete Backup PVC
apps/ocp.sh deleteMegaBackupPvc
# Recreate Backup PVC
apps/ocp.sh recreateMegaBackupPvc|
Important
|
Take care that all backups are saved before deleting or recreating the backup persistence volume. Jenkins requires the backup pvc to exist because Jenkins uses it as a volume. |
# Create jenkins
apps/ocp.sh createJenkins
# Delete jenkins
apps/ocp.sh deleteJenkins
# Delete/Create jenkins
apps/ocp.sh recreateJenkins
# Create jenkins pvc
apps/ocp.sh createJenkinsPvc
# Delete jenkins pvc
apps/ocp.sh deleteJenkinsPvc
# Delete/Create jenkins pvc
apps/ocp.sh recreateJenkinsPvc
# Create maven pvc
apps/ocp.sh createMavenPvc
# Delete maven pvc
apps/ocp.sh deleteMavenPvc
# Delete/Create maven pvc
apps/ocp.sh recreateMavenPvc# Create the mega db image streams
apps/ocp.sh createMegaDbIs
# Delete the mega db image streams
apps/ocp.sh deleteMegaDbIs
# Delete/Create the mega db image streams
apps/ocp.sh recreateMegaDbIs
# Create the mega db persistence volume
apps/ocp.sh createMegaDbPvc
# Delete the mega db persistence volume
apps/ocp.sh deleteMegaDbPvc
# Delete/Create the mega db persistence volume
apps/ocp.sh recreateMegaDbPvc
# Create the mega db
apps/ocp.sh createMegaDb
# Delete the mega db
apps/ocp.sh deleteMegaDb
# Delete/Create the mega db
apps/ocp.sh recreateMegaDb|
Important
|
Ensure that the properties in ocp/jenkins.properties and env vars in apps/ocp.sh are properly setup for your needs.
|
# Create secrets
STAGE=[prod|test] apps/ocp.sh createMegaSecrets|
Important
|
Ensure that the configuration is properly setup for the selected stage and that you have set the proper project context |
After Jenkins has newly deployed the build pipelines tries to scan the github repository for branches whereby it cannot consume the assigend credential the firt time the scan is executed. Therefore, after a deployment you need to skip the first execution of the scan and restart it. The second time the crednetial is consume correctly.