-
Notifications
You must be signed in to change notification settings - Fork 15
Specific instructions to run with Kubernetes cluster
A few possible ways:
- Install CVMFS on every Kubernetes node and mount.
- At CERN, use cvmfs-csi as a k8s pvc and sc to mount in container.
After finished Installation and Configuration, please install kubernetes python client:
pip install kubernetes
Using the Job kind to prevent Pod from occupying resources after it completed.
The following list shows parameters need to be modified:
| Name | Description |
|---|---|
| metadata.name | Job name |
| spec.template.spec.containers.name | Container name |
| spec.template.spec.containers.image | `atlasadc/atlas-grid-centos6` or `atlasadc/atlas-grid-slc6` |
The command content will be executed to setup container environment and then pull pilot to run.
For example,
apiVersion: batch/v1
kind: Job
metadata:
name: atlasadc-job
spec:
template:
spec:
containers:
- name: atlas-grid-centos6
image: atlasadc/atlas-grid-centos6
volumeMounts:
- name: cvmfs
mountPath: /cvmfs
imagePullPolicy: IfNotPresent
command:
- "sh"
- "-c"
- >
curl -4 https://bootstrap.pypa.io/2.6/get-pip.py -o /root/get-pip.py;
if [ -s /root/get-pip.py ]; then
python /root/get-pip.py;
pip install requests subprocess32;
fi;
echo -e "export computingSite=$computingSite\nexport pandaQueueName=$pandaQueueName\nexport proxyContent='$proxyContent'\nexport workerID=$workerID\nexport logs_frontend_w=$logs_frontend_w\nexport logs_frontend_r=$logs_frontend_r\n" > /etc/profile.d/job-setup.sh;
groupadd -g 1308 atlasprd;
useradd -u 41000 -g 1308 atlasprd;
su - atlasprd -c "cd /home/atlasprd; curl -4 https://raw.githubusercontent.com/mightqxc/panda-harvester/k8s/pandaharvester/harvestercloud/k8s_startup_script.py -o /home/atlasprd/k8s_startup_script.py; python /home/atlasprd/k8s_startup_script.py";
securityContext:
allowPrivilegeEscalation: false
restartPolicy: Never
volumes:
- name: cvmfs
hostPath:
path: /cvmfs
type: Directory
Set more than one container in YAML also works, but note that one worker is mapped to one container.
The second container (or more) setup in YAML will be ignored.
If you run job on the Kubernetes cluster at CERN, please refer to CERN cloud document to setup CVMFS.
See: http://clouddocs.web.cern.ch/clouddocs/containers/tutorials/cvmfs.html
An example of setting CVMFS on nodes,
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-cvmfs-atlas
provisioner: csi-cvmfsplugin
parameters:
repository: atlas.cern.ch
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-cvmfs-sft
provisioner: csi-cvmfsplugin
parameters:
repository: sft.cern.ch
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-cvmfs-grid
provisioner: csi-cvmfsplugin
parameters:
repository: grid.cern.ch
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-cvmfs-atlas-pvc
spec:
accessModes:
- ReadOnlyMany
resources:
requests:
storage: 1Gi
storageClassName: csi-cvmfs-atlas
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-cvmfs-sft-pvc
spec:
accessModes:
- ReadOnlyMany
resources:
requests:
storage: 1Gi
storageClassName: csi-cvmfs-sft
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-cvmfs-grid-pvc
spec:
accessModes:
- ReadOnlyMany
resources:
requests:
storage: 1Gi
storageClassName: csi-cvmfs-grid
and job YAML,
apiVersion: batch/v1
kind: Job
metadata:
name: atlasadc-job
spec:
template:
spec:
containers:
- name: atlas-grid-centos6
image: atlasadc/atlas-grid-centos6
volumeMounts:
- name: atlas
mountPath: /cvmfs/atlas.cern.ch
- name: sft
mountPath: /cvmfs/sft.cern.ch
- name: grid
mountPath: /cvmfs/grid.cern.ch
imagePullPolicy: IfNotPresent
command:
- "sh"
- "-c"
- >
curl -4 https://bootstrap.pypa.io/2.6/get-pip.py -o /root/get-pip.py;
if [ -s /root/get-pip.py ]; then
python /root/get-pip.py;
pip install requests subprocess32;
fi;
echo -e "export computingSite=$computingSite\nexport pandaQueueName=$pandaQueueName\nexport proxyContent='$proxyContent'\nexport workerID=$workerID\nexport logs_frontend_w=$logs_frontend_w\nexport logs_frontend_r=$logs_frontend_r\n" > /etc/profile.d/job-setup.sh;
groupadd -g 1308 atlasprd;
useradd -u 41000 -g 1308 atlasprd;
su - atlasprd -c "cd /home/atlasprd; curl -4 https://raw.githubusercontent.com/mightqxc/panda-harvester/k8s/pandaharvester/harvestercloud/k8s_startup_script.py -o /home/atlasprd/k8s_startup_script.py; python /home/atlasprd/k8s_startup_script.py";
securityContext:
allowPrivilegeEscalation: false
restartPolicy: Never
volumes:
- name: atlas
persistentVolumeClaim:
claimName: csi-cvmfs-atlas-pvc
readOnly: true
- name: sft
persistentVolumeClaim:
claimName: csi-cvmfs-sft-pvc
readOnly: true
- name: grid
persistentVolumeClaim:
claimName: csi-cvmfs-grid-pvc
readOnly: true
backoffLimit: 0
Before start Kubernetes plugin, the module and class name should be set in $PANDA_HOME/etc/panda/panda_queueconfig.json.
Also some parameters need to be adjusted:
| Name | Description |
|---|---|
| proxySecretPath | Path of the proxy file inside container. Can work with the k8s secret managing proxy |
| x509UserProxy | Proxy file path on Harvester node to pass to container. Only works if proxySecretPath NOT set |
| cpuAdjustRatio | Set ratio to adjust resource of CPU before pod creating (default is 100) |
| memoryAdjustRatio | Set ratio to adjust resource of memory before pod creating (default is 100) |
| k8s_yaml_file | YAML file path which creates a kubernetes job |
| k8s_config_file | Configuration file path for Kubernetes client authentication |
| k8s_namespace | If you want to distinguish multiple teams or projects on cluster. See:https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/#when-to-use-multiple-namespaces |
For example,
"ANALY_TAIWAN_TEST": {
…...
"submitter": {
"name":"K8sSubmitter",
"module":"pandaharvester.harvestersubmitter.k8s_submitter",
"x509UserProxy": "/root/atlas-production.proxy"
"cpuAdjustRatio": 90,
"memoryAdjustRatio": 100
…...
"monitor": {
"name":"K8sMonitor",
"module":"pandaharvester.harvestermonitor.k8s_monitor"
},
"sweeper": {
"name": "K8sSweeper",
"module": "pandaharvester.harvestersweeper.k8s_sweeper"
},
"common": {
"k8s_yaml_file": "/home/harvesteruser/atlas_job.yaml",
"k8s_config_file": "/home/harvesteruser/.kube/config",
"k8s_namespace": "default"
}
},
Now Harvester has a credential manager plugin k8s_secret_cred_manager to create/update a k8s secret object.
One can thus export proxy files into containers via k8s secret, and configure harvester to use k8s_secret_cred_manager to update the proxy periodically.
One needs to create a configuration file in JSON format for k8s_secret_credmanager.
Important keys are k8s_namespace, k8s_config_file, proxy_files.
Note the proxy files listed in proxy_files must be updated periodically by harvester no_voms_credmanager or in other ways. Thus, k8s_secret_credmanager can update the newest proxy files to k8s secret.
Example of config json file of k8s_secret_credmanager (/opt/harvester_k8s/k8s_secret_cred_manager_config.json in the example above):
{
"k8s_namespace": "",
"k8s_config_file": "/opt/harvester_k8s/kubeconf",
"proxy_files": ["/data/atlpan/atlas.prod.proxy", "/data/atlpan/atlas.pilot.proxy"]
}
In panda_harvester.cfg, one needs to add lines of k8s_secret_credmanager in credmanager block.
Here moduleName is pandaharvester.harvestercredmanager.k8s_secret_cred_manager and className is K8sSecretCredManager.
Put the path of configuration file of k8s_secret_credmanager mentioned above at certFile.
Many other attributes are useless for k8s_secret_credmanager.
Example of credmanager block in panda_harvester.cfg:
[credmanager]
# module name
moduleName =
...
pandaharvester.harvestercredmanager.k8s_secret_cred_manager
# class name
className =
...
K8sSecretCredManager
# original certificate file to generate new short-lived certificate
certFile =
...
/opt/harvester_k8s/k8s_secret_cred_manager_config.json
# the name of short-lived certificate
outCertFile =
...
useless_string
# voms
voms =
...
useless_string
# sleep interval in sec
sleepTime = 1800
When using k8s_secret_cred_manager, a k8s secret object with name proxy-secret will be created.
In the yaml file, one needs to add a volume of secret with secretName: proxy-secret , and mount it in the container, with a proper mountPath.
Example of yaml file of k8s job:
apiVersion: batch/v1
kind: Job
metadata:
name: atlasadc-job
spec:
template:
spec:
containers:
- name: atlas-grid-slc6
image: atlasadc/atlas-grid-slc6
volumeMounts:
- name: atlas
mountPath: /cvmfs/atlas.cern.ch
- name: atlas-condb
mountPath: /cvmfs/atlas-condb.cern.ch
- name: atlas-nightlies
mountPath: /cvmfs/atlas-nightlies.cern.ch
- name: sft
mountPath: /cvmfs/sft.cern.ch
- name: grid
mountPath: /cvmfs/grid.cern.ch
- name: proxy-secret
mountPath: /proxy
imagePullPolicy: IfNotPresent
#resources:
# requests:
# memory: "1.5Gi"
command:
- "sh"
- "-c"
- >
curl -4 https://bootstrap.pypa.io/2.6/get-pip.py -o /root/get-pip.py;
if [ -s /root/get-pip.py ]; then
python /root/get-pip.py;
pip install requests subprocess32;
fi;
echo -e "export computingSite=$computingSite\nexport pandaQueueName=$pandaQueueName\nexport proxySecretPath=$proxySecretPath\nexport proxyContent='$proxyContent'\nexport workerID=$workerID\nexport logs_frontend_w=$logs_frontend_w\nexport logs_frontend_r=$logs_frontend_r\nexport PANDA_JSID=$PANDA_JSID\nexport resourceType=$resourceType\n" > /etc/profile.d/job-setup.sh;
groupadd -g 1308 atlasprd;
useradd -u 41000 -g 1308 atlasprd;
su - atlasprd -c "cd /home/atlasprd; curl -4 https://raw.githubusercontent.com/PanDAWMS/panda-harvester/flin/pandaharvester/harvestercloud/k8s_startup_script.py -o /home/atlasprd/k8s_startup_script.py; python /home/atlasprd/k8s_startup_script.py";
securityContext:
allowPrivilegeEscalation: false
restartPolicy: Never
volumes:
- name: atlas
persistentVolumeClaim:
claimName: csi-cvmfs-atlas-pvc
readOnly: true
- name: atlas-condb
persistentVolumeClaim:
claimName: csi-cvmfs-atlas-condb-pvc
readOnly: true
- name: atlas-nightlies
persistentVolumeClaim:
claimName: csi-cvmfs-atlas-nightlies-pvc
readOnly: true
- name: sft
persistentVolumeClaim:
claimName: csi-cvmfs-sft-pvc
readOnly: true
- name: grid
persistentVolumeClaim:
claimName: csi-cvmfs-grid-pvc
readOnly: true
- name: proxy-secret
secret:
secretName: proxy-secret
backoffLimit: 0
In queue configuration submitter block, one needs to add the line of proxySecretPath. Note the value of proxySecretPath must be the proxy file path inside the container, basically corresponding to the mountPath setup in yaml and proxy_files defined in configuration json of k8s_secret_cred_manager.
Example of queue configuration json file:
"CERN-EXTENSION_K8S_HARVESTER": {
"queueStatus": "online",
"prodSourceLabel": "managed",
"nQueueLimitWorker": 100,
"maxWorkers": 1000,
"maxNewWorkersPerCycle": 30,
"runMode":"slave",
"mapType": "NoJob",
"truePilot": true,
"preparator": {
"name": "DummyPreparator",
"module": "pandaharvester.harvesterpreparator.dummy_preparator"
},
"submitter": {
"name":"K8sSubmitter",
"module":"pandaharvester.harvestersubmitter.k8s_submitter",
"proxySecretPath":"/proxy/atlas.prod.proxy",
"x509UserProxy": "/data/atlpan/x509up_u25606_production"
},
"workerMaker": {
"name": "SimpleWorkerMaker",
"module": "pandaharvester.harvesterworkermaker.simple_worker_maker"
},
"messenger": {
"name": "SharedFileMessenger",
"module": "pandaharvester.harvestermessenger.shared_file_messenger",
"accessPoint": "/data/atlpan/harvester_wdirs/${harvesterID}/${_workerID_3.2}/${_workerID_1.0}/${workerID}"
},
"stager": {
"name": "DummyStager",
"module": "pandaharvester.harvesterstager.dummy_stager"
},
"monitor": {
"name":"K8sMonitor",
"module":"pandaharvester.harvestermonitor.k8s_monitor"
},
"sweeper": {
"name": "K8sSweeper",
"module": "pandaharvester.harvestersweeper.k8s_sweeper"
},
"common": {
"k8s_yaml_file": "/opt/harvester_k8s/k8s_atlas_job_prod_secret.yaml",
"k8s_config_file": "/opt/harvester_k8s/kubeconf",
"k8s_namespace": ""
}
}
K8s default scheduling spreads the pods across the nodes with a Round Robin algorithm. This can cause single core pods to spread across all nodes and preventing multi core pods to be scheduled. You can define a custom scheduling policy. Here is the example that worked for us:
- On the master node define the policy file, including priority stratagy "{"name" : "MostRequestedPriority", "weight" : 1}" at /etc/kubernetes/scheduler-policy.json
{
"kind" : "Policy",
"apiVersion" : "v1",
"predicates" : [
{"name" : "GeneralPredicates"},
{"name" : "MatchInterPodAffinity"},
{"name" : "NoDiskConflict"},
{"name" : "NoVolumeZoneConflict"},
{"name" : "PodToleratesNodeTaints"}
],
"priorities" : [
{"name" : "MostRequestedPriority", "weight" : 1},
{"name" : "InterPodAffinityPriority", "weight" : 2}
]
}
- In /etc/kubernetes/scheduler refer to the policy config file in KUBE_SCHEDULER_ARGS:
KUBE_SCHEDULER_ARGS="--leader-elect=true --policy-config-file /etc/kubernetes/scheduler-policy.json"
- Then restart scheduler to make the changes take effect:
$ systemctl restart kube-scheduler.service
Authored by Mandy Yang, FaHui Lin
| Getting started |
|---|
| Installation and configuration |
| Testing and running |
| Debugging |
| Work with Middleware |
| Admin FAQ |
| Development guides |
|---|
| Development workflow |
| Tagging |
| Production & commissioning |
|---|
| Condor experiences |
| Commissioning on the grid |
| Production servers |
| Service monitoring |
| Auto Queue Configuration with AGIS |
| GCE setup |
| Kubernetes setup |
| SSH+RPC middleware setup |