LM
/
nwl-ci
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

doc/nwl-ci: added documentation with nexus artifact uploader 

Signed-off-by: Marc Mattmüller <marc.mattmueller@netmodule.com>
This commit is contained in:
Marc Mattmüller 2023-06-19 20:56:15 +02:00
parent 1aea7bf218
commit 8e54299b58
1 changed files with 276 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

276
doc/src/setup/nwl-ci.rst
View File

@ -716,6 +716,282 @@ With these commits we are able to build a NWL image that can be used for further
Deploying Artifacts to Nexus
****************************
There is a `Nexus <https://artifactory.gad.local/>`_ available within the HAC infrastructure. For this proof of concept
I will use the maven-releases repository. To upload artifacts from a pipeline to this Nexus-repository we do not want to
grant admin priviledges. Thus, I created a new role and user:
* new role
- ID = ci-uploader
- Name = custom-ci-artifact-uploader
- Role description = Role for the CI to upload artifacts
- Priviledges Given
+ nx-repository-admin-maven2-maven-releases-browse
+ nx-repository-admin-maven2-maven-releases-edit
+ nx-repository-admin-maven2-maven-releases-read
+ nx-repository-view-maven2-maven-releases-*
* new user
- ID = ci-build-user
- First name = CI
- Last name = Build User
- Email = marc.mattmueller@netmodule.com
- Status = Active
- Roles Granted:
+ custom-ci-artifact-uploader
To be able to upload an artifact from a build pipeline we can use the Jenkins plugin *nexus-artifact-uploader*. If this
plugin is installed on the Jenkins Controller, then we can use the following parameters and snippets using this
*Nexus Artifact Uploader* plugin:
.. code-block:: bash
# parameter definition:
env.NEXUS_VERSION = "nexus3"
env.NEXUS_PROTOCOL = "https"
env.NEXUS_URL = "https://artifactory.gad.local:443"
env.NEXUS_REPOSITORY = "maven-releases"
env.NEXUS_ARTIFACT_COPIER_URL = "${env.NEXUS_PROTOCOL}://${env.NEXUS_URL}/repository/${env.NEXUS_REPOSITORY}"
# step to upload an artifact:
nexusArtifactUploader(
nexusVersion: "${NEXUS_VERSION}",
protocol: "${NEXUS_PROTOCOL}",
nexusUrl: "${NEXUS_URL}",
groupId: "${groupId}",
version: "${version}",
repository: "${NEXUS_REPOSITORY}",
credentialsId: "nexus_uploader_credentials",
artifacts: [
[artifactId: "${artifactId}",
classifier: "",
file: "${artifactfilepath}",
type: "${artifacttype}"]
]
);
# whereas the following variables could be:
# groupId="nwl.sdk"
# artifactId="cn9130-cf-pro"
# version="latest"
# artifactfilepath="./build/tmp/deploy/sdk/nwl-cn9130-cf-pro-x86_64-sdk.sh"
# artifacttype="sh"
Building a new NWL Instance for Nexus Uploads
==============================================
For uploading an artifact to Nexus we have to install the plugin and to add credentials to the Jenkins Controller
instance. All the steps are split in the following subsections.
Plugin Installation
-------------------
The Nexus Plugin is currently not installed in the Jenkins Controller instances of HAC. Thus we need to adapt the
Dockerfile of the *build-docker* repository as the seen in this change set:
.. code-block:: bash
diff --git a/jenkins-ci/Dockerfile b/jenkins-ci/Dockerfile
index f446d48..b30ee58 100644
--- a/jenkins-ci/Dockerfile
+++ b/jenkins-ci/Dockerfile
@@ -68,7 +68,7 @@ RUN install-plugins.sh cloudbees-folder antisamy-markup-formatter \
pipeline-graph-analysis pipeline-milestone-step workflow-multibranch pipeline-utility-steps \
ssh-agent job-dsl cvs config-file-provider ant matrix-project pipeline-maven maven-plugin \
permissive-script-security uno-choice jdk-tool throttle-concurrents sidebar-link \
- generic-webhook-trigger publish-over-cifs metrics
+ generic-webhook-trigger publish-over-cifs metrics nexus-artifact-uploader
RUN JENKINS_VERSION=`java -jar /usr/share/jenkins/jenkins.war --version` && \
echo ${JENKINS_VERSION} > /usr/share/jenkins/ref/jenkins.install.UpgradeWizard.state && \
The Dockerfile is now ready. In the next section we add the credential to the Jenkins Controller instance that are used
to upload an artifact to Nexus.
Adding Nexus Credentials
------------------------
As in the previous section add the needed credentials. To add the credentials two files in separate repositories are
involved:
* repository *build-docker*
- jenkins-ci/scripts/credentials.groovy
* repository *build-admin*
- config/jenkins.xml
.. note::
For security reasons we do not adapt the config/jenkins.xml on the repository but we do the adaptions on the cloned
repository on the server.
See the changes in the *build-docker* repository as follows:
.. code-block:: bash
diff --git a/jenkins-ci/scripts/credentials.groovy b/jenkins-ci/scripts/credentials.groovy
index d2207b6..046b309 100644
--- a/jenkins-ci/scripts/credentials.groovy
+++ b/jenkins-ci/scripts/credentials.groovy
@@ -25,4 +25,10 @@ Credentials c = (Credentials) new UsernamePasswordCredentialsImpl(CredentialsSco
store.addCredentials(domain, c);
+Credentials nexus = (Credentials) new UsernamePasswordCredentialsImpl(CredentialsScope.GLOBAL,
+ "nexus_uploader_credentials", "",
+ config.jenkins.nexus.@user.text(), config.jenkins.nexus.@uploaderPw.text());
+
+store.addCredentials(domain, nexus);
+
Jenkins.instance.save();
**On the server itself** we adapt the jenkins.xml according the following steps:
* log into the server
* enter the ci directory: ``cd ~/work/ci``
* perform the changes according this git diff snippet:
.. code-block:: bash
diff --git a/config/jenkins.xml b/config/jenkins.xml
index fa0eeed..f852dea 100644
--- a/config/jenkins.xml
+++ b/config/jenkins.xml
@@ -5,6 +5,8 @@
<ldap managerDn="GA_ContinousIntegration@eu.GAD.local" server="ldaps://denec1adc003p.gad.local:3269"/>
+ <nexus name="CI_NexusArtifacts" user="ci-build-user" uploaderPw="4ciArtifacts" email="GA_ContinuousIntegration@belden.com"/>
+
<smtp server="host.docker.internal" suffix="@belden.com"/>
<executors count="8"/>
Push Changes and Build new Image
---------------------------------
Let's push the changes of the *build-docker* repository and build the new image according similar as above. For a better
readability see the steps below:
* Rebuild the docker images on my local machine with tag 0.2.0:
.. code-block:: bash
DOCKER_BUILDKIT=1 ./build.sh nwl 0.2.0
* Upload the essential images to the server:
.. code-block:: bash
docker save nwl-env-ci:latest | bzip2 | pv | ssh user@10.115.101.98 docker load
The image nwl-env-ci:latest already exists, renaming the old one with ID sha256:e28a607cbbfb19dddf766e9404572811475fe8fc533a1737b2dc325ecbc06e6e to empty string
Loaded image: nwl-env-ci:latest
docker save nwl-jenkins-ci:latest | bzip2 | pv | ssh user@10.115.101.98 docker load
The image nwl-jenkins-ci:latest already exists, renaming the old one with ID sha256:c7666cf7a03e5e1096325f26e83f8bde1cbe102cdce3fbb5242e6ab9e08eb89f to empty string
Loaded image: nwl-jenkins-ci:latest
* Switching back to the server
- Tag the new images to differntiate them from the others:
.. code-block:: bash
docker image tag nwl-env-ci:latest nwl-env-ci:0.2.0
docker image tag nwl-jenkins-ci:latest nwl-jenkins-ci:0.2.0
- Remove the residing parts of the previous instance:
.. code-block:: bash
# stop and destroy the current running instance
./manage.sh --name=nwl_0_1_2 destroy
# remove the residing file system content
rm -rf instances/nwl/main
- Create and start the new instance:
.. code-block:: bash
./manage.sh --image=nwl-env-ci:0.2.0 --branch=main \
--name=nwl_0_2_0 --platform=nwl \
--config=/home/user/work/ci/config/config.xml \
--revision=0.2.0 --maintainer=TeamCHBE create
Creating new instance...
Done!
# check the entry:
./manage.sh -p
+-----------+----------------------------+-------+---------+--------+----------+------------+----------+------------------+--------------+---------+
| name | host | port | status | branch | revision | maintainer | platform | image | container | display |
+-----------+----------------------------+-------+---------+--------+----------+------------+----------+------------------+--------------+---------+
| nwl_0_2_0 | netmodule-03.tcn.gad.local | 32780 | running | main | 0.2.0 | TeamCHBE | nwl | nwl-env-ci:0.2.0 | c52994e54431 | NULL |
+-----------+----------------------------+-------+---------+--------+----------+------------+----------+------------------+--------------+---------+
* Entering Jenkins in the `browser <https://10.115.101.98:32780/>`_ shows us now the desired effect and we have to
build a NWL image:
- The first run built the images properly but the upload failed with the following error:
.. code-block:: bash
Uploading artifact NWL-cn9130-cf-pro.zip started....
GroupId: nwl.nwl-image-testable
ArtifactId: cn9130-cf-pro
Classifier:
Type: zip
Version: latest
File: NWL-cn9130-cf-pro.zip
Repository:maven-releases
Uploading: https://artifactory.gad.local:443/repository/maven-releases/nwl/nwl-image-testable/cn9130-cf-pro/latest/cn9130-cf-pro-latest.zip
Failed to deploy artifacts: Could not transfer artifact nwl.nwl-image-testable:cn9130-cf-pro:zip:latest from/to maven-releases (https://artifactory.gad.local:443/repository/maven-releases): transfer failed for https://artifactory.gad.local:443/repository/maven-releases/nwl/nwl-image-testable/cn9130-cf-pro/latest/cn9130-cf-pro-latest.zip, status: 413 Request Entity Too Large
The days where we implemented Nexus into the CI environment at NetModule, we faced the same issue. The reason
was the reverse proxy configuration that influences the data uploaded to Nexus.
Therfore this configuration limit needs to be increased according the information of this
`link <https://help.sonatype.com/repomanager3/planning-your-implementation/run-behind-a-reverse-proxy>`_.
- At the moment where this error occured I did not want to touch the configuration of Nexus and/or proxy but I
to prove that the upload is working. Thus, I replayed the failing job and adapted the pipeline to upload a small
file (a zip file containing the efibootguardaa64.efi). See the output here:
.. code-block:: bash
deploying /var/jenkins_home/jobs/build-pipeline/branches/develop/workspace/nwl/toDeploy/testPkg.zip as nwl.nwl-image-testable.cn9130-cf-pro to Nexus...
[Pipeline] nexusArtifactUploader
Uploading artifact testPkg.zip started....
GroupId: nwl.nwl-image-testable
ArtifactId: cn9130-cf-pro
Classifier:
Type: zip
Version: latest
File: testPkg.zip
Repository:maven-releases
Uploading: https://artifactory.gad.local:443/repository/maven-releases/nwl/nwl-image-testable/cn9130-cf-pro/latest/cn9130-cf-pro-latest.zip
16 % completed (4.1 kB / 24 kB).
33 % completed (8.2 kB / 24 kB).
50 % completed (12 kB / 24 kB).
67 % completed (16 kB / 24 kB).
84 % completed (20 kB / 24 kB).
100 % completed (24 kB / 24 kB).
Uploaded: https://artifactory.gad.local:443/repository/maven-releases/nwl/nwl-image-testable/cn9130-cf-pro/latest/cn9130-cf-pro-latest.zip (24 kB at 144 kB/s)
Uploading artifact testPkg.zip completed.
.. |coreOsCiChain| image:: ./media/nwl-ci-jenkins-dashboard.png .. |coreOsCiChain| image:: ./media/nwl-ci-jenkins-dashboard.png
:width: 700px :width: 700px