doc/nwl-ci: added documentation with nexus artifact uploader
Signed-off-by: Marc Mattmüller <marc.mattmueller@netmodule.com>
This commit is contained in:
Marc Mattmüller
parent
1aea7bf218
commit
8e54299b58
|
|
@ -716,6 +716,282 @@ With these commits we are able to build a NWL image that can be used for further
|
|||
|
||||
|
||||
|
||||
Deploying Artifacts to Nexus
|
||||
****************************
|
||||
There is a `Nexus <https://artifactory.gad.local/>`_ available within the HAC infrastructure. For this proof of concept
|
||||
I will use the maven-releases repository. To upload artifacts from a pipeline to this Nexus-repository we do not want to
|
||||
grant admin priviledges. Thus, I created a new role and user:
|
||||
|
||||
* new role
|
||||
|
||||
- ID = ci-uploader
|
||||
- Name = custom-ci-artifact-uploader
|
||||
- Role description = Role for the CI to upload artifacts
|
||||
- Priviledges Given
|
||||
|
||||
+ nx-repository-admin-maven2-maven-releases-browse
|
||||
+ nx-repository-admin-maven2-maven-releases-edit
|
||||
+ nx-repository-admin-maven2-maven-releases-read
|
||||
+ nx-repository-view-maven2-maven-releases-*
|
||||
|
||||
* new user
|
||||
|
||||
- ID = ci-build-user
|
||||
- First name = CI
|
||||
- Last name = Build User
|
||||
- Email = marc.mattmueller@netmodule.com
|
||||
- Status = Active
|
||||
- Roles Granted:
|
||||
|
||||
+ custom-ci-artifact-uploader
|
||||
|
||||
|
||||
To be able to upload an artifact from a build pipeline we can use the Jenkins plugin *nexus-artifact-uploader*. If this
|
||||
plugin is installed on the Jenkins Controller, then we can use the following parameters and snippets using this
|
||||
*Nexus Artifact Uploader* plugin:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# parameter definition:
|
||||
env.NEXUS_VERSION = "nexus3"
|
||||
env.NEXUS_PROTOCOL = "https"
|
||||
env.NEXUS_URL = "https://artifactory.gad.local:443"
|
||||
env.NEXUS_REPOSITORY = "maven-releases"
|
||||
env.NEXUS_ARTIFACT_COPIER_URL = "${env.NEXUS_PROTOCOL}://${env.NEXUS_URL}/repository/${env.NEXUS_REPOSITORY}"
|
||||
|
||||
# step to upload an artifact:
|
||||
nexusArtifactUploader(
|
||||
nexusVersion: "${NEXUS_VERSION}",
|
||||
protocol: "${NEXUS_PROTOCOL}",
|
||||
nexusUrl: "${NEXUS_URL}",
|
||||
groupId: "${groupId}",
|
||||
version: "${version}",
|
||||
repository: "${NEXUS_REPOSITORY}",
|
||||
credentialsId: "nexus_uploader_credentials",
|
||||
artifacts: [
|
||||
[artifactId: "${artifactId}",
|
||||
classifier: "",
|
||||
file: "${artifactfilepath}",
|
||||
type: "${artifacttype}"]
|
||||
]
|
||||
);
|
||||
|
||||
# whereas the following variables could be:
|
||||
# groupId="nwl.sdk"
|
||||
# artifactId="cn9130-cf-pro"
|
||||
# version="latest"
|
||||
# artifactfilepath="./build/tmp/deploy/sdk/nwl-cn9130-cf-pro-x86_64-sdk.sh"
|
||||
# artifacttype="sh"
|
||||
|
||||
|
||||
Building a new NWL Instance for Nexus Uploads
|
||||
==============================================
|
||||
For uploading an artifact to Nexus we have to install the plugin and to add credentials to the Jenkins Controller
|
||||
instance. All the steps are split in the following subsections.
|
||||
|
||||
|
||||
Plugin Installation
|
||||
-------------------
|
||||
The Nexus Plugin is currently not installed in the Jenkins Controller instances of HAC. Thus we need to adapt the
|
||||
Dockerfile of the *build-docker* repository as the seen in this change set:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
diff --git a/jenkins-ci/Dockerfile b/jenkins-ci/Dockerfile
|
||||
index f446d48..b30ee58 100644
|
||||
--- a/jenkins-ci/Dockerfile
|
||||
+++ b/jenkins-ci/Dockerfile
|
||||
@@ -68,7 +68,7 @@ RUN install-plugins.sh cloudbees-folder antisamy-markup-formatter \
|
||||
pipeline-graph-analysis pipeline-milestone-step workflow-multibranch pipeline-utility-steps \
|
||||
ssh-agent job-dsl cvs config-file-provider ant matrix-project pipeline-maven maven-plugin \
|
||||
permissive-script-security uno-choice jdk-tool throttle-concurrents sidebar-link \
|
||||
- generic-webhook-trigger publish-over-cifs metrics
|
||||
+ generic-webhook-trigger publish-over-cifs metrics nexus-artifact-uploader
|
||||
|
||||
RUN JENKINS_VERSION=`java -jar /usr/share/jenkins/jenkins.war --version` && \
|
||||
echo ${JENKINS_VERSION} > /usr/share/jenkins/ref/jenkins.install.UpgradeWizard.state && \
|
||||
|
||||
The Dockerfile is now ready. In the next section we add the credential to the Jenkins Controller instance that are used
|
||||
to upload an artifact to Nexus.
|
||||
|
||||
|
||||
Adding Nexus Credentials
|
||||
------------------------
|
||||
As in the previous section add the needed credentials. To add the credentials two files in separate repositories are
|
||||
involved:
|
||||
|
||||
* repository *build-docker*
|
||||
|
||||
- jenkins-ci/scripts/credentials.groovy
|
||||
|
||||
* repository *build-admin*
|
||||
|
||||
- config/jenkins.xml
|
||||
|
||||
.. note::
|
||||
For security reasons we do not adapt the config/jenkins.xml on the repository but we do the adaptions on the cloned
|
||||
repository on the server.
|
||||
|
||||
|
||||
See the changes in the *build-docker* repository as follows:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
diff --git a/jenkins-ci/scripts/credentials.groovy b/jenkins-ci/scripts/credentials.groovy
|
||||
index d2207b6..046b309 100644
|
||||
--- a/jenkins-ci/scripts/credentials.groovy
|
||||
+++ b/jenkins-ci/scripts/credentials.groovy
|
||||
@@ -25,4 +25,10 @@ Credentials c = (Credentials) new UsernamePasswordCredentialsImpl(CredentialsSco
|
||||
|
||||
store.addCredentials(domain, c);
|
||||
|
||||
+Credentials nexus = (Credentials) new UsernamePasswordCredentialsImpl(CredentialsScope.GLOBAL,
|
||||
+ "nexus_uploader_credentials", "",
|
||||
+ config.jenkins.nexus.@user.text(), config.jenkins.nexus.@uploaderPw.text());
|
||||
+
|
||||
+store.addCredentials(domain, nexus);
|
||||
+
|
||||
Jenkins.instance.save();
|
||||
|
||||
|
||||
**On the server itself** we adapt the jenkins.xml according the following steps:
|
||||
|
||||
* log into the server
|
||||
* enter the ci directory: ``cd ~/work/ci``
|
||||
* perform the changes according this git diff snippet:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
diff --git a/config/jenkins.xml b/config/jenkins.xml
|
||||
index fa0eeed..f852dea 100644
|
||||
--- a/config/jenkins.xml
|
||||
+++ b/config/jenkins.xml
|
||||
@@ -5,6 +5,8 @@
|
||||
|
||||
<ldap managerDn="GA_ContinousIntegration@eu.GAD.local" server="ldaps://denec1adc003p.gad.local:3269"/>
|
||||
|
||||
+ <nexus name="CI_NexusArtifacts" user="ci-build-user" uploaderPw="4ciArtifacts" email="GA_ContinuousIntegration@belden.com"/>
|
||||
+
|
||||
<smtp server="host.docker.internal" suffix="@belden.com"/>
|
||||
|
||||
<executors count="8"/>
|
||||
|
||||
|
||||
Push Changes and Build new Image
|
||||
---------------------------------
|
||||
Let's push the changes of the *build-docker* repository and build the new image according similar as above. For a better
|
||||
readability see the steps below:
|
||||
|
||||
* Rebuild the docker images on my local machine with tag 0.2.0:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
DOCKER_BUILDKIT=1 ./build.sh nwl 0.2.0
|
||||
|
||||
* Upload the essential images to the server:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker save nwl-env-ci:latest | bzip2 | pv | ssh user@10.115.101.98 docker load
|
||||
The image nwl-env-ci:latest already exists, renaming the old one with ID sha256:e28a607cbbfb19dddf766e9404572811475fe8fc533a1737b2dc325ecbc06e6e to empty string
|
||||
Loaded image: nwl-env-ci:latest
|
||||
|
||||
docker save nwl-jenkins-ci:latest | bzip2 | pv | ssh user@10.115.101.98 docker load
|
||||
The image nwl-jenkins-ci:latest already exists, renaming the old one with ID sha256:c7666cf7a03e5e1096325f26e83f8bde1cbe102cdce3fbb5242e6ab9e08eb89f to empty string
|
||||
Loaded image: nwl-jenkins-ci:latest
|
||||
|
||||
* Switching back to the server
|
||||
|
||||
- Tag the new images to differntiate them from the others:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker image tag nwl-env-ci:latest nwl-env-ci:0.2.0
|
||||
docker image tag nwl-jenkins-ci:latest nwl-jenkins-ci:0.2.0
|
||||
|
||||
- Remove the residing parts of the previous instance:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# stop and destroy the current running instance
|
||||
./manage.sh --name=nwl_0_1_2 destroy
|
||||
|
||||
# remove the residing file system content
|
||||
rm -rf instances/nwl/main
|
||||
|
||||
- Create and start the new instance:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
./manage.sh --image=nwl-env-ci:0.2.0 --branch=main \
|
||||
--name=nwl_0_2_0 --platform=nwl \
|
||||
--config=/home/user/work/ci/config/config.xml \
|
||||
--revision=0.2.0 --maintainer=TeamCHBE create
|
||||
Creating new instance...
|
||||
Done!
|
||||
|
||||
# check the entry:
|
||||
./manage.sh -p
|
||||
+-----------+----------------------------+-------+---------+--------+----------+------------+----------+------------------+--------------+---------+
|
||||
| name | host | port | status | branch | revision | maintainer | platform | image | container | display |
|
||||
+-----------+----------------------------+-------+---------+--------+----------+------------+----------+------------------+--------------+---------+
|
||||
| nwl_0_2_0 | netmodule-03.tcn.gad.local | 32780 | running | main | 0.2.0 | TeamCHBE | nwl | nwl-env-ci:0.2.0 | c52994e54431 | NULL |
|
||||
+-----------+----------------------------+-------+---------+--------+----------+------------+----------+------------------+--------------+---------+
|
||||
|
||||
* Entering Jenkins in the `browser <https://10.115.101.98:32780/>`_ shows us now the desired effect and we have to
|
||||
build a NWL image:
|
||||
|
||||
- The first run built the images properly but the upload failed with the following error:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
Uploading artifact NWL-cn9130-cf-pro.zip started....
|
||||
GroupId: nwl.nwl-image-testable
|
||||
ArtifactId: cn9130-cf-pro
|
||||
Classifier:
|
||||
Type: zip
|
||||
Version: latest
|
||||
File: NWL-cn9130-cf-pro.zip
|
||||
Repository:maven-releases
|
||||
Uploading: https://artifactory.gad.local:443/repository/maven-releases/nwl/nwl-image-testable/cn9130-cf-pro/latest/cn9130-cf-pro-latest.zip
|
||||
Failed to deploy artifacts: Could not transfer artifact nwl.nwl-image-testable:cn9130-cf-pro:zip:latest from/to maven-releases (https://artifactory.gad.local:443/repository/maven-releases): transfer failed for https://artifactory.gad.local:443/repository/maven-releases/nwl/nwl-image-testable/cn9130-cf-pro/latest/cn9130-cf-pro-latest.zip, status: 413 Request Entity Too Large
|
||||
|
||||
The days where we implemented Nexus into the CI environment at NetModule, we faced the same issue. The reason
|
||||
was the reverse proxy configuration that influences the data uploaded to Nexus.
|
||||
|
||||
Therfore this configuration limit needs to be increased according the information of this
|
||||
`link <https://help.sonatype.com/repomanager3/planning-your-implementation/run-behind-a-reverse-proxy>`_.
|
||||
|
||||
- At the moment where this error occured I did not want to touch the configuration of Nexus and/or proxy but I
|
||||
to prove that the upload is working. Thus, I replayed the failing job and adapted the pipeline to upload a small
|
||||
file (a zip file containing the efibootguardaa64.efi). See the output here:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
deploying /var/jenkins_home/jobs/build-pipeline/branches/develop/workspace/nwl/toDeploy/testPkg.zip as nwl.nwl-image-testable.cn9130-cf-pro to Nexus...
|
||||
[Pipeline] nexusArtifactUploader
|
||||
Uploading artifact testPkg.zip started....
|
||||
GroupId: nwl.nwl-image-testable
|
||||
ArtifactId: cn9130-cf-pro
|
||||
Classifier:
|
||||
Type: zip
|
||||
Version: latest
|
||||
File: testPkg.zip
|
||||
Repository:maven-releases
|
||||
Uploading: https://artifactory.gad.local:443/repository/maven-releases/nwl/nwl-image-testable/cn9130-cf-pro/latest/cn9130-cf-pro-latest.zip
|
||||
16 % completed (4.1 kB / 24 kB).
|
||||
33 % completed (8.2 kB / 24 kB).
|
||||
50 % completed (12 kB / 24 kB).
|
||||
67 % completed (16 kB / 24 kB).
|
||||
84 % completed (20 kB / 24 kB).
|
||||
100 % completed (24 kB / 24 kB).
|
||||
Uploaded: https://artifactory.gad.local:443/repository/maven-releases/nwl/nwl-image-testable/cn9130-cf-pro/latest/cn9130-cf-pro-latest.zip (24 kB at 144 kB/s)
|
||||
Uploading artifact testPkg.zip completed.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
.. |coreOsCiChain| image:: ./media/nwl-ci-jenkins-dashboard.png
|
||||
:width: 700px
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue