********************************
Setting up a CoreOS based distro
********************************

This chapter explains how to setup a distro based on CoreOS.

Repository structures
#####################

OpenEmbedded is a flexible tool, but we encourage each of our users to adopt
the same structure as CoreOS. In this chapter, replace each usage of `PRODUCT`
or `product` by a unique name related to your product.

.. code-block::

    product/
    ├── build/ (ignored by git)
    ├── documentation/
    ├── layers/
    |   └── coreos/ (submodule)
    |   |   └── bitbake/ (submodule)
    |   |   └── layers/
    |   |   |   ├── openembedded-core (submodule)
    |   |   |   ├── meta-belden-coreos
    |   |   |   ├── meta-belden-coreos-bsp
    |   |   |   └── ...
    |   |   └── ...
    |   ├── meta-product/
    |   ├── meta-other-layers/
    |   └── ...
    ├── scripts/
    ├── templates/
    ├── product-init-build-env
    ├── .gitignore

Creating the structures
#######################

.. code-block:: sh

    ~$  mkdir product
    ~$  cd product
    ~/product$ git init
    ~/product$ git submodule init

    ~/product$ mkdir layers
    ~/product$ mkdir scripts

    ~/product$ git submodule add -b <branch> ssh://git@bitbucket.gad.local:7999/ico/coreos.git layers/coreos
    ~/product$ git submodule update --init --recursive

    ~/product$ cp -r layers/coreos/templates ./templates
    ~/product$ cp layers/coreos/.gitignore ./.gitignore

    ~/product$ touch product-init-build-env
    ~/product$ chmod +x product-init-build-env
    ~/product$ nano product-init-build-env

.. note::

    By copying the .gitignore file of CoreOS, the build directory in the the product
    repository will not be tracked by Git, which is the recommended approach as using
    `devtool modify` modifies the local `bblayers.conf`. Instead we recommend to
    keep the template directory up to date so that a sane configuration can be
    created when fetching the repository for the first time.

Then you can enter the following inside the product-init-build-env file:

.. code-block:: sh

    #!/bin/sh

    # This script is used to setup the OE Build Environment
    # Normally this is called as '. ./product-init-build-env <builddir>'

    # On some shell, we can get the path of this script when sources. Otherwise we
    # use the current directory as a fallback
    if [ -z "$PRODUCT_ROOT" ]; then
        if [ -n "$BASH_SOURCE" ]; then
            PRODUCT_ROOT=$(dirname "$BASH_SOURCE")
        elif [ -n "$ZSH_NAME" ]; then
            PRODUCT_ROOT=$(dirname "$0")
        else
            PRODUCT_ROOT="$(pwd)"
        fi
    fi

    # Get a non relative path to the root directory
    PRODUCT_ROOT=$(readlink -f "${PRODUCT_ROOT}")

    # CoreOS init settings
    COREOS_ROOT="${PRODUCT_ROOT}/layers/coreos"
    TEMPLATECONF="${PRODUCT_ROOT}/templates"

    # Call the coreos-init-build-env scripts of CoreOS
    . "${COREOS_ROOT}/coreos-init-build-env" "${1:-$PRODUCT_ROOT/build}"

    # From here the scripts and functions defined by CoreOS and
    # OpenEmbedded-Core are available

    # Add support for ##PRODUCTS_LAYERSDIR## inside of bblayer template
    coreos-bblayers-envsub PRODUCT_LAYERSDIR "${PRODUCT_ROOT}/layers"

    # Add the scripts directory of the product to the path
    coreos_path_add "${PRODUCT_ROOT}/scripts"

Using your new project
######################

.. code-block:: sh

    ~product$  source product-init-build-env

Creating your product layers
############################

You can create a new layer and add it to your active bblayers.conf file like
this:

.. code-block:: sh

    ~product/build$ bitbake-layers create-layer ../layers/meta-belden-bsp
    ~product/build$ bitbake-layers add-layer ../layers/meta-product

Don't forget to update your templates `projects/templates/bblayers.conf.sample`
file. Inside this file use `##PRODUCT_LAYERSDIR##/meta-product` to have a
machine agnostic path.

Optional: Change some git settings
##################################

If you want to always `--recurse-submodules` when using `git pull`, you can
change your `submodule.recurse` git setting, either locally or globally

.. code-block:: sh

    ~/product$ git config submodule.recurse true # Only inside of product repo
    ~/product$ git config --global submodule.recurse true # Set it for all repos


Create your own distro based on CoreOS
######################################

Create a new file inside configuration file inside
`product/layers/meta-product/conf/distro`. For a distro named `product`, you will create
`product/layers/meta-product/conf/distro/product.conf`.

Open this file and enter the following:

.. code-block:: ini

    # This should come at the beginning of the file, to ensure that you use
    # CoreOS defaults
    require conf/distro/belden-coreos.conf

    # This should always be set in your own configuration file, to not use the
    # values of CoreOS
    DISTRO = "product"
    DISTRO_NAME = "Product Linux Distribution"
    MAINTAINER = "Belden Product Team"

    # You may want to add a version and a codename to your distro instead of
    # using the version and codename of CoreOS
    DISTRO_VERSION = "2022.05"
    DISTRO_CODENAME = "ProductOS Summer 2022 Edition"

    # Here you can override settings from the CoreOS distro or from
    # OpenEmbedded-core. But keep in mind that the CoreOS team doesn't support
    # all the features of OpenEmbedded-Core. We have added some checks for some
    # of the settings that we don't allow to change or that we don't support.
    # See the coreos-sanity.bbclass file for more info.

Then you can activate the distro by setting the `DISTRO` to `product` inside
your `product/build/conf/local.conf` file. You should also set it in the 
`product/templates/local.conf.sample` file so that it will be set as the default
when create the build environment for the first time.

What to do next
###############

How do I...
############

...add a PRODUCT_ROOT variable usable in recipes files?
*******************************************************

Add this line inside your meta-product layer configuration file at 
`product/layers/meta-product/conf/layer.conf`:

.. code-block:: ini

    # Set a variable to get to the top of the metadata location
    PRODUCT_ROOT = '${@os.path.normpath("${LAYERDIR}/../../")}'

... add PRODUCT_METADATA_BRANCH and PRODUCT_METADATA_REVISION variables to get the current git branch and git sha of the PRODUCT repository?
*********************************************************************************************************************************************

Create the file `product/layers/meta-product/classes/product_metadata_scm.bbclass`
and copy the content of the coreos_metadata_scm.bbclass file. Replacing all
reference to COREOS by PRODUCT should works.

... start fast and easy development
***********************************

By adding `debug-tweaks` to `EXTRA_IMAGE_FEATURES` the image is made suitable
for development. This allows for example root login with no password.
For a complete list of the functionality that is added or removed by using
`debug-tweaks` have a look at the official documentation.

Following CoreOS specific functionality was added to `debug-tweaks`:

* disables the read-only filesystem.

.. warning::

    This is for development only and must not be used for production images.

... set a root password
***********************

If you have `debug-tweaks` set in `EXTRA_IMAGE_FEATURES` you will not be asked for
a root password when logging in. If `debug-tweaks` is not set (should not be set in
the final product) you cannot login with root anymore. Therefore you need to set a
root password with:

.. code-block:: ini

    IMAGE_CLASSES += "extrausers"

    PASSWD='\$5\$sj6q14XssP2LRRFr\$U1EcE5DS/viWXWGdK1eRseoPzX6bSe5C9kWlKUXibl.'
    EXTRA_USERS_PARAMS = "\
        usermod -p '${PASSWD}' root; \
        "

The password needs to be provided as a hash and can be created on the host with
following command:

.. code-block:: bash

    printf "%q\n" $(mkpasswd -m sha256crypt root)

.. warning::

    This is for development only if you do not use `debug-tweaks`. For releases
    this would be a real security problem.

... configure a overlay filesystem
**********************************

Especially when you have a read-only filesystem you might want to have some
directories to be writeable. This can be achieved by using a overlay filesystem.
It is distinguished between two scenarios:

1. The directory is located somewhere under `/etc`
2. The directory is located under all other directories (except `/etc`)

The main difference for directories located under `/etc` is that they are mostly
config files that are used during the init process. However the init process
itself usually mounts the overlay filesystem. Therefore another mechanism is
needed which mounts the overlay before the actual init. This is solved by
replacing the actual init with a script that mounts the overlay filesystem and
then starts the actual init binary. But don't worry Yocto handles this for you.

Following are the steps to easily add a overlay filesystem:

**Overlay filesystem for directories under `/etc`**

1. Create a partition (in the wic file) and specify the mount point.

.. code-block:: bash

    part /mnt/overlay --fstype=ext4 --rootfs-dir=${IMAGE_ROOTFS}/mnt/overlay --label overlay --align 1024 --ondisk mmcblk1 --size 128M

2. Add `overlayfs-etc` to your `IMAGE_FEATURES` in the image file (e.g. coreos-image-minimal.bb)

.. code-block:: bash

    IMAGE_FEATURES += "overlayfs-etc"

3. Provide overlay filesystem details in the machine config file (e.g. cn9130-cex7.conf)

.. code-block:: bash

    OVERLAYFS_ETC_MOUNT_POINT = "/mnt/overlay"
    OVERLAYFS_ETC_DEVICE = "/dev/mmcblk1p5"
    OVERLAYFS_ETC_FSTYPE ?= "ext4"

4. Specify the directory that will be provided through the overlay filesystem in a recipe or bbappend file

.. code-block:: bash

    OVERLAYFS_WRITABLE_PATHS[overlay] += "/etc/ssh"

More detailed information is available under the official Yocto Project
documentation under `overlayfs-etc <https://docs.yoctoproject.org/4.0.4/ref-manual/classes.html#overlayfs-etc-bbclass>`_.

**Overlay filesystem for other directories**

1. Create a partition (in the wic file) and specify the mount point.

.. code-block:: bash

    part /mnt/overlay --fstype=ext4 --rootfs-dir=${IMAGE_ROOTFS}/mnt/overlay --label overlay --align 1024 --ondisk mmcblk1 --size 128M

2. Add `overlayfs` to your `DISTRO_FEATURES` in the distro configuration file (e.g. belden-coreos.conf)

.. code-block:: bash

    DISTRO_FEATURES += "overlayfs"

3. Specify the mount points in the machine configuration (e.g. cn9130-cex7.conf)

.. code-block:: bash

    OVERLAYFS_MOUNT_POINT[overlay] = "/mnt/overlay"

4. Specify the directory that will be provided through the overlay filesystem in a recipe or bbappend file

.. code-block:: bash

    inherit overlayfs
    OVERLAYFS_WRITABLE_PATHS[overlay] += "/etc/ssh"

More detailed information is available under the official Yocto Project
documentation under `overlayfs <https://docs.yoctoproject.org/4.0.4/ref-manual/classes.html#overlayfs-bbclass>`_.

.. note::
    The overlayfs QA check is looking for a systemd mount unit which is not
    needed if you use wic. Therefore just disable the QA check with:

    .. code-block:: bash

        OVERLAYFS_QA_SKIP[overlay] = "mount-configured"

Alternative repository structure
################################

It's also possible but not recommended to clone CoreOS without any submodule, to
create a more flat structure. But then you have to ensure and manage the 
Bitbake et OpenEmbedded-Core version by yourself.

.. important::

    CoreOS is only tested with the version of Bitbake and OpenEmbedded-Core used
    in the CoreOS repository as submodule. By doing this you have to ensure that
    you project stay in sync with CoreOS regarding CoreOS version and
    corresponding Bitbake and OpenEmbedded-Core version.


.. code-block::

    product/
    ├── build/ (ignored by git)
    ├── bitbake/ (submodule)
    ├── documentation/
    ├── layers/
    |   ├── openembedded-core (submodule)
    |   └── coreos/ (cloned without submodule)
    |   |   ├── layers/
    |   |   |   ├── meta-belden-coreos
    |   |   |   ├── meta-belden-coreos-bsp
    |   |   |   └── ...
    |   |   └── ...
    |   ├── meta-product/
    |   ├── meta-other-layers/
    |   └── ...
    ├── scripts/
    ├── templates/
    ├── product-init-build-env
    ├── .gitignore

Setting this structure is out of the scope for this documentation, but as a
hint, to implement it you have to set in `product-init-build-env`:

- `BITBAKEDIR` to the path of the Bitbake repository
- `OEROOT` to the path of the OpenEmbedded-Core repository

.. important::

    Calling directly oe-init-build-env from OpenEmbedded-Core is not supported!
    Ensure that your product-init-build-env call coreos-init-build-env egal if
    you use the recommended or alternative repository structures.