From f096d36a9fdebf503c333f3fa8086a0ca8a7fb76 Mon Sep 17 00:00:00 2001 From: Boekhorst Date: Wed, 25 Sep 2024 14:33:36 +0200 Subject: [PATCH 1/4] feat: add guide docs for rookify Signed-off-by: Boekhorst --- docs/guides/configuration-guide/rookify.md | 136 +++++++++++++++++++ docs/guides/deploy-guide/rookify.md | 94 +++++++++++++ docs/guides/deploy-guide/services/rookify.md | 17 +++ docs/guides/operations-guide/rookify.md | 83 +++++++++++ 4 files changed, 330 insertions(+) create mode 100644 docs/guides/configuration-guide/rookify.md create mode 100644 docs/guides/deploy-guide/rookify.md create mode 100644 docs/guides/deploy-guide/services/rookify.md create mode 100644 docs/guides/operations-guide/rookify.md diff --git a/docs/guides/configuration-guide/rookify.md b/docs/guides/configuration-guide/rookify.md new file mode 100644 index 0000000000..c6eddf9874 --- /dev/null +++ b/docs/guides/configuration-guide/rookify.md @@ -0,0 +1,136 @@ +--- +sidebar_label: Configure Rookify: Migrate to Rook from Ceph Ansible (technical preview) +sidebar_position: 31 +--- + +# Configure Rookify: Migrate from Ceph Ansible to Rook (Technical Preview) + +:::warning + +Rookify is developed to migrate from Ceph-Ansible to Rook _in place_ and _without downtime_. +Nevertheless, it is **strongly advised** to test Rookify in a controlled environment beforehand, such as the [OSISM testbed](https://github.com/osism/testbed). Additionally, ensure that all precautionary backups are taken, and any other necessary safety measures are in place. + +::: + +The [Rookify GitHub repository](https://github.com/SovereignCloudStack/rookify) includes a README.md that provides a condensed summary of the information covered here. + +## Config.yaml + +The primary configuration file for Rookify is `config.yaml`. The repository contains an example file for general use, as well as one specifically tailored to the OSISM testbed setup: + +- [config.example.yaml](https://github.com/SovereignCloudStack/rookify/blob/main/config.example.yaml) +- [config.example.osism.yaml](https://github.com/SovereignCloudStack/rookify/blob/main/config.example.osism.yaml) + +### Parameters + +#### General + +```yaml title="config.example.yaml" +general: + machine_pickle_file: data.pickle +``` + +The general section allows for optional definition of a pickle file, which allows for saving the state of the migration as serialized objects on disk. The pickle filed can be named as pleased. + +#### Logging + +```yaml title="config.example.yaml" +logging: + level: INFO # level at which logging should start + format: + time: "%Y-%m-%d %H:%M.%S" # other example: "iso" + renderer: console # or: json +``` + +The `logging` section allows for specification of `structlog`. The `level` parameter can be set to all python [log-levels](https://docs.python.org/3/library/logging.html#logging-levels), i.e. `NOTSET, DEBUG, INFO, WARNING, ERROR, CRITICAl`, but it is recommended to use `INFO`. + +#### Ceph + +```yaml title="config.example.yaml" + +ceph: + config: ./.ceph/ceph.conf + keyring: ./.ceph/ceph.client.admin.keyring +``` + +The `ceph` section specifies the `ceph.conf` and `ceph.client.admin.keyring` of the target systems (i.e. the system where Ceph-Ansible needs to be migrated to Rook). + +#### SSH + +```yaml title="config.example.yaml" +# fill in correct path to private key +ssh: + private_key: /home/USER/.ssh/cloud.private + hosts: + testbed-node-0: + address: 192.168.16.10 + user: dragon + testbed-node-1: + address: 192.168.16.11 + user: dragon + testbed-node-2: + address: 192.168.16.12 + user: dragon +``` + +The `ssh` section requires specification of the `private_key` and `hosts`. The `hosts` section specifies the hostnames or aliases (e.g. keys like `testbed-node-0`), their ip-addresses (e.g. if rookify connects to the target systems per VPN add ips starting with `192.186...`) and user for login. If you are using the OSISM testbed, make sure that the private key does not contain any `EOF` or other strings, i.e. make sure the keys are 'clean' to avoid connection errors. + +#### Kubernetes + +```yaml title="config.example.yaml" + +kubernetes: + config: ./k8s/config +``` + +The `kubernetes` section specifies the kubernetes configuration (e.g. if you use kubectl it is located in `~/.kube/config`) for rookifies kubernetes library. Rookify needs to connect to the kubernetes cluster on the target systems in order to use Rook. + +```yaml title="config.example.yaml" +rook: + cluster: + name: osism-ceph + namespace: rook-ceph + ceph: + image: quay.io/ceph/ceph:v18.2.1 +``` + +The `rook` sections requires some information about the Rook installation on the target system. Concerning the `cluster` Rookify needs to know the cluster-name and cluster-namespace. Rookify also needs to know the ceph version used by Rookify, i.e. the image version of the ceph container. + +_Note_: Rookify does not install Rook for you. You need to provide a running Rook, i.e. a Rook Operator, on you target system. + +_Note_: For OSISM specific migrations, Rookify needs to have some additional information, i.e. the respective labels for the rook resources: + +```yaml title="config.example.osism.yaml" +rook: + cluster: + name: osism-ceph + namespace: rook-ceph + mds_placement_label: node-role.osism.tech/rook-mds + mgr_placement_label: node-role.osism.tech/rook-mgr + mon_placement_label: node-role.osism.tech/rook-mon + osd_placement_label: node-role.osism.tech/rook-osd + rgw_placement_label: node-role.osism.tech/rook-rgw + ceph: + image: quay.io/ceph/ceph:v18.2.1 +``` + + +#### Migration_modules + +```yaml title="config.example.yaml" +migration_modules: +- analyze_ceph +- create_rook_cluster +- migrate_mons +- migrate_osds +- migrate_osd_pools +- migrate_mds +- migrate_mds_pools +- migrate_mgrs +- migrate_rgws +- migrate_rgw_pools +``` + +Rookify is designed to use a modular structure. It has contains various modules to migrate parts of Ceph-Ansible to Rook. The `migration_modules` section specifies which modules need to be run for the migration. Rookify contains more modules, take a look at the [`src/rookify/module`](https://github.com/SovereignCloudStack/rookify/tree/main/src/rookify/modules) directory to see the ones that are currently implemented. + +_NOTE_: Many modules are dependent of each other. This means that some models will implicitly run other modules. For example: the `analyze_ceph` module specified above, will be run by all the modules. This means, that one does not need to specify it. It was added here only for reasons of clearity. diff --git a/docs/guides/deploy-guide/rookify.md b/docs/guides/deploy-guide/rookify.md new file mode 100644 index 0000000000..68849236fb --- /dev/null +++ b/docs/guides/deploy-guide/rookify.md @@ -0,0 +1,94 @@ +--- +sidebar_label: Deploy Rookify: Migrate to Rook from Ceph Ansible (technical preview) +sidebar_position: 51 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Deploy Rookify: Migrate to Rook from Ceph Ansible (technical preview) + +:::warning + +Rookify is developed to migrate from Ceph-Ansible to Rook _in place_ and _without downtime_. +Nevertheless, it is **strongly advised** to test Rookify in a controlled environment beforehand, such as the [OSISM testbed](https://github.com/osism/testbed). Additionally, ensure that all precautionary backups are taken, and any other necessary safety measures are in place. + +::: + +Rookify is now available in the SCS Reference Implementation (OSISM) and can be deployed as shown in the [paragraph below](#using-the-scs-reference-implementation-osism). + +The [Rookify GitHub repository](https://github.com/SovereignCloudStack/rookify) includes a README.md that provides a condensed summary of the information covered here. + +## Using the SCS Reference Implementation (OSISM) + +:::info + +Rookify will be available in OSISM and will be deployable usin osism-commands based on ansible configurations. + +::: + +But is is not available yet ;) + +## Manual Installation + +### Download or Clone the Repository + +Clone or download Rookify from the [repository](https://github.com/SovereignCloudStack/rookify). + +:::tip + +Checkout the included options of the added `Makefile` by simply typing `make`. + +::: + +## Install and Run Locally + +1. Navigate to the tool directory: + +```bash +cd rookify +``` + +2. To install Rookify locally, pythons virtualenv will be used (Note: This will install pre-commit in your local user context): + +``` +make setup +``` + +This command also checks if you have the required python library for `radoslib` installed. Make sure to have it installed on your linux distribution. + +:::tip + +Before running rookify, first check all options by using `rookify --help` + +::: + +To run rookify you can either run it directly from within pythons virtualenv or with help of the make file: + +```bash +# directly +./.venv/bin/rookify --help +# using make +make run-local-rookify +``` + +## Install and Run from within a Container + +1. Navigate to the tool directory: + +2. To install Rookify into a container, podman or docker can be used (Note: in both cases pythons library for `radoslib` needs to be installed locally): + +``` +make check-radoslib +make up +``` + +This command uses `docker compose`, so make sure you have that installed as well. + +To run rookify you can either enter the container and run rookify from there or use `make run-rookify`. + +:::note + +Before running rookify, it can be usefull to check all options by using `rookify --help` + +::: diff --git a/docs/guides/deploy-guide/services/rookify.md b/docs/guides/deploy-guide/services/rookify.md new file mode 100644 index 0000000000..b67efbc918 --- /dev/null +++ b/docs/guides/deploy-guide/services/rookify.md @@ -0,0 +1,17 @@ +--- +sidebar_label: Migrate Ceph-Ansible via Rookify (technical preview) +sidebar_position: 51 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Migrate Ceph-Ansible via Rookify (technical preview) + +:::warning + +This is a technical preview and not recommended for production use yet. + +::: + +Rookify is now also available in OSISM. diff --git a/docs/guides/operations-guide/rookify.md b/docs/guides/operations-guide/rookify.md new file mode 100644 index 0000000000..cc05c4888a --- /dev/null +++ b/docs/guides/operations-guide/rookify.md @@ -0,0 +1,83 @@ +--- +sidebar_label: Rookify Operation: Migrate to Rook from Ceph Ansible (technical preview) +--- + +# Use Rookify: Migrate to Rook from Ceph Ansible (technical preview) + +:::warning + +Rookify is developed to migrate from Ceph-Ansible to Rook _in place_ and _without downtime_. +Nevertheless, it is **strongly advised** to test Rookify in a controlled environment beforehand, such as the [OSISM testbed](https://github.com/osism/testbed). Additionally, ensure that all precautionary backups are taken, and any other necessary safety measures are in place. + +::: + +The [Rookify GitHub repository](https://github.com/SovereignCloudStack/rookify) includes a README.md that provides a condensed summary of the information covered here. + +## Consider using a pickle file + +In order to have a trackable state of process you can add a pickle file. Specify this option in the `config.yaml`: + +```yaml title="config.example.yaml" +general: + machine_pickle_file: data.pickle +``` + +Now you will be able to to view the state of progress by running `rookify --show-state`. + +:::warning + Rookify will take data.pickle as a source of truth for its operations. If you want to start a clean migration, be sure to delete the file. +::: + +## Rookify CLI + +### Run + +:::warning + Currenlty rookify executes per default +::: + +Currenlty rookify executes per default. This means: if you run rookify like so `.venv/bin/rookify`, it will start the migration as configured in `config.yaml`. + +### --dry-run + +:::tip + Run preflight-mode to check if Rookify can connect to your target systems +::: + +Rookify has a `preflight-mode` to check if basic commands and the connection to the target systems are working. +You can always run `--dry-run` mode, without any danger that migration processes will be executed. + +### --help + +Run `--help` to see the various options the CLI provides. + +### --show + +Run `--show` if you used a pickel-file (see the configuration-guide concerning the general section). This will show the status of your migration process. + +## Debugging and Testing + +If you suspect that something is not working in Rookify itself, you can start setting logging to `DEBUG`. + +If you suspect that something is not working in Rookify itself, you could start by running the tests. For this you need to have access to rookifies code. + +### Set logging to debug level + +Edit the `config.yaml` and set level to "DEBUG": + +```yaml title="config.example.yaml" +logging: + level: DEBUG + format: + time: "%Y-%m-%d %H:%M.%S" # other example: "iso" + renderer: console # or: json +``` + +You can also set the other formatting options, as indicated by the comments, for further options take a look a the documentation of [structlog](https://www.structlog.org/en/stable/standard-library.html). + +### Run tests + +Make sure you can reach Rookifies code on you system. Then you can either: + +1. run `make run-tests-locally` from the working directory of the rookify repository. If you prefer to use a containerized setup, use `make run-tests`. +2. run `.venv/bin/python -m pytest` from the virtual environment of you installation From 87666a70b2b85d02e1516af3b44a07c9b6127938 Mon Sep 17 00:00:00 2001 From: Boekhorst Date: Wed, 25 Sep 2024 14:55:28 +0200 Subject: [PATCH 2/4] fix: remote special characters from titles Signed-off-by: Boekhorst --- docs/guides/configuration-guide/rookify.md | 2 +- docs/guides/deploy-guide/rookify.md | 2 +- docs/guides/operations-guide/rookify.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/guides/configuration-guide/rookify.md b/docs/guides/configuration-guide/rookify.md index c6eddf9874..db725259e7 100644 --- a/docs/guides/configuration-guide/rookify.md +++ b/docs/guides/configuration-guide/rookify.md @@ -1,5 +1,5 @@ --- -sidebar_label: Configure Rookify: Migrate to Rook from Ceph Ansible (technical preview) +sidebar_label: Rookify Ceph Ansible (technical preview) sidebar_position: 31 --- diff --git a/docs/guides/deploy-guide/rookify.md b/docs/guides/deploy-guide/rookify.md index 68849236fb..8b2ea7a5b4 100644 --- a/docs/guides/deploy-guide/rookify.md +++ b/docs/guides/deploy-guide/rookify.md @@ -1,5 +1,5 @@ --- -sidebar_label: Deploy Rookify: Migrate to Rook from Ceph Ansible (technical preview) +sidebar_label: Rookify Ceph Ansible (technical preview) sidebar_position: 51 --- diff --git a/docs/guides/operations-guide/rookify.md b/docs/guides/operations-guide/rookify.md index cc05c4888a..85b35bf56e 100644 --- a/docs/guides/operations-guide/rookify.md +++ b/docs/guides/operations-guide/rookify.md @@ -1,5 +1,5 @@ --- -sidebar_label: Rookify Operation: Migrate to Rook from Ceph Ansible (technical preview) +sidebar_label: Rookify Ceph Ansible (technical preview) --- # Use Rookify: Migrate to Rook from Ceph Ansible (technical preview) From a9382c00c33aaa7d613d700e96e4d97f074a0a76 Mon Sep 17 00:00:00 2001 From: Boekhorst Date: Wed, 25 Sep 2024 15:16:16 +0200 Subject: [PATCH 3/4] fix: add prerequisites and reduce text, feat: add troubleshoot section Signed-off-by: Boekhorst --- docs/guides/deploy-guide/rookify.md | 33 +++++++++++--------- docs/guides/troubleshooting-guide/rookify.md | 32 +++++++++++++++++++ 2 files changed, 50 insertions(+), 15 deletions(-) create mode 100644 docs/guides/troubleshooting-guide/rookify.md diff --git a/docs/guides/deploy-guide/rookify.md b/docs/guides/deploy-guide/rookify.md index 8b2ea7a5b4..648efd97a3 100644 --- a/docs/guides/deploy-guide/rookify.md +++ b/docs/guides/deploy-guide/rookify.md @@ -15,19 +15,22 @@ Nevertheless, it is **strongly advised** to test Rookify in a controlled environ ::: -Rookify is now available in the SCS Reference Implementation (OSISM) and can be deployed as shown in the [paragraph below](#using-the-scs-reference-implementation-osism). +Currenlty Rookify is best install it on your own machine and then connected through vpn to you target system (the one where Ceph-Ansible needs to be rookified ;) ). -The [Rookify GitHub repository](https://github.com/SovereignCloudStack/rookify) includes a README.md that provides a condensed summary of the information covered here. - -## Using the SCS Reference Implementation (OSISM) - -:::info +Rookify runs `in place`, which means that there are no parallel nodes necesssary. As mentioned above, Rookify is developed to migrate from Ceph-Ansible to Rook _in place_ and _without downtime_ but infrastructures are complicated, so precautionary backups and safety measure are very much advised. -Rookify will be available in OSISM and will be deployable usin osism-commands based on ansible configurations. +The [Rookify GitHub repository](https://github.com/SovereignCloudStack/rookify) includes a README.md that provides a condensed summary of the information covered here. -::: +## Prerequisites && Requirements -But is is not available yet ;) +- A functioning Ceph cluster deployed via traditional methods. +- Access to a Kubernetes cluster with sufficient resources to host the + migrated Ceph cluster. + - Kubernetes nodes should be rolled out at least on the OSD nodes +- Rook operator version 1.13 or higher installed in the Kubernetes clu +ster. +- _local development enivornment_ requires radoslib version 2.0.0 inst +alled ## Manual Installation @@ -35,12 +38,6 @@ But is is not available yet ;) Clone or download Rookify from the [repository](https://github.com/SovereignCloudStack/rookify). -:::tip - -Checkout the included options of the added `Makefile` by simply typing `make`. - -::: - ## Install and Run Locally 1. Navigate to the tool directory: @@ -51,6 +48,12 @@ cd rookify 2. To install Rookify locally, pythons virtualenv will be used (Note: This will install pre-commit in your local user context): +:::tip + +Checkout the included options of the added `Makefile` by simply typing `make`. + +::: + ``` make setup ``` diff --git a/docs/guides/troubleshooting-guide/rookify.md b/docs/guides/troubleshooting-guide/rookify.md new file mode 100644 index 0000000000..306d813e00 --- /dev/null +++ b/docs/guides/troubleshooting-guide/rookify.md @@ -0,0 +1,32 @@ +--- +sidebar_label: Rookify +sidebar_position: 40 +--- + +# Rookify (technical preview) + +:::warning + +Rookify is developed to migrate from Ceph-Ansible to Rook _in place_ and _without downtime_. +Nevertheless, it is **strongly advised** to test Rookify in a controlled environment beforehand, such as the [OSISM testbed](https://github.com/osism/testbed). Additionally, ensure that all precautionary backups are taken, and any other necessary safety measures are in place. + +::: + +The [Rookify GitHub repository](https://github.com/SovereignCloudStack/rookify) includes a README.md that provides a condensed summary of the information covered here. + +## ssh-issues + + +**"Failed to load private key" ** + + +- make sure the id-rsa keys are "clean" and do not contain unexpected strings like "\<\ Date: Wed, 25 Sep 2024 15:17:48 +0200 Subject: [PATCH 4/4] fix: reduce titles in sidebar Signed-off-by: Boekhorst --- docs/guides/configuration-guide/rookify.md | 2 +- docs/guides/deploy-guide/rookify.md | 2 +- docs/guides/operations-guide/rookify.md | 2 +- docs/guides/troubleshooting-guide/rookify.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/guides/configuration-guide/rookify.md b/docs/guides/configuration-guide/rookify.md index db725259e7..e73d70be0d 100644 --- a/docs/guides/configuration-guide/rookify.md +++ b/docs/guides/configuration-guide/rookify.md @@ -1,5 +1,5 @@ --- -sidebar_label: Rookify Ceph Ansible (technical preview) +sidebar_label: Rookify (technical preview) sidebar_position: 31 --- diff --git a/docs/guides/deploy-guide/rookify.md b/docs/guides/deploy-guide/rookify.md index 648efd97a3..38d6763d30 100644 --- a/docs/guides/deploy-guide/rookify.md +++ b/docs/guides/deploy-guide/rookify.md @@ -1,5 +1,5 @@ --- -sidebar_label: Rookify Ceph Ansible (technical preview) +sidebar_label: Rookify (technical preview) sidebar_position: 51 --- diff --git a/docs/guides/operations-guide/rookify.md b/docs/guides/operations-guide/rookify.md index 85b35bf56e..0fb8aff1e2 100644 --- a/docs/guides/operations-guide/rookify.md +++ b/docs/guides/operations-guide/rookify.md @@ -1,5 +1,5 @@ --- -sidebar_label: Rookify Ceph Ansible (technical preview) +sidebar_label: Rookify (technical preview) --- # Use Rookify: Migrate to Rook from Ceph Ansible (technical preview) diff --git a/docs/guides/troubleshooting-guide/rookify.md b/docs/guides/troubleshooting-guide/rookify.md index 306d813e00..5e8c0e8252 100644 --- a/docs/guides/troubleshooting-guide/rookify.md +++ b/docs/guides/troubleshooting-guide/rookify.md @@ -1,5 +1,5 @@ --- -sidebar_label: Rookify +sidebar_label: Rookify (technical preview) sidebar_position: 40 ---