Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data migration and Migrating from 3.x to 4.x in Kubernetes #1722

Merged
merged 7 commits into from
Nov 28, 2024

Conversation

henokgetachew
Copy link
Contributor

Description

This guide answers:

  • How do you use couchdb-migration in Kubernetes environments?
  • How do you migrate a 3.x instance to 4.x without using docker-compose as an intermediary?

License

The software is provided under AGPL-3.0. Contributions to this project are accepted under the same license.

Copy link
Contributor

@mrjones-plip mrjones-plip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! We're in strong need of more k8s docs - yay!

I've made some suggestions - since this is endemic to Medic, we should move this to the Medic section so it's a peer of the existing EKS docs

content/en/hosting/4.x/data-migration-k8s.md Outdated Show resolved Hide resolved
content/en/hosting/4.x/data-migration-k8s.md Outdated Show resolved Hide resolved
content/en/hosting/4.x/data-migration-k8s.md Outdated Show resolved Hide resolved
content/en/hosting/4.x/data-migration-k8s.md Outdated Show resolved Hide resolved
content/en/hosting/4.x/data-migration-k8s.md Outdated Show resolved Hide resolved
content/en/hosting/4.x/data-migration-k8s.md Outdated Show resolved Hide resolved
Comment on lines 91 to 96
# Get the PVC name
kubectl get pvc -n $NAMESPACE

# Get the volume ID from the PVC
VOLUME_ID=$(kubectl get pvc <your-3x-pvc-name> -n $NAMESPACE -o jsonpath='{.spec.volumeName}')
EBS_VOLUME_ID=$(kubectl get pv $VOLUME_ID -o jsonpath='{.spec.awsElasticBlockStore.volumeID}' | cut -d'/' -f4)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think because this section is so long, it's likely having them manually set <your-3x-pvc-name> to be an env var. let's break this out into it's own step to find the the PVC name and set the three env vars. the next larger section can then just by copy and pasted blindly

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get your point. I also considered doing that. The issue is that there will be more than one pvc and we pretty much want the user to identify the pvc that has the data.

content/en/hosting/4.x/data-migration-k8s.md Outdated Show resolved Hide resolved
content/en/hosting/4.x/data-migration-k8s.md Outdated Show resolved Hide resolved
Comment on lines 213 to 237
First verify CouchDB is running properly:
```shell
# Check pod status
kubectl get pods -n $NAMESPACE

# For single node check CouchDB is up
kubectl exec -it -n $NAMESPACE $(kubectl get pod -n $NAMESPACE -l cht.service=couchdb -o name) -- \
curl -s http://localhost:5984/_up

# For clustered setup (check all nodes)
kubectl exec -it -n $NAMESPACE $(kubectl get pod -n $NAMESPACE -l cht.service=couchdb-1 -o name) -- \
curl -s http://localhost:5984/_up
```

Access the new CouchDB pod based on your deployment type.

For single node:
```shell
kubectl exec -it -n $NAMESPACE $(kubectl get pod -n $NAMESPACE -l cht.service=couchdb -o name) -- bash
```

For clustered setup (always use couchdb-1):
```shell
kubectl exec -it -n $NAMESPACE $(kubectl get pod -n $NAMESPACE -l cht.service=couchdb-1 -o name) -- bash
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

love how we've set $NAMESPACE so all these command work just copy and paste - nice work!!

Copy link
Contributor

@mrjones-plip mrjones-plip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops! prior feedback I submitted as "comment" as opposed to "request changes"

Copy link
Contributor Author

@henokgetachew henokgetachew left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes pushed.

Comment on lines 91 to 96
# Get the PVC name
kubectl get pvc -n $NAMESPACE

# Get the volume ID from the PVC
VOLUME_ID=$(kubectl get pvc <your-3x-pvc-name> -n $NAMESPACE -o jsonpath='{.spec.volumeName}')
EBS_VOLUME_ID=$(kubectl get pv $VOLUME_ID -o jsonpath='{.spec.awsElasticBlockStore.volumeID}' | cut -d'/' -f4)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get your point. I also considered doing that. The issue is that there will be more than one pvc and we pretty much want the user to identify the pvc that has the data.

content/en/hosting/4.x/data-migration-k8s.md Outdated Show resolved Hide resolved
Copy link
Member

@dianabarsan dianabarsan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Such a shame that we can't spin up a container with the migration tool. Is there absolutely no way we can do that in k8s?

content/en/hosting/4.x/data-migration-k8s.md Outdated Show resolved Hide resolved
content/en/hosting/4.x/data-migration-k8s.md Outdated Show resolved Hide resolved

Pre-index views to minimize downtime:
```shell
./bin/pre-index-views <desired CHT version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With npm link this would just be pre-index-views 4.2.2

@henokgetachew
Copy link
Contributor Author

Is there absolutely no way we can do that in k8s?

We could use k8s jobs but if we go that route, we need to build automation around it to avoid making the guide a lot more complicated.

I have made the changes for using npm link

Copy link
Contributor

@mrjones-plip mrjones-plip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! Thanks for all the changes.

While I told you to to use native numbers, we loose the linkable subheads and clickable TOC on the right - so I did the work of reverting the changes I requested - only fair!

As well, the YAML file has a LOT of changes that are needed, and unless we call them all out, literally having a list of what you need to change, it's too easy to miss one and then the deployment fails. I listed out every item that needs to be changed - but feel free to update nomenclature if you feel it doesn't make sense.

approving, but please update the 4 sections that literally say Henok what are dev and prod values ;)

thanks again!

Copy link
Member

@dianabarsan dianabarsan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think my only comment here is to add a partial for the sections that are common between these two PRs. More details here: #1723 (review)

@henokgetachew
Copy link
Contributor Author

henokgetachew commented Nov 27, 2024

@dianabarsan there is actually very little that is shared between the two PRs. This one is strictly from medic-eks hosted CHT 3.x to medic-eks hosted CHT 4.x. The order of the steps might be similar but there is very little that's the same. The other PR however definitely can use a partial as the first steps are identical.

Copy link
Member

@dianabarsan dianabarsan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool then! Let's get this going!

@henokgetachew henokgetachew merged commit 79a44cd into main Nov 28, 2024
2 checks passed
@henokgetachew henokgetachew deleted the data-migration-in-k8s branch November 28, 2024 14:38
henokgetachew added a commit that referenced this pull request Dec 2, 2024
* Data migration in k8s
---------------
Co-authored-by: mrjones-plip <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants