From 84de4b18dfa37ed98778f5a57bfc2155dc91d52e Mon Sep 17 00:00:00 2001
From: 10sharmashivam <10sharmashivam@gmail.com>
Date: Fri, 18 Oct 2024 06:57:30 +0530
Subject: [PATCH] MPI Operator troubleshooting steps added in docs

Signed-off-by: 10sharmashivam <10sharmashivam@gmail.com>
---
 docs/deployment/plugins/k8s/index.rst | 36 +++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/docs/deployment/plugins/k8s/index.rst b/docs/deployment/plugins/k8s/index.rst
index 64fbb41136..ad5113c132 100644
--- a/docs/deployment/plugins/k8s/index.rst
+++ b/docs/deployment/plugins/k8s/index.rst
@@ -1035,3 +1035,39 @@ Wait for the upgrade to complete. You can check the status of the deployment pod
   kubectl get pods -n flyte
 
 Once all the components are up and running, go to the `examples section <https://docs.flyte.org/en/latest/flytesnacks/integrations.html#native-backend-plugins>`__ to learn more about how to use Flyte backend plugins.
+
+Troubleshooting MPI Operator Installation
+-----------------------------------------
+
+This section covers common issues encountered during the setup of the MPI operator for distributed training jobs on Flyte.
+
+**Worker Pods Failing to Start (Insufficient Resources)**
+
+MPI worker pods may fail to start or exhibit scheduling issues, leading to job timeouts or failures. This often occurs due to resource constraints (CPU, memory, or GPU) in the cluster.
+
+1. Adjust Resource Requests:
+Ensure that each worker pod has sufficient resources. You can adjust the resource requests in your task definition:
+
+.. code-block:: bash
+
+      requests=Resources(cpu="<your_cpu_request>", mem="<your_mem_request>")
+
+Modify the CPU and memory values according to your cluster's available resources. This helps prevent pod scheduling failures caused by resource constraints.
+
+2. Check Pod Logs for Errors:
+If the worker pods still fail to start, check the logs for any related errors:
+
+.. code-block:: bash
+
+      kubectl logs <pod-name> -n <namespace>
+
+Look for resource allocation or worker communication errors.
+
+**Workflow Registration Method Errors (Timeouts or Deadlocks)**
+
+If your MPI workflow hangs or times out, it may be caused by an incorrect workflow registration method.
+
+Troubleshooting Steps:
+
+1. 1.	Verify Registration Method:
+When using a custom image, refer to the Flyte documentation on `Registering workflows <https://docs.flyte.org/en/latest/user_guide/flyte_fundamentals/registering_workflows.html#registration-patterns>`__ to ensure you're following the correct registration method.