samba client pod is not coming up after k apply -f client-test-pod.yaml

[mwaykole]

steps :

1. create a minikue deployment
2. clone the repo and make deploy
3. $ k config set-context --current --namespace=samba-operator-system
4. $ # k get pods
   NAME READY STATUS RESTARTS AGE
   samba-operator-controller-manager-844d976b7b-nlgqb 2/2 Running 0

---

[a@dhcp47-98 files]$ k apply -f client-test-pod.yaml
pod/smbclient created

[a@dhcp47]$ k get pods
NAME READY STATUS RESTARTS AGE
samba-ad-server-86b7dd9856-shvxp 1/1 Running 0 46m
samba-operator-controller-manager-844d976b7b-nlgqb 2/2 Running 0 19h
smbclient 0/1 ContainerCreating 0 30m -->> pod is not comming up

---

events:

35m Normal Scheduled pod/smbclient Successfully assigned samba-operator-system/smbclient to minikube
34m Warning FailedMount pod/smbclient MountVolume.SetUp failed for volume "data" : configmap "sample-data1" not found
27m Warning FailedMount pod/smbclient MountVolume.SetUp failed for volume "data" : configmap "sample-data1" not found
25m Warning FailedMount pod/smbclient Unable to attach or mount volumes: unmounted volumes=[kube-api-access-45s5r data], unattached volumes=[kube-api-access-45s5r data]: timed out waiting for the condition
4m36s Normal Scheduled pod/smbclient Successfully assigned samba-operator-system/smbclient to minikube
2m28s Warning FailedMount pod/smbclient MountVolume.SetUp failed for volume "data" : configmap "sample-data1" not found
2m33s Warning FailedMount pod/smbclient Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data kube-api-access-88s6g]: timed out waiting for the condition
87s Normal Scheduled pod/smbclient Successfully assigned samba-operator-system/smbclient to minikube
24s Warning FailedMount pod/smbclient MountVolume.SetUp failed for volume "data" : configmap "sample-data1" not found
16s Warning FailedMount pod/smbclient Unable to attach or mount volumes: unmounted volumes=[data kube-api-access-88s6g], unattached volumes=[data kube-api-access-88s6g]: timed out waiting for the condition

---

Hi plz, let me know if I missed any steps ,

[phlogistonjohn]

Hello there. I'm happy to see someone new trying out samba operator!

The yaml manifest client-test-pod.yaml depends on the config map sample-data1. This config map is currently kept at tests/files/data1.yaml. Create the configmap using something like kubectl apply -f tests/files/data1.yaml and then the client test pod should start running.

[mwaykole]

Hi Thanks the above problem is solved , but now

---

samba-ad-server-86b7dd9856-h8dh5                     1/1     Running    0          27m
samba-operator-controller-manager-844d976b7b-hb94m   2/2     Running    0          2d19h
smbclient                                            1/1     Running    0          2d19h
tshare1-7dbf77d95f-vgd4f                             1/1     Running    0          68m
tshare2-85464f98c4-njldh                             0/4     Init:0/2   0          4m41s

---

tshare2-85464f98c4-njldh <<<<-- is not coming up ? in events also not seen any errors

[mwaykole]

Events:
Type Reason Age From Message

---

Normal Scheduled 2m5s default-scheduler Successfully assigned samba-operator-system/tshare2-8bf9f566d-k2gw6 to minikube
Warning FailedMount 61s (x8 over 2m5s) kubelet MountVolume.SetUp failed for volume "join-data-0" : references non-existent secret key: join2.json
Warning FailedMount 2s kubelet Unable to attach or mount volumes: unmounted volumes=[join-data-0], unattached volumes=[join-data-0 samba-wb-sockets-dir tshare2-pvc-smb svcwatch samba-container-config samba-state-dir kube-api-access-fsdfm]: timed out waiting for the condition

[phlogistonjohn]

If you're using the same YAML files from tests/files, much like the client pod, these have dependencies. In the case of our SmbShares, these are based on data files defined in the SmbSecurityConfig. Take a look at the file smbsecurityconfig1.yaml for tshare1 and smbsecurityconfig2.yaml for tshare2. Both refer to a secret. These secrets must be loaded in the same namespace as the pod.

For tshare2 this is kept in the yaml file joinsecret1.yaml. Use kubectl to load that file and see if that resolves your problem.

[mwaykole]

I am trying to use something like

---
apiVersion: v1
kind: Secret
metadata:
  namespace: samba-operator-system
  name: join1
type: Opaque
stringData:
  # Change the value below to match the username and password for a user that
  # can join systems your test AD Domain
  join.json: |
    {"username": "admin", "password": "admin"}
  # join2.json: |
  #   {"username": "Administrator", "password": "admin"}
---
apiVersion: samba-operator.samba.org/v1alpha1
kind: SmbSecurityConfig
metadata:
  name: adsec1
spec:
  mode: active-directory
  realm: domain1.sink.test
  joinSources:
  - userJoin:
      secret: join1
      key: join1.json
  dns:
    register: cluster-ip

is it correct?

[phlogistonjohn]

Looks mostly OK, but I do see that the secret resource specifies a namespace but the SmbSecurityConfig does not. So depending on what your kubectl command line is the resources may or may not be created in the same namespace. It's not wrong - but I can't say if you'll get the expected result.

Also, what password you use depends on your AD if you're using the same AD server container that the tests do, the password is different. But it's totally OK if want to change the password. So again, I can't say if it's right or wrong. But it is different.

[mrrajan]

@phlogistonjohn I am facing the same issue as @milindw96 mentioned. I am using the files from /test/files. Standalone pods are up and running without issues. But the pods with active-directory mode on smbsecurityconfig stuck at init state.
Please find the list of resources created for "tshare2" below:

Please clarify any prerequisite needs to be done before running the samba-operator for AD.

[phlogistonjohn]

Hi @rravi6121 I suspect that the issue is similar too, the AD domain member instances has more dependencies that the standalone and more opportunities to get stuck. :-)

Can you please run kubectl describe pod <podname>?

PS. It's generally better to copy and paste text from your terminal over a screenshot image. When you paste you can surround the text in three backticks (```) on their own line before and after your paste to make it clear what you pasted. An example:

Hello world!

[mrrajan]

Thanks @phlogistonjohn, Please find the output of "kubectl describe pod" below:

Name:         tshare2-5b78b7764-5rwxc
Namespace:    samba-operator-system
Priority:     0
Node:         minikube/<<ipaddress>>
Start Time:   Thu, 22 Jul 2021 10:38:30 +0530
Labels:       app=samba
              app.kubernetes.io/component=smbd
              app.kubernetes.io/instance=samba-tshare2
              app.kubernetes.io/managed-by=samba-operator
              app.kubernetes.io/name=samba
              app.kubernetes.io/part-of=samba
              pod-template-hash=5b78b7764
              samba-operator.samba.org/service=tshare2
Annotations:  kubectl.kubernetes.io/default-container: samba
              kubectl.kubernetes.io/default-logs-container: samba
Status:       Pending
IP:           10.244.0.5
IPs:
  IP:           10.244.0.5
Controlled By:  ReplicaSet/tshare2-5b78b7764
Init Containers:
  init:
    Container ID:  cri-o://82b5401b6a519a95fd2db99344256950cf8779b0ebc4dededa25128a705cc80a
    Image:         quay.io/samba.org/samba-server:latest
    Image ID:      quay.io/samba.org/samba-server@sha256:4ef1fd9f02b6e5cef4d32c4b4388dde1411fcab651eb2bb1b2fa33423d8c0bd3
    Port:          <none>
    Host Port:     <none>
    Args:
      init
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 23 Jul 2021 09:39:52 +0530
      Finished:     Fri, 23 Jul 2021 09:39:52 +0530
    Ready:          True
    Restart Count:  0
    Environment:
      SAMBA_CONTAINER_ID:  tshare2
      SAMBACC_CONFIG:      /etc/container-config/config.json
    Mounts:
      /etc/container-config from samba-container-config (rw)
      /var/lib/samba from samba-state-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-plb9l (ro)
  must-join:
    Container ID:  cri-o://ae28980851302223f03597962c8e8c339e05568aa02611b29bc28f762d67380c
    Image:         quay.io/samba.org/samba-server:latest
    Image ID:      quay.io/samba.org/samba-server@sha256:4ef1fd9f02b6e5cef4d32c4b4388dde1411fcab651eb2bb1b2fa33423d8c0bd3
    Port:          <none>
    Host Port:     <none>
    Args:
      must-join
    State:          Running
      Started:      Fri, 23 Jul 2021 09:40:08 +0530
    Ready:          False
    Restart Count:  0
    Environment:
      SAMBA_CONTAINER_ID:  tshare2
      SAMBACC_CONFIG:      /etc/container-config/config.json
      SAMBACC_JOIN_FILES:  /var/tmp/join/0/join.json:/var/tmp/join/1/join.json
    Mounts:
      /etc/container-config from samba-container-config (rw)
      /var/lib/samba from samba-state-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-plb9l (ro)
      /var/tmp/join/0 from join-data-0 (rw)
      /var/tmp/join/1 from join-data-1 (rw)
Containers:
  samba:
    Container ID:  
    Image:         quay.io/samba.org/samba-server:latest
    Image ID:      
    Port:          445/TCP
    Host Port:     0/TCP
    Args:
      run
      smbd
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Liveness:       tcp-socket :445 delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      SAMBA_CONTAINER_ID:  tshare2
      SAMBACC_CONFIG:      /etc/container-config/config.json
    Mounts:
      /etc/container-config from samba-container-config (rw)
      /mnt/aa5ccaa6-0445-4e89-a0ae-108ca118a769 from tshare2-pvc-smb (rw)
      /run/samba/winbindd from samba-wb-sockets-dir (rw)
      /var/lib/samba from samba-state-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-plb9l (ro)
  wb:
    Container ID:  
    Image:         quay.io/samba.org/samba-server:latest
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      run
      winbindd
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Liveness:       exec [samba-container check winbind] delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      SAMBA_CONTAINER_ID:  tshare2
      SAMBACC_CONFIG:      /etc/container-config/config.json
    Mounts:
      /etc/container-config from samba-container-config (rw)
      /run/samba/winbindd from samba-wb-sockets-dir (rw)
      /var/lib/samba from samba-state-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-plb9l (ro)
  dns-register:
    Container ID:  
    Image:         quay.io/samba.org/samba-server:latest
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      dns-register
      --watch
      --target=internal
      /var/lib/svcwatch/status.json
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      SAMBA_CONTAINER_ID:  tshare2
      SAMBACC_CONFIG:      /etc/container-config/config.json
    Mounts:
      /etc/container-config from samba-container-config (rw)
      /run/samba/winbindd from samba-wb-sockets-dir (rw)
      /var/lib/samba from samba-state-dir (rw)
      /var/lib/svcwatch from svcwatch (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-plb9l (ro)
  svc-watch:
    Container ID:   
    Image:          quay.io/samba.org/svcwatch:latest
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      DESTINATION_PATH:     /var/lib/svcwatch/status.json
      SERVICE_LABEL_KEY:    samba-operator.samba.org/service
      SERVICE_LABEL_VALUE:   (v1:metadata.labels['samba-operator.samba.org/service'])
      SERVICE_NAMESPACE:    samba-operator-system (v1:metadata.namespace)
    Mounts:
      /var/lib/svcwatch from svcwatch (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-plb9l (ro)
Conditions:
  Type              Status
  Initialized       False 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  samba-container-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      samba-container-config
    Optional:  false
  samba-state-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  tshare2-pvc-smb:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  tshare2-pvc
    ReadOnly:   false
  samba-wb-sockets-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  join-data-0:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  join1
    Optional:    false
  join-data-1:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  join1
    Optional:    false
  svcwatch:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  default-token-plb9l:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-plb9l
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason   Age    From     Message
  ----    ------   ----   ----     -------
  Normal  Pulling  6m36s  kubelet  Pulling image "quay.io/samba.org/samba-server:latest"
  Normal  Pulled   6m32s  kubelet  Successfully pulled image "quay.io/samba.org/samba-server:latest" in 4.612177804s
  Normal  Created  6m31s  kubelet  Created container init
  Normal  Started  6m31s  kubelet  Started container init
  Normal  Pulling  6m31s  kubelet  Pulling image "quay.io/samba.org/samba-server:latest"
  Normal  Pulled   6m15s  kubelet  Successfully pulled image "quay.io/samba.org/samba-server:latest" in 15.324456624s
  Normal  Created  6m15s  kubelet  Created container must-join
  Normal  Started  6m15s  kubelet  Started container must-join

[phlogistonjohn]

Thanks. Based on that output I suspect that "must-join" is unable to join the pod to active directory. The best thing we can do next is to see why must-join is not proceeding. That's usually because the join information doesn't match AD or AD is unreachable.

Run kubectl logs tshare2-5b78b7764-5rwxc -c must-join to get the logs from the must-join container. Then if you don't have any sensitive data in it, share your join secret with kubectl get secret join1 -o yaml. And lastly it would help if you described how you set up AD - is it just the AD pod yaml from our tests/files directory? And if so, did you deploy it using the script in ./tests/test-deploy-ad-server.sh and/ or did you configure the coredns config for your cluster?

[mrrajan]

Thanks @phlogistonjohn I have deployed AD pod using yaml with "kubectl apply -f" command and that might be the reason for the issue.

I tried deploying with shell script ./tests/test-deploy-ad-server.sh and the pod is up and running without issues

NAME                                                     READY   STATUS    RESTARTS   AGE
pod/samba-ad-server-86b7dd9856-7rwc5                     1/1     Running   0          45m
pod/samba-operator-controller-manager-844d976b7b-cx4qj   2/2     Running   0          59m
pod/smbclient                                            1/1     Running   0          55m
pod/tshare1-7f47b774b4-d4plg                             1/1     Running   0          33m
pod/tshare2-76f95db4d8-xlt42                             4/4     Running   0          31m

Steps followed:

1. Install and Start minikube. Check the status and make sure minikube running without issues
2. Git clone samba-operator repository and navigate into it.
3. Run "make deploy" from the "samba-operator" folder
4. Run "kubectl config set-context --current --namespace=samba-operator-system" to change the minikube namespace to "samba-operator-system"
5. Run "kubectl apply -f" command to deploy yaml files with kind Secret, ConfigMap, SmbCommonConfig and SmbSecurityConfig from "./test/files" folder. Make sure Secrets and ConfigMaps deployed before SmbSecurityConfig
6. Run shell script "./tests/test-deploy-ad-server.sh" to deploy samba-ad-server
7. Deploy Smbshare by running "kubectl apply -f ./tests/tshare2.yaml"

[mrrajan]

Hi @phlogistonjohn ,
I am trying to connect one of my windows AD server with SINK. But must-join step on the smbshare pod fails with the below error message and smbshare pod stuck at init status

Enter Administrator's password:
Failed to join domain: failed to find DC for domain WINDOWS2K16 - The object was not found.
Enter Administrator's password:
Failed to join domain: failed to find DC for domain WINDOWS2K16 - The object was not found.
ERROR: failed 2 join attempts
  - failed to run ['net', 'ads', 'join', '--no-dns-updates', '-U', 'Administrator']
  - failed to run ['net', 'ads', 'join', '--no-dns-updates', '-U', 'Administrator']

I am trying to manually deploy the pods with the configuration yaml files and not using the shell script on tests folder.

Please find the yaml files below:

joinsecret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: join1
type: Opaque
stringData:
  # Change the value below to match the username and password for a user that
  # can join systems your test AD Domain
  join.json: |
    {"username": "Administrator", "password": "P4ssw0rd"}
  join2.json: |
    {"username": "Administrator", "password": "Passw0rd"}
  joinad.json: |
    {"username": "Administrator", "password": "redhat@1"}

smbsecurityconfig.yaml

--
apiVersion: samba-operator.samba.org/v1alpha1
kind: SmbSecurityConfig
metadata:
  name: adsec2
spec:
  mode: active-directory
  realm: Window2k16.DOMAIN.sinktest
  joinSources:
  - userJoin:
      secret: join1
  - userJoin:
      secret: join1
      key: joinad.json
  dns:
    register: cluster-ip

samba-ad-server-deployment.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: samba-ad-server
  labels:
    app: samba-ad
spec:
  replicas: 1
  selector:
    matchLabels:
      app: samba-ad
  template:
    metadata:
      labels:
        app: samba-ad
    spec:
      hostname: w2K16
      containers:
      - name: samba
        image: quay.io/samba.org/samba-ad-server:latest
        securityContext:
          capabilities:
            add: ["SYS_ADMIN"]
        ports:
        - containerPort: 53
          name: dns
        - containerPort: 135
          name: epm
          protocol: TCP
        - containerPort: 137
          name: netbios-ns
          protocol: UDP
        - containerPort: 138
          name: netbios-dgram
          protocol: UDP
        - containerPort: 139
          name: netbios-session
          protocol: TCP
        - containerPort: 389
          name: ldap
        - containerPort: 445
          name: smb
          protocol: TCP
        - containerPort: 464
          name: kerberos
        - containerPort: 636
          name: ldaps
          protocol: TCP
        - containerPort: 3268
          name: gc
          protocol: TCP
        - containerPort: 3269
          name: gc-ssl
          protocol: TCP

smbshare.yaml

---
apiVersion: samba-operator.samba.org/v1alpha1
kind: SmbShare
metadata:
  name: tsharead
spec:
  shareName: "My Kingdom"
  readOnly: false
  securityConfig: adsec2
  storage:
    pvc:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi

coredns-snippet.template

   domain1.sink.test:53 {
        errors
        cache 30
        forward . AD_SERVER_IP
    }
    Window2K16.DOMAIN.sinktest:53 {
        errors
        cache 30
        forward . AD_SERVER_IP
    }

/etc/hosts/ file inside samba-ad-server pod

127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
fe00::0	ip6-mcastprefix
fe00::1	ip6-allnodes
fe00::2	ip6-allrouters
172.17.0.5	dc1
##.##.##.###	w2k16

Windows server manager

Please let me know whether I have missed anything and help me on this.

[phlogistonjohn]

Hi @phlogistonjohn ,
I am trying to connect one of my windows AD server with SINK. But must-join step on the smbshare pod fails with the below error message and smbshare pod stuck at init status

Enter Administrator's password:
Failed to join domain: failed to find DC for domain WINDOWS2K16 - The object was not found.
Enter Administrator's password:
Failed to join domain: failed to find DC for domain WINDOWS2K16 - The object was not found.
ERROR: failed 2 join attempts
  - failed to run ['net', 'ads', 'join', '--no-dns-updates', '-U', 'Administrator']
  - failed to run ['net', 'ads', 'join', '--no-dns-updates', '-U', 'Administrator']

This seems like it could be related to DNS within the pod unable to reach the AD DNS. I think that the first aspect of this we ought to debug.

I am trying to manually deploy the pods with the configuration yaml files and not using the shell script on tests folder.

Understood.

Please find the yaml files below:

joinsecret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: join1
type: Opaque
stringData:
  # Change the value below to match the username and password for a user that
  # can join systems your test AD Domain
  join.json: |
    {"username": "Administrator", "password": "P4ssw0rd"}
  join2.json: |
    {"username": "Administrator", "password": "Passw0rd"}
  joinad.json: |
    {"username": "Administrator", "password": "redhat@1"}

it shouldn't be a problem to have the "extra" items in this secret, but if "joinad.json" is the one that matches your ad it might help simplify things to remove "join.json" and "join2.json".

smbsecurityconfig.yaml

--
apiVersion: samba-operator.samba.org/v1alpha1
kind: SmbSecurityConfig
metadata:
  name: adsec2
spec:
  mode: active-directory
  realm: Window2k16.DOMAIN.sinktest
  joinSources:
  - userJoin:
      secret: join1
  - userJoin:
      secret: join1
      key: joinad.json
  dns:
    register: cluster-ip

Can I have you confirm that Window2k16.DOMAIN.sinktest is indeed the fqdn of the AD domain on your windows system? I'll also make a (very small) nitpick that "sinktest" isn't really good TLD to use - you can read up online about the issues with making up TLDs for internal networks but I'll just note the rfc that defines the ".test" domain which is what we use for our custom deployed AD.

samba-ad-server-deployment.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: samba-ad-server
  labels:
    app: samba-ad
spec:
  replicas: 1
  selector:
    matchLabels:
      app: samba-ad
  template:
    metadata:
      labels:
        app: samba-ad
    spec:
      hostname: w2K16
      containers:
      - name: samba
        image: quay.io/samba.org/samba-ad-server:latest
        securityContext:
          capabilities:
            add: ["SYS_ADMIN"]
        ports:
        - containerPort: 53
          name: dns
        - containerPort: 135
          name: epm
          protocol: TCP
        - containerPort: 137
          name: netbios-ns
          protocol: UDP
        - containerPort: 138
          name: netbios-dgram
          protocol: UDP
        - containerPort: 139
          name: netbios-session
          protocol: TCP
        - containerPort: 389
          name: ldap
        - containerPort: 445
          name: smb
          protocol: TCP
        - containerPort: 464
          name: kerberos
        - containerPort: 636
          name: ldaps
          protocol: TCP
        - containerPort: 3268
          name: gc
          protocol: TCP
        - containerPort: 3269
          name: gc-ssl
          protocol: TCP

This is the first thing that jumps out at me as questionable. If you're using a pre-existing windows based AD, I'm not sure what creating a samba based AD in the k8s cluster will accomplish. At best, its just probably unnecessary. At worst, this could be confusing the DNS situation.

smbshare.yaml

---
apiVersion: samba-operator.samba.org/v1alpha1
kind: SmbShare
metadata:
  name: tsharead
spec:
  shareName: "My Kingdom"
  readOnly: false
  securityConfig: adsec2
  storage:
    pvc:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi

coredns-snippet.template

   domain1.sink.test:53 {
        errors
        cache 30
        forward . AD_SERVER_IP
    }
    Window2K16.DOMAIN.sinktest:53 {
        errors
        cache 30
        forward . AD_SERVER_IP
    }

Here's the next thing I'm not sure about. This might work but depends greatly on how the windows AD is set up. First, it might not be needed if your windows based domain is reachable from the node. If not, this could be useful if the pods can reach "AD_SERVER_IP" for the address(es) of the windows based AD.

/etc/hosts/ file inside samba-ad-server pod

127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
fe00::0	ip6-mcastprefix
fe00::1	ip6-allnodes
fe00::2	ip6-allrouters
172.17.0.5	dc1
##.##.##.###	w2k16

Just to make sure I'm not missing something: you're not trying to create a trust between the samba-based AD and the windows-based AD right? If not, I don't think this config will have much of an impact on the must-join container.

Windows server manager

Please let me know whether I have missed anything and help me on this.

Looking at your screenshot, one thing that jumps out at me is that the computer name is "Window2K16" - the domain is "DOMAIN.sinktest". So the first thing that I would do is change smbsecurityconfig.yaml to have realm: DOMAIN.sinktest. As this value only expects the name of the domain/realm - not the name of a computer in said domain.

If that alone doesn't do it, I would next start checking that the IP of the windows AD DC is reachable from k8s. Check if the nodes can access that IP and if the pods can access that IP. After that I'd check DNS resolution from the nodes and pods. You can even create a test pod with a DNS tool such as dig and check that DOMAIN.sinktest is resolvable.

Lastly, in an ideal world we would be using DNS domains and forests tied into org/global dns. It something difficult to do for our integration test, but we ought to get more experience with set ups that are a bit more real world. After we've resolved your immediate issues I'd like to discuss our options to try and test AD configurations that don't require altering the in-cluster coredns configuration. :-)

[phlogistonjohn]

Hi, I'm happy to help, but it feels like a lot of what in in this issue is general setup and debugging help rather than a defect/bug. Do you mind if I first convert this issue to a github discussion?

[mrrajan]

Sure @phlogistonjohn , It wont be a problem.

[mrrajan]

@phlogistonjohn I have gone through the debugging steps you have shared, I have setup a pod with dig and the windows AD server is reachable, But the error persist.

So I doubt the configuration that I have done. Please find the configurations which I followed below and let me know if anything missed out.

1. Modified the below three files from /tests/files folder

	modified:   joinsecret1.yaml
	modified:   smbsecurityconfig2.yaml
	modified:   smbshare2.yaml

• updated windows AD server credentials on joinsecrect1.yaml

apiVersion: v1
kind: Secret
metadata:
  name: join1
type: Opaque
stringData:
  # Change the value below to match the username and password for a user that
  # can join systems your test AD Domain
  join.json: |
    {"username": "Administrator", "password": "P4ssw0rd"}
  join2.json: |
    {"username": "Administrator", "password": "********"}

• updated smbsecurityconfig.yaml file with realm name

---
apiVersion: samba-operator.samba.org/v1alpha1
kind: SmbSecurityConfig
metadata:
  name: adsec1
spec:
  mode: active-directory
  realm: mysambanetwork.qe
  joinSources:
  - userJoin:
      secret: join1
  - userJoin:
      secret: join1
      key: join2.json
  dns:
    register: cluster-ip

• updated smbshare2.yaml for Computer name on sharename

---
apiVersion: samba-operator.samba.org/v1alpha1
kind: SmbShare
metadata:
  name: tshare2
spec:
  shareName: "MYSAMBANETWORK"
  readOnly: false
  securityConfig: adsec1
  storage:
    pvc:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi

2. Updated the /etc/hosts file on the working machine with the IP address of the windows AD server
3. kubectl apply -f for joinsecrect and smbsecurityconfig2.yaml files
4. Executed the shell script to bring samba-ad-server pod online
5. kubectl apply -f for smbshare2.yaml file

[phlogistonjohn]

@phlogistonjohn I have gone through the debugging steps you have shared, I have setup a pod with dig and the windows AD server is reachable, But the error persist.

Please show the results of the dig command(s) you ran please...

So I doubt the configuration that I have done. Please find the configurations which I followed below and let me know if anything missed out.

1. Modified the below three files from /tests/files folder

	modified:   joinsecret1.yaml
	modified:   smbsecurityconfig2.yaml
	modified:   smbshare2.yaml

* updated windows AD server credentials on joinsecrect1.yaml

apiVersion: v1
kind: Secret
metadata:
  name: join1
type: Opaque
stringData:
  # Change the value below to match the username and password for a user that
  # can join systems your test AD Domain
  join.json: |
    {"username": "Administrator", "password": "P4ssw0rd"}
  join2.json: |
    {"username": "Administrator", "password": "********"}

I'm unclear if "P4ssw0rd" is your password or the join2.json is. I understand if you can't share the actual password but let's at least eliminate any additional possible sources of confusion and ensure we have 1 valid entry in the secret with the right data.

* updated smbsecurityconfig.yaml file with realm name

---
apiVersion: samba-operator.samba.org/v1alpha1
kind: SmbSecurityConfig
metadata:
  name: adsec1
spec:
  mode: active-directory
  realm: mysambanetwork.qe
  joinSources:
  - userJoin:
      secret: join1
  - userJoin:
      secret: join1
      key: join2.json
  dns:
    register: cluster-ip

Without seeing anything else I'll have to trust you have specified the correct value for the realm. Again, posting the results of your dig command would help.

* updated smbshare2.yaml for Computer name on sharename

---
apiVersion: samba-operator.samba.org/v1alpha1
kind: SmbShare
metadata:
  name: tshare2
spec:
  shareName: "MYSAMBANETWORK"
  readOnly: false
  securityConfig: adsec1
  storage:
    pvc:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi

1. Updated the /etc/hosts file on the working machine with the IP address of the windows AD server

To what? Can you share what the /etc/hosts file is and what the "working machine" is?

2. kubectl apply -f for joinsecrect and smbsecurityconfig2.yaml files

3. Executed the shell script to bring samba-ad-server pod online

Why? What do you hope to accomplish by deploying the samba-ad-server pod if you already have a windows AD?

4. kubectl apply -f for smbshare2.yaml file

I think we still need more details of where and what the debugging commands, such as dig reported. Since you've changed a bunch of things reporting the logs from the must-join pod will also be helpful.

If you want to schedule an interactive debugging session some time, let me know. You can reach out to me thru work chat too.

[mrrajan]

The issue with must-join failure is coredns of k8s cluster couldnt connect to the outside world where the windows AD lives in.
To check whether k8s cluster able to reach out to your domain, setup a pod with "gcr.io/kubernetes-e2e-test-images/dnsutils:1.3" image on the same k8s cluster and run dig command. if the query status is "NXDOMAIN", then we need to teach our coredns of our cluster to reach out the windows AD DC.

To do this, copy the file https://github.com/samba-in-kubernetes/samba-operator/blob/master/tests/files/coredns-snippet.template. Replace "domain1.sink.test" to your realmname and change "AD_SERVER_IP" to the actual IP of the Domain Controller on this template.

Run "kubectl edit cm -n kube-system coredns", it will open a editor with coredns configuration. Copy the content from template file created on the above step and paste it as a subsection of "Corefile: |" section. Wait for 30s and the changes would get effective. With this, k8s cluster reach outs to the AD DC through port 53.

Test whether the domain reachable from the dnsutils pod and the smbshare pod would invoked without issues!

Thanks @phlogistonjohn for these debugging/resolution steps. With this we can close this discussion!