User accounts and groups

[Zarquan]

We have a variety of different user accounts and groups, which we will probably need to bring together to make the consistent.

• The Ansible and Kubernetes deployments have different user account names and UIDs.
• Nigel's personal account on the gateway machine has a specific account name and UID.
• The Zeppelin and Spark components may have different user account names and UIDs.

This thread is to discuss how many accounts we need and how to manage them.

There are different ways to manage the accounts:

1. Each user gets a separate Unix account and UID. - like the nch account set up for Nigel.
   • This means we have to propagate the UID all the way through the chain so that the Zeppelin and Spark jobs inherit the correct UID for that user.
   • We might end up needing to use something like an LDAP database to manage all the different user accounts and group memberships.
2. All the processes use the same Unix account and UID for data processing, and we manage data access by controlling at a different level.
   • A variant of this is to have a zeppelin account and a spark account, which are members of the same Unix group.

[stvoutsin]

This is what our user/accounts setup looks like in our current Ansible-based deploy:

Unix accounts:
On the Zeppelin & Master nodes, we have 3 accounts setup, the fedora (default) account, and one for nch & stv. These two accounts were created manually, and the fedora account is the only account with sudo access.
On the worker nodes we just have the fedora account.

Zeppelin:
In Zeppelin we have two accounts setup currently, an admin and the gaiauser accounts. These are created/managed by the shiro.ini configuration file.

Yarn/Spark//Zeppelin Interaction:
When running a Spark job via Zeppelin, the job is created as the fedora user, rather than the Zeppelin user who is logged in.

As Zeppelin & Spark are ran as the fedora user, this means that the process is owned by fedora. So when Zeppelin tries to write to local disk for example, this is done using the fedora acount, which means that the folder needs to be setup for read/write access to fedora.
Zeppelin users share the SparkContext with the default configuration (unless we decide to change this), so jobs from all users go through a single Spark application.

[Zarquan]

Could we do something with the Unix groups? Perhaps add all the accounts to the same group, and make the user data directories sticky read/write to that group?