Skip to content

Commit 342a071

Browse files
rameeshmRamesh ManiCopilot
authored
RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker (#660)
* RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker Signed-off-by: Ramesh Mani <[email protected]> * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker - Review comments adddress, hadoop and hive ssh issue while startup addressed, removed not need configs * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker - changes to use ranger base image for user creation, fix issue with usage of ranger base image in other containers * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker - addressed review comment on issue related to base immage * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker - address review comment on the switch user statement * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker - address review comment * Update dev-support/ranger-docker/scripts/hive-site-oracle.xml Co-authored-by: Copilot <[email protected]> * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker - fixed Ozone version --------- Signed-off-by: Ramesh Mani <[email protected]> Co-authored-by: Ramesh Mani <[email protected]> Co-authored-by: Copilot <[email protected]>
1 parent a61d9c0 commit 342a071

20 files changed

+748
-41
lines changed

‎dev-support/ranger-docker/.env‎

Lines changed: 48 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -11,46 +11,66 @@ RANGER_BASE_IMAGE=apache/ranger-base
1111
RANGER_BASE_VERSION=20250707-1-8
1212
# Java version used to build Apache Ranger is present as suffix: -8, valid values for suffix: -8, -11, -17
1313
RANGER_BASE_BUILD_VERSION=20250707-1-8
14+
RANGER_VERSION=3.0.0-SNAPSHOT
1415

15-
# third party image versions
16-
MARIADB_VERSION=10.7.3
17-
POSTGRES_VERSION=12
18-
ORACLE_VERSION=23.6
19-
SQLSERVER_VERSION=2019-latest
20-
ENABLE_DB_MOUNT=true
21-
ZK_VERSION=3.9.2
22-
SOLR_VERSION=8.11.3
23-
24-
# service versions
16+
# Hadoop Configuration
2517
HADOOP_VERSION=3.3.6
26-
HBASE_VERSION=2.6.0
18+
HDFS_PLUGIN_VERSION=3.0.0-SNAPSHOT
19+
YARN_PLUGIN_VERSION=3.0.0-SNAPSHOT
20+
21+
# Hive Configuration
22+
HIVE_HADOOP_VERSION=3.3.6
2723
HIVE_VERSION=4.0.1
28-
HIVE_HADOOP_VERSION=3.1.1
24+
HIVE_PLUGIN_VERSION=3.0.0-SNAPSHOT
25+
26+
# Tez Configuration
27+
TEZ_VERSION=0.10.4
28+
29+
# HBase Configuration
30+
HBASE_VERSION=2.6.0
31+
HBASE_PLUGIN_VERSION=3.0.0-SNAPSHOT
32+
33+
# Kafka Configuration
2934
KAFKA_VERSION=2.8.2
35+
KAFKA_PLUGIN_VERSION=3.0.0-SNAPSHOT
36+
37+
# Knox Configuration
3038
KNOX_VERSION=2.0.0
31-
TRINO_VERSION=377
32-
OZONE_VERSION=1.4.0
33-
OZONE_RUNNER_VERSION=20230615-1
34-
OZONE_RUNNER_IMAGE=apache/ozone-runner
35-
OZONE_OPTS=
39+
KNOX_PLUGIN_VERSION=3.0.0-SNAPSHOT
3640

37-
# versions of ranger services
38-
RANGER_VERSION=3.0.0-SNAPSHOT
41+
# KMS Configuration
3942
KMS_VERSION=3.0.0-SNAPSHOT
43+
44+
# Usersync Configuration
4045
USERSYNC_VERSION=3.0.0-SNAPSHOT
46+
47+
# Tagsync Configuration
4148
TAGSYNC_VERSION=3.0.0-SNAPSHOT
4249

43-
# plugin versions
44-
HDFS_PLUGIN_VERSION=3.0.0-SNAPSHOT
45-
YARN_PLUGIN_VERSION=3.0.0-SNAPSHOT
46-
HIVE_PLUGIN_VERSION=3.0.0-SNAPSHOT
47-
HBASE_PLUGIN_VERSION=3.0.0-SNAPSHOT
48-
KAFKA_PLUGIN_VERSION=3.0.0-SNAPSHOT
49-
KNOX_PLUGIN_VERSION=3.0.0-SNAPSHOT
50-
TRINO_PLUGIN_VERSION=3.0.0-SNAPSHOT
50+
# Solr Configuration
51+
SOLR_VERSION=8.11.2
52+
53+
# Zookeeper Configuration
54+
ZK_VERSION=3.8.4
55+
56+
# Database Versions
57+
POSTGRES_VERSION=13.16
58+
MARIADB_VERSION=10.6
59+
ORACLE_VERSION=21.3.0-xe
60+
SQLSERVER_VERSION=2022-latest
61+
62+
# Ozone Configuration
63+
OZONE_VERSION=1.4.0
5164
OZONE_PLUGIN_VERSION=3.0.0-SNAPSHOT
65+
OZONE_RUNNER_IMAGE=apache/ozone-runner
66+
OZONE_RUNNER_VERSION=20230615-1
67+
68+
# Trino Configuration
69+
TRINO_VERSION=435
70+
TRINO_PLUGIN_VERSION=3.0.0-SNAPSHOT
5271

53-
# To enable debug logs
72+
# Debug Configuration
5473
DEBUG_ADMIN=false
5574
DEBUG_USERSYNC=false
5675
DEBUG_TAGSYNC=false
76+
ENABLE_FILE_SYNC_SOURCE=false

‎dev-support/ranger-docker/Dockerfile.ranger-hadoop‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,14 @@ FROM ${RANGER_BASE_IMAGE}:${RANGER_BASE_VERSION}
2121
ARG HADOOP_VERSION
2222
ARG HDFS_PLUGIN_VERSION
2323
ARG YARN_PLUGIN_VERSION
24+
ARG TEZ_VERSION
2425

2526

2627
COPY ./dist/version /home/ranger/dist/
2728
COPY ./dist/ranger-${HDFS_PLUGIN_VERSION}-hdfs-plugin.tar.gz /home/ranger/dist/
2829
COPY ./dist/ranger-${YARN_PLUGIN_VERSION}-yarn-plugin.tar.gz /home/ranger/dist/
2930
COPY ./downloads/hadoop-${HADOOP_VERSION}.tar.gz /home/ranger/dist/
31+
COPY ./downloads/apache-tez-${TEZ_VERSION}-bin.tar.gz /home/ranger/dist/
3032

3133
COPY ./scripts/ranger-hadoop-setup.sh /home/ranger/scripts/
3234
COPY ./scripts/ranger-hadoop.sh /home/ranger/scripts/
@@ -37,6 +39,9 @@ COPY ./scripts/ranger-yarn-plugin-install.properties /home/ranger/scripts/
3739
RUN tar xvfz /home/ranger/dist/hadoop-${HADOOP_VERSION}.tar.gz --directory=/opt/ && \
3840
ln -s /opt/hadoop-${HADOOP_VERSION} /opt/hadoop && \
3941
rm -f /home/ranger/dist/hadoop-${HADOOP_VERSION}.tar.gz && \
42+
tar xvfz /home/ranger/dist/apache-tez-${TEZ_VERSION}-bin.tar.gz --directory=/opt/ && \
43+
ln -s /opt/apache-tez-${TEZ_VERSION}-bin /opt/tez && \
44+
rm -f /home/ranger/dist/apache-tez-${TEZ_VERSION}-bin.tar.gz && \
4045
tar xvfz /home/ranger/dist/ranger-${HDFS_PLUGIN_VERSION}-hdfs-plugin.tar.gz --directory=/opt/ranger && \
4146
ln -s /opt/ranger/ranger-${HDFS_PLUGIN_VERSION}-hdfs-plugin /opt/ranger/ranger-hdfs-plugin && \
4247
rm -f /home/ranger/dist/ranger-${HDFS_PLUGIN_VERSION}-hdfs-plugin.tar.gz && \

‎dev-support/ranger-docker/Dockerfile.ranger-hive‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1414
# See the License for the specific language governing permissions and
1515
# limitations under the License.
16-
ARG RANGER_DB_TYPE
16+
# Only the ARGs needed for this build stage
1717

1818
ARG RANGER_BASE_IMAGE
1919
ARG RANGER_BASE_VERSION
@@ -22,13 +22,15 @@ FROM ${RANGER_BASE_IMAGE}:${RANGER_BASE_VERSION}
2222
ARG HIVE_VERSION
2323
ARG HIVE_HADOOP_VERSION
2424
ARG HIVE_PLUGIN_VERSION
25+
ARG TEZ_VERSION
2526
ARG RANGER_DB_TYPE
2627

2728

2829
COPY ./dist/version /home/ranger/dist/
2930
COPY ./dist/ranger-${HIVE_PLUGIN_VERSION}-hive-plugin.tar.gz /home/ranger/dist/
3031
COPY ./downloads/apache-hive-${HIVE_VERSION}-bin.tar.gz /home/ranger/dist/
3132
COPY ./downloads/hadoop-${HIVE_HADOOP_VERSION}.tar.gz /home/ranger/dist/
33+
COPY ./downloads/apache-tez-${TEZ_VERSION}-bin.tar.gz /home/ranger/dist/
3234
COPY ./downloads/postgresql-42.2.16.jre7.jar /home/ranger/dist/
3335
COPY ./downloads/mysql-connector-java-8.0.28.jar /home/ranger/dist/
3436
COPY ./downloads/ojdbc8.jar /home/ranger/dist/
@@ -38,7 +40,7 @@ COPY ./scripts/ranger-hive.sh /home/ranger/scripts/
3840
COPY ./scripts/ranger-hive-plugin-install.properties /home/ranger/scripts/
3941
COPY ./scripts/hive-site-${RANGER_DB_TYPE}.xml /home/ranger/scripts/hive-site.xml
4042

41-
RUN tar xvfz /home/ranger/dist/apache-hive-${HIVE_VERSION}-bin.tar.gz --directory=/opt/ && \
43+
RUN cd /opt && tar xzf /home/ranger/dist/apache-hive-${HIVE_VERSION}-bin.tar.gz && \
4244
ln -s /opt/apache-hive-${HIVE_VERSION}-bin /opt/hive && \
4345
rm -f /home/ranger/dist/apache-hive-${HIVE_VERSION}-bin.tar.gz && \
4446
mv /home/ranger/dist/postgresql-42.2.16.jre7.jar /opt/hive/lib/ && \
@@ -47,6 +49,9 @@ RUN tar xvfz /home/ranger/dist/apache-hive-${HIVE_VERSION}-bin.tar.gz --director
4749
tar xvfz /home/ranger/dist/hadoop-${HIVE_HADOOP_VERSION}.tar.gz --directory=/opt/ && \
4850
ln -s /opt/hadoop-${HIVE_HADOOP_VERSION} /opt/hadoop && \
4951
rm -f /home/ranger/dist/hadoop-${HIVE_HADOOP_VERSION}.tar.gz && \
52+
tar xvfz /home/ranger/dist/apache-tez-${TEZ_VERSION}-bin.tar.gz --directory=/opt/ && \
53+
ln -s /opt/apache-tez-${TEZ_VERSION}-bin /opt/tez && \
54+
rm -f /home/ranger/dist/apache-tez-${TEZ_VERSION}-bin.tar.gz && \
5055
tar xvfz /home/ranger/dist/ranger-${HIVE_PLUGIN_VERSION}-hive-plugin.tar.gz --directory=/opt/ranger && \
5156
ln -s /opt/ranger/ranger-${HIVE_PLUGIN_VERSION}-hive-plugin /opt/ranger/ranger-hive-plugin && \
5257
rm -f /home/ranger/dist/ranger-${HIVE_PLUGIN_VERSION}-hive-plugin.tar.gz && \
@@ -55,7 +60,8 @@ RUN tar xvfz /home/ranger/dist/apache-hive-${HIVE_VERSION}-bin.tar.gz --director
5560

5661
ENV HIVE_HOME=/opt/hive
5762
ENV HADOOP_HOME=/opt/hadoop
58-
ENV PATH=/usr/java/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/hive/bin:/opt/hadoop/bin
59-
63+
ENV TEZ_HOME=/opt/tez
64+
ENV TEZ_CONF_DIR=/opt/tez/conf
65+
ENV PATH=/usr/java/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/hive/bin:/opt/hadoop/bin:/opt/tez/bin
6066

6167
ENTRYPOINT [ "/home/ranger/scripts/ranger-hive.sh" ]

‎dev-support/ranger-docker/Dockerfile.ranger-kafka‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ RUN tar xvfz /home/ranger/dist/kafka_2.12-${KAFKA_VERSION}.tgz --directory=/opt/
3939
cp -f /home/ranger/scripts/ranger-kafka-plugin-install.properties /opt/ranger/ranger-kafka-plugin/install.properties && \
4040
chmod 744 ${RANGER_SCRIPTS}/ranger-kafka-setup.sh ${RANGER_SCRIPTS}/ranger-kafka.sh
4141

42+
4243
ENV KAFKA_HOME=/opt/kafka
4344
ENV PATH=/usr/java/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/kafka/bin
4445

‎dev-support/ranger-docker/Dockerfile.ranger-kms‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1414
# See the License for the specific language governing permissions and
1515
# limitations under the License.
16+
1617
ARG RANGER_DB_TYPE
1718
ARG RANGER_BASE_IMAGE
1819
ARG RANGER_BASE_VERSION
@@ -38,7 +39,7 @@ RUN tar xvfz /home/ranger/dist/ranger-${KMS_VERSION}-kms.tar.gz --directory=${RA
3839
ln -s /etc/init.d/ranger-kms /etc/rc3.d/S88ranger-kms && \
3940
ln -s /etc/init.d/ranger-kms /etc/rc3.d/K90ranger-kms && \
4041
ln -s ${RANGER_HOME}/kms/ranger-kms-services.sh /usr/bin/ranger-kms-services.sh && \
41-
chown -R rangerkms:ranger ${RANGER_HOME}/kms/ ${RANGER_SCRIPTS}/ /var/run/ranger_kms/ /var/log/ranger/ && \
42+
chown -R rangerkms:ranger ${RANGER_HOME}/kms/ ${RANGER_SCRIPTS}/ /var/run/ranger_kms/ /var/log/ranger/ /etc/ranger && \
4243
chmod 744 ${RANGER_SCRIPTS}/ranger-kms.sh
4344

4445
FROM ranger-kms AS ranger_postgres

‎dev-support/ranger-docker/Dockerfile.ranger-usersync‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -31,10 +31,9 @@ RUN tar xvfz /home/ranger/dist/ranger-${USERSYNC_VERSION}-usersync.tar.gz --dire
3131
ln -s ${RANGER_HOME}/ranger-${USERSYNC_VERSION}-usersync ${RANGER_HOME}/usersync && \
3232
rm -f /home/ranger/dist/ranger-${USERSYNC_VERSION}-usersync.tar.gz && \
3333
cp -f ${RANGER_SCRIPTS}/ranger-usersync-install.properties ${RANGER_HOME}/usersync/install.properties && \
34-
mkdir -p /var/run/ranger /var/log/ranger/usersync /etc/ranger && \
35-
mkdir /etc/init.d || true && \
36-
mkdir /etc/rc2.d || true && \
37-
mkdir /etc/rc3.d || true && \
34+
mkdir -p /var/run/ranger /var/log/ranger/usersync && \
35+
mkdir -p /etc/ranger && \
36+
mkdir -p /etc/init.d /etc/rc2.d /etc/rc3.d && \
3837
touch /etc/init.d/ranger-usersync && \
3938
ln -s /etc/init.d/ranger-usersync /etc/rc2.d/S99ranger-usersync && \
4039
ln -s /etc/init.d/ranger-usersync /etc/rc2.d/K00ranger-usersync && \

‎dev-support/ranger-docker/config/my.cnf‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,5 +19,7 @@
1919
collation-server = utf8_unicode_ci
2020
init-connect = 'SET NAMES utf8'
2121
character_set_server = utf8
22+
# Enable hostname resolution for user authentication (fixes ranger-db user issue)
23+
skip-name-resolve = 0
2224
# transaction-isolation = REPEATABLE-READ
2325
# transaction-isolation = READ-COMMITTED

‎dev-support/ranger-docker/docker-compose.ranger-db.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ services:
2525
args:
2626
- MARIADB_VERSION=${MARIADB_VERSION}
2727
image: ranger-mysql
28-
command: --default-authentication-plugin=mysql_native_password
28+
command: --default-authentication-plugin=mysql_native_password --skip-name-resolve=0
2929
container_name: ranger-mysql
3030
hostname: ranger-db.example.com
3131
ports:

‎dev-support/ranger-docker/docker-compose.ranger-hadoop.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ services:
99
- HADOOP_VERSION=${HADOOP_VERSION}
1010
- HDFS_PLUGIN_VERSION=${HDFS_PLUGIN_VERSION}
1111
- YARN_PLUGIN_VERSION=${YARN_PLUGIN_VERSION}
12+
- TEZ_VERSION=${TEZ_VERSION}
1213
image: ranger-hadoop
1314
container_name: ranger-hadoop
1415
hostname: ranger-hadoop.example.com
@@ -32,6 +33,7 @@ services:
3233
- HADOOP_VERSION
3334
- HDFS_PLUGIN_VERSION
3435
- YARN_PLUGIN_VERSION
36+
- TEZ_VERSION
3537

3638
networks:
3739
ranger:

‎dev-support/ranger-docker/docker-compose.ranger-hive.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ services:
99
- HIVE_HADOOP_VERSION=${HIVE_HADOOP_VERSION}
1010
- HIVE_VERSION=${HIVE_VERSION}
1111
- HIVE_PLUGIN_VERSION=${HIVE_PLUGIN_VERSION}
12+
- TEZ_VERSION=${TEZ_VERSION}
1213
- RANGER_DB_TYPE=${RANGER_DB_TYPE}
1314
image: ranger-hive
1415
container_name: ranger-hive
@@ -19,6 +20,7 @@ services:
1920
- ranger
2021
ports:
2122
- "10000:10000"
23+
- "9083:9083"
2224
depends_on:
2325
ranger:
2426
condition: service_started
@@ -30,6 +32,7 @@ services:
3032
- HIVE_HADOOP_VERSION
3133
- HIVE_VERSION
3234
- HIVE_PLUGIN_VERSION
35+
- TEZ_VERSION
3336
- RANGER_DB_TYPE
3437

3538
networks:

0 commit comments

Comments
 (0)