Skip to content

BIGTOP-3908 Upgrade Spark Packages for PySpark Requres Python3 #1087

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

BIGTOP-3908 Upgrade Spark Packages for PySpark Requres Python3 #1087

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion bigtop-packages/src/common/spark/install_spark.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ cat > $PREFIX/$BIN_DIR/pyspark <<EOF
# Autodetect JAVA_HOME if not defined
. /usr/lib/bigtop-utils/bigtop-detect-javahome

export PYSPARK_PYTHON=python
export PYSPARK_PYTHON=python3

exec $LIB_DIR/bin/pyspark "\$@"
EOF
Expand Down
20 changes: 1 addition & 19 deletions bigtop-packages/src/rpm/spark/SPECS/spark.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,11 +116,7 @@ Server for Spark worker
%package -n %{spark_pkg_name}-python
Summary: Python client for Spark
Group: Development/Libraries
%if 0%{?rhel} >= 8
Requires: %{spark_pkg_name}-core = %{version}-%{release}, python2
%else
Requires: %{spark_pkg_name}-core = %{version}-%{release}, python
%endif
Requires: %{spark_pkg_name}-core = %{version}-%{release}, python36
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, bigtop uses Spark 3.2, but since Spark 3.3, the minimal python version is 3.7, so can we use a higher version to make it more future-proofing?
https://spark.apache.org/docs/3.3.2/api/python/getting_started/install.html

Copy link
Contributor Author

@vivostar vivostar Mar 26, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for your suggestions, I think we can use a higher python3 version.


%description -n %{spark_pkg_name}-python
Includes PySpark, an interactive Python shell for Spark, and related libraries
Expand Down Expand Up @@ -183,19 +179,6 @@ bash $RPM_SOURCE_DIR/do-component-build
%__rm -rf $RPM_BUILD_ROOT
%__install -d -m 0755 $RPM_BUILD_ROOT/%{initd_dir}/

%if 0%{?rhel} >= 8
PYSPARK_PYTHON=python2 bash $RPM_SOURCE_DIR/install_spark.sh \
--build-dir=`pwd` \
--source-dir=$RPM_SOURCE_DIR \
--prefix=$RPM_BUILD_ROOT \
--doc-dir=%{doc_spark} \
--lib-dir=%{usr_lib_spark} \
--var-dir=%{var_lib_spark} \
--bin-dir=%{bin_dir} \
--man-dir=%{man_dir} \
--etc-default=%{etc_default} \
--etc-spark=%{etc_spark}
%else
bash $RPM_SOURCE_DIR/install_spark.sh \
--build-dir=`pwd` \
--source-dir=$RPM_SOURCE_DIR \
Expand All @@ -207,7 +190,6 @@ bash $RPM_SOURCE_DIR/install_spark.sh \
--man-dir=%{man_dir} \
--etc-default=%{etc_default} \
--etc-spark=%{etc_spark}
%endif

%__rm -f $RPM_BUILD_ROOT/%{usr_lib_spark}/jars/hadoop-*.jar

Expand Down