Skip to content

Shibboth clustering

Jump to bottom
Yuriy Movchan edited this page Oct 20, 2023 · 26 revisions

Background

The main problem with IDP clustering is related to IDP development model. The target of this update is to try to make session serializable. Also in the session objects, there are several application level services which make this task much harder.

Idea

  1. Update IDP code to allow serialize session attributes.
  2. Fix issue in spring-webflow serialization code which led to NPE.
  3. Write jetty plugin to allow serialize big data and skip un-serialized (not required) objects.

Thought

  1. During serialization our jetty serialization plugin log Skipping serialization of CLASS. It's not major issue because variables with objects like class org.glassfish.json.JsonBuilderFactoryImpl, net.shibboleth.utilities.java.support.logic.PredicateSupport$$Lambda$535/0x000000010077e040@24a9de9c, etc IDP classes initialize on creation.

Code an latest patches

The code and version below are for CE 4.5.2 and Jetty 10.0.15

git clone https://git.shibboleth.net/git/java-opensaml
git checkout 4.3.0
git clone https://git.shibboleth.net/git/java-identity-provider
git checkout 4.3.1
git clone https://git.shibboleth.net/git/java-support
git checkout 8.4.0

Apply patches to projects above: https://github.com/GluuFederation/oxShibboleth/tree/version_4.5.3/webapp-cluster/src/main/patches After applying all these patches it's possible to build any module of these projects with command like: mvn -Dmaven.test.skip=true clean package install

Build

There is prebuild idp.war already: https://maven.gluu.org/maven/org/gluu/oxshibbolethIdpCluster/4.5.3-SNAPSHOT/oxshibbolethIdpCluster-4.5.3-SNAPSHOT.war We build it in next module.

This module build next libraries for final war file:

spring-webflow-4.5.3-SNAPSHOT.jar

idp-attribute-api-4.3.1.jar, idp-authn-api-4.3.1.jar, idp-saml-api-4.3.1.jar, idp-ui-4.3.1.jar,

java-support-8.4.0.jar,

opensaml-core-4.3.0.jar, opensaml-messaging-api-4.3.0.jar, opensaml-saml-api-4.3.0.jar, opensaml-saml-impl-4.3.0.jar,
opensaml-security-api-4.3.0.jar, opensaml-storage-api-4.3.0.jar, opensaml-storage-impl-4.3.0.jar,
opensaml-xmlsec-api-4.3.0.jar, opensaml-xmlsec-impl-4.3.0.jar

CE deployment

  1. Install and configure IDP.

  2. Download jetty module session-store-jdbc-extended configuration and unpack it into /opt/jetty-10.0/jetty-home-10.0.15. META-INF folder is not needed and must be omitted during unpack. Make sure that jetty and plugin version is the same.

  3. Download jetty module session-store-jdbc-extended library and put it into /opt/jetty-10.0/jetty-home-10.0.15/lib. Make sure that jetty and plugin version is the same.

  4. Add session-store-jdbc-extended and other required modules

su - jetty
/opt/jre/bin/java -jar /opt/jetty/start.jar jetty.home=/opt/jetty jetty.base=/opt/gluu/jetty/idp --add-module=session-store-jdbc-extended
/opt/jre/bin/java -jar /opt/jetty/start.jar jetty.home=/opt/jetty jetty.base=/opt/gluu/jetty/idp --add-module=session-cache-null
/opt/jre/bin/java -jar /opt/jetty/start.jar jetty.home=/opt/jetty jetty.base=/opt/gluu/jetty/idp --add-module=ext
  1. Update /opt/gluu/jetty/idp/start.d/session-store-jdbc-extended.ini
...
jetty.session.savePeriod.seconds=10
#jetty.session.lockTime.millis=0
#jetty.session.delayTime.millis=0
#jetty.session.serialization.compress.data=false

...
db-connection-type=driver
jetty.session.jdbc.driverClass=com.mysql.jdbc.Driver
jetty.session.jdbc.driverUrl=jdbc:mysql://127.0.0.1:3306/session?user=session&password=secret
...

...
jetty.session.jdbc.blobType=mediumblob
...
  1. Update /opt/gluu/jetty/idp/start.d/session-cache-null.ini
...
jetty.session.saveOnCreate=true
#jetty.session.removeUnloadableSessions=false
jetty.session.flushOnResponseCommit=true
...
  1. Add MySQL JDBC driver
cd /opt/gluu/jetty/idp/lib/ext
wget https://repo1.maven.org/maven2/com/mysql/mysql-connector-j/8.0.33/mysql-connector-j-8.0.33.jar
  1. Isntall MySQL server if needed
apt install mysql-server
  1. Add user to access sessions schema

mysql -u root -p

CREATE DATABASE session;
CREATE USER 'session'@'localhost' IDENTIFIED WITH mysql_native_password BY 'secret';
GRANT ALL PRIVILEGES ON session.* TO 'session'@'localhost' WITH GRANT OPTION;
FLUSH PRIVILEGES;
EXIT;
  1. service idp restart
  2. tail -f 2023_08_18.jetty.log
  3. Open https://samltest.id/start-idp-test and trigger login.

Jetty module session-store-jdbc-extended has additional properties:

  • jetty.session.lockTime.millis allow to configure delay between session persistence and loading.

Benchmarking

In this testing test server has 8 CPU and 32 GiB of memory.

  1. Test with 2 IDP instances with sessions serialization and session data compression on:
  • Number of threads 70
  • Loop count: 250

  • CPU usage: 61.7

Detailed server load

  1. Test with 2 IDP instances with sessions serialization and session data compression off:
  • Number of threads: 70
  • Loop count: 250

  • CPU usage: 51.2

Detailed server load

  1. Test with 2 IDP instances with sessions serialization and session data compression off:
  • Number of threads: 90
  • Loop count: 250

  • CPU usage: 64.2

Detailed server load

  1. Test with 2 IDP instances without sessions serialization but with sticky sessions:
  • Number of threads: 70
  • Loop count: 250

  • CPU usage: Min: 29.8 Mean: 31.2 Max 32.5

Detailed server load

Result description

  1. IDP with session serialization works well but this adds additional CPU usage +45-50%.
  2. Req/sec in both approach almost the same.
  3. The main issue with IDP session is session objects size. Serialized data size is 70 KiB (compresses 30 KiB). As result serialization/de-serialization requires a lot of CPU resources. It's possible to significantly reduce CPU load by implementing Externalizable interface in serializable objects. This should allows to get similar CPU load picture in compare to tests with sticky sessions.
  4. According to MySQL CPU usage sessions serialization not adds big DB load. Also it's possible to store IDP session data and oxAuth session data in second DB if needed.

Support

This update add few classes to IDP code. Most updates are related to adding java.io.Serializable to required session beans. Hence code support should be simple unless IDP team will not apply major changes to projects. During update to next IPD version we need to update these properties to conform actual versions of this release:

		<shibboleth.version>4.3.1</shibboleth.version>
		<opensaml.version>4.3.0</opensaml.version>
		<java-support.version>8.4.0</java-support.version>
		<spring-webflow.version>2.5.1.RELEASE</spring-webflow.version>

		<shibboleth.dist.sha256>e71e0a948499a0c383d03e35e4bab3779881b72d0ceb5a03dbb776c04230c613</shibboleth.dist.sha256>

What next

  1. Currently we use java default session serialization which is not optimal for big session data. It do scan all properties of each bean even if we not need to serialize them, etc... Instead of it session classes should implement java.io.Externalizable which will make this process much faster and will less required CPU resources.
Clone this wiki locally