July 2023 APAC community sync

[jojochuang]

Jul 13

Sammi, ChenXi, Weichiu, George, Yiyang

• Yiyang: HDDS-8974. Introduce detailed lock information #5020 to support Cost Based Scheduler.
• ChenXI: HDDS-8869. Make DN DeleteBlocksCommandHandler wait for the lock can timeout. #4913 allow DN lock holder to timeout.
• Process for 1.4.0 release.
• George: 1.3.1 release planning.
• Weichiu: CommunityOverCode proposal submission.

[jojochuang]

July 20

Weichiu, Yiyang, George

• Yiyang: https://github.com/apache/hadoop/pull/5863/files
  ** multipart upload needs optimization.
• Weichiu: potential OOM due to codec de/serialization.
  ** Q3 roadmap discussion.

[ChenSammi]

July 28th
Attender: Shopee(Symious, Ivan), DiDi(Hongbin, Guangbao and several other DiDi engineers), Qihoo 360(Guo Hao), Cloudera(Sammi)

Agenda:
Cloudera

• Give an update of community status and features’ progress, such as snapshot, quota, EC, block token etc.

Shopee

• Previous seeing a lot bellowing failure during SCM leader transfer,
  Inconsistent read for blockID=conID: 1 locID: 111677748019200007 bcsId: 0 length=2 position=1 numBytesToRead=1 numBytesRead=-1

  Is root caused and resolved by HDDS-8973

• 1.4.0 release progress discussion.

DiDi

• A non production cluster was upgraded to the community master branch recently. After upgrading, it took around 30 min to boot up. It was doing the bucket usage calculation of 259 million keys. DiDi would like to know if there is any way to reduce the time before next time they upgrade the Ozone production cluster.

• Find memory leak issue in Ozone s3g service, the same problem as RATIS-1705

• DiDi ‘s HDFS cluster is adding 1PB high density storage DN. They have agreed to share this experience later once it’s finished. Ozone can learn from it how to support the high density node in future.

• DN decommission. A DN with 50GB data is under decommissioned for more than 3 days. The decommission is still going on. Two suggestions for them,
  - Check SCM log to find if there is any unhealthy container
  - Increase “hdds.datanode.replication.streams.limit”

Qihoo 360

• They are developing one big file to save multiple small files for an EC type container.
• They recently synced their internal branch with Ozone master branch. They find some complexity during the code merge. The major two are
  • The concrete class KeyValueContainer is used instead of the interface Container in many places. As they implemented a new Container type with a new implementation, it costs some time for them to complete the code merge.
  • They use Apache Cassandra to replace the rocksdb. The replacement is mainly based on the Table interface. With the new snapshot feature, the coupling of upper level logic and rocksdb is tighter. This part of code merge is going on.
• The plan to submit some refactor patch later for the above two related modules.