Set filter to preloading data source for partition pruning #7382

zhli1142015 · 2023-11-02T07:32:14Z

netlify · 2023-11-02T07:32:20Z

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`2e7e7d6`
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/659f52eb47a58800080e3b07

Yuhta

Thanks for the fix, is it possible to add a unit test?

Yuhta · 2023-11-02T15:48:27Z

velox/exec/TableScan.cpp

@@ -92,6 +92,7 @@ RowVectorPtr TableScan::getOutput() {

      if (!split.hasConnectorSplit()) {
        noMoreSplits_ = true;
+        pendingDynamicFilters_.clear();


I think it's easier to follow that we clear the pending filters after passing them to data source

pendingDynamicFilters_ is used in preload to intitalize different data source object for preloading splits, I think it should be clear after no more splits. Thanks.

Emm I see what you want to do. But merging filters on every split could cause performance regression. I think we need to do it a little differently. Maybe when we do the preload, we should pass in the scan spec from the current dataSource_ to use in preloading data source as well.

Hello @Yuhta ,
I tried this proposal, the scanSpec_ can't be used by multi threads same time, even only the filter_ field in ScanSpec.
For your concern about merging filters on every split, I add two APIs in DataSource to get and set the filter for the specific outputChannel, then we only need to set the merged filters to the preloading data sources. This should avoid this problem.
Please help to chekc again, thanks.

velox/exec/TableScan.cpp

zhli1142015 · 2023-11-06T03:19:21Z

Thanks for the fix, is it possible to add a unit test?

Add a UT to assert the split skipping happens in preloaded splits. Thanks.

zhli1142015 · 2023-11-07T08:38:37Z

Hello @Yuhta , could you please check again?

velox/exec/tests/HashJoinTest.cpp

Yuhta · 2023-11-20T21:58:34Z

velox/exec/TableScan.cpp

@@ -256,6 +259,7 @@ void TableScan::preload(std::shared_ptr<connector::ConnectorSplit> split) {
       ctx = operatorCtx_->createConnectorQueryCtx(
           split->connectorId, planNodeId(), connectorPool_),
       task = operatorCtx_->task(),
+       &mergedDynamicFilters = mergedDynamicFilters_,


Can we make this a shared_ptr to be safe?

Changed, thanks.

Yuhta · 2023-11-20T22:05:48Z

velox/connectors/hive/HiveDataSource.h

+  std::unique_ptr<common::Filter> getFilter(
+      column_index_t outputChannel) override {
+    auto& fieldSpec = scanSpec_->getChildByChannel(outputChannel);
+    return fieldSpec.filter()->clone();


The filter should be MT safe as far as I see, all the methods should be read only. Can you try using a shared_ptr<const Filter> to avoid deep copy?

Yes, you are right, changed.

zhli1142015 · 2023-12-01T14:50:08Z

Hello @Yuhta ,
I addressed all comments, could you please continue to take a look?
Thanks.

facebook-github-bot · 2023-12-05T18:04:24Z

@Yuhta has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

zhli1142015 · 2023-12-06T07:08:59Z

The fuzzer failure looks realted to #7856

I20231206 04:56:20.786527 29572 ExpressionVerifier.cpp:97] Executing expression 0 : try(translate(regex_replace("c0",md5("c1"),Y`%2&fl*ZC`d`"F@-j/T2),"c0",^>XIvcHADU#jT[eE&6[JR1gy$vj!4W%9~RUE/W`>Jq,hRy4xUWdOSg-jI))
E20231206 04:56:20.811030 29572 Exceptions.h:69] Line: ../../velox/expression/tests/ExpressionVerifier.cpp:77, Function:operator(), Expression: left->equalValueAt(right.get(), row, row) Different results at idx '0': '[0->71] null' vs. '^>XI>cHADU#jT[eE&6[JJ1gy$vA!y#%9#RUEyg`>JARhgy4jU[dOeg-jj-E`EjJ&1J19`', Source: RUNTIME, ErrorCode: INVALID_STATE

mbasmanova · 2023-12-06T16:39:09Z

velox/connectors/Connector.h

@@ -226,6 +226,17 @@ class DataSource {
  virtual int64_t estimatedRowSize() {
    return kUnknownRowSize;
  }
+
+  virtual std::shared_ptr<common::Filter> getFilter(


This PR changes the Connector API. Please, update PR description to describe the changes and add comments for the newly introduced APIs.

Updated, thanks.

mbasmanova · 2023-12-06T16:40:01Z

velox/exec/tests/HashJoinTest.cpp

+        }
+      })
+      .run();
+  FLAGS_split_preload_per_driver = oldSplitPreload;


This is an anti-pattern. Use RAII.

Updated, thanks.

mbasmanova · 2023-12-07T12:13:22Z

velox/connectors/Connector.h

+  // Sets the filter to the column identified by @param outputChannel.
+  // This can be called when passing the column filters to a preloading data
+  // source. The purpose of this is to avoid the overhead of filters merging.
+  virtual void setColumnFilter(


How is this different from addDynamicFilter? Having these two APIs is confusing. Please, update PR description to provide overall context for this change.

The problem to solve in this PR is passing the dynamic filters to preloading data sources, which are used for partition pruning.
Ininitially I use addDynamicFilter to passe the filters from dataSource_ to preloading data sources, Yuta mentioned merging the filters for every split would cause perf regression, #7382 (comment). Then I added these two methods to get and set filters directly (without merging). Please let me know if any suggestion about this.

@zhli1142015 Thank you for this additional context. Is there a GitHub issue about this? Sounds like we need to iterate on the design a bit and it will be easier to do in a GitHub issue.

@zhli1142015 For example, it is still not clear to me why we need a new setColumnFilter when we already have addDynamicFilter. These 2 APIs seem to provide exactly same information.

#7381, here is the issue I opened.

@zhli1142015 Please, open a GitHub issue and describe the use case you are looking to enabling and the details of the design you are proposing. Thanks.

mbasmanova · 2023-12-07T12:20:50Z

velox/exec/tests/utils/FlagUpdater.h

+
+// Generic RAII class to update GFlag.
+template <typename T>
+class FlagUpdater {


Maybe ScopedGFlag?

mbasmanova · 2023-12-07T12:22:09Z

velox/exec/tests/HashJoinTest.cpp

+  const int32_t numSplits = 5;
+  std::vector<RowVectorPtr> probeVectors;
+  probeVectors.reserve(numSplits);
+  FlagUpdater(FLAGS_split_preload_per_driver, 3);


Are there other tests that use split_preload_per_driver flag and therefore need to be updated to use the new RAII class? If so, would you extract the RAII into a separate PR and updating existing tests?

Does this need to be a GFlag? Can it be a configuration property instead? CC: @xiaoxmeng

This flag is used also in PrintPlanWithStatsTest and TableScanTest. I think this should be converted to a configuration, but I think we can't remove the flag as it's already used in Gluten, https://github.com/oap-project/gluten/blob/main/cpp/velox/compute/VeloxBackend.cc#L53C15-L53C39

Got it. Let's make a new PR to replace this gflag with a configuration property. Then come back to this PR.

Sure, get it.

#7931, thanks.

zhli1142015 · 2023-12-11T05:37:08Z

The failure should be not related to this change, thanks.

I20231211 03:18:16.575408 29965 AggregationFuzzer.cpp:870] ==============================> Started iteration 30 (seed: 3731594752)
terminate called after throwing an instance of 'facebook::velox::dwio::common::exception::LoggedException'
  what():  Exception: VeloxException
Error Source: RUNTIME
Error Code: UNKNOWN
Reason: not supported yet HUGEINT
Retriable: False
Function: create
File: ../../velox/dwio/dwrf/writer/ColumnWriter.cpp
Line: 2082
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxException5State4makeIZNS1_C4EPKcmS5_St17basic_string_viewIcSt11char_traitsIcEES9_S9_S9_bNS1_4TypeES9_EUlRT_E_EESt10shared_ptrIKS2_ESA_SB_
# 2  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 3  _ZN8facebook5velox4dwio6common9exception15LoggedExceptionC1EPKcmS6_S6_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESE_SE_
# 4  _ZN8facebook5velox4dwrf16BaseColumnWriter6createERNS1_13WriterContextERKNS0_4dwio6common10TypeWithIdEjSt8functionIFvRNS1_12IndexBuilderEEE
# 5  _ZN8facebook5velox4dwrf16BaseColumnWriter6createERNS1_13WriterContextERKNS0_4dwio6common10TypeWithIdEjSt8functionIFvRNS1_12IndexBuilderEEE
# 6  _ZN8facebook5velox4dwrf6WriterC2ESt10unique_ptrINS0_4dwio6common8FileSinkESt14default_deleteIS6_EERKNS1_13WriterOptionsESt10shared_ptrINS0_6memory10MemoryPoolEE
# 7  _ZN8facebook5velox4dwrf6WriterC1ESt10unique_ptrINS0_4dwio6common8FileSinkESt14default_deleteIS6_EERKNS1_13WriterOptionsE
# 8  _ZN8facebook5velox4exec4test12_GLOBAL__N_112_GLOBAL__N_111writeToFileERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS0_10BaseVectorEEPNS0_6memory10MemoryPoolE
# 9  _ZN8facebook5velox4exec4test12_GLOBAL__N_117AggregationFuzzer10makeSplitsERKSt6vectorISt10shared_ptrINS0_9RowVectorEESaIS8_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
# 10 _ZN8facebook5velox4exec4test12_GLOBAL__N_117AggregationFuzzer17verifyAggregationERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaISB_EESF_SF_RKS5_ISt10shared_ptrINS0_9RowVectorEESaISI_EEbRKS5_ISG_INS2_14ResultVerifierEESaISO_EE
# 11 _ZN8facebook5velox4exec4test12_GLOBAL__N_117AggregationFuzzer2goEv
# 12 _ZN8facebook5velox4exec4test15aggregateFuzzerESt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt6vectorISt10shared_ptrINS1_26AggregateFunctionSignatureEESaISD_EESt4hashIS9_ESt8equal_toIS9_ESaISt4pairIKS9_SF_EEEmRKS3_IS9_SB_INS2_14ResultVerifierEESH_SJ_SaISK_ISL_SQ_EEERKS3_IS9_SB_INS2_14InputGeneratorEESH_SJ_SaISK_ISL_SX_EEENS0_12VectorFuzzer7Options18TimestampPrecisionERKS3_IS9_S9_SH_SJ_SaISK_ISL_S9_EEERKSt8optionalIS9_ESt10unique_ptrINS2_20ReferenceQueryRunnerESt14default_deleteIS1G_EE
# 13 _ZN8facebook5velox4exec4test23AggregationFuzzerRunner9runFuzzerEmRKSt8optionalINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEESt10unique_ptrINS2_20ReferenceQueryRunnerESt14default_deleteISF_EERKNS3_7OptionsE
# 14 _ZN8facebook5velox4exec4test23AggregationFuzzerRunner3runEmSt10unique_ptrINS2_20ReferenceQueryRunnerESt14default_deleteIS5_EERKNS3_7OptionsE
# 15 main
# 16 __libc_start_main
# 17 _start

zhli1142015 · 2023-12-11T08:32:45Z

@mbasmanova and @Yuhta , could you please help to review this PR? I made the change according to the comment: #7381 (comment)

Yuhta · 2023-12-15T18:43:22Z

velox/connectors/Connector.h

+  void addDynamicFilter(
+      column_index_t outputChannel,
+      const std::shared_ptr<common::Filter>& filter,
+      const bool isMergedFilter) {


nit: No need to const qualify this

Yuhta · 2023-12-15T18:46:58Z

velox/dwio/common/ScanSpec.h

-  ScanSpec& getChildByChannel(column_index_t channel);
+  ScanSpec& getChildByChannel(column_index_t channel) const;
+
+  std::shared_ptr<common::Filter> getFilterByChannel(


const std::shared_ptr<common::Filter>&

Yuhta · 2023-12-15T18:59:31Z

velox/connectors/hive/HiveDataSource.cpp

@@ -476,6 +476,18 @@ HiveDataSource::HiveDataSource(

  readerOpts_.setFileSchema(hiveTableHandle_->dataColumns());
  ioStats_ = std::make_shared<io::IoStatistics>();
+  for (const auto& entry : tableHandle->dynamicFilters()) {


We need to be careful about reseting the filters on ScanSpec. This should be done very rarely as it requires reset of the stats and order of the filters. We should keep the filters in scan spec as the golden source and only change it when really needed, and in table handle just keep some references/shared_ptr to them, not the reverse.

Yuhta · 2023-12-18T18:51:06Z

velox/dwio/common/ScanSpec.h

+    getChildByChannel(channel).addFilter(filter);
+  }
+
+  void setChildFilterByChannel(column_index_t channel, common::Filter& filter) {


Take shared_ptr to avoid deep copy

Updated, thanks.

Yuhta · 2023-12-18T18:52:54Z

velox/connectors/hive/HiveDataSource.h

@@ -71,6 +71,11 @@ class HiveDataSource : public DataSource {

  int64_t estimatedRowSize() override;

+  std::shared_ptr<common::Filter> getColumnFilter(


const std::shared_ptr<common::Filter>&

Updated, thanks.

zhli1142015 · 2024-01-10T00:46:44Z

@zhli1142015

I have no evidence to prove that this is an issue. On the contrary, within our internal repo, we use the existing addDynamicFilter, and in our testing, we haven't observed any problems. The reason for doing this is because of this comment #7382 (comment). I assumed this approach was taken due to some presto limitation not spark side. If we can call existing addDynamicFilter for preloading sources, then change should be simple and understandable enough.

Thank you for clarifying.

But merging filters on every split could cause performance regression.

@Yuhta Jimmy, looks like this may not be an issue. Trying to optimize for this hypothetical problem leads to pretty convoluted code. Perhaps, we don't need that can use a simpler alternative instead.

Hello @mbasmanova ,
I change this PR back to utilize existing addDynamicFilter fucntion, could you please help check?
Thanks.

mbasmanova · 2024-01-10T01:00:22Z

velox/exec/TableScan.cpp

@@ -256,6 +256,7 @@ void TableScan::preload(std::shared_ptr<connector::ConnectorSplit> split) {
       ctx = operatorCtx_->createConnectorQueryCtx(
           split->connectorId, planNodeId(), connectorPool_),
       task = operatorCtx_->task(),
+       &pendingDynamicFilters = pendingDynamicFilters_,


Do we need a reference here? We pass everything else by value.

Thanks for catching this, we should pass by value here

mbasmanova · 2024-01-10T01:05:24Z

velox/exec/tests/HashJoinTest.cpp

+  ColumnHandleMap assignments = {
+      {"n1_0", regularColumn("c0", BIGINT())},
+      {"n1_1", regularColumn("c1", BIGINT())},
+      {"n1_2", partitionKey("k", BIGINT())}};


Is it important to use partition keys columns in this test? Would you explain why?

Yes, we expect dynamic filters on the partition column to be generated. This way, we can verify whether the dynamic filter is passed to the preloaded data source by checking if the split is skipped during processing.

mbasmanova · 2024-01-10T01:07:17Z

velox/exec/tests/HashJoinTest.cpp

+      .verifier([&](const std::shared_ptr<Task>& task, bool hasSpill) {
+        SCOPED_TRACE(fmt::format("hasSpill:{}", hasSpill));
+        if (!hasSpill) {
+          ASSERT_EQ(1, getFiltersProduced(task, 1).sum);


It is error prone to use hard-coded operator indices here. Can we use plan node IDs instead? We have probeScanId and we can also capture join node ID. We can then get stats via toPlanStats.

std::unordered_map<core::PlanNodeId, PlanNodeStats> toPlanStats( const TaskStats& taskStats);

mbasmanova · 2024-01-10T01:08:18Z

velox/exec/tests/HashJoinTest.cpp

+    };
+  };
+  auto outputType =
+      ROW({"n1_0", "n1_1", "n1_2"}, {BIGINT(), BIGINT(), BIGINT()});


These column names are hard to read. By convention, we use t0, t1, ... and u0, u1, ..... You can specify column names as first argument of makeRowVector.

Updated, thanks.

mbasmanova

Looks good % some questions about the test.

mbasmanova

Looks good % some minor comments.

mbasmanova · 2024-01-10T11:55:39Z

velox/exec/tests/HashJoinTest.cpp

+      std::vector<exec::Split> probeSplits;
+      for (int32_t i = 0; i < numSplits; ++i) {
+        auto value = std::to_string(i);
+        auto split = facebook::velox::exec::test::HiveConnectorSplitBuilder(


drop facebook::velox::exec::test::

mbasmanova · 2024-01-10T11:57:39Z

velox/exec/tests/HashJoinTest.cpp

+  HashJoinBuilder(*pool_, duckDbQueryRunner_, driverExecutor_.get())
+      .planNode(std::move(op))
+      .config(core::QueryConfig::kMaxSplitPreloadPerDriver, "3")
+      .makeInputSplits(makeInputSplits(probeScanId))


this code is confusing as makeInputSplits is repeated twice; would you use .inputSplits(probeSplits) instead and remove makeInputSplits lambda from this test?

Updated, thanks.

mbasmanova

Thanks.

mbasmanova · 2024-01-10T13:04:29Z

velox/exec/tests/HashJoinTest.cpp

+  // Executing the join with p1=b0, we expect a dynamic filter for p1 to prune
+  // the entire file/split. There are total of five splits, and all except the
+  // first one are expected to be pruned. The result 'preloadedSplits' > 1
+  // confirms the successful push of dynamic filters to the preloading data


Nice comment.

mbasmanova · 2024-01-10T13:04:49Z

velox/exec/tests/HashJoinTest.cpp

+  // first one are expected to be pruned. The result 'preloadedSplits' > 1
+  // confirms the successful push of dynamic filters to the preloading data
+  // source.
+  core::PlanNodeId probeScanId, joinNodeId;


define each variable separately

Updated, thanks.

mbasmanova · 2024-01-10T13:05:15Z

velox/exec/tests/HashJoinTest.cpp

+          .capturePlanNodeId(joinNodeId)
+          .project({"p0"})
+          .planNode();
+  SplitInput splits = {{probeScanId, probeSplits}};


no need for this variable

.splits({{probeScanId, probeSplits}})

facebook-github-bot · 2024-01-10T13:06:13Z

@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2024-01-10T13:45:11Z

@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

zhli1142015 · 2024-01-10T23:43:04Z

Hello @mbasmanova ,
Looks CI failed in internal tests, please let me know if any fix needed, thanks.

mbasmanova · 2024-01-11T00:31:41Z

@zhli1142015 Would you rebase?

Add dynamic filter for preloading. Velox's dynamic filters would be evaluated in `addSplit` method to skip the whole split: https://msdata.visualstudio.com/A365/_git/Velox?path=/velox/connectors/hive/HiveConnector.cpp&version=GBmain&line=550&lineEnd=551&lineStartColumn=1&lineEndColumn=1&lineStyle=plain&_a=contents, but it was missed in preloading method. This change impacts data loading in query59. Before ![image (2).png](https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_apis/git/repositories/28288850-d04d-499e-821b-6adc4e4d2331/pullRequests/1106399/attachments/image%20%282%29.png) After ![image.png](https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_apis/git/repositories/28288850-d04d-499e-821b-6adc4e4d2331/pullRequests/1106399/attachments/image.png) Related work items: #2529082

zhli1142015 · 2024-01-11T02:32:05Z

@zhli1142015 Would you rebase?

Done, thanks.

facebook-github-bot · 2024-01-11T02:36:24Z

@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2024-01-11T12:41:13Z

@mbasmanova merged this pull request in ef97b5f.

conbench-facebook · 2024-01-11T13:05:16Z

Conbench analyzed the 1 benchmark run on commit ef97b5f7.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

…ncubator#7382) Summary: Fixes facebookincubator#7381 Pull Request resolved: facebookincubator#7382 Reviewed By: xiaoxmeng Differential Revision: D51857062 Pulled By: mbasmanova fbshipit-source-id: bc97ee22fcc5b2fbde37666929fdc718fd876ef9

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 2, 2023

Yuhta reviewed Nov 2, 2023

View reviewed changes

zhli1142015 requested a review from Yuhta November 6, 2023 03:19

zhli1142015 force-pushed the minor-fix-for-preloading branch from 31e1c06 to 3c81cf6 Compare November 11, 2023 14:19

Yuhta reviewed Nov 20, 2023

View reviewed changes

zhli1142015 requested a review from Yuhta November 29, 2023 01:34

zhli1142015 force-pushed the minor-fix-for-preloading branch from 02ae0ca to e2560ed Compare December 6, 2023 04:20

mbasmanova reviewed Dec 6, 2023

View reviewed changes

zhli1142015 changed the title ~~Add dynamic filter for preloading data source~~ Set filter to preloading data source for partition pruning Dec 7, 2023

mbasmanova reviewed Dec 7, 2023

View reviewed changes

zhli1142015 force-pushed the minor-fix-for-preloading branch 2 times, most recently from 775bad0 to fd58569 Compare December 11, 2023 02:31

zhli1142015 force-pushed the minor-fix-for-preloading branch from fd58569 to f43a912 Compare December 12, 2023 06:10

zhli1142015 requested a review from mbasmanova December 12, 2023 10:58

Yuhta reviewed Dec 15, 2023

View reviewed changes

zhli1142015 force-pushed the minor-fix-for-preloading branch 2 times, most recently from 7c025fc to 18d392a Compare December 18, 2023 02:35

Yuhta reviewed Dec 18, 2023

View reviewed changes

zhli1142015 force-pushed the minor-fix-for-preloading branch from 18d392a to 393334f Compare December 19, 2023 03:20

zhli1142015 force-pushed the minor-fix-for-preloading branch from 80d272c to 2181731 Compare January 5, 2024 07:02

zhli1142015 requested a review from mbasmanova January 10, 2024 00:44

mbasmanova reviewed Jan 10, 2024

View reviewed changes

zhli1142015 requested a review from mbasmanova January 10, 2024 10:27

mbasmanova reviewed Jan 10, 2024

View reviewed changes

zhli1142015 requested a review from mbasmanova January 10, 2024 13:04

mbasmanova approved these changes Jan 10, 2024

View reviewed changes

zhli1142015 added 5 commits January 11, 2024 10:14

address comments

4163267

address comments

ba62481

address comments

f177780

address comments

2e7e7d6

zhli1142015 force-pushed the minor-fix-for-preloading branch from 1a18867 to 2e7e7d6 Compare January 11, 2024 02:31

facebook-github-bot closed this in ef97b5f Jan 11, 2024

facebook-github-bot added the Merged label Jan 11, 2024

		@@ -71,6 +71,11 @@ class HiveDataSource : public DataSource {

		int64_t estimatedRowSize() override;

		std::shared_ptr<common::Filter> getColumnFilter(

Set filter to preloading data source for partition pruning #7382

Set filter to preloading data source for partition pruning #7382

Conversation

zhli1142015 commented Nov 2, 2023

netlify bot commented Nov 2, 2023 • edited Loading

✅ Deploy Preview for meta-velox canceled.

Yuhta left a comment

Choose a reason for hiding this comment

Yuhta Nov 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhli1142015 commented Nov 6, 2023

zhli1142015 commented Nov 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhli1142015 commented Dec 1, 2023

facebook-github-bot commented Dec 5, 2023

zhli1142015 commented Dec 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbasmanova Dec 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhli1142015 commented Dec 11, 2023

zhli1142015 commented Dec 11, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Yuhta Dec 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhli1142015 commented Jan 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbasmanova left a comment

Choose a reason for hiding this comment

mbasmanova left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbasmanova left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

facebook-github-bot commented Jan 10, 2024

facebook-github-bot commented Jan 10, 2024

zhli1142015 commented Jan 10, 2024

mbasmanova commented Jan 11, 2024

zhli1142015 commented Jan 11, 2024

facebook-github-bot commented Jan 11, 2024

facebook-github-bot commented Jan 11, 2024

netlify bot commented Nov 2, 2023 •

edited

Loading

Yuhta Nov 2, 2023 •

edited

Loading

mbasmanova Dec 7, 2023 •

edited

Loading

Yuhta Dec 15, 2023 •

edited

Loading