crash when data size exceeds int32

[SiNZeRo]

maybe related to this code:

LightGBM/src/io/cuda/cuda_row_data.cpp

Line 326 in 4f47547

const size_t data_offset = offset + data_index * num_columns_in_cur_partition;

[jameslamb]

Thanks for using LightGBM.

I've updated the link in your question to one that's anchored to a specific commit, so even if that file you've linked to is altered, anyone reading this in the future will know what line you meant. If you don't know how to do that, see https://docs.github.com/en/repositories/working-with-files/using-files/getting-permanent-links-to-files#press-y-to-permalink-to-a-file-in-a-specific-commit.

Typically a report like "this crashed" without any other details is very difficult for us to investigate. Since you didn't provide any details like a minimal, reproducible example, I'm going to interpret this as a question.... "does training with device_type=cuda support using training data with more than max(int32) (2,147,483,647) rows"?

I SUSPECT that the answer is "no", given that even CPU-based LightGBM does not support more than int32 rows in the input data: #5454 .

@guolinke or @shiyu1994 can you please comment?

[SiNZeRo]

Thanks for using LightGBM.

I've updated the link in your question to one that's anchored to a specific commit, so even if that file you've linked to is altered, anyone reading this in the future will know what line you meant. If you don't know how to do that, see https://docs.github.com/en/repositories/working-with-files/using-files/getting-permanent-links-to-files#press-y-to-permalink-to-a-file-in-a-specific-commit.

Typically a report like "this crashed" without any other details is very difficult for us to investigate. Since you didn't provide any details like a minimal, reproducible example, I'm going to interpret this as a question.... "does training with device_type=cuda support using training data with more than max(int32) (2,147,483,647) rows"?

I SUSPECT that the answer is "no", given that even CPU-based LightGBM does not support more than int32 rows in the input data: #5454 .

@guolinke or @shiyu1994 can you please comment?

thanks for the updates.

sorry for misleading you, actually datasize means num_rows * num_feats.

somewhat related to this pr: #5167

[jameslamb]

closed via #5706

[jameslamb]

thanks for the help @SiNZeRo !

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.