Skip to content

Commit f6e4126

Browse files
committed
Add progress tracking and memory estimation for JDBC module
This commit introduces progress tracking functionality for the JDBC module, including both simple and detailed modes configurable via system properties. Added new tests, configuration class (`JdbcConfig`), and memory estimation utilities to ensure efficient handling of large datasets.
1 parent 93d60d3 commit f6e4126

File tree

9 files changed

+1646
-2
lines changed

9 files changed

+1646
-2
lines changed
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
package org.jetbrains.kotlinx.dataframe.io
2+
3+
/**
4+
* Internal configuration for JDBC module behavior.
5+
* Controlled via system properties or environment variables.
6+
*
7+
* Example usage:
8+
* ```kotlin
9+
* // Enable progress with detailed statistics
10+
* System.setProperty("dataframe.jdbc.progress", "true")
11+
* System.setProperty("dataframe.jdbc.progress.detailed", "true")
12+
*
13+
* // Or via environment variables
14+
* export DATAFRAME_JDBC_PROGRESS=true
15+
* export DATAFRAME_JDBC_PROGRESS_DETAILED=true
16+
* ```
17+
*/
18+
internal object JdbcConfig {
19+
/**
20+
* Enable progress logging during data loading.
21+
*
22+
* When `true`, enables DEBUG-level progress messages and memory estimates.
23+
* When `false` (default), only respects logger level (e.g., DEBUG).
24+
*
25+
* System property: `-Ddataframe.jdbc.progress=true`
26+
* Environment variable: `DATAFRAME_JDBC_PROGRESS=true`
27+
* Default: `false`
28+
*/
29+
@JvmStatic
30+
val PROGRESS_ENABLED: Boolean by lazy {
31+
System.getProperty("dataframe.jdbc.progress")?.toBoolean()
32+
?: System.getenv("DATAFRAME_JDBC_PROGRESS")?.toBoolean()
33+
?: false
34+
}
35+
36+
/**
37+
* Enable detailed progress logging with statistics.
38+
*
39+
* When `true`: Shows row count, percentage, speed (rows/sec), memory usage
40+
* When `false`: Shows only basic "Loaded X rows" messages
41+
*
42+
* Only takes effect when progress is enabled.
43+
*
44+
* System property: `-Ddataframe.jdbc.progress.detailed=true`
45+
* Environment variable: `DATAFRAME_JDBC_PROGRESS_DETAILED=true`
46+
* Default: `true`
47+
*/
48+
@JvmStatic
49+
val PROGRESS_DETAILED: Boolean by lazy {
50+
System.getProperty("dataframe.jdbc.progress.detailed")?.toBoolean()
51+
?: System.getenv("DATAFRAME_JDBC_PROGRESS_DETAILED")?.toBoolean()
52+
?: true
53+
}
54+
55+
/**
56+
* Report progress every N rows.
57+
*
58+
* System property: `-Ddataframe.jdbc.progress.interval=5000`
59+
* Environment variable: `DATAFRAME_JDBC_PROGRESS_INTERVAL=5000`
60+
* Default: `1000`
61+
*/
62+
@JvmStatic
63+
val PROGRESS_INTERVAL: Int by lazy {
64+
System.getProperty("dataframe.jdbc.progress.interval")?.toIntOrNull()
65+
?: System.getenv("DATAFRAME_JDBC_PROGRESS_INTERVAL")?.toIntOrNull()
66+
?: 1000
67+
}
68+
}

0 commit comments

Comments
 (0)