Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DATA-2672] Refactor data capture and data sync #4161

Merged
merged 60 commits into from
Sep 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
f051419
Refactor data capture and data sync
dgottlieb Jun 28, 2024
29f89d4
wip
nicksanford Aug 22, 2024
5dc1af4
update docs
nicksanford Aug 23, 2024
4f84b33
wip
nicksanford Aug 23, 2024
e5039ae
wip
nicksanford Aug 23, 2024
71cbf13
wip
nicksanford Aug 23, 2024
34f2253
add more tests
nicksanford Aug 27, 2024
415ddad
wip
nicksanford Aug 27, 2024
542ea1f
lint
nicksanford Aug 28, 2024
a9e0ad5
comment
nicksanford Aug 28, 2024
a651e33
wip
nicksanford Aug 28, 2024
1941050
wip
nicksanford Aug 28, 2024
4c93917
wip
nicksanford Aug 28, 2024
b4dc58a
improve logging
nicksanford Aug 29, 2024
8e5f29a
wip
nicksanford Aug 30, 2024
4fcca0c
wip
nicksanford Sep 4, 2024
23535f2
add loggign
nicksanford Sep 5, 2024
30cf3c8
reduce log spam
nicksanford Sep 5, 2024
9d07e83
fix logs
nicksanford Sep 5, 2024
e4e7e17
fix DATA-3114
nicksanford Sep 5, 2024
3db16f1
improve loggin
nicksanford Sep 5, 2024
479b188
Update services/datamanager/builtin/builtin.go
nicksanford Sep 6, 2024
bf7817e
Update services/datamanager/builtin/capture/file_count_logger.go
nicksanford Sep 6, 2024
9e4c14e
feedback
nicksanford Sep 6, 2024
742d647
feedback
nicksanford Sep 6, 2024
1bc1d3f
feedback
nicksanford Sep 6, 2024
e088062
docs
nicksanford Sep 6, 2024
90b3892
wip
nicksanford Sep 6, 2024
f4789f2
Update services/datamanager/builtin/sync/upload_data_capture_file.go
nicksanford Sep 6, 2024
95c15d5
wip
nicksanford Sep 6, 2024
a2882bc
wip
nicksanford Sep 6, 2024
efd1dd2
wip
nicksanford Sep 9, 2024
c8a1cde
add comments
nicksanford Sep 9, 2024
6b74c5a
improve logging
nicksanford Sep 9, 2024
f06df5e
wip
nicksanford Sep 9, 2024
0f21967
wip
nicksanford Sep 9, 2024
48312f8
wip
nicksanford Sep 9, 2024
86224dc
improve logging
nicksanford Sep 9, 2024
5d4a988
wip
nicksanford Sep 9, 2024
ef9881e
log
nicksanford Sep 9, 2024
f865268
add disk summarizer
nicksanford Sep 11, 2024
3a4edb2
add upload stats
nicksanford Sep 12, 2024
2c0a115
add more stats
nicksanford Sep 13, 2024
25d87d8
format
nicksanford Sep 13, 2024
d8c736f
add sync stats
nicksanford Sep 16, 2024
feb5d83
wip
nicksanford Sep 16, 2024
2d3d827
change max_sync_threads default from 1000 NumCpu
nicksanford Sep 16, 2024
96967b9
change to runtime.NumCpu() / 2
nicksanford Sep 16, 2024
d8c34c6
feedback
nicksanford Sep 16, 2024
2362808
feedback
nicksanford Sep 16, 2024
4e73481
log which subdirectory you are syncing
nicksanford Sep 17, 2024
62480ce
change data collectors to only create 1 goroutine
nicksanford Sep 17, 2024
7a3ee83
DATA-2506
nicksanford Sep 20, 2024
514b10c
wip
nicksanford Sep 20, 2024
5642540
lint
nicksanford Sep 20, 2024
d7f339a
fix collector deadlock
nicksanford Sep 20, 2024
8507242
wip
nicksanford Sep 20, 2024
b1871b6
wip
nicksanford Sep 23, 2024
cd05725
fix rare race
nicksanford Sep 24, 2024
03edc6f
better logging
nicksanford Sep 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions data/bytes_format_utils.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
package data

import "fmt"

const (
_ = 1 << (10 * iota)
kib
mib
gib
tib
)

// FormatBytesI64 formats an int64 representing bytes
// as an easily human parsable string.
func FormatBytesI64(b int64) string {
switch {
case b > tib:
return fmt.Sprintf("%.2f TB", float64(b)/tib)
case b > gib:
return fmt.Sprintf("%.2f GB", float64(b)/gib)
case b > mib:
return fmt.Sprintf("%.2f MB", float64(b)/mib)
case b > kib:
return fmt.Sprintf("%.2f KB", float64(b)/kib)
default:
return fmt.Sprintf("%d Bytes", b)
}
}

// FormatBytesU64 formats an uint64 representing bytes
// as an easily human parsable string.
func FormatBytesU64(b uint64) string {
switch {
case b > tib:
return fmt.Sprintf("%.2f TB", float64(b)/tib)
case b > gib:
return fmt.Sprintf("%.2f GB", float64(b)/gib)
case b > mib:
return fmt.Sprintf("%.2f MB", float64(b)/mib)
case b > kib:
return fmt.Sprintf("%.2f KB", float64(b)/kib)
default:
return fmt.Sprintf("%d Bytes", b)
}
}
Loading
Loading