Support for faster processing

[s4sukan]

I am using this driver to get the data from MySQL, apply some business logic and put it back to another MySQL[Downstream].

This process takes up some IO but memory usage is fairly less, I have use parallel threads to improve but all taking up IO and not using memory. Is there a way that we can make use of memory as a buffer?

[dveeden]

Can you share some more details on how you are using this and how you measure the IO? Is this filesystem I/O, network I/O or something else?

[s4sukan]

It's filesystem IO, it's local VM not EBS mount.

It will connect to MySQL using this package --> read logs --> apply business logic --> Write the data to destination MySQL.

If you look at the top command output, it doesn't use the memory. Do we have any buffers to do these operations in memory. I am talking about the business logic which involves some lookups(on another table) based on events in the binary logs.

top - 14:11:34 up 25 days, 21:52,  1 user,  load average: 20.11, 20.00, 20.78
Tasks: 345 total,   2 running, 343 sleeping,   0 stopped,   0 zombie
%Cpu(s): 26.3 us, 25.5 sy,  0.0 ni, 44.6 id,  0.0 wa,  1.7 hi,  1.9 si,  0.0 st
MiB Mem :  31982.1 total,  18200.0 free,   6809.5 used,   6972.7 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  23141.7 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  2255 root      20   0 9077636 849364 291160 S   7.3   2.6 911:11.89 s1-agent
   960 root      20   0  628612 422556 297236 S   0.7   1.3  52:25.31 systemd-journal
  1319 root      20   0  975532 400608  81544 S   0.3   1.2  86:04.67 rsyslogd
 26256 root      20   0 3411028 271060   4784 S  17.5   0.8 446:51.69 teststreamer
 34385 root      20   0 4217200 268076   4776 S  23.2   0.8 672:18.25 teststreamer
 36552 root      20   0 3617532 137716   7232 S  50.0   0.4 837:02.08 teststreamer
 29930 root      20   0 4084744 137440   7440 S  70.5   0.4 874:51.57 teststreamer
 37444 root      20   0 3959264 136520   4868 S 124.2   0.4   1645:23 teststreamer
 30256 root      20   0 4019508 135312   7384 S  47.4   0.4 620:15.32 teststreamer
 38648 root      20   0 4152540 130912   4988 S  72.2   0.4   1012:47 teststreamer
 31838 root      20   0 3752292 120664   7408 S  43.7   0.4 656:53.96 teststreamer
 36836 root      20   0 3682532 119040   7336 S  52.0   0.4 903:37.65 teststreamer
 33450 root      20   0 3617148 118388   7452 S  84.1   0.4 916:47.48 teststreamer
 28278 root      20   0 3617660 118020   7312 S  75.8   0.4 895:42.12 teststreamer
 37424 root      20   0 3686608 110724   7312 S  57.6   0.3 770:16.87 teststreamer
 34657 root      20   0 3547260 109816   4884 S  30.8   0.3 547:07.24 teststreamer
107594 root      20   0 3885580 101024   7320 S  62.9   0.3 922:12.41 teststreamer

[lance6716]

You can run golang's built-in profiling to see the bottleneck. And this package does not use much filesystem IO, can you check what's the call stack?