Changing logging levels

[zhenghh04]

In this PR, we changed the per step output from info to debug to reduce the logging overhead.

We also add support for changing logging level

[hariharan-devarajan]

We should add error logging as an option for pure performance mode.
For the final per metric we should use a different logging, which outputs PRINT messages to STDOUT and redirects to the dlio.log file.

The other logging is optional from the benchmark.

[hariharan-devarajan]

@zhenghh04 Can we not print the file name and line number for info logging. We should only do that for debug logging. This will significantly reduce log size.

[zhenghh04]

@hariharan-devarajan , made the changes as you suggested. Please review it again.

[hariharan-devarajan]

We are almost there. I see we have some code which got reverted from the mock logging.

[zhenghh04]

@hariharan-devarajan , I am hesitating between using workflow.log_level to control the log level, vs using DLIO_LOG_LEVEL environment variable to control it. I slightly lean towards to the latter one, which is more common in other apps. What is your preference?

[hariharan-devarajan]

@hariharan-devarajan , I am hesitating between using workflow.log_level to control the log level, vs using DLIO_LOG_LEVEL environment variable to control it. I slightly lean towards to the latter one, which is more common in other apps. What is your preference?

I see a value in both. But I prefer the environment variable, too. By default, we should have WARN level, not info. Then we can make DLIO_LOG_LEVEL to higher logging levels like INFO and DEBUG.

We should switch the per epoch time to print and include the sample rate per epoch there.

Then, per step becomes info.

And variable logging becomes debug.

[zhenghh04]

@hariharan-devarajan After reading the documentation more: https://docs.python.org/3/library/logging.html#levels, I feel that we should have default logging level to be info, and everything should go through logging, not through print.

What do you think?

[hariharan-devarajan]

So printing is different than logging in my opinion.

The things that tell us about benchmark high-level like initialize, progress, and metrics.

Logging here is for internal parts.

I wrote a logger in the past on c++ in which I added print as the highest log above error.

For benchmarks I feel it makes sense to have this.

[hariharan-devarajan]

Now if u want to use logging for printing, I would create two loggers. One for printing and one for logging internal stuff.

The printing could always be info, internal, and cannot change by benchmark parameters.