Support for setting different DLIO_LOG_LEVEL

[zhenghh04]

In this PR, we changed the per step output from info to debug to reduce the logging overhead.

We added support for changing logging level through environment variable DLIO_LOG_LEVEL which can be set to be error, warning, output, info, debug. By default we set as output.

All the per-step log is put >= info level. The output level only contains per epoch logs. With this, the log file size is significantly reduced.

In the debug level, the log format contains source file and line number, while other levels do not.

[hariharan-devarajan]

We should add error logging as an option for pure performance mode.
For the final per metric we should use a different logging, which outputs PRINT messages to STDOUT and redirects to the dlio.log file.

The other logging is optional from the benchmark.

[hariharan-devarajan]

@zhenghh04 Can we not print the file name and line number for info logging. We should only do that for debug logging. This will significantly reduce log size.

[zhenghh04]

@hariharan-devarajan , made the changes as you suggested. Please review it again.

[hariharan-devarajan]

We are almost there. I see we have some code which got reverted from the mock logging.

[zhenghh04]

@hariharan-devarajan , I am hesitating between using workflow.log_level to control the log level, vs using DLIO_LOG_LEVEL environment variable to control it. I slightly lean towards to the latter one, which is more common in other apps. What is your preference?

[hariharan-devarajan]

@hariharan-devarajan , I am hesitating between using workflow.log_level to control the log level, vs using DLIO_LOG_LEVEL environment variable to control it. I slightly lean towards to the latter one, which is more common in other apps. What is your preference?

I see a value in both. But I prefer the environment variable, too. By default, we should have WARN level, not info. Then we can make DLIO_LOG_LEVEL to higher logging levels like INFO and DEBUG.

We should switch the per epoch time to print and include the sample rate per epoch there.

Then, per step becomes info.

And variable logging becomes debug.

[zhenghh04]

@hariharan-devarajan After reading the documentation more: https://docs.python.org/3/library/logging.html#levels, I feel that we should have default logging level to be info, and everything should go through logging, not through print.

What do you think?

[hariharan-devarajan]

So printing is different than logging in my opinion.

The things that tell us about benchmark high-level like initialize, progress, and metrics.

Logging here is for internal parts.

I wrote a logger in the past on c++ in which I added print as the highest log above error.

For benchmarks I feel it makes sense to have this.

[hariharan-devarajan]

Now if u want to use logging for printing, I would create two loggers. One for printing and one for logging internal stuff.

The printing could always be info, internal, and cannot change by benchmark parameters.

[zhenghh04]

@hariharan-devarajan How about we just keep the logging system as this PR? We can have another PR to improve it?

I need this feature for now to have an easy way to reduce some I/O overhead from the logging.

[zhenghh04]

We should add error logging as an option for pure performance mode. For the final per metric we should use a different logging, which outputs PRINT messages to STDOUT and redirects to the dlio.log file.

The other logging is optional from the benchmark.

Now it is doing that

[zhenghh04]

@zhenghh04 Can we not print the file name and line number for info logging. We should only do that for debug logging. This will significantly reduce log size.

This is done!

[zhenghh04]

Now if u want to use logging for printing, I would create two loggers. One for printing and one for logging internal stuff.

The printing could always be info, internal, and cannot change by benchmark parameters.

Done!

[zhenghh04]

@hariharan-devarajan This PR is ready for review again.

[hariharan-devarajan]

The documentation still refers to debug in workflow.

The rest looks good.

https://github.com/search?q=repo%3Aargonne-lcf%2Fdlio_benchmark+debug+path%3A*.rst&type=code

[zhenghh04]

The documentation still refers to debug in workflow.

The rest looks good.

https://github.com/search?q=repo%3Aargonne-lcf%2Fdlio_benchmark+debug+path%3A*.rst&type=code

Just fixed this.

[hariharan-devarajan]

Looks good.