Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Network writing on remote storage is too long #3233

Open
alicecaron opened this issue Nov 29, 2024 · 2 comments
Open

Network writing on remote storage is too long #3233

alicecaron opened this issue Nov 29, 2024 · 2 comments

Comments

@alicecaron
Copy link
Contributor

alicecaron commented Nov 29, 2024

Describe the current behavior

Writing a Network into a remote storage folder takes a lot more time than writing it into a local folder (x66times than in a local folder).

Describe the expected behavior

Writing a Network into a remote storage folder should takes more or less the same time than writing it into a local folder.

Describe the steps

In a Java environment (jdk17) in developpement mode, with an access to both a local and remote storage accross network.

  • Create a java class with dependency on powsybl-core (current version 6.6.0-SNAPSHOT, works also in 6.5.1).
import com.powsybl.iidm.network.*;
import java.nio.file.Paths;
import java.util.Properties;

public final class WriteNetworkSandbox {

    public static void main(String[] args) {
        String filepath = "<path to network file>.xiidm";
        Network network = Network.read(new File(filepath).getPath());

        String remoteDestPath = "<path to remote folder>/output.xiidm";
        String localDestPath = "<path to local folder>/output.xiidm";

        writeNetwork(network, localDestPath);
        writeNetwork(network, remoteDestPath);
    }

    private static void writeNetwork(Network network, String destinationFile) {
        Properties properties = new Properties();
        properties.put("iidm.export.xml.version", "1.0");
        String format = "XIIDM";
        long startTime = System.currentTimeMillis();
        network.write(format, properties, Paths.get(destinationFile));
        System.out.println("Network writing duration in " + destinationFile + " : " + (System.currentTimeMillis() - startTime) + "ms");
    }

    private WriteNetworkSandbox() {
    }
}
  • run this example code
  • output should display duration of the 2 runs (local and remote) and remote one should be way bigger in ms than the local one when the bug is not fixed.

Output example:

Network writing duration in /local/path/output.xiidm : 1716ms
Network writing duration in /remote/path/output.xiidm : 67112ms

Environment

It is reproductible in a java with jdk17 development environment, having access to a remote folder.

Relevant Log Output

No response

Extra Information

The bug may be a regression since refactor on the Datasource part. Missing a bufferized output stream somewhere?

@olperr1
Copy link
Member

olperr1 commented Dec 18, 2024

The output stream is already buffered.
Using an unbuffered output stream takes much longer: ~ 3min on a local storage (vs ~4s), and maybe more than an hour — I stopped my test at ~40% in 30min — on a remote storage (vs ~17s).

@olperr1
Copy link
Member

olperr1 commented Dec 18, 2024

Format Local storage Remote storage
XIIDM 3,441ms 18,928ms
JIIDM 2,687ms 17,437ms
BIIDM 1,830ms 1,494ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: TODO
Development

No branches or pull requests

2 participants