Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide Stream output for transformers #1176

Open
snuyanzin opened this issue Apr 26, 2024 · 5 comments
Open

Provide Stream output for transformers #1176

snuyanzin opened this issue Apr 26, 2024 · 5 comments

Comments

@snuyanzin
Copy link
Collaborator

The problem with current method like net.datafaker.transformations.JsonTransformer#generate(net.datafaker.transformations.Schema<IN,?>, int)
it generates the whole String and then returns it. As a result for bigger numbers it consumes larger amount of memory and e.g.
such test fails with OutOfMemory

@Test
    void test2() {
        BaseFaker faker = new BaseFaker(new Random(10L));
        Schema<Object, ?> schema = Schema.of(
            field("Text", () -> faker.name().firstName()),
            field("Bool", () -> faker.name().lastName())
        );

        JsonTransformer<Object> transformer = JsonTransformer.builder().build();
        String json = transformer.generate(schema, 50_000_000);
        System.out.println(json);
    }

There is not so much we can do about this method since anyway with such approach we need somehow to store that giant string value.

Another approach is instead of generation the final string value we could generate a stream of values and return it.

@snuyanzin
Copy link
Collaborator Author

partially covered with #1177
however not all formats are supported yet

@RVRhub
Copy link
Contributor

RVRhub commented Jun 1, 2024

Do I understand correctly that this proposal suggests implementing a solution similar to the one used for JsonTransformer, but also for:

  • XmlTransformer
  • JavaObjectTransformer
  • SqlTransformer

@snuyanzin
Copy link
Collaborator Author

yes you are right

@gatear
Copy link
Contributor

gatear commented Jun 20, 2024

My PR covers the SQL transformer support #1264

@gatear
Copy link
Contributor

gatear commented Jul 25, 2024

Also a PR for JavaObjectTransformer #1313

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants