Modules referenced in parent directories cause resource issues/errors

[brandonpollack23]

What happened?

When convertering

• https://github.com/terraform-aws-modules/terraform-aws-vpc/examples/complete
• https://github.com/terraform-aws-modules/terraform-aws-s3-bucket/examples/object

the program executes until it is killed by the OS for consuming too much memory.

Upon profiling, increasing memory pressure on the heap can be observed, indicating a memory leak.

trace

• pprof memory profile for vpc
• pprof memory profile for s3 object example

(open with go tool pprof -http :8080 FILE)

Example

NA

Output of pulumi about

NA

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

This is a related bug to #21

[brandonpollack23]

Notes:

If I exit the program and do not generate the pp files into a tempdir before running binding/conversion it converts from terraform just fine. The issue is in our binding (perhaps in fetching/converting deps).

Here is the pcl project code which results in an oom if trying to convert it to anything (even just into pcl again!)

https://drive.google.com/open?id=1-2zY0BuTO3tTYeX7Ij6rbVZmat5b7_VV&usp=drive_fs

Here is the profile:

[Zaid-Ajaj]

As described in pulumi/pulumi#17538, the problem was not really a memory leak but rather an infinite recursion because of the the CLI didn't check if a component was binding the same directory it was in and the converter is unable to convert referenced modules that live outside of its entry point. Consider this structure:

| -- parentFolder (dir)
      | -- main.tf
      | -- examples (dir)
               | -- first_example (dir)
                          |-- main.tf

if you were running the converter from first_example and that example had a module reference that looked like this:

module first_example {
  source "../.."
}

then we fail to convert the top-level module because the the destination directory starts at / (current directory). Honestly I am not sure how we supported this before because that would require copying the top-evel structure into the cwd of the converter (where pulumi convert is running from.

[brandonpollack23]

As Zaid mentioned, #17538 fixes the out of resources issue but now has the following error:

================================================================================
                The Pulumi CLI encountered a code generation error. This is a bug!
                We would appreciate a report: https://github.com/pulumi/pulumi/issues/
                Please provide all of the below text in your report.
                ================================================================================
                Pulumi Version:   3.136.2-dev.0
                error: main.pp:18,1-21,2: /tmp/pulumi-convert196531652/vpc_5.13.0/main.pp:97,5-115: cannot iterate over a value of type number; , and 43 other diagnostic(s); /tmp/pulumi-convert196531652/vpc_5.13.0/main.pp:97,5-115: cannot iterate over a value of type number; , and 43 other diagnostic(s)
                error: main.pp:27,1-33,2: cannot bind component mainSg from the same directory as the parent program; cannot bind component mainSg from the same directory as the parent program
                error: main.pp:39,1-236,2: cannot bind component completeSg from the same directory as the parent program; cannot bind component completeSg from the same directory as the parent program
                error: main.pp:242,1-274,2: cannot bind component ipv4Ipv6Example from the same directory as the parent program; cannot bind component ipv4Ipv6Example from the same directory as the parent program
                error: main.pp:280,1-288,2: cannot bind component fixedNameSg from the same directory as the parent program; cannot bind component fixedNameSg from the same directory as the parent program
                error: main.pp:294,1-302,2: cannot bind component onlyRules from the same directory as the parent program; cannot bind component onlyRules from the same directory as the parent program
                error: main.pp:322,1-338,2: cannot bind component prefixList from the same directory as the parent program; cannot bind component prefixList from the same directory as the parent program
                error: main.pp:355,1-371,2: cannot bind component prefixListSg from the same directory as the parent program; cannot bind component prefixListSg from the same directory as the parent program
                error: could not generate output program

So the next step is to execute the copying Zaid mentioned the parent module dirs.

[brandonpollack23]

Since @Zaid-Ajaj is handling the circular dep issue I'll assign this to you for tracking purposes