Move vulkan codegen generators 2 #1874

MarkY-LunarG · 2024-11-12T13:49:55Z

This builds on #1851. For this review, start at commit 216cd68

ci-tester-lunarg · 2024-11-12T13:49:58Z

CI gfxreconstruct build queued with queue ID 300829.

ci-tester-lunarg · 2024-11-12T13:55:01Z

CI gfxreconstruct build # 5337 running.

ci-tester-lunarg · 2024-11-12T14:29:16Z

CI gfxreconstruct build # 5337 passed.

ci-tester-lunarg · 2024-11-12T16:18:30Z

CI gfxreconstruct build queued with queue ID 300955.

ci-tester-lunarg · 2024-11-12T16:20:06Z

CI gfxreconstruct build # 5338 running.

ci-tester-lunarg · 2024-11-12T16:58:28Z

CI gfxreconstruct build # 5338 passed.

jzulauf-lunarg

Review is WIP: book mark review progress at: codegen: Start using ApiData content b7f0c70

framework/generated/khronos_generators/khronos_base_decoder_body_generator.py

framework/generated/khronos_generators/khronos_base_struct_decoders_body_generator.py

framework/generated/khronos_generators/vulkan_generators/base_generator.py

framework/generated/khronos_generators/khronos_base_generator.py

framework/generated/khronos_generators/vulkan_generators/base_generator.py

framework/generated/khronos_generators/vulkan_generators/decode_pnext_struct_generator.py

framework/generated/khronos_generators/vulkan_generators/encode_pnext_struct_generator.py

framework/generated/khronos_generators/vulkan_generators/vulkan_stype_util_generator.py

framework/generated/khronos_generators/khronos_base_generator.py

jzulauf-lunarg

Review WIP: Bookmark codegen: Rename extraVulkanHeaders to extra_headers
278e534

framework/generated/khronos_generators/khronos_base_generator.py

framework/generated/khronos_generators/vulkan_generators/decode_pnext_struct_generator.py

framework/generated/generated_encode_pnext_struct.cpp

framework/generated/khronos_generators/vulkan_generators/encode_pnext_struct_generator.py

jzulauf-lunarg

Review WIP: Bookmark codegen: Update struct->json body af283a2

framework/generated/khronos_generators/vulkan_generators/decode_pnext_struct_generator.py

jzulauf-lunarg · 2024-11-19T23:13:21Z

framework/generated/khronos_generators/khronos_struct_type_util_generator.py

+                )
+
+        self.newline()
+        write('GFXRECON_END_NAMESPACE(util)', file=self.outFile)


Shouldn't this part be part of an end file function?

We can fix in the future. Nothing is broken by leaving this in the khronos files for now.

jzulauf-lunarg · 2024-11-19T23:19:34Z

framework/generated/khronos_generators/khronos_base_generator.py

@@ -347,6 +347,8 @@ def __init__(
        self.handle_names = set()  # Set of current API's handle typenames
        self.dispatchable_handle_names = set()  # Set of current API's dispatchable handle typenames
        self.flags_types = dict()  # Map of flags types
+        self.flags_type_aliases = dict()  # Map of flags type aliases


Comment says "move" but this doesn't seem to have been deleted from anywhere.

jzulauf-lunarg · 2024-11-19T23:21:51Z

...ework/generated/khronos_generators/vulkan_generators/vulkan_struct_to_json_body_generator.py

+                    ''')
+                body = remove_trailing_newlines(indent_cpp_code(body))
+                write(body, file=self.outFile)
+


move up yapf enable

jzulauf-lunarg · 2024-11-19T23:51:38Z

...ework/generated/khronos_generators/vulkan_generators/vulkan_struct_to_json_body_generator.py

    def make_pnext_body(self):
        body = ''
-        for struct in self.pnext_extension_structs:
+        for struct in self.all_extended_structs:


Not clear why these two are equivalent

jzulauf-lunarg · 2024-11-19T23:52:34Z

...ork/generated/khronos_generators/vulkan_generators/vulkan_struct_to_json_header_generator.py

-            err_file=err_file,
-            warn_file=warn_file,
-            diag_file=diag_file
+            self, err_file=err_file, warn_file=warn_file, diag_file=diag_file


why the reformatting?

jzulauf-lunarg · 2024-11-20T00:20:06Z

...ork/generated/khronos_generators/vulkan_generators/vulkan_struct_to_json_header_generator.py

-            GFXRECON_BEGIN_NAMESPACE(decode)
-            ''')
-        body += "\n"
+            GFXRECON_BEGIN_NAMESPACE(decode)''')


Does this not change whitespace in the output? Seems like it should, but there's no h/cpp diff in the commit. Hmmm...

jzulauf-lunarg · 2024-11-20T00:21:54Z

framework/generated/khronos_generators/khronos_base_generator.py

@@ -107,7 +107,8 @@ class ApiData():
        base_in_struct              - The base input structure defined in this Khronos API
        base_out_struct             - The base output structure defined in this Khronos API
        extended_struct_variable    - The extended struct varible name used in this Khronos API
-        extended_struct_func_prefix - The function prefix to use for extended struct functions for this Khronos API.=
+        extended_struct_func_prefix - The function prefix to use for extended struct functions for this Khronos API.


trailing "." not consistent w/ comments above.

jzulauf-lunarg · 2024-11-20T00:27:59Z

...ework/generated/khronos_generators/vulkan_generators/vulkan_struct_to_json_body_generator.py

@@ -25,6 +25,7 @@
 from base_generator import *
 from reformat_code import format_cpp_code, indent_cpp_code, remove_leading_empty_lines, remove_trailing_newlines

+


stray newline

jzulauf-lunarg

Not clear if this is generate_vulkan.py bisectable, at least one stray commit would indicate it isn't. A couple places where the two parts a "move" are in different commits. For changes where the contents of a generate_feature are being moved to endFile time, way cleaner to rename the functions/variables accessed in place. Better for review and maintenance.

jzulauf-lunarg · 2024-11-20T16:41:01Z

...ework/generated/khronos_generators/vulkan_generators/vulkan_struct_to_json_body_generator.py

@@ -160,25 +159,20 @@ def writeBodyContents(self):
                body = remove_trailing_newlines(indent_cpp_code(body))
                write(body, file=self.outFile)

+        stype_var = self.getStructTypeVarName()


Another way to do this would be to get the ApiData as a dictionary, pass that to format, using named {} fields

jzulauf-lunarg · 2024-11-20T18:11:09Z

framework/generated/generated_vulkan_struct_to_json.cpp

@@ -14013,7 +14013,7 @@ void FieldToJson(nlohmann::ordered_json& jdata, const Decoded_VkPipelineColorWri

        FieldToJson(jdata["sType"], decoded_value.sType, options);
        FieldToJson(jdata["attachmentCount"], decoded_value.attachmentCount, options);
-        FieldToJson(jdata["pColorWriteEnables"], meta_struct.pColorWriteEnables, options);
+        Bool32ToJson(jdata["pColorWriteEnables"], &meta_struct.pColorWriteEnables, options);


This is actually an array, I wonder if this is correct in the original or this update... Hmmmm.

jzulauf-lunarg · 2024-11-20T18:19:10Z

framework/generated/khronos_generators/khronos_struct_to_json_body_generator.py

+            # Determine the appropriate type.  If it's an alias, get
+            # the original type
+            value_type = value.base_type
+            if (self.is_struct(value.base_type) and value.base_type in self.all_struct_aliases):


Is alias lookup a common operation?

jzulauf-lunarg · 2024-11-20T18:59:45Z

.../generated/khronos_generators/vulkan_generators/vulkan_api_call_encoders_header_generator.py

@@ -82,6 +82,16 @@ def beginFile(self, gen_opts):

    def endFile(self):
        """Method override."""
+        for cmd in self.get_all_filtered_cmd_names():
+            info = self.all_cmd_params[cmd]
+            proto = info[1]


Not for this PR: What an awful way to store and retrieve this information, a tuple or array, with different types of contents at fixed indices...

jzulauf-lunarg · 2024-11-20T19:11:37Z

framework/generated/khronos_generators/vulkan_generators/layer_func_table_generator.py

@@ -126,17 +138,3 @@ def need_feature_generation(self):
        if self.feature_cmd_params:
            return True
        return False
-
-    def generate_feature(self):


Easier to review if you simply rename the function/variables in situ, and then call from endFile

jzulauf-lunarg · 2024-11-20T19:12:41Z

...nerated/khronos_generators/vulkan_generators/vulkan_struct_handle_wrappers_body_generator.py

@@ -233,37 +262,6 @@ def need_feature_generation(self):
            return True
        return False

-    def generate_feature(self):


Same comment. Rename the function, touch up the variable names in place, and then call from endfile. Easier to review, more maintainable.

Move the Vulkan generator scripts under a khronos_generators folder to prepare it for for sharing with other khronos APIs.

DX12 was still relying on one file in the base_generators folder. Make a copy in the DX12 folder, and remove all references to the Vulkan and base_generators folders in the generate_dx12.py script.

Move the reformat_code.py into the base of khronos_generators.

Move common items from the vulkan_generators/base_generator.py to the khronos_generators/khronos_base_generator.py

Move more common type parsing into the khronos_base_generator from the vulkan/base_generator.

base_generator code still has vulkan-specific items in it. This starts the process of pulling those out.

Remove more vulkan-specific items in the base_generator code path.

Modify the handling of 'pNext' extended struct variables in the base_generators to be more Khronos-generic instead of Vulkan-specific.

First phase moving of base_generator files to have a khronos identifier

Use the ApiData for the structure type variable names and enum names.

This is a first step to moving the content into a separate file.

Move the actual methods for generating the decode of the pnext items into a khronos generic file.

Move the encoding of the pnext items into separate methods.

Move the pNext encode functionality into the Khronos generator since it is now API agnostic.

Update the writeCommonVulkanHeaders method so that it is in the base Khronos api directory with a generic name.

Move the structure type util generation into the Khronos base.

Move the generation of the struct to json code for the header to endFile time instead of generate_feature time.

Update the struct to json code so that more of the common functionality sits in the khronos generators folder.

Change the struct to json body code so that the bulk of the functionality is in separate methods.

Update the struct to json generation to use the extended structure information and the structure type information from the khronos base.

Move the common struct to json body code into a Khronos file now.

Update the type used in the struct body generation to take into account an alias.

Add the ability to generate the correct information when we encounter a parent/child structure in the JSON output.

Move the generation of content to endFile time for the vulkan_command_buffer_util generation.

Move code generation to endFile time for the Vulkan API call encoder functionality.

Update more Vulkan python scripts to generate at endFile time instead of generate_feature time.

Generate the Vulkan struct handle wrapping content at endFile time instead of generate_feature time.

ci-tester-lunarg · 2024-11-21T20:09:31Z

CI gfxreconstruct build queued with queue ID 307849.

ci-tester-lunarg · 2024-11-21T20:10:00Z

CI gfxreconstruct build # 5368 running.

ci-tester-lunarg · 2024-11-21T20:38:17Z

CI gfxreconstruct build # 5368 passed.

MarkY-LunarG requested a review from jzulauf-lunarg November 12, 2024 13:49

MarkY-LunarG changed the title ~~Marky move vulkan codegen generators 2~~ Move vulkan codegen generators 2 Nov 15, 2024

jzulauf-lunarg reviewed Nov 19, 2024

View reviewed changes

jzulauf-lunarg reviewed Nov 20, 2024

View reviewed changes

jzulauf-lunarg requested changes Nov 20, 2024

View reviewed changes

MarkY-LunarG added 18 commits November 21, 2024 12:37

codegen vulkan: Move vulkan_generator scripts

5e238b4

Move the Vulkan generator scripts under a khronos_generators folder to prepare it for for sharing with other khronos APIs.

codegen dx12: Copy reformat_code.py

d0d2fad

DX12 was still relying on one file in the base_generators folder. Make a copy in the DX12 folder, and remove all references to the Vulkan and base_generators folders in the generate_dx12.py script.

codegen: Move base_generators under khronos_generators

7a11e55

codegen: Move common file into khronos_generator

60a2ed4

Move the reformat_code.py into the base of khronos_generators.

codegen: Move common items from vulkan to khronos

d7608d1

Move common items from the vulkan_generators/base_generator.py to the khronos_generators/khronos_base_generator.py

codegen dx12: Remove duplicate item

931ca07

codegen: Move more from vulkan to khronos

f425d15

Move more common type parsing into the khronos_base_generator from the vulkan/base_generator.

codgen: Automatically determine dispatchable handles

3c0c4f0

codegen: Modify base_generators to remove Vulkan

7c635b2

base_generator code still has vulkan-specific items in it. This starts the process of pulling those out.

codegen: Remove more vulkan in base_generators

f983c7d

Remove more vulkan-specific items in the base_generator code path.

codegen: Modidfy extended struct handling

7cfe191

Modify the handling of 'pNext' extended struct variables in the base_generators to be more Khronos-generic instead of Vulkan-specific.

codegen: Move base_generator scripts up to khronos

ab912be

codegen: Start renaming base files

7e5c081

First phase moving of base_generator files to have a khronos identifier

codegen: Rename base_replay_consumer_body

e16046a

codegen: Rename BaseDecoderBody to match filename

b26951c

codegen: Updated BaseStructDecoderBody

04d34e8

codegen: Rename BaseStructDecodersHeader

317354b

codgen: Rename BaseStructHandleMappersBody

753c2af

MarkY-LunarG added 26 commits November 21, 2024 15:02

codegen: pnext encode use ApiData for sType

7cc1a11

Use the ApiData for the structure type variable names and enum names.

codegen: Rename extraVulkanHeaders to extra_headers

04a3051

codegen: Move pnext decode to separate methods

9514b68

This is a first step to moving the content into a separate file.

codegen: Move decode pnext into khronos

75f12aa

Move the actual methods for generating the decode of the pnext items into a khronos generic file.

codegen: Move encode pnext gen into sep methods

272dfca

Move the encoding of the pnext items into separate methods.

codegen: Move pnext encode into Khronos

e53094a

Move the pNext encode functionality into the Khronos generator since it is now API agnostic.

codegen: Update writeVulkanHeaders method name

6d2836d

Update the writeCommonVulkanHeaders method so that it is in the base Khronos api directory with a generic name.

codegen: Make stype functionality generic

c46976e

codegen: Move struct type gen into Khronos

b8b4fb7

Move the structure type util generation into the Khronos base.

codegen: Move flag detection logic to Khronos

5f88548

codegen: struct->json header move to endFile

f6c3234

Move the generation of the struct to json code for the header to endFile time instead of generate_feature time.

codegen: struct->json body gen at endFile time

db6a323

codegen: struct->json body use khronos types

da44f90

codegen: Update struct->json header

07dd940

Update the struct to json code so that more of the common functionality sits in the khronos generators folder.

codegen: Add APIData bool types

a5ef5b4

codegen: struct->json migration change

2b0e525

Change the struct to json body code so that the bulk of the functionality is in separate methods.

codegen: Update struct->json body

4eeff22

Update the struct to json generation to use the extended structure information and the structure type information from the khronos base.

codegen: Move common struct->json body to Khronos

7944b3f

Move the common struct to json body code into a Khronos file now.

codegen: Add JSON support for boolean arrays

dd9a1c8

codegen: Update types in khronos struct->json

a23cca8

Update the type used in the struct body generation to take into account an alias.

codegen: struct->json body add parent/child func

135ed50

Add the ability to generate the correct information when we encounter a parent/child structure in the JSON output.

codegen: Add parent->child to struct->json header

9e6ec9c

codegen: Move generation to endFile for cmd buf utils

2fdf5a4

Move the generation of content to endFile time for the vulkan_command_buffer_util generation.

codegen: Move generation to endFile for Api encoder

7d937df

Move code generation to endFile time for the Vulkan API call encoder functionality.

codegen: Update more Vulkan to generate at endFile

49c747f

Update more Vulkan python scripts to generate at endFile time instead of generate_feature time.

codegen: Generate vulkan handle wrapping to endFile

b999a29

Generate the Vulkan struct handle wrapping content at endFile time instead of generate_feature time.

MarkY-LunarG force-pushed the marky-move-vulkan-codegen-generators-2 branch from 375b952 to b999a29 Compare November 21, 2024 20:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move vulkan codegen generators 2 #1874

Move vulkan codegen generators 2 #1874

MarkY-LunarG commented Nov 12, 2024

ci-tester-lunarg commented Nov 12, 2024

ci-tester-lunarg commented Nov 12, 2024

ci-tester-lunarg commented Nov 12, 2024

ci-tester-lunarg commented Nov 12, 2024

ci-tester-lunarg commented Nov 12, 2024

ci-tester-lunarg commented Nov 12, 2024

jzulauf-lunarg left a comment •

edited

Loading

jzulauf-lunarg left a comment

jzulauf-lunarg left a comment

jzulauf-lunarg Nov 19, 2024

MarkY-LunarG Nov 21, 2024

jzulauf-lunarg Nov 19, 2024

jzulauf-lunarg Nov 19, 2024

jzulauf-lunarg Nov 19, 2024

jzulauf-lunarg Nov 19, 2024

jzulauf-lunarg Nov 20, 2024

jzulauf-lunarg Nov 20, 2024

jzulauf-lunarg Nov 20, 2024

jzulauf-lunarg left a comment

jzulauf-lunarg Nov 20, 2024

jzulauf-lunarg Nov 20, 2024

jzulauf-lunarg Nov 20, 2024

jzulauf-lunarg Nov 20, 2024

jzulauf-lunarg Nov 20, 2024

jzulauf-lunarg Nov 20, 2024

ci-tester-lunarg commented Nov 21, 2024

ci-tester-lunarg commented Nov 21, 2024

ci-tester-lunarg commented Nov 21, 2024

		@@ -25,6 +25,7 @@
		from base_generator import *
		from reformat_code import format_cpp_code, indent_cpp_code, remove_leading_empty_lines, remove_trailing_newlines

Move vulkan codegen generators 2 #1874

Are you sure you want to change the base?

Move vulkan codegen generators 2 #1874

Conversation

MarkY-LunarG commented Nov 12, 2024

ci-tester-lunarg commented Nov 12, 2024

ci-tester-lunarg commented Nov 12, 2024

ci-tester-lunarg commented Nov 12, 2024

ci-tester-lunarg commented Nov 12, 2024

ci-tester-lunarg commented Nov 12, 2024

ci-tester-lunarg commented Nov 12, 2024

jzulauf-lunarg left a comment • edited Loading

Choose a reason for hiding this comment

jzulauf-lunarg left a comment

Choose a reason for hiding this comment

jzulauf-lunarg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jzulauf-lunarg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ci-tester-lunarg commented Nov 21, 2024

ci-tester-lunarg commented Nov 21, 2024

ci-tester-lunarg commented Nov 21, 2024

jzulauf-lunarg left a comment •

edited

Loading