support golang

[fgeller]

hey! great presentation at emacsconf - thanks for pouring time into this project :)
i'd love to try it for my daily work, but i use mainly golang. i'd like to add support but it doesn't seem as straight-forward. would you like me to collect questions here or do you prefer another channel?

[ethan-leba]

Glad you liked it! Yep, in GH issues works perfectly fine. Not sure how much you've surmised so far, but here's the basic rundown for adding a new lang:

1. Clone tree-sitter-langs somewhere
2. Run cask emacs --script dev/tree-edit-generate-grammars.el <tree-sitter-langs>/repos/<language>/src/grammar.json <language> <language-mode> in tree-edit root
3. Copy tree-edit-java.el to tree-edit-<language>.el and replace code as needed (a template should probably be autogenerated)
4. Add language to tree-edit-language-alist
5. See what breaks!

As you can probably tell this project is still pretty rocky so something may explode along the way, but this should be a fun trial run :) Let me know if anything comes up!

[fgeller]

alright, let's start the process :)

i got till step 2, then the script fails when parsing the grammar -- might be a quick one for you, else i'll start digging:

tree-edit % cask emacs --script dev/tree-edit-generate-grammars.el /home/fgeller/src/github.com/emacs-tree-sitter/tree-sitter-langs/repos/go/src/grammar.json go go-mode                                                                                                                                                                                                                                                                                                                                                                                          
Parsing go grammar at /home/fgeller/src/github.com/emacs-tree-sitter/tree-sitter-langs/repos/go/src/grammar.json.                                                                                                                                                                                                                                                                                                                                                                                                                                                 
failed to parse source_file                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
failed to parse _top_level_declaration                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
failed to parse package_clause                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
failed to parse import_declaration                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
failed to parse import_spec                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
failed to parse dot                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
failed to parse blank_identifier                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
failed to parse import_spec_list                                                                                                                                                                                                                                                                                                                                                              
failed to parse _declaration                                                                                                                                                                                                                                                                                                                                                                  
failed to parse const_declaration                                                                                                                                                                                                                                                                                                                                                             
failed to parse const_spec                                                                                                                                                                                                                                                                                                                                                                    
failed to parse var_declaration                                                                                                                                                                                                                                                                                                                                                               
failed to parse var_spec                                                                                                                                                                                                                                                                                                                                                                      
failed to parse function_declaration                                                                                                                                                                                                                                                                                                                                                          
failed to parse method_declaration                                                                                                                                                                                                                                                                                                                                                            
failed to parse parameter_list                                                                                                                                                                                                                                                                                                                                                                
failed to parse parameter_declaration                                                                                                                                                                                                                                                                                                                                                         
failed to parse variadic_parameter_declaration                                                                                                                                                                                                                                                                                                                                                
failed to parse type_alias                                                                                                                                                                                                                                                                                                                                                                    
failed to parse type_declaration                                                                                                                                                                                                                                                                                                                                                              
failed to parse type_spec                                                                                                                                                                                                                                                                                                                                                                     
failed to parse field_name_list                                                                                                                                                                                                                                                                                                                                                               
failed to parse expression_list                                                                                                                                                                                                                                                                                                                                                               
failed to parse _type                                                                                                                                                                                                                                                                                                                                                                         
failed to parse parenthesized_type                                                                                                                                                                                                                                                                                                                                                            
failed to parse _simple_type                                                                                                                                                                                                                                                                                                                                                                  
failed to parse pointer_type                                                                                                                                                                                                                                                                                                                                                                  
failed to parse array_type                                                                                                                                                                                                                                                                                                                                                                    
failed to parse implicit_length_array_type                                                                                                                                                                                                                                                                                                                                                    
failed to parse slice_type                                                                                                                                                                                                                                                                                                                                                                    
failed to parse struct_type                                                                                                                                                                                                                                                                                                                                                                   
failed to parse field_declaration_list                                                                                                                                                                                                                                                                                                                                                        
failed to parse field_declaration                                                                                                                                                                                                                                                                                                                                                             
failed to parse interface_type                                                                                                                                                                                                                                                                                                                                                                
failed to parse method_spec_list                                                                                                                                                                                                                                                                                                                                                              
failed to parse method_spec                                                                                                                                                                                                                                                                                                                                                                   
failed to parse map_type                                                                                                                                                                                                                                                                                                                                                                      
failed to parse channel_type                                                                                                                                                                                                                                                                                                                                                                  
failed to parse function_type                                                                                                                                                                                                                                                                                                                                                                 
failed to parse block                                                                                                                                                                                                                                                                                                                                                                         
failed to parse _statement_list                                                                                                                                                                                                                                                                                                                                                               
failed to parse _statement                                                                                                                                                                                                                                                                                                                                                                    
failed to parse empty_statement                                                                                                                                                                                                                                                                                                                                                               
failed to parse _simple_statement                                                                                                                                                                                                                                                                                                                                                             
failed to parse send_statement                                                                                                                                                                                                                                                                                                                                                                
failed to parse receive_statement                                                                                                                                                                                                                                                                                                                                                             
failed to parse inc_statement                                                                                                                                                                                                                                                                                                                                                                 
failed to parse dec_statement                                                                                                                                                                                                                                                                                                                                                                 
failed to parse assignment_statement                                                                                                                                                                                                                                                                                                                                                          
failed to parse short_var_declaration                                                                                                                                                                                                                                                                                                                                                         
failed to parse labeled_statement                                                                                                                                                                                                                                                                                                                                                             
failed to parse empty_labeled_statement                                                                                                                                                                                                                                                                                                                                                       
failed to parse fallthrough_statement                                                                                                                                                                                                                                                                                                                                                         
failed to parse break_statement                                                                                                                                                                                                                                                                                                                                                               
failed to parse continue_statement                                                                                                                                                                                                                                                                                                                                                            
failed to parse goto_statement                                                                                                                                                                                                                                                                                                                                                                
failed to parse return_statement                                                                                                                                                                                                                                                                                                                                                              
failed to parse go_statement                                                                                                                                                                                                                                                                                                                                                                  
failed to parse defer_statement                                                                                                                                                                                                                                                                                                                                                               
failed to parse if_statement                                                                                                                                                                                                                                                                                                                                                                  
failed to parse for_statement                                                                                                                                                                                                                                                                                                                                                                 
failed to parse for_clause                                                                                                                                                                                                                                                                                                                                                                    
failed to parse range_clause                                                                                                                                                                                                                                                                                                                                                                  
failed to parse expression_switch_statement                                                                                                                                                                                                                                                                                                                                                   
failed to parse expression_case                                                                                                                                                                                                                                                                                                                                                               
failed to parse default_case                                                                                                                                                                                                                                                                                                                                                                  
failed to parse type_switch_statement                                                                                                                                                                                                                                                                                                                                                         
failed to parse _type_switch_header                                                                                                                                                                                                                                                                                                                                                           
failed to parse type_case                                                                                                                                                                                                                                                                                                                                                                     
failed to parse select_statement                                                                                                                                                                                                                                                                                                                                                              
failed to parse communication_case                                                                                                                                                                                                                                                                                                                                                            
failed to parse _expression                                                                                                                                                                                                                                                                                                                                                                   
failed to parse parenthesized_expression                                                                                                                                                                                                                                                                                                                                                      
failed to parse call_expression                                                                                                                                                                                                                                                                                                                                                               
failed to parse variadic_argument                                                                                                                                                                                                                                                                                                                                                             
failed to parse special_argument_list                               
failed to parse argument_list                                       
failed to parse selector_expression                                 
failed to parse index_expression                                    
failed to parse slice_expression                                    
failed to parse type_assertion_expression                           
failed to parse type_conversion_expression                                                                                                                                                                                                     
failed to parse composite_literal                                                                                                                                                                                                                                                                                                                                                                                                                                                             
failed to parse literal_value                                       
failed to parse keyed_element                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
failed to parse element                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
failed to parse func_literal                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
failed to parse unary_expression                                                                                                                                                                                                                                                                                                                                                                                                                                                              
failed to parse binary_expression                                                                                                                                                                                                                                                                                                                                                                                                                                                             
failed to parse qualified_type                                                                                                                                                                                                                                                                                                                                                                                                                                                                
failed to parse identifier                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
failed to parse _type_identifier                                                                                                                                                                                                                                                                                                                                                                                                                                                              
failed to parse _field_identifier                                                                                                                                                                                                                                                                                                                                                                                                                                                             
failed to parse _package_identifier                                                                                                                                                                                                                                                                                                                                                                                                                                                           
failed to parse _string_literal                                                                                                                                                                                                                                                                                                                                                                                                                                                               
failed to parse raw_string_literal                                                                                                                                                                                                                                                                                                                                                                                                                                                            
failed to parse interpreted_string_literal                                                                                                                                                                                                                                                                                                                                                                                                                                                    
failed to parse escape_sequence                                                                                                                                                                                                                                                                                                                                                                                                                                                               
failed to parse int_literal                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
failed to parse float_literal                                       
failed to parse imaginary_literal                                   
failed to parse rune_literal                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
failed to parse nil                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
failed to parse true                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
failed to parse false                                                                                                                                                                                                                          
failed to parse comment                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
Debugger entered--Lisp error: (error "Expected regex node, found ((type . TOKEN) (conten...")                                                                                                                                                                                                                                                                                                                                                                                                 
  error("Expected regex node, found %s" ((type . "TOKEN") (content (type . "SEQ") (members ((type . "CHOICE") (members ((type . "PATTERN") (value . "\\p{L}")) ((type . "STRING") (value . "_")))) ((type . "REPEAT") (content (type . "CHOICE") (members (... ...) (... ...))))))))                                                                                                                                                                                                          
  (let nil (error "Expected regex node, found %s" identifier-node)) 
  (if (equal x314 '"PATTERN") (let* ((x315 (cdr-safe identifier-node))) (if (consp x315) (let* ((x316 (car-safe x315))) (if (consp x316) (let* ((x317 ...)) (if (eq x317 ...) (let* ... ...) (let nil ...))) (let nil (error "Expected regex node, found %s" identifier-node)))) (let nil (error "Expected regex node, found %s" identifier-node)))) (let nil (error "Expected regex node, found %s" identifier-node)))
  (let* ((x314 (cdr-safe x312))) (if (equal x314 '"PATTERN") (let* ((x315 (cdr-safe identifier-node))) (if (consp x315) (let* ((x316 (car-safe x315))) (if (consp x316) (let* (...) (if ... ... ...)) (let nil (error "Expected regex node, found %s" identifier-node)))) (let nil (error "Expected regex node, found %s" identifier-node)))) (let nil (error "Expected regex node, found %s" identifier-node))))
  (if (eq x313 'type) (let* ((x314 (cdr-safe x312))) (if (equal x314 '"PATTERN") (let* ((x315 (cdr-safe identifier-node))) (if (consp x315) (let* ((x316 ...)) (if (consp x316) (let* ... ...) (let nil ...))) (let nil (error "Expected regex node, found %s" identifier-node)))) (let nil (error "Expected regex node, found %s" identifier-node)))) (let nil (error "Expected regex node, found %s" identifier-node)))
  (let* ((x313 (car-safe x312))) (if (eq x313 'type) (let* ((x314 (cdr-safe x312))) (if (equal x314 '"PATTERN") (let* ((x315 (cdr-safe identifier-node))) (if (consp x315) (let* (...) (if ... ... ...)) (let nil (error "Expected regex node, found %s" identifier-node)))) (let nil (error "Expected regex node, found %s" identifier-node)))) (let nil (error "Expected regex node, found %s" identifier-node))))
  (if (consp x312) (let* ((x313 (car-safe x312))) (if (eq x313 'type) (let* ((x314 (cdr-safe x312))) (if (equal x314 '"PATTERN") (let* ((x315 ...)) (if (consp x315) (let* ... ...) (let nil ...))) (let nil (error "Expected regex node, found %s" identifier-node)))) (let nil (error "Expected regex node, found %s" identifier-node)))) (let nil (error "Expected regex node, found %s" identifier-node)))
  (let* ((x312 (car-safe identifier-node))) (if (consp x312) (let* ((x313 (car-safe x312))) (if (eq x313 'type) (let* ((x314 (cdr-safe x312))) (if (equal x314 '"PATTERN") (let* (...) (if ... ... ...)) (let nil (error "Expected regex node, found %s" identifier-node)))) (let nil (error "Expected regex node, found %s" identifier-node)))) (let nil (error "Expected regex node, found %s" identifier-node))))
  (if (consp identifier-node) (let* ((x312 (car-safe identifier-node))) (if (consp x312) (let* ((x313 (car-safe x312))) (if (eq x313 'type) (let* ((x314 ...)) (if (equal x314 ...) (let* ... ...) (let nil ...))) (let nil (error "Expected regex node, found %s" identifier-node)))) (let nil (error "Expected regex node, found %s" identifier-node)))) (let nil (error "Expected regex node, found %s" identifier-node)))
  (let ((identifier-node (alist-get (intern (alist-get 'word grammar)) (alist-get 'rules grammar)))) (if (consp identifier-node) (let* ((x312 (car-safe identifier-node))) (if (consp x312) (let* ((x313 (car-safe x312))) (if (eq x313 'type) (let* (...) (if ... ... ...)) (let nil (error "Expected regex node, found %s" identifier-node)))) (let nil (error "Expected regex node, found %s" identifier-node)))) (let nil (error "Expected regex node, found %s" identifier-node))))
  tree-edit--extract-word-regex(((name . "go") (word . "identifier") (rules (source_file (type . "REPEAT") (content (type . "CHOICE") (members ... ...))) (_top_level_declaration (type . "CHOICE") (members (... ...) (... ...) (... ...) (... ...))) (package_clause (type . "SEQ") (members (... ...) (... ...))) (import_declaration (type . "SEQ") (members (... ...) (... ...))) (import_spec (type . "SEQ") (members (... ...) (... ... ...))) (dot (type . "STRING") (value . ".")) (blank_identifier (type . "STRING") (value . "_")) (import_spec_list (type . "SEQ") (members (... ...) (... ...) (... ...))) (_declaration (type . "CHOICE") (members (... ...) (... ...) (... ...))) (const_declaration (type . "SEQ") (members (... ...) (... ...))) (const_spec (type . "PREC_LEFT") (value . 0) (content (type . "SEQ") (members ... ...))) (var_declaration (type . "SEQ") (members (... ...) (... ...))) (var_spec (type . "SEQ") (members (... ... ...) (... ...))) (function_declaration (type . "PREC_RIGHT") (value . 1) (content (type . "SEQ") (members ... ... ... ... ...))) (method_declaration (type . "PREC_RIGHT") (value . 1) (content (type . "SEQ") (members ... ... ... ... ... ...))) (parameter_list (type . "SEQ") (members (... ...) (... ...) (... ...))) (parameter_declaration (type . "SEQ") (members (... ... ...) (... ... ...))) (variadic_parameter_declaration (type . "SEQ") (members (... ... ...) (... ...) (... ... ...))) (type_alias (type . "SEQ") (members (... ... ...) (... ...) (... ... ...))) (type_declaration (type . "SEQ") (members (... ...) (... ...))) (type_spec (type . "SEQ") (members (... ... ...) (... ... ...))) (field_name_list (type . "SEQ") (members (... ...) (... ...))) (expression_list (type . "SEQ") (members (... ...) (... ...))) (_type (type . "CHOICE") (members (... ...) (... ...))) (parenthesized_type (type . "SEQ") (members (... ...) (... ...) (... ...))) (_simple_type (type . "CHOICE") (members (... ... ...) (... ...) (... ...) (... ...) (... ...) (... ...) (... ...) (... ...) (... ...) (... ...))) (pointer_type (type . "PREC") (value . 6) (content (type . "SEQ") (members ... ...))) (array_type (type . "SEQ") (members (... ...) (... ... ...) (... ...) (... ... ...))) (implicit_length_array_type (type . "SEQ") (members (... ...) (... ...) (... ...) (... ... ...))) (slice_type (type . "SEQ") (members (... ...) (... ...) (... ... ...))) (struct_type (type . "SEQ") (members (... ...) (... ...))) (field_declaration_list (type . "SEQ") (members (... ...) (... ...) (... ...))) (field_declaration (type . "SEQ") (members (... ...) (... ... ...))) (interface_type (type . "SEQ") (members (... ...) (... ...))) (method_spec_list (type . "SEQ") (members (... ...) (... ...) (... ...))) (method_spec (type . "SEQ") (members (... ... ...) (... ... ...) (... ... ...))) (map_type (type . "SEQ") (members (... ...) (... ...) (... ... ...) (... ...) (... ... ...))) ...) (extras ((type . "SYMBOL") (name . "comment")) ((type . "PATTERN") (value . "\\s"))) (conflicts ("_simple_type" "_expression") ("qualified_type" "_expression") ("func_literal" "function_type") ("function_type") ("parameter_declaration" "_simple_type")) (precedences) (externals) (inline "_type" "_type_identifier" "_field_identifier" "_package_identifier" "_top_level_declaration" "_string_literal") (supertypes "_expression" "_type" "_simple_type" "_statement" "_simple_statement")))
  (prin1-to-string (tree-edit--extract-word-regex raw-grammar))
  (format tree-edit--grammar-template name mode (list 'quote (prin1-to-string grammar)) (prin1-to-string (tree-edit--extract-word-regex raw-grammar)) (list 'quote (prin1-to-string supertypes)) (list 'quote (prin1-to-string (tree-edit--invert-supertypes supertypes))) (list 'quote (prin1-to-string (tree-edit--generate-alias-names grammar))) (list 'quote (prin1-to-string (tree-edit--generate-containing-types grammar))))
  (let* ((raw-grammar (let ((json-array-type 'list)) (json-read-file path))) (grammar (map-apply #'(lambda (type node) (cons type (tree-edit--process-grammar node))) (alist-get 'rules raw-grammar))) (grammar (tree-edit--extract-grammar grammar)) (supertypes (tree-edit--generate-supertypes grammar))) (format tree-edit--grammar-template name mode (list 'quote (prin1-to-string grammar)) (prin1-to-string (tree-edit--extract-word-regex raw-grammar)) (list 'quote (prin1-to-string supertypes)) (list 'quote (prin1-to-string (tree-edit--invert-supertypes supertypes))) (list 'quote (prin1-to-string (tree-edit--generate-alias-names grammar))) (list 'quote (prin1-to-string (tree-edit--generate-containing-types grammar)))))
  tree-edit--generate-grammar-file("/home/fgeller/src/github.com/emacs-tree-sitter/tre..." "go" "go-mode")
  (let ((time-start (float-time)) (templated-string (tree-edit--generate-grammar-file path name mode))) (insert templated-string) (message (format "Wrote grammar to %s after %s seconds." file-name (round (- (float-time) time-start)))))
  (save-current-buffer (set-buffer temp-buffer) (let ((time-start (float-time)) (templated-string (tree-edit--generate-grammar-file path name mode))) (insert templated-string) (message (format "Wrote grammar to %s after %s seconds." file-name (round (- (float-time) time-start))))))
  (prog1 (save-current-buffer (set-buffer temp-buffer) (let ((time-start (float-time)) (templated-string (tree-edit--generate-grammar-file path name mode))) (insert templated-string) (message (format "Wrote grammar to %s after %s seconds." file-name (round (- (float-time) time-start)))))) (save-current-buffer (set-buffer temp-buffer) (write-region nil nil temp-file nil 0)))
  (unwind-protect (prog1 (save-current-buffer (set-buffer temp-buffer) (let ((time-start (float-time)) (templated-string (tree-edit--generate-grammar-file path name mode))) (insert templated-string) (message (format "Wrote grammar to %s after %s seconds." file-name (round (- ... time-start)))))) (save-current-buffer (set-buffer temp-buffer) (write-region nil nil temp-file nil 0))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))
  (let ((temp-file (format "tree-edit-%s-grammar.el" name)) (temp-buffer (generate-new-buffer " *temp file*" t))) (unwind-protect (prog1 (save-current-buffer (set-buffer temp-buffer) (let ((time-start (float-time)) (templated-string (tree-edit--generate-grammar-file path name mode))) (insert templated-string) (message (format "Wrote grammar to %s after %s seconds." file-name (round ...))))) (save-current-buffer (set-buffer temp-buffer) (write-region nil nil temp-file nil 0))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))
  (let* ((--dash-source-12-- command-line-args-left) (path (car-safe (prog1 --dash-source-12-- (setq --dash-source-12-- (cdr --dash-source-12--))))) (name (car-safe (prog1 --dash-source-12-- (setq --dash-source-12-- (cdr --dash-source-12--))))) (mode (car --dash-source-12--)) (file-name (format "tree-edit-%s-grammar.el" name))) (message (format "Parsing %s grammar at %s." name path)) (let ((temp-file (format "tree-edit-%s-grammar.el" name)) (temp-buffer (generate-new-buffer " *temp file*" t))) (unwind-protect (prog1 (save-current-buffer (set-buffer temp-buffer) (let ((time-start ...) (templated-string ...)) (insert templated-string) (message (format "Wrote grammar to %s after %s seconds." file-name ...)))) (save-current-buffer (set-buffer temp-buffer) (write-region nil nil temp-file nil 0))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))))
  load-with-code-conversion("/home/fgeller/src/github.com/fgeller/dots/.emacs.d..." "/home/fgeller/src/github.com/fgeller/dots/.emacs.d..." nil t)
  command-line-1(("-scriptload" "dev/tree-edit-generate-grammars.el" "/home/fgeller/src/github.com/emacs-tree-sitter/tre..." "go" "go-mode"))
  command-line()
  normal-top-level()

[ethan-leba]

Tried this out, it looks like you ran into two things here:

1. The code in main doesn't handle the immediate token type, but luckily I have a commit for that hanging around
2. Tree-edit expects identifiers to be expressed as a regex, but the go grammar doesn't use a regex for it.

I pushed fixes for these two things to a branch, hopefully that should sort those issues out! I'll have to merge fixes for these two into main at some point, but working off of that branch should suffice for now.

[fgeller]

thanks, that worked! i was able to generate the grammar and complete the 4 steps above.
next i need to update tree-edit-go.el but i wanted to start by getting basic motion to work without evil.
need a bit more time to figure out how the tree and current node are maintained (would appreciate pointers but mostly wanted to post a reply and say thanks :))

[fgeller]

pushing my changes on top of the go-test branch to my fork

[ethan-leba]

thanks, that worked!

Awesome! :)

i wanted to start by getting basic motion to work without evil.

Yeah, I understand that using evil is a hard sell for those who aren't already fully bought into the ecosystem, and that's something I'd like to address. I've tried to keep the tree-edit library focused on being a library, but there's probably alot of code in evil-tree-edit that would be useful for other packages that want to wrap tree-edit. Maybe some could be integrated? Or I could just point folks to copypasta from evil-tree-edit, need to think on that.

I think a modal solution is the best fit for tree-edit though, so maybe something using hydra to emulate an evil state is a more emacsy solution. Do you have any thoughts on a more emacs idiomatic approach to this? I'm frankly as evil as they come, so I probably don't have the best insight on this.

[fgeller]

i prefer modal editing myself, no need to convince me ;) i just don't use evil-mode but something custom 🤷
that said, i'd probably try to have the motion and edit commands as the main exports of the tree-edit lib . so it'd be easy to use it with a evil/hydra/custom key bindings approach.