Adding and deleting Languages

[ntitze-del]

Thank you for the Plugin. It was exactly what I needed for my requirements.

I have languages which are not in the highlight.php supported languages and want to add them to the plugin. How would I do that?

On a side note is it possible to delete the json files of languages I don't need and update the Plugin so they do not show in the block editor anymore?

[westonruter]

You can't currently remove or add languages from the list, but you can change which languages are used for autodetection. See https://github.com/westonruter/syntax-highlighting-code-block/wiki#limiting-auto-detection-languages

As for adding new languages, @allejo is there a way to register new language definitions?

[allejo]

As for adding new languages, @allejo is there a way to register new language definitions?

You can register new languages by calling Highlighter::registerLanguage(). However, they must be translated to a JSON format for highlight.php to be able to handle them.

is it possible to delete the json files of languages I don't need and update the Plugin so they do not show in the block editor anymore?

You can't currently remove or add languages from the list, but you can change which languages are used for autodetection. See westonruter/syntax-highlighting-code-block/wiki#limiting-auto-detection-languages

Adding on to Weston's comment. At the moment, you would have to build your own copy of the plugin after manually modifying language-names.js (which, would probably be more effort than it's worth).

[ntitze-del]

You can register new languages by calling Highlighter::registerLanguage(). However, they must be translated to a JSON format for highlight.php to be able to handle them.

Where would I call the function? Do I need to use an add_action hook in the functions.php? I am not really familiar with WordPress hooks.

And do I understand it correctly that if I would register the language. It will not show in the block editor but the highlighting works(the not showing up is no problem for me)?

[westonruter]

Do you have the language translated into the necessary JSON format? What language are you wanting to add?

[ntitze-del]

Sorry for my late reply. I want to add Cobol. I have a definition but not yet translated into JSON.

[westonruter]

A patch like this would be required, on the PHP side:

--- a/syntax-highlighting-code-block.php
+++ b/syntax-highlighting-code-block.php
@@ -465,6 +465,13 @@ function render_block( $attributes, $content ) {
 	$auto_detect_languages = apply_filters( 'syntax_highlighting_code_block_auto_detect_languages', [] );
 
 	$transient_key = 'syntax-highlighted-' . md5( wp_json_encode( $attributes ) . implode( '', $auto_detect_languages ) . $matches['content'] . PLUGIN_VERSION );
+
+	/**
+	 * Filters the transient key which is used for storing the syntax-highlighted code block.
+	 *
+	 * @param string $transient_key Transient key.
+	 */
+	$transient_key = apply_filters( 'syntax_highlighting_code_block_transient_key', $transient_key, $attributes );
 	$highlighted   = get_transient( $transient_key );
 
 	if ( ! DEVELOPMENT_MODE && $highlighted && isset( $highlighted['content'] ) ) {
@@ -486,6 +493,13 @@ function render_block( $attributes, $content ) {
 			$highlighter->setAutodetectLanguages( $auto_detect_languages );
 		}
 
+		/**
+		 * Fires after the Highlighter has been instantiated for the Syntax Highlighting Code Block.
+		 *
+		 * @param \Highlight\Highlighter() $highlighter Highlighter.
+		 */
+		do_action( 'syntax_highlighting_code_block_highlighter_init', $highlighter );
+
 		$language = $attributes['language'];
 		$content  = html_entity_decode( $matches['content'], ENT_QUOTES );

You'd have to filter syntax_highlighting_code_block_transient_key to make sure it varies to account for any changes you make to your custom languages. Then you'd need to register the language via the Highlighter instance that is passed to this syntax_highlighting_code_block_highlighter_init action.

This doesn't account for adding Cobol to the list of available languages in the block settings, but if leaving it on auto-detect then that would work.

For adding the language to the dropdown, we'd need to move the language names into PHP instead of JS, and then export them from PHP to JS for use in the editor. This is also needed for #37 to be able to show the name of the language for a given code block.

[allejo]

What are your thoughts on adding a new filter, like syntax_highlighting_code_block_custom_languages and it'd return an array of languages (and their paths) that gets merged with get_language_names()?

syntax-highlighting-code-block/syntax-highlighting-code-block.php

Lines 410 to 417 in 1b572ab

	/**
	* Language names.
	*
	* @return array Mapping slug to name.
	*/
	function get_language_names() {
	return require __DIR__ . '/language-names.php';
	}

[westonruter]

I like that. Should each entry also include a version? If someone registers a custom language, then they should indicate a version which is used as part of the cache key. The entire array of languages, including the versions and paths, would then be serialized and hashed to then be made part of the $transient_key. Then there would be no need for the syntax_highlighting_code_block_highlighter_init action as we'd handle the language registration for them.

[allejo]

I like having a version key 👍 Then users would be able to bump their custom language definitions and clear cached highlights for their languages.

[allejo]

For language additions, would this be good for the structure?

[
    'cobal' => [
        'slug' => 'cobal',
        'name' => 'COBAL',
        'path' => '/path/to/cobal.json',
        'version' => 1,
    ],
]

Considering there are ~185 languages, instead of being able to remove languages, would it make more sense to specify which languages to include?

[
    'php',
    'cobal',
    'javascript',
    'css',
]

Should adding languages and "removing" them be two separate filters or one?

[westonruter]

We've had requests to be able to remove languages, as I recall. So I think we could populate the array with all of the languages bundled with highlight.php and provide the version being the version of that library. Then users can either remove languages they don't want, add new languages they do, or even replace a language definition with something else.

For example:

add_filter( 'syntax_highlighting_code_block_languages', function ( $languages ) {
	// Remove undesired languages.
	unset( $languages['brainfuck'] );

	// Replace an existing language.
	$languages['javascript']['path']    = plugin_dir_path( __FILE__ ) . '/my-javascript.json';
	$languages['javascript']['version'] = 2;

	// Add a new language.
	$languages['cobol'] = [
		'name' => 'COBAL',
		'path' => plugin_dir_path( __FILE__ ) . '/cobol.json',
		'version' => 2,
	];

	return $languages;
} );

Would that work for highlight.php?

[ntitze-del]

A patch like this would be required, on the PHP side:

--- a/syntax-highlighting-code-block.php
+++ b/syntax-highlighting-code-block.php
@@ -465,6 +465,13 @@ function render_block( $attributes, $content ) {
 	$auto_detect_languages = apply_filters( 'syntax_highlighting_code_block_auto_detect_languages', [] );
 
 	$transient_key = 'syntax-highlighted-' . md5( wp_json_encode( $attributes ) . implode( '', $auto_detect_languages ) . $matches['content'] . PLUGIN_VERSION );
+
+	/**
+	 * Filters the transient key which is used for storing the syntax-highlighted code block.
+	 *
+	 * @param string $transient_key Transient key.
+	 */
+	$transient_key = apply_filters( 'syntax_highlighting_code_block_transient_key', $transient_key, $attributes );
 	$highlighted   = get_transient( $transient_key );
 
 	if ( ! DEVELOPMENT_MODE && $highlighted && isset( $highlighted['content'] ) ) {
@@ -486,6 +493,13 @@ function render_block( $attributes, $content ) {
 			$highlighter->setAutodetectLanguages( $auto_detect_languages );
 		}
 
+		/**
+		 * Fires after the Highlighter has been instantiated for the Syntax Highlighting Code Block.
+		 *
+		 * @param \Highlight\Highlighter() $highlighter Highlighter.
+		 */
+		do_action( 'syntax_highlighting_code_block_highlighter_init', $highlighter );
+
 		$language = $attributes['language'];
 		$content  = html_entity_decode( $matches['content'], ENT_QUOTES );

...

I added your solution and before the do_action I registered the language:

$highlighter->registerLanguage("cobol", __DIR__ . '/vendor/scrivo/highlight-php/Highlight/languages/cobol.json');

Where I added a small test cobol.json to see if everything works and it did. Additionally I am using the language filter for auto-detection now. So now to translate the whole language definition. And finding a better location to place the custom languages.

Thanks and I will follow the changes you mentioned about changing the adding and deleting of languages. From my side this can be closed but I naturally hope that the solution to have custom languages in the dropdown menu can be explored/implemented.

[allejo]

add_filter( 'syntax_highlighting_code_block_languages', function ( $languages ) {
	// Remove undesired languages.
	unset( $languages['brainfuck'] );

	// Replace an existing language.
	$languages['javascript']['path']    = plugin_dir_path( __FILE__ ) . '/my-javascript.json';
	$languages['javascript']['version'] = 2;

	// Add a new language.
	$languages['cobol'] = [
		'name' => 'COBAL',
		'path' => plugin_dir_path( __FILE__ ) . '/cobol.json',
		'version' => 2,
	];

	return $languages;
} );

Would that work for highlight.php?

Mostly possible. registerLanguage() has the $overwrite parameter, which lets you re-register a language definition. However, this won't "remove" the language from the Highlighter's internals. There's no performance boost in doing so, the only performance boost you can get is setting the auto-detect languages.

https://github.com/scrivo/highlight.php/blob/006e334dbf8e0a30573174e2cb6e11682b224f15/Highlight/Highlighter.php#L145-L155

My only minor concern is can we get a diff of $languages or see what's been touched? When a Highlighter object is created, it will automatically register all 185 languages available. But, if $languages is modified, will we be calling registerLanguage again for each language? e.g. In your example, will we be re-registering 184 languages + COBAL?

[allejo]

So now to translate the whole language definition. And finding a better location to place the custom languages.

@ntitze-del there are no official docs on how to generate JSON files for highlight.php but you don't need to manually translate it either! Take a look at the tools/process.sh script in the highlight.php repository. It's as simple as putting your JS file inside of a highlight.js 9.18.x folder, build it for node, and then run those utilities in that bash script.

[westonruter]

My only minor concern is can we get a diff of $languages or see what's been touched? When a Highlighter object is created, it will automatically register all 185 languages available. But, if $languages is modified, will we be calling registerLanguage again for each language? e.g. In your example, will we be re-registering 184 languages + COBAL?

Humm. Good point. Yes, I like the idea of getting a diff. If one of the language entries is identical before/after filtering, then we can just skip over it. Otherwise, if it is different then we can call registerLanguage again with the $overwrite parameter.

If a language has been filtered out of the list or is set to null, then we can remove it. But question: I don't see there being any unregisterLanguage method. It seems what is needed is to be able to pass in our own language list by invoking \Highlight\Highlighter::registerLanguages(), but this is currently a private method. Perhaps we should be able to pass in our own $languages to the Highlighter constructor to bypass the logic used to scan the filesystem for the languages to register?

If the constructor allowed passing an argument to skip registering any language, then we could follow-up with calling registerLanguage for the filtered set that we have.

[allejo]

highlight.js 9.18.x has two different behaviors, register all languages by default or opt-in to register each language as needed. I can justify translating this behavior to the constructor in highlight.php since there's no equivalent of importing differently in PHP.

[allejo]

I've tagged highlight.php v9.18.1.4 with the ability to opt-out of the auto language registration (argument in the Highlighter constructor). I've also added some other utility functions that should make our lives easier on this end.

[westonruter]

Great. Updated in #215. So now we can proceed with making use of the new changes.