If the language is similar enough to a language already supported by SciTE, you might be able to copy the properties file for that language and change to it to match your new language.
If this does not work, you can write a piece of code that SciTE calls a "lexer".
-
a lexer defines how to add syntax highlighting and/or folding
-
it's less complex than truly parsing the file that the user is editing
-
in recent versions of SciTE, a lexer can be written in the Lua language (simpler and doesn't require a C++ compiler)
-
documentation on Writing a lexer in the Lua language
-
Andreas Tscharner's tutorial for adding Syntax Highlighting & Code Folding, and example code in C++