290 CodeIntel OOP Protocol

This document details the protocol used for out-of-process code intelligence in Komodo 8.5. This can be used to implement additional language support via extensions. This was initially implemented in bug 93455.

Note that the protocl defined here is not considered stable; changes are expected in the future. For Komodo 9, we expect to change how the buffers are handled so that an explicit request from codeintel back to Komodo is required to obtain the buffer, so that discarded scan requests would not need to transmit file contents.

Framing

Frames

A frame is a single request or response. Each frame consists of, in order:

• A run of decimal digits, [0-9], indicating the frame size in bytes (ASCII-encoded).
• A JSON object (dict). Any strings in the object are UTF-8 encoded. (As in, the raw bytestream of the frame is in UTF-8.)

Example

2{}

An empty frame with no data.

Requests

Each request (client -> codeintel process) must have the following keys:

• command {String}: The command to execute
• req_id {String}: Request identifier; this must be sent back with each response.

Example

38{"command":"ping","req_id":"d428baa4"}

A (ficticious) ping command.

Responses

Each response (codeintel process -> client) must have the following keys:

• req_id {String}: The request identifier sent as part of the request. If this is an unsolicited response, this is not provided.
• success {Boolean}: Whether the command succeeded. If this key is missing, the request has not yet been completed, and this is merely a status update.

Failures

If the request failed, also expect:

• message {String}: Some sort of message about the error.

Example

36{"req_id":"d428baa4","success":true}

A response to the previous command. :

63{"req_id":"d428baa4","success":false,"message":"I am not here"}

A failure version of the previous response. Only one response should be sent for the request, not both.

Unsolicited responses

Sometimes the codeintel process might decide to send unsolicited responses; for example, it might want to send status messages about stdlib preloading. This should only be for messages that are not targeted towards a particular Komodo window.

Notes

• All positions are specified in UTF-8 byte counts. Position 0 is the first character.

  (Historical note: Komodo 8.0 was in character counts.)

• For cases specifying ranges:

  • The start position is the position of the first character in the range (i.e. inclusive).
  • The end position is after the last character in the range (i.e. exclusive).

  For example, given the buffer "Komodo", range [2,4) is the string "mo".

(Further examples will not be in wire protocol format, but in normal JSON format instead.)

Commands

Commands are determined by the command key in the JSON frame. All arguments are siblings of the command key.

abort

Abort some outstanding request as soon as possible. Arguments:

• id {String}: The request to abort.

add-dirs

Add catalog, lexer, or module directories. All of the parameters are optional.

Arguments

• catalog-dirs {Array of String} (Optional): Catalog directories to add. Providing this argument will cause catalogs to be rescanned.
• lexer-dirs {Array of String} (Optional): Lexer directories to add. This must be called before the corresponding modules are registered.
• module-dirs {Array of String} (Optional): Module directories to add. This must be called before the module can be used; in practice, this means right at startup.

load-extension

Load a Python-based extension into the codeintel process. See the Extensions section below for details.

Arguments

• module-path {String}: The directory containing the Python extension to load.
• module-name {String}: The name of the module to load.

Note

The module must export a registerExtension method that takes no arguments; it will be called on initial module load.

database-info

Provide information about the code intelligence database. No arguments.

Return value

• state {String}: The state of the database. May be one of the following strings:

  • ready: The database is ready.
  • preload-needed: The database needs to have the standard libraries pre-loaded before use.
  • upgrade-needed: The database must be upgraded before use.
  • upgrade-blocked: The database format is obsolete and cannot be upgraded. It must be reset before use.
  • broken: The database is broken in an unspecified way; it must be reset before use.

• state-detail {String}: Extra details about the state; may be missing for some states.

database-reset

Rebuild the database, destroying any existing information. No arguments. No command-specific response.

database-preload

Preload the code-intelligence database with standard catalog data.

Arguments

• languages {Object} (Optional): A dictionary of languages to preload. The key is the language name; the value is the language version desired, or null to indicate all versions. If not given, a default list of languages is preloaded.
• catalogs {Array}: the list of catalogs to load. Each item is a string consisting of the name of the catalog that must be preloaded.

Example

    {"command": "database-preload",
     "languages": {"Python": "2.6",
                   "JavaScript": null
                  },
     "catalogs": ["pywin32", "pywin32 (Python3)"]
    }

Note

This method will have multiple status update messages before success. Each update message will contain the following data:

• progress {Number}: An estimate of the current progress in unspecified units.
• total {Number}: An estimate of the total in the same units required before the operation completes. If 0, the total is undetermined.
• message {String}: A short description of what is currently being loaded.

database-upgrade

Start the database upgrade process. No arguments.

Note

This command will succeed immediately if the database does not need upgrading; it is harmless to run on an already-current database. It is an error to run on a database in a state other than ready or upgrade-needed.

get-languages

Get the list of languages.

Arguments

• type {String}: The type of languages to get. Accepted values:
  • cpln: Get languages which support completion.
  • citadel: Get citadel languages.
  • xml: Get XML languages.
  • multilang: Get languages which may contain multiple sub-languages. (e.g. RHTML, which can contain Ruby and JavaScript)
  • stdlib-supported: Get languages which have built-in standard library databases available.

Return value

• languages {Array of String}: Languages which support code completion; language names are strings, such as "Python" and "Python3".

get-language-info

Get information about a specific language

Arguments

• language {String}: The language to look up.

Return value

• completion-fillup-chars {String}: Characters which cause auto-complete fillup (automatic selection of autocomplete).
• completion-stop-chars {String}: Characters which abort auto-completion.

get-available-catalogs

Get the list of catalogs available for selection

Return value

• catalogs {Array of Object}: The available catalogs. They have the following properties:
  • name {String}: The name of the catalog (either the name attribute as given by the <codeintel> tag where available, or the file base name).
  • lang {String}: The language the catalog applies to. (The lang attribute of the first <file> tag in the catalog.)
  • description {String}: A description of the catalog. (The description attribute of the <codeintel> tag.)
  • cix_path {String}: The path to the .cix file for the catalog.
  • selection {String}: The opaque identifier to use for selections.

Note

This should not be run before finishing with the add-dirs commands.

set-xml-catalogs

Set the available XML catalogs. Existing catalogs are unloaded.

Arguments

• catalogs {Array of String}: The paths of the catalog files.

get-xml-catalogs

Get a list of the loadad XML catalog information

Return value

• public {Array of String}: The public ids of the loaded XML catalogs.
• system {Array of String}: The system ids of the loaded XML catalogs.
• namespaces {Array of String}: The namespaces of the loaded XML catalogs.

set-environment

Set the global environment and preferences. Any of the arguments can be omitted, in which case the old values are used for that setting.

Arguments

• env {Object} (Optional): A String to String mapping of the enviroment to use. This will override any current environment being used.
• prefs {Array of Object} (Optional): A list of mappings of the preferences to use. Each element in the array represents one preference depth level. Each level is a JS object used as a dictionary; the key is the preference name, and the value is the preference value at that level. The preference levels are ordered from most important/specific to least important/specific.

Example

    {"command": "set-environment",
      "env": {
        "PYTHONPATH": "/dev/shm/python/path"
      },
      "prefs": {
        [{
          "pythonExtraPaths": ["/extra/path", "/from/file"]
         }, {
          "pythonExtraPaths": ["/extra/path/from/project"],
         }, {
          "pythonExtraPaths": ["/global", "/extra"]
         }]
      }
    }

eval

Evaluate a trigger created by trg-from-pos. The return value depends on the trigger type.

Arguments

• trg {Object}: The serialized trigger from trg-from-pos.
• silent {Boolean} (Optional): If true, evaluation errors/warnings will not be logged. Defaults to false.

Completion return value

• cplns {Array of 2-Tuples}: Available completions; each completion entry is a 2-tuple of (type, name); both are strings.
• retrigger {Boolean}: If true, there may be more completions available after the current completion response has been applied.

Calltip return value

• calltip {String}: The calltip.
• args {Array} (Optional): Arguments for the calltip, if available. Each argument is a dictionary with the keys:
  • name {String}: The name of the argument.
  • default: The default value.
  • start {Number}: The start position within the calltip for this argument; null if not available.
  • end {Number}: The end position within the calltip for this argument; null if not available.

Definition return value

• defns {Array of Object}: A list of definitions. Each definition has the attributes as given in koICodeIntelDefinition.

quit

Terminate the codeintel process.

Commands Dealing with Buffers

These commands all take a shared set of arguments that indicate a buffer, in addition to any command-specific arguments. The shared arguments are:

• path {String}: The path of the file. For unsaved files, it should be something like <Unsaved>/Text1.txt.
• language {String} (Optional): The language for this file. If not given, should attempt to detect from the path.
• encoding {String} (Optional): The chracter encoding of the file on disk. If not given, some autodetection is attempted. Ignored if text is given (since the protocol is always sent in UTF-8).
• text {String} (Optional): The text to use for the file contents. If given, the file on disk is not read.
• env {Object} (Optional): The file-specific environment. This should contain either an env child or a prefs child (or both). This is in the format of the arguments to set-environment. If not supplied, the previous environment used for this file, if any, is used. Setting the children to None clears the environment (and uses the global environment).

Commands in this subset follow:

calltip-arg-range

Get the calltip argument range (i.e. what argument is currently being entered).

Arguments

• trg_pos {Number}: The trigger position.
• calltip {String}: The full calltip text.
• curr_pos {Number}: The position in the buffer for the argument.

Return value

• start {Number}: The start position of the argument; -1 if the text has exited the call region (i.e. the user typed beyond the call).
• end {Number}: The end position of the argument; -1 if the text has exited the call region (i.e. the user typed beyond the call).

get-sections

Get all sections in the document. This command has no arguments beyond those typical of commands dealing with buffers.

Return value

• sections {Array of Object}: All sections in the document. Each section has the following keys:

  • title {String} The title of the section.
  • line {Number} The starting line number of the section (1-based).
  • lang {String} The language the section belongs to. Used for multi-lang documents.

  Sections may also have the following optional keys:

  • type {String}: The type of the section; same as those used in autocomplete.
  • level {Number}: Level of the section; the topmost level is 0, and its children are 1, etc.
  • id {String}: A section identifier; may be not unique.

scan-document

Proactively scan a document.

Arguments

• priority {Number}: The priority of the request. The smaller the number, the higher priority (i.e. the sooner it will be scanned). See the PRIORITY_* constants in src/codeintel/lib/codeintel2/common.py.
• mtime {String} (Optional): If given, the last modified time of the file; the file will not be re-scanned if the last scan was started after the last modified time. If not given, the last modified time reported by the file system is used. If not given, but text is given, the current time is used (and the file system is ignored).

trg-from-pos

Create a trigger from a given position.

Arguments

• pos {Number}: The position to generate the trigger from.
• type {String} (Optional): If given and is defn, attempts to locate a definition instead.
• curr-pos {Number} (Optional): The current position; if this is given, this acts like preceding_trg_from_pos.
• implicit {Boolean} (Optional): Whether the this is implicit. Defaults to true.

Return value

• trg {Object}: A serialized trigger as an object (dictionary); this may have arbitrary structure and should only be used to be passed back to other commands requiring a trigger.
  • Note that the trigger contains information on the file path/language/etc.

Unsolicited Responses

scan-complete

Fired when a scan is completed.

Arguments

• path {String}: The path of the file that was scanned.
• language {String}: The language that was scanned. Useful for multi-language files, as multiple scan-complete events would be generated.

report-message

Fired to report a generic codeintel message. If the type is known to the client, it may wish to do special handling for particular types.

Arguments

• message {String}: The message to report
• type {String}: Message type, to support better reporting of particular messages.

scan-progress messages

These messages are for reporting scanning progress; they have the following additional fields:

• total {Number}: The total number of directories that will be scanned.
• completed {Number}: The number of directories that have been scanned.

logging messages

These messages are from the Python logging module. Only log levels that are enabled will be reported.

• name {String}: The name of the logger.
• level {Number}: The logging level; it uses the scale found in the Python logging module.

report-error

Fired to report messages to the error log.

Arguments

• message {String}: The error message to log.

global-prefs-observe

Add or remove global pref observers

Arguments

• add {Array of String}: The preference names to start observing
• remove {Array of String}: The preference names to stop observing

Initialization

On startup, the codeintel process will start with a frame with no contents, "2{}", to indicate successful startup.

Termination

The codeintel process will gracefully shut down when encountering a quit command. It will also shut down if the control connection has been closed, as it cannot be re-established.

Extensions

The load-extension command can be used to load codeintel extensions in the codeintel process. There is no stable API defined at this point; it is initially being used for code browser tree building.

Within the codeintel2.oop.driver module, the Driver class has a static method named registerCommandHandler() which accepts an instance of CommandHandler. This instance can have a supportedCommands iterable property which yields strings of commands this handler supports. The instance further has a canHandleRequest(request) method that allows it to accept or reject specific requests; once accepted, its handleRequest(request, driver) method will be called to actually process the request. The handler may send results back via driver.send().

Komodo finds codeintel extensions by enumerating the codeintel-command-extension category registered with the XPCOM category manager; each result is a contract ID that will be createInstaced and then unwrapped to a normal Python object. This object is then iterated over for (module-path, module-name) tuples as used in load-extension.