LSP Architecture

Abstract

The language server is a single-process pygls application. It owns a Workspace of open TextDocumentState records, re-parses on every mutation, and dispatches LSP methods to pure-ish feature handlers that take (params, source) rather than holding server globals. Capabilities are declared once at initialize; transport defaults to STDIO for editor integration.

Conceptual model

Interface surface

Process entry

# Editable install
pip install -e ".[lsp]"
python -m pynescript.langserver   # same as pyne-lsp / pynescript-lsp
make run-lsp

Entry: src/pynescript/langserver/__main__.py constructs PynescriptLanguageServer() and calls server.start_io() (STDIO JSON-RPC).

pyproject.toml registers:

pyne-lsp = "pynescript.langserver.__main__:main"
pynescript-lsp = "pynescript.langserver.__main__:main"   # alias

Server class

PynescriptLanguageServer (server.py):

• Subclasses pygls.lsp.server.LanguageServer with name="Pynescript" and version=__version__ from src/pynescript/__about__.py (0.6.31). initialize reports serverInfo.name = "Pynescript Language Server" with the same version.
• Instantiates self.pine_workspace = Workspace().
• Registers handlers in setup_method_handlers() via @self.feature(...).

Document sync

Sync options (config.get_server_capabilities):

• TextDocumentSyncKind.Incremental
• open_close=True
• save with include_text=True
• will_save / will_save_wait_until off

Capability set (declared)

From config.py (get_server_capabilities):

• Diagnostic provider (identifier="pynescript-diagnostics", workspace_diagnostics=True; push diagnostics carry the document version)
• Completion: trigger ".", resolve_provider=True
• Hover, definition, references, document highlight
• Linked editing (linked_editing_range_provider=True) — same-document user ident ranges; keywords / builtin namespaces refused
• Rename with prepareProvider (every open workspace document that contains the ident; keywords / builtin namespaces refused)
• Document + workspace symbols
• Full + range formatting, plus on-type (documentOnTypeFormattingProvider, first_trigger_character="\n")
• Folding ranges (folding_range_provider=True) for blocks, type/enum/function bodies, and //#region
• Selection ranges (selection_range_provider=True) from ident / smallest AST node to enclosing statement and block
• Color provider (color_provider=True) — textDocument/documentColor + colorPresentation for #hex, named color.*, color.rgb / color.new / color(), and literal color.from_gradient
• Document links (DocumentLinkOptions(resolve_provider=True)) — http:// / https:// URLs in comments (//, /* */, hash # …) and string literals. documentLink/resolve fills tooltip with the URL when tooltip is unset
• Code action (QuickFix + RefactorRewrite): Convert to Pine v6, W001 insert //@version=6, W002 replace old pragma, C002 format-on-long-line + executeCommand pynescript.convertToV6
• Inlay hints (InlayHintOptions(resolve_provider=False)): display-only parameter-name hints on builtin calls (ta.sma(close, 14) → source: / length:). No inlayHint/resolve, no data, no textEdits
• Semantic tokens full and range (range=true, full=true; legend of standard token types/modifiers)
• Signature help (( / ,)

File filters (get_filter_options): *.pine, *.pinev5, *.pinev6 under language id pinescript. .pyne is not in this helper — VS Code still maps it via the extension contribution (extensions: .pyne first).

Internals

Workspace

Workspace (workspace.py) maps uri → TextDocumentState:

TextDocumentState
  uri, source, version
  ast | None
  diagnostics: list[LintWarning]
  parse_error, parse_error_line

_parse_and_lint:

1. parse(source, filename=uri) — on success, lint_script(source, filename=uri).
2. On exception: clear AST, store parse_error string, extract line via regex line[:\s]+(\d+).

Incremental edits: _apply_text_edit splits on \n, pads the line list when the range sits at/past EOF (append without a trailing newline), clamps columns, then replaces the range. Out-of-range positions no longer leave a stale buffer. Whole-document change events replace source wholesale. Identical text after apply skips re-parse/lint (version still updates).

Position encoding

LSP positions count UTF-16 code units; Python strings count code points. They differ only for non-BMP characters (emoji, some CJK extensions). The server converts at the boundary:

• request Position/Range values go UTF-16 → code points before a feature handler runs;
• result ranges go back to UTF-16 on the way out (including nested WorkspaceEdit / Location objects, which use each document's own lines);
• diagnostics and incremental didChange edits convert too;
• SemanticTokens columns and lengths are converted before delta encoding.

ASCII/LF buffers skip the recursive conversion pass entirely. Handlers therefore keep using Python string indices internally.

Diagnostics conversion (_lint_warnings_to_diagnostics):

• Map severity strings → DiagnosticSeverity.
• Source tag "pyne", code from lint rule.
• Append synthetic E001 error for parse failures at the extracted line.

Feature dispatch pattern

Handlers are functions, not server methods, e.g.:

@self.feature(lsp.TEXT_DOCUMENT_COMPLETION)
def text_completion(params):
    source = self.pine_workspace.get_source(params.text_document.uri)
    return completion_feature.handle_completion(params, source)

Definition / references / symbols / hover / completion / inlay / semantic tokens (full and range) / linked editing / document links / formatting (including on-type) receive the version-matched AST (tree=doc.valid_ast) and, where useful, the scope index (analysis=doc.valid_analysis). valid_ast returns None while a debounced parse for a newer buffer is in flight, so features never combine new text with old positions. Pass tree=None when the last parse failed. Omit tree only in isolated tests (handler then parses from source).

Pull diagnostics

• textDocument/diagnostic → RelatedFullDocumentDiagnosticReport with result_id=f"{uri}-{version}".
• workspace/diagnostic → one full report per open document from get_all_diagnostics().

Workspace symbols

workspace/symbol walks open documents through the shared document-outline collector (functions/methods, types, enums + members, imports, variables) and then queries WorkspaceIndex for closed .pyne/.pine/.pinev5/.pinev6/.pinescript files under the workspace roots. The closed-file index is bounded (file count/size, ignored build/VCS/dependency directories, mtime cache, 2 s search budget). Filter is case-insensitive substring on params.query.

Invariants and edge cases

1. Parse failure is non-fatal for the process. AST-dependent features return None or []; diagnostics still surface E001.
2. Source of truth is the open buffer, not disk. Save only re-publishes; it does not re-read files.
3. Incremental sync pads/clamps rather than dropping the edit. Ranges at EOF grow the line list; columns are clamped to the current line length.
4. Version is stored and used in pull-diagnostic result_id; the server does not reject out-of-order versions itself.
5. Nuitka onefile embeds providers data and may rely on encrypted metadata — see builtin metadata. Architecture of handlers is identical to the pure-Python path.

Worked example — minimal initialize

Client → server (conceptual):

{
  "method": "initialize",
  "params": {
    "capabilities": {},
    "clientInfo": { "name": "example", "version": "1" }
  }
}

Server returns InitializeResult with serverInfo.name = "Pynescript Language Server" and the full ServerCapabilities table from get_server_capabilities().

Failure modes

See also

• Diagnostics
• Linked editing
• Formatting — full, range, and on-type
• Semantic tokens — full and range
• Inlay hints — types, param names, resolve
• Document links — http(s) URLs; documentLink/resolve fills tooltip
• Document color — #hex / color.* / literal color calls
• Builtin metadata
• VS Code extension
• Nuitka build