lexer

Build our lexer in addition to utilizing already built ones.

Pygments, IPython, prompt_toolkit, Jinja2 and Sphinx all come with their own concepts of lexers which doesn’t include the built-in modules.:

• parser
• token
• tokenizer
• re
• ast

So it’d be tough to say we’re at a lack of tools!

Configurable Lexers

The lexers defined here were originally created to stand in as objects able to merge the seemingly disjoint APIs of IPython, traitlets, prompt_toolkit and pygments.

Todo

The IPython.terminal.lexer.IPythonPTLexer also should have a few of these attributes as well.

And here’s how you join the bridge.

PygmentsLexer(Lexer):

Lexer that calls a pygments lexer.

Example:

from pygments.lexers.html import HtmlLexer lexer = PygmentsLexer(HtmlLexer)

Note: Don’t forget to also load a Pygments compatible style. E.g.:

from prompt_toolkit.styles.from_pygments import style_from_pygments_cls from pygments.styles import get_style_by_name style = style_from_pygments_cls(get_style_by_name(‘monokai’))

Parameters

• pygments_lexer_cls – A Lexer from Pygments.

• sync_from_start – Start lexing at the start of the document. This will always give the best results, but it will be slow for bigger documents. (When the last part of the document is display, then the whole document will be lexed by Pygments on every key stroke.) It is recommended to disable this for inputs that are expected to be more than 1,000 lines.

• syntax_sync – SyntaxSync object.

Lexer Autogenerated Docs

our_style()

get_lexer()

pygments_tokens()

A list of Pygments style tokens. In case you need that.

class IPythonConfigurableLexer(**kwargs)

Bases: traitlets.config.configurable.LoggingConfigurable

lexer

A trait whose value must be an instance of a specified class.

The value can also be an instance of a subclass of the specified class.

Subclasses can declare default classes by overriding the klass attribute

name = None

Name of the lexer

aliases = []

Shortcuts for the lexer

filenames = []

File name globs

alias_filenames = []

Secondary file name globs

mimetypes = []

MIME types

priority = 0

Priority, should multiple lexers match and no content is provided

__init__(shell=None, original_lexer=None, **kwargs)

shell

A trait whose value must be an instance of a specified class.

The value can also be an instance of a subclass of the specified class.

Subclasses can declare default classes by overriding the klass attribute

class Colorizer(pylexer=None, formatter=None)

Bases: object

Make the pygments function ‘highlight’ even easier to work with.

Additionally utilize __slots__ to conserve memory.

__init__(pylexer=None, formatter=None)

pylexer

A PythonLexer from Pygments

formatter

Format tokens with ANSI color sequences, for output in a true-color terminal or console. Like in TerminalFormatter color sequences are terminated at newlines, so that paging the output works correctly.

New in version 2.1.

Options accepted:

style

The style to use, can be a string or a Style subclass (default: 'default').

highlight(code)

class MyPythonLexer

Bases: IPython.terminal.ptutils.IPythonPTLexer

EXTRA_KEYWORDS = {'!'}

get_tokens_unprocessed(text)

get_titlebar_text()

Return (style, text) tuples for startup.

generate_and_print_hsplit()

Generate a prompt_toolkit.layout.container.HSplit.

Originally was in 33_bottom_toolbar but moved here so we can use the lexer.