public sealed class CStyleTokenizerA tokenizer for the curly-brace languages, given their reserved words.
No guide page documents this yet — the page shows what the code says about itself.
Remarks
One implementation for Raven, C# and VCSS because their lexical shapes agree — // to the end of the line, /* */ across them, quoted strings with backslash escapes, numbers, and a set of words. What differs between them is grammar, which a highlighter does not read.
⚠ The only state it carries is "inside a block comment". A multi-line string — C#'s """, a template literal — would need another, and reporting one as several broken strings is wrong in a way that is visible. Said out loud: this handles the languages named above, and a language with multi-line literals wants its own tokenizer, which is what the interface is for.
Fields and properties (2)
public static CStyleTokenizer RavenRaven's reserved words.
public static CStyleTokenizer CSharpC#'s, near enough for an in-editor script.
Methods (2)
public CStyleTokenizer(IEnumerable<string> keywords, IEnumerable<string>? types = null)Creates a tokenizer over two word sets.
public int Tokenize(string line, int state, List<CodeToken> into)Splits one line into runs.
Used by (6)
- CodeEditorTestsVixen.Ui.Controls.Advanced.Tests
- CodeTokenizerTestsVixen.Ui.Controls.Advanced.Tests
- CodeAnalysisTestsVixen.Editor.AssetEditors.Tests
- ShaderDocumentVixen.Editor.AssetEditors
- ShaderGraphViewVixen.Editor.AssetEditors
- StyleSheetDocumentVixen.Editor.AssetEditors