Tkinter / Pygments: Strange results in highlighting

Question

I try to highlight texts according to tokens. Tokens can be highlighted by the below codes but there are some unwanted results. Some examples:

Suppose to, i typed:

a = "a"

Both a's are highlighted with the same color despite the first 'a' is Token.Name, the second 'a' is Token.Literal.String.Double

Another unwanted case is, when i typed "if", the word is highlighted and if i continue to add some letters to the word "if", the color of the word is changed as expected. However when i delete some letters of this word until the word becomes "if" again, this word is not highlighted as before.

Can you help me to understand this problem here?

Codes:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

from pygments import lex
from pygments.token import Token
from pygments.lexers import Python3Lexer
try:
    import Tkinter as tk
except ImportError:
    import tkinter as tk

ROOT = tk.Tk()
TEXT = tk.Text(master=ROOT, fg="white", bg="black", font="TkDefaultFont 10")
TEXT.pack(fill="both", expand=True)


def tag(event):

    def colorize(word, color):
        index = []
        index1 = TEXT.search(word, "1.0", "end")
        while index1:
            index2 = ".".join([index1.split(".")[0], str(int(index1.split(".")[1]) + len(word))])
            index.append((index1, index2))
            index1 = TEXT.search(word, index2, "end")
        for i, j in index:
            TEXT.tag_add(word, i, j)
            TEXT.tag_configure(word, foreground=color)

    for token, content in lex(TEXT.get("1.0", "end"), Python3Lexer()):
        if token == Token.Literal.Number.Integer:
            colorize(content, color="purple")
        elif token == Token.Keyword:
            colorize(content, color="orange")
        elif token == Token.Operator.Word:
            colorize(content, color="red")
        elif token == Token.Name.Builtin:
            colorize(content, color="blue")
        elif token == Token.Comment.Hashbang or token == Token.Comment.Single:
            colorize(content, color="grey")
        elif token == Token.Keyword.Namespace:
            colorize(content, color="yellow")
        elif token == Token.Namespace:
            colorize(content, color="green")
        elif token == Token.Punctuation:
            colorize(content, color="brown")
        elif token == Token.Literal.String.Double:
            colorize(content, color="cyan")
        elif token == Token.Name:
            colorize(content, color="white")


TEXT.bind("<KeyRelease>", tag)
ROOT.mainloop()

`

What have you done to debug this? Fo example, have you printed out the value of the variables as the loop runs to verify they are what you think they should be? — Bryan Oakley, Jun 10 '18 at 22:58
I put a print statement in the for loop that prints the content and token. — dildeolupbiten, Jun 11 '18 at 10:56
So, are you claiming all of the content and tokens are correct, and the indexes are correct, but it’s coloring the wrong thing? — Bryan Oakley, Jun 11 '18 at 12:18
I just want to say, once a word was recognised as a token, this word is highlighted in a different word. For example if program can recognise `"a"` as Token.Literal.String.Double so this chars are highlighted. But not only this string literal but also all a's in the Text widget. Here you can see on the screenshots. https://forum.yazbel.com/uploads/default/original/1X/09f8af59804472fe8f6bd891d0c6aefc4e58af73.png https://forum.yazbel.com/uploads/default/original/1X/ae2374523e84f5c80e6ba6c3784cf8534a94efa5.png — dildeolupbiten, Jun 11 '18 at 12:35

score 0 · Answer 1 · answered Jun 12 '18 at 14:52

The problem was solved via to the below codes.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import sys
if sys.version_info.major == 2:
    exit()
elif sys.version_info.major == 3:
    import builtins as builtins
    import tkinter as tk
import io
import tokenize
import keyword


root = tk.Tk()
text = tk.Text(master=root, fg="white", bg="black", font="TkDefaultFont 10")
text.pack(fill="both", expand=True)

count = 0


def colorize(*args):
    global count
    row1, col1 = args[0].startz
    row1, col1 = str(row1), str(col1)
    row2, col2 = args[0].end
    row2, col2 = str(row2), str(col2)
    start = ".".join((row1, col1))
    end = ".".join((row2, col2))
    text.tag_add(str(count), start, end)
    try:
        text.tag_config(str(count), foreground=args[1], font=args[2])
    except IndexError:
        text.tag_config(str(count), foreground=args[1])
    count += 1


def search(event):
    try:
        for i in tokenize.tokenize(io.BytesIO(text.get("1.0", "end").encode("utf-8")).readline):
            if i.type == 1:
                if i.string in keyword.kwlist:
                    colorize(i, "orange")
                elif i.string in dir(builtins):
                    colorize(i, "blue")
                else:
                    colorize(i, "white")
            elif i.type == 2:
                colorize(i, "cyan")
            elif i.type == 3:
                colorize(i, "purple")
            elif i.type == 53:
                if i.string == "," or i.string == "." or i.string == ":":
                    colorize(i, "orange")
                elif i.string == "(" or i.string == ")" or i.string == "[" \
                        or i.string == "]" or i.string == "{" or i.string == "}":
                    colorize(i, "darkred")
                else:
                    colorize(i, "green")
            elif i.type == 57:
                colorize(i, "grey", "TkDefaultFont 10 italic")
    except tokenize.TokenError:
        pass


text.bind("<KeyRelease>", search)
root.mainloop()

Tkinter / Pygments: Strange results in highlighting

1 Answers1