Kv Parser

Parses source code contains kv rules and contexts.

class kivy.lang.compiler.ast_parse.KvException

Bases: builtins.Exception

Raised when something goes wrong with kv compilation or loading.

class kivy.lang.compiler.ast_parse.KvCompilerParserException

Bases: kivy.lang.compiler.ast_parse.KvException

Raised when something goes wrong with kv parsing.

kivy.lang.compiler.ast_parse.parse_expr_ast(expr_str)

Takes a expression string (i.e. can be represented by ast.Expr type), e.g. “self.x”, and returns the AST node that directly corresponds to that expression (i.e. it’s not wrapped in a Module like ast.parse).

E.g.:

>>> ast.dump(parse_expr_ast('self.x'))
"Attribute(value=Name(id='self', ctx=Load()), attr='x', ctx=Load())"

class kivy.lang.compiler.ast_parse.BindSubGraph(nodes, terminal_nodes)

Bases: builtins.object

A minimal rebind/leaf constrained subgraph from the bind graph.

bind_store_rebind_nodes_indices = {}

The nodes that are in the nodes of the subgraph, and maps to the index in callback_names and bind_store_indices where this subgraph was bound to in the bind store. I.e. the index in bind store, where each node bind a rebind function that is the rebind created for this subgraph.

n_rebind_deps = 0

The number of rebind nodes in nodes.

rebind_callback_name = None

There is one rebind callback for each subtree, provided there are at least one rebind attribute as a node in nodes (i.e. n_rebind_deps is non-zero) in which case there are none.

class kivy.lang.compiler.ast_parse.ASTBindNodeRef(is_attribute)

Bases: _ast.AST

Created by ParseKvBindTransformer as it walks the AST graph to represent an original ast.AST node that we could bind to (e.g. a attribute access of a rebind, or leaf node) or a dependency of such a node (e.g. a local or global or Name etc.).

E.g. given self.x + self.y, when visiting its ast nodes, we create corresponding ASTBindNodeRef nodes for self, x, and y.

Subsequently, depends and depends_on_me describes a newly constructed ast graph that represents a subgraph of the original ast graph, that contains this node.

E.g. self.x + other.x represents two independent subgraphs, one with ASTBindNodeRef nodes representing self and self.x, and the other with nodes representing other and other.x.

After parsing, the graphs created by these nodes are self contained in the sense they

count = 0

The number of unique leaves (leaf_rule is not None, not counting leaf_rule value duplicates) that this node is an ancestor of or 1 if it’s a leaf_rule. E.g. in self.obj.x + self.obj.y, self.obj has a count of two as does self. y and x both have a count of one.

:atr:`count` is not the same as len(depends_on_me). Although for the the rule self.x + self.y.z, the len and count will be the same for self (2). For rule self.x[self.x.y].z, there’s only one leaf rule so count will be one, yet depends_on_me will contain two items for the node representing self.x.

Similarly, if we have two rules, each saying self.x, then self will have a count of two.

depends = []

List of ASTBindNodeRef instances that represent ast nodes that are parents of this node. The ASTBindNodeRef.depends_on_me of such parents must include this node.

depends_on_me = []

List of ASTBindNodeRef instances that represent ast nodes that are children of this node. The ASTBindNodeRef.depends of such children must include this node.

static group_by_required_deps_ordered(nodes)

Groups the nodes into groups based on their deps. All nodes with the same deps are grouped together. The order of nodes is preserved as given in nodes such that the first occurrence of nodes with unique deps are ordered in the order they occur in nodes.

is_attribute = False

Whether the ASTBindNodeRef represents an ast.Attribute node. These are nodes that we potentially bind to either because it’s a rebind node or because it’s a leaf.

leaf_rule = None

If is_attribute and it’s a leaf node in the ast expression, e.g. x in self.x or (self.y + self.z).x, then leaf_rule is set to the value of rule passed to ParseKvBindTransformer.update_graph(). That indicates that it’s a leaf. Otherwise it is None.

The exact value of leaf_rule is not important, we just use it to check if it’s None.

proxy = False

If the node’s child is an is_attribute, then it indicates whether the object represented by this node must not hold a direct reference to the other objects after binding. When True, it will hold a proxy to the objects.

E.g. if a rule is bound to self.x.y, then we bind to self.x and potentially to self, depending on the rebind value of self.x. Either way, if proxy is True for self and/or self.x, neither will hold a direct ref to any of the binding objects or callbacks. This is to prevent the self and/or self.x objects from keeping the other objects from being garbage collected.

proxy may be set after graph parsing is done and is not used during parsing (ParseKvBindTransformer.update_graph()) in any way.

rebind = False

If the node is an is_attribute, and if it’s not a leaf_rule then it indicates whether this node will be rebound.

E.g. if a rule is bound to self.x.y, then we would need to rebind the rule to the x attribute if rebind is True. In this case, if x changes, all the nodes that are children of x will be re-evaluated and e.g. the rule will be rebound so then when the y value of the new x changes the rule will be dispatched.

rebind may be set after graph parsing is done and is not used during parsing (ParseKvBindTransformer.update_graph()) in any way.

ref_node = None

The AST node that this instance represents.

src = ''

The generate_source() generated source representing the ref_node. E.g. for the graph self.x.y + self.z, if the ref_node represents the x Attribute node, then src will be “self.x”.

In the bind graph, this is the primary method we use currently to identify node duplicates. I.e. ParseKvBindTransformer.src_node_map keeps a mapping from src to the ASTBindNodeRef instance that represents it for non-leaf_rule nodes. Then, if we encounter a source fragment we have seen before, we reuse that node and increment count, unless it’s a leaf_rule, then we don’t re-use but always create a new one (except if we have seen this leaf already in that rule as tracked by ParseKvBindTransformer.nodes_under_leaf).

subgraph_for_terminal_node = None

The subgraph the node appears in as a terminal node. It can only be one. Otherwise, either it has shared deps, in which case it would be one subgraph as each contains all the deps of every terminal node. Otherwise, they don’t share deps, which makes no sense.

Also, nodes that are not rebind, will occur only in one subgraph, so this is also set to the subgraph by populate_bind_stores_and_indices_and_callback_names. Nodes that are not rebind and occur in multiple subgraphs, e.g. a number, such nodes never care what subgraph it’s in, because it has no rebind in its parent graph so the subgraph is the last one it occurred in and.

subgraphs_to_bind_store_idx_and_name = {}

Maps a subgraph to the index in callback_names and bind_store_indices, indicating the index in these lists that corresponds to the bind for that subgraph.

class kivy.lang.compiler.ast_parse.ASTStrPlaceholder(*largs, **kwargs)

Bases: _ast.AST

An AST node that represents a list of strings that is added to the source code when generated to source. They are implicitly indented to the indentation level of the node they are contained in.

Useful for dumping strings rather than having to create AST nodes for all the source code manually. E.g.:

>>> node = ASTStrPlaceholder()
>>> node.src_lines = ['self.x', 'name = self.y']
>>> generate_source(node)
'self.x

name = self.y’

>>> # then
>>> if_node = ast.parse('if self.z:\n    55')
>>> ast.dump(if_node)
"Module(body=[If(test=Attribute(value=Name(id='self', ctx=Load()), attr='z', ctx=Load()), body=[Expr(value=Num(n=55))], orelse=[])])"
>>> if_node.body[0].body[0].value = node
>>> generate_source(if_node)
'if self.z:\n\n    self.x\n    name = self.y'

class kivy.lang.compiler.ast_parse.ASTNodeList(*largs, **kwargs)

Bases: _ast.AST

AST node that represents a list of AST nodes.

E.g.:

>>> node1 = ast.parse('self.x')
>>> node2 = ast.parse('self.y')
>>> node_list = ASTNodeList()
>>> node_list.nodes = [node1, node2]
>>> generate_source(node_list)
'self.x\nself.y'

Or:

>>> node1 = ast.parse('if self.x:\n    55')
>>> node2 = ast.parse('name = self.y')
>>> node_list = ASTNodeList()
>>> node_list.nodes = [node1, node2]
>>> generate_source(node_list)
'if self.x:\n    55\nname = self.y'

nodes = []

List of ast nodes represented by this node.

class kivy.lang.compiler.ast_parse.RefSourceGenerator(indent_with, add_line_information=False, pretty_string=<function pretty_string>, len=<built-in function len>, isinstance=<built-in function isinstance>, callable=<built-in function callable>)

Bases: astor.code_gen.SourceGenerator

Generates the source code from the custom AST nodes.

Used with generate_source() to add custom AST nodes.

kivy.lang.compiler.ast_parse.generate_source(node)

Generates the source code string represented by the node based on astor.to_source.

As opposed to astor.to_source, it understands how to handle our custom AST classes and removes any trailing newlines.

E.g.:

>>> node = ast.parse('self.x + self.y + len("hello")')
>>> generate_source(node)
"self.x + self.y + len('hello')"

class kivy.lang.compiler.ast_parse.ParseCheckWhitelisted(whitelist)

Bases: ast.NodeTransformer

Takes a expression node and checks whether any of the expression nodes that are part of the node’s graph are not in the whitelist.

For every node encountered, we map the node in node_has_illegal_parent to whether the node or any of its parents are not whitelisted.

E.g. self.x would map self and self.x to False, while (self.x + self.y).z would map self, self.x, self.y to False and (self.x + self.y) and (self.x + self.y).z to True (assuming only Attribute and Name are whitelisted).

check_node_graph(node)

Gets called with a node describing a single expression. Doesn’t accept non-expression nodes, e.g. an assign, Module etc. It then populates node_has_illegal_parent.

generic_visit(node)

Called if no explicit visitor function exists for a node.

has_illegal_parent = False

Set by node to True if it or any of its parents are not whitelisted.

node_has_illegal_parent = {}

Maps each node in check_node_graph() node’s graph to whether it is or has a non-whitelisted parent.

whitelist = set()

Set of ast class names that are whitelisted. These are strings with the exact class names, e.g. “Attribute”.

class kivy.lang.compiler.ast_parse.ParseKvBindTransformer(kv_syntax=None)

Bases: ast.NodeTransformer

Transforms and computes all the nodes that should be bound to. It processes all the nodes and create the graphs representing the bind rules.

Algorithm: given ast for e.g. (self.x + self.y).z + 55, the leaves are `z and 55 and the roots are self, 55. For every node we visit, it first processes its parents and then itself. E.g. (self.x + self.y) is processed before (self.x + self.y).z, and self.x, self.y before (self.x + self.y).

The idea is to find the first node starting from leaves and working up the parent graph that are an Attribute, e.g. (self.x + self.y).z. Once we have that, we may need to rebind to all intermediate attribute nodes in its parent graph. E.g. for (self.x + self.y).z, we may need to rebind to self.x and self.y. We create ASTBindNodeRef for each node we encounter once we encounter the first Attribute.

So e.g. for self.x + self.q, we create 3 ASTBindNodeRef nodes - for self, self.x, and self.q. Their summation does not happen “under” a Attribute access, so it is ignored.

ASTBindNodeRef instances describe subgraphs within the larger ast graph. As can be seen, ASTBindNodeRef points to nodes that depend on it (children) and that it depends on (parents)- that defines the subgraph that we may need to rebind to.

Nodes that evaluate to the same string, e.g. in self.x + self.x, self.x occurs twice, but in the graph we create, we only create one node for self and one node for self.x. Similarly, for self.x + self.y, only 3 nodes are created.

Examples so far, all created one subgraph - but we can have more than one. E.g. self.x + other.x creates two independent graphs, one for self.x and one for other.x, each composed of two nodes because they share no common nodes. But e.g. (self.x + other.y).z will create one subgraph with 6 nodes, a node for each operation because they all happen under the same Attribute leaf so it describes one graph.

The key thing to remember is that parent nodes are processed before children. This order determines the linear order of the nodes as they get added to nodes_by_rule. This means that when a node is encountered in the nodes_by_rule lists, all their parents are present in the lists at indices previous to that node. In other words, if we walk the lists linearly, we encounter a node only after all its dependencies. This key property is used by the compiler to help reason about the graph order.

There are two syntax flavors, “minimal” and the the full syntax. The “minimal” version only rebinds to nodes that are attribute access or slice access. E.g. self.x[self.y].z. The full syntax, allows also e.g. (self.x + other.y).z and would also rebind to the z property of (self.x + other.y). The “minimal” syntax in this case would only rebind to self.x and other.y.

This is used by calling update_graph() once for each rule, where we pass the ast nodes for a rule to that function at once. The idea is to create graphs that represent bindings across all the rules, while also tracking which rule a (leaf) node came from. E.g. if we had two rules - self.x + self.y, and self.x.z + 55 + self.y, then we would create one graph: self -> x, self -> y, x -> z that describes the bindings for the rules. Additionally, we would create a leaf from x, y for the first rule, and z, y for the second. You can see more in the visualization module.

Currently, all comprehensions and lambda are treated as opaque so we don’t bind to anything within it.

current_child_node = None

As we walk the bind graph nodes, for each expression node we create a ASTBindNodeRef that references the node and add it to nodes_by_rule. The node is stored here while we explore its parent tree. Then, all the direct parent’s of this node will point to this node using their ASTBindNodeRef.depend_on_me and this node’s ASTBindNodeRef.depends will point to the parents.

current_rule = None

During the processing of a rule in update_graph(), we store here the rule parameter passed to update_graph(). Its value is used to set ASTBindNodeRef.leaf_rule for leaf nodes.

generic_visit(node, is_attribute=False, is_final_attribute=False)

Called if no explicit visitor function exists for a node.

illegal_parent_check = None

Stores the ParseCheckWhitelisted we use to check for whitelisted nodes when not all classes are whitelisted. Otherwise, it’s None.

leaf_nodes_in_rule = set()

For each rule, while the rule is processed it keeps track of the leaf nodes in this rule we encountered already. This prevent creating duplicate copies of e.g. self.x + self.x for the same rule. So if we already saw this exact leaf in the rule, we don’t add a node for it again.

nodes_by_rule = []

For each rule (each time we call update_graph()), we add a new list here that contains all the ASTBindNodeRef that were created for this rule. Existing nodes that are re-used from previous rules are not added.

The order is such that if we were to linearly walk across all the lists in order in nodes_by_rule, when we a encounter a node, all the parents of the node, up to the roots would have been encountered by the time we reach the node. This means that the node deps are always encountered first. But, unrelated nodes may also be present before reaching the node, even if they are not parents.

nodes_under_leaf = set()

For every rule and for every leaf, this stores all the nodes that we encountered while processing the parents of the leaf.

src_node_map = {}

For each non leaf ASTBindNodeRef node we create (i.e. ASTBindNodeRef.leaf_rule is None), we map the source code representation of the node (ASTBindNodeRef.src) to the node. E.g. for self.x.y we create three reference nodes with the corresponding representation, “self”, “self.x”, and “self.x.y”.

This helps find duplicates across all rules, e.g. for self.x + self.x, we only create two nodes for “self” and “self.x” the first time we encounter them. The second time, we just re-use the existing node for self and increment its ASTBindNodeRef.count. The second time we encounter self.x in this rule, we don’t create a new leaf because leaf_nodes_in_rule tracks the already seen leaf nodes per rule.

src_nodes = []

During each rule, it creates the list of ASTBindNodeRef instances that were created due to this rule. Re-used nodes from previous rules are not added.

update_graph(nodes, rule)

Adds the nodes of the the rule to the existing bind graph.

It creates a new rule list in nodes_by_rule and all nodes newly created due to this rule are added to it.

whitelist_opts = {'minimal': {'Num', 'Name', 'Attribute', 'Subscript', 'Str', 'Bytes', 'NameConstant'}}

For each syntax flavor, the values is a set of the names of AST node classes that is allowed to be “under” a Attribute. If we encounter a non-whitelisted class and illegal_parent_check is not None, the nodes between the Attribute and illegal node would be dropped from the bind graph.

class kivy.lang.compiler.ast_parse.ParseKvFunctionTransformer(kv_syntax=None, context_classes={}, rule_classes={})

Bases: ast.NodeTransformer

Transforms and parses the kv decorated function and creates all the graphs required for the generation of compiled source code.

Notes:

The most important things is that we cannot allow a kv rule to be conditionally executed because we’d be lying to the user.

E.g. a return statement is forbidden within a kv context.

There’s another problem to avoid, having callbacks use variables that are not available anymore or has changed. Therefore, we capture all the variables, local, nonlocal, and global defined before the rule and provide it to the callback (side note, that improves speed). Therefore, some statements, e.g. a function def contains local variables which is too hard to trace, so we don’t allow it within a rule at all as we cannot figure out which variables need to be captured (yet).

This transformer is not allowed to change nodes (except trivially), except for changing kv rule and ctx from with to plain assignment, and similarly from x @= y or x ^= y to x = y.

generic_visit(node)

Called if no explicit visitor function exists for a node.

visit(node)

Visit a node.

class kivy.lang.compiler.ast_parse.DoNothingAST

Bases: _ast.AST

An AST node indicating that nothing should generated in the source code.

class kivy.lang.compiler.ast_parse.ParseExpressionNameLoadTransformer

Bases: ast.NodeTransformer

AST transformer that finds all the Name nodes in the graph.

class kivy.lang.compiler.ast_parse.VerifyKvCaptureOnExitTransformer(first_pass_rules, first_pass_contexts, **kwargs)

Bases: kivy.lang.compiler.ast_parse.ParseKvFunctionTransformer

AST transformer that verifies that captured variables are not modified and are read only, for the case where the bindings occur upon KvContext exit.

class kivy.lang.compiler.ast_parse.VerifyKvCaptureOnEnterTransformer(first_pass_rules, first_pass_contexts, **kwargs)

Bases: kivy.lang.compiler.ast_parse.VerifyKvCaptureOnExitTransformer

AST transformer that verifies that captured variables are not modified and are read only, for the case where the bindings occur upon KvContext enter.

kivy.lang.compiler.ast_parse.verify_readonly_nodes(graph, transformer, bind_on_enter)

Runs the appropriate transformer that verifies that captured and read only variables are not modified.