How to use compile_commands.json with clang python bindings?

Question

I have the following script that attempts to print out all the AST nodes in a given C++ file. This works fine when using it on a simple file with trivial includes (header file in the same directory, etc).

#!/usr/bin/env python
from argparse import ArgumentParser, FileType
from clang import cindex


def node_info(node):
    return {'kind': node.kind,
            'usr': node.get_usr(),
            'spelling': node.spelling,
            'location': node.location,
            'file': node.location.file.name,
            'extent.start': node.extent.start,
            'extent.end': node.extent.end,
            'is_definition': node.is_definition()
            }


def get_nodes_in_file(node, filename, ls=None):
    ls = ls if ls is not None else []
    for n in node.get_children():
        if n.location.file is not None and n.location.file.name == filename:
            ls.append(n)
            get_nodes_in_file(n, filename, ls)
    return ls


def main():
    arg_parser = ArgumentParser()
    arg_parser.add_argument('source_file', type=FileType('r+'),
                            help='C++ source file to parse.')
    arg_parser.add_argument('compilation_database', type=FileType('r+'),
                            help='The compile_commands.json to use to parse the source file.')
    args = arg_parser.parse_args()
    compilation_database_path = args.compilation_database.name
    source_file_path = args.source_file.name
    clang_args = ['-x', 'c++', '-std=c++11', '-p', compilation_database_path]
    index = cindex.Index.create()
    translation_unit = index.parse(source_file_path, clang_args)
    file_nodes = get_nodes_in_file(translation_unit.cursor, source_file_path)
    print [p.spelling for p in file_nodes]


if __name__ == '__main__':
    main()

However, I get a clang.cindex.TranslationUnitLoadError: Error parsing translation unit. when I run the script and provide a valid C++ file that has a compile_commands.json file in its parent directory. This code runs and builds fine using CMake with clang, but I can't seem to figure out how to pass the argument for pointing to the compile_commands.json correctly.

I also had difficulty finding this option in the clang documentation and could not get -ast-dump to work. However, clang-check works fine by just passing the file path!

Tamás Szelei · Accepted Answer · 2018-09-15T09:53:20.273

13

Your own accepted answer is incorrect. libclang does support compilation databases and so does cindex.py, the libclang python binding.

The main source of confusion might be that the compilation flags that libclang knows/uses are only a subset of all arguments that can be passed to the clang frontend. The compilation database is supported but does not work automatically: it must be loaded and queried manually. Something like this should work:

#!/usr/bin/env python
from argparse import ArgumentParser, FileType
from clang import cindex

compilation_database_path = args.compilation_database.name
source_file_path = args.source_file.name
index = cindex.Index.create()

# Step 1: load the compilation database
compdb = cindex.CompilationDatabase.fromDirectory(compilation_database_path)

# Step 2: query compilation flags
try:
    file_args = compdb.getCompileCommands(source_file_path)
    translation_unit = index.parse(source_file_path, file_args)
    file_nodes = get_nodes_in_file(translation_unit.cursor, source_file_path)
    print [p.spelling for p in file_nodes]
except CompilationDatabaseError:
    print 'Could not load compilation flags for', source_file_path

edited Sep 15 '18 at 09:53

answered Jul 13 '16 at 13:40

Tamás Szelei

23,169
18
105
180

1

What version of clang was this first introduced? When I wrote this question I was using clang 3.4 of both libclang and cindex.py. – Lucas Jul 13 '16 at 20:16
No idea, but I'm pretty sure that at the time it was 3.8 already which had it for ages. – Tamás Szelei Jul 13 '16 at 23:24
Looks like it's in 3.4 as well, must have missed it. – Lucas Jul 14 '16 at 14:36
Great answer. Thank you so much, I'm sure that will help many people. – static_rtti Jul 14 '16 at 17:23
Thanks @static_rtti for the bounty and kind words. – Tamás Szelei Jul 14 '16 at 17:46
@TamásSzelei : could you add imports to make your snippet easier to use? – static_rtti Jul 14 '16 at 21:27
@static_rtti Sure. – Tamás Szelei Jul 15 '16 at 10:13
I think this way of creating the arguments has become deprecated. If anyone can update the answer to reflect the changes to the python API that would be awesome. I copy pasted this as is, and the file_args type seems incompatible with the expected type of the args parameter of the parsing method. – Makogan Jul 12 '21 at 04:57

score 0 · Answer 2 · answered Apr 18 '16 at 15:17

0

From what I can tell Libclang does not support the compilation database but Libtooling does. To get around this I took the path to the compile_commands.json as an argument and ended up parsing it myself to find the file of interest and the relevant includes (the -I and -isystem includes).

answered Apr 18 '16 at 15:17

Lucas

2,514
4
27
37

My answer is not correct, please see the accepted answer. – Lucas Sep 08 '16 at 17:30

score 0 · Answer 3 · answered Jul 12 '21 at 05:07

The accepted answer seems to be deprecated, at minimum it did not work for me, I had to do this:

import clang.cindex


def main():
    index = clang.cindex.Index.create()

    compdb = clang.cindex.CompilationDatabase.fromDirectory(
        "dir/")

    source_file_path = 'path/to/file.cpp'
    commands = compdb.getCompileCommands(source_file_path)

    file_args = []
    for command in commands:
        for argument in command.arguments:
            file_args.append(argument)
    file_args = file_args[3:-3]
    print(file_args)
    translation_unit = index.parse(source_file_path, args=file_args)

    comment_tokens = GetDoxygenCommentTokens(translation_unit)


if __name__ == "__main__":
    main()

Basically I had to iterate over the commands and the arguments to create a string, and then eliminate some g++ specific flags.

How to use compile_commands.json with clang python bindings?

3 Answers3

Linked