Skip to content

Python plugin implementation #781

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 156 commits into
base: master
Choose a base branch
from
Open

Python plugin implementation #781

wants to merge 156 commits into from

Conversation

wbqpk3
Copy link
Collaborator

@wbqpk3 wbqpk3 commented Feb 27, 2025

This PR implements the Python plugin for CodeCompass.

Plugin components:

Parser:

  • The main parser component is pyparser written in Python, uses the Jedi static analysis library. Moreover, pyparser also parses the Python AST in order to achieve better results.
  • During parsing, the pyparser module gets loaded using the Python C API
  • The nodes are inserted to a database table named PYName

Service:

  • PythonService implements most Language Service functionality with a few missing
  • PythonDiagram class to generate the following diagrams: function call, class overview, usage, module dependency diagram.

WebGUI:

  • It mainly targets the legacy WebGUI, however Info Tree functionality works fine in the new WebGUI

Tests:

  • Implemented test for the Parser and Service components.

Documentation:

  • Added documentation to doc/pythonplugin.md.

The missing part is the diagrams in the new WebGUI, however the diagrams component file needs some refactoring to use the new language-service instead of CppService (similarly to #759).
I think we should implement that in a separate PR.

@wbqpk3 wbqpk3 added Kind: Enhancement 🌟 Plugin: Python Issues related to the parsing and presentation of Python projects. labels Feb 27, 2025
@mcserep
Copy link
Collaborator

mcserep commented Mar 4, 2025

@wbqpk3 Nice and large 😅 work, I will try to find some time to review it.

Meanwhile please rebase it to master, as some used GitHub Actions had to be updated for the CI to run. So we can see whether the CI passes or not.

@wbqpk3
Copy link
Collaborator Author

wbqpk3 commented Mar 4, 2025

@mcserep Rebased to master.

@mcserep mcserep requested review from mcserep and Copilot April 8, 2025 04:14
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 40 out of 46 changed files in this pull request and generated 5 comments.

Files not reviewed (6)
  • Config.cmake: Language not supported
  • plugins/python/CMakeLists.txt: Language not supported
  • plugins/python/model/CMakeLists.txt: Language not supported
  • plugins/python/parser/.gitignore: Language not supported
  • plugins/python/parser/CMakeLists.txt: Language not supported
  • plugins/python/parser/requirements.txt: Language not supported

Comment on lines +28 to +31
except:
if self.config.debug:
log(f"{bcolors.FAIL}Failed to find definition! (file = {str(name.module_path)} line = {name.line} column = {name.column})")
if self.config.stack_trace:
Copy link
Preview

Copilot AI Apr 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Avoid using a bare except; replace 'except:' with 'except Exception as e:' to prevent unintentionally catching system-exiting exceptions.

Suggested change
except:
if self.config.debug:
log(f"{bcolors.FAIL}Failed to find definition! (file = {str(name.module_path)} line = {name.line} column = {name.column})")
if self.config.stack_trace:
except Exception as e:
if self.config.debug:
log(f"{bcolors.FAIL}Failed to find definition! (file = {str(name.module_path)} line = {name.line} column = {name.column}) Exception: {e}")

Copilot uses AI. Check for mistakes.

Comment on lines +56 to +59
except:
if self.config.debug:
log(f"{bcolors.FAIL}Failed to find references! (file = {str(x.module_path)} line = {x.line} column = {x.column})")
if self.config.stack_trace:
Copy link
Preview

Copilot AI Apr 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Avoid a bare except block here; use 'except Exception as e:' to ensure only anticipated exceptions are caught.

Suggested change
except:
if self.config.debug:
log(f"{bcolors.FAIL}Failed to find references! (file = {str(x.module_path)} line = {x.line} column = {x.column})")
if self.config.stack_trace:
except Exception as e:
if self.config.debug:
log(f"{bcolors.FAIL}Failed to find references! (file = {str(x.module_path)} line = {x.line} column = {x.column}) Exception: {str(e)}")

Copilot uses AI. Check for mistakes.

Comment on lines +1 to +8
from dataclasses import dataclass

@dataclass
class ParseResult:
path: str
status: str = "full"
nodes = []
imports = []
Copy link
Preview

Copilot AI Apr 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a mutable default (an empty list) in a dataclass can lead to unexpected side effects; consider using field(default_factory=list).

Suggested change
from dataclasses import dataclass
@dataclass
class ParseResult:
path: str
status: str = "full"
nodes = []
imports = []
from dataclasses import dataclass, field
@dataclass
class ParseResult:
path: str
status: str = "full"
nodes: list = field(default_factory=list)
imports: list = field(default_factory=list)

Copilot uses AI. Check for mistakes.

Comment on lines +1 to +8
from dataclasses import dataclass

@dataclass
class ParseResult:
path: str
status: str = "full"
nodes = []
imports = []
Copy link
Preview

Copilot AI Apr 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Avoid using a mutable default value by replacing the direct assignment with field(default_factory=list) to ensure instance isolation.

Suggested change
from dataclasses import dataclass
@dataclass
class ParseResult:
path: str
status: str = "full"
nodes = []
imports = []
from dataclasses import dataclass, field
@dataclass
class ParseResult:
path: str
status: str = "full"
nodes: list = field(default_factory=list)
imports: list = field(default_factory=list)

Copilot uses AI. Check for mistakes.

try:
tree = ast.parse(source)
self.astNodes = list(ast.walk(tree))
except:
Copy link
Preview

Copilot AI Apr 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace the bare except with 'except Exception:' to catch only the expected exceptions during AST parsing.

Suggested change
except:
except Exception:

Copilot uses AI. Check for mistakes.

@mcserep
Copy link
Collaborator

mcserep commented Apr 8, 2025

Copilot reviewed 40 out of 46 changed files in this pull request and generated 5 comments.

Files not reviewed (6)

@wbqpk3 I just tried out this Copilot-based review feature of GitHub. The suggestions maybe worth a check, but feel free to ignore them in case they look unimportant.

@mcserep
Copy link
Collaborator

mcserep commented Apr 8, 2025

@wbqpk3 The CI fails for Ubuntu 20.04. While we should drop support for Ubuntu 20.04, it should still be checked why the build fails there.

[ 84%] Building CXX object plugins/python/test/CMakeFiles/pythonservicetest.dir/src/pythonservicetest.cpp.o
In file included from /home/runner/work/CodeCompass/CodeCompass/plugins/python/service/include/service/pythonservice.h:15,
from /home/runner/work/CodeCompass/CodeCompass/plugins/python/test/src/pythonservicetest.cpp:4:
/home/runner/cc-build/service/language/gen-cpp/LanguageService.h:10:10: fatal error: thrift/TDispatchProcessor.h: No such file or directory
10 | #include <thrift/TDispatchProcessor.h>
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [plugins/python/test/CMakeFiles/pythonservicetest.dir/build.make:93: plugins/python/test/CMakeFiles/pythonservicetest.dir/src/pythonservicetest.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:3021: plugins/python/test/CMakeFiles/pythonservicetest.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

@mcserep
Copy link
Collaborator

mcserep commented Apr 8, 2025

@wbqpk3 The CI fails for Ubuntu 20.04. While we should drop support for Ubuntu 20.04, it should still be checked why the build fails there.

UPDATE: CI jobs for Ubuntu 20.04 already started to skip, due to image unavailability. I have disabled the CI build for Ubuntu 20.04 in b9e6733, and added an issue #789 to deprecate it completely.

Fixing this CI fail is not required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Kind: Enhancement 🌟 Plugin: Python Issues related to the parsing and presentation of Python projects.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants