Introduces MAGIC command parsing for MATLAB Kernels.

Author: Shushant Singh

README.md

@@ -1,6 +1,6 @@
 # MATLAB Integration _for Jupyter_
 
-[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/mathworks/jupyter-matlab-proxy/run-tests.yml?branch=main&logo=github)](https://github.com/mathworks/jupyter-matlab-proxy/actions) [![PyPI badge](https://img.shields.io/pypi/v/jupyter-matlab-proxy.svg?logo=pypi)](https://pypi.python.org/pypi/jupyter-matlab-proxy) [![codecov](https://codecov.io/gh/mathworks/jupyter-matlab-proxy/branch/main/graph/badge.svg?token=ZW3SESKCSS)](https://codecov.io/gh/mathworks/jupyter-matlab-proxy) [![Downloads](https://static.pepy.tech/personalized-badge/jupyter-matlab-proxy?period=month&units=international_system&left_color=grey&right_color=blue&left_text=PyPI%20downloads/month)](https://pepy.tech/project/jupyter-matlab-proxy)
+[![PyPI badge](https://img.shields.io/pypi/v/jupyter-matlab-proxy.svg?logo=pypi)](https://pypi.python.org/pypi/jupyter-matlab-proxy) [![codecov](https://codecov.io/gh/mathworks/jupyter-matlab-proxy/branch/main/graph/badge.svg?token=ZW3SESKCSS)](https://codecov.io/gh/mathworks/jupyter-matlab-proxy) [![Downloads](https://static.pepy.tech/personalized-badge/jupyter-matlab-proxy?period=month&units=international_system&left_color=grey&right_color=blue&left_text=PyPI%20downloads/month)](https://pepy.tech/project/jupyter-matlab-proxy)
 
 
 
@@ -34,7 +34,7 @@ From your Jupyter notebook or JupyterLab, you can also open the MATLAB developme
   - Linux®
   - MacOS
   - Windows® (supported from [v0.6.0](https://github.com/mathworks/jupyter-matlab-proxy/releases/tag/v0.6.0)).
-  - Windows Subsystem for Linux (WSL 2) [Installation Guide](./install_guides/wsl2/README.md).
+  - Windows Subsystem for Linux (WSL 2) [Installation Guide](https://github.com/mathworks/jupyter-matlab-proxy/blob/main/install_guides/wsl2/README.md).
 
 * Python versions: 3.8 | 3.9 | 3.10 | 3.11
 
@@ -148,11 +148,16 @@ This opens a Jupyter notebook that supports MATLAB.
 <p align="center"><img width="600" src="https://github.com/mathworks/jupyter-matlab-proxy/raw/main/img/jupyterlab-notebook.png"></p>
 
 
-- When you execute MATLAB code in a notebook for the first time, enter your MATLAB license information in the dialog box that appears. See [Licensing](https://github.com/mathworks/matlab-proxy/blob/main/MATLAB-Licensing-Info.md) for details. The MATLAB session can take a few minutes to start.
-- Multiple notebooks running on a Jupyter server share the underlying MATLAB process, so executing code in one notebook affects the workspace in others. If you work in several notebooks simultaneously, be aware that they share a workspace.
-- With MATLAB R2022b and later, you can define a local function at the end of the cell where you want to call it:
+### Notes
+
+- **Licensing:** When you execute MATLAB code in a notebook for the first time, enter your MATLAB license information in the dialog box that appears. For details, see [Licensing](https://github.com/mathworks/matlab-proxy/blob/main/MATLAB-Licensing-Info.md). The MATLAB session can take a few minutes to start.
+- **Multiple notebooks:** Multiple notebooks running on a Jupyter server share the underlying MATLAB process, so executing code in one notebook affects the workspace in others. If you work in several notebooks simultaneously, be aware that they share a workspace.
+- **Local functions:** with MATLAB R2022b and later, you can define a local function at the end of the cell where you want to call it:
     <p><img width="350" src="https://github.com/mathworks/jupyter-matlab-proxy/raw/main/img/local_functions.png"></p>
-For technical details about how the MATLAB kernel works, see [MATLAB Kernel for Jupyter](https://github.com/mathworks/jupyter-matlab-proxy/blob/main/src/jupyter_matlab_kernel/README.md).
+
+- **Magic Commands:** You can use predefined magic commands in a Jupyter notebook with the MATLAB kernel, and you can also implement your own. To see a list of predefined magic commands, run `%%lsmagic`. For details about using magic commands, see [Magic Commands for MATLAB Kernel](https://github.com/mathworks/jupyter-matlab-proxy/blob/main/src/jupyter_matlab_kernel/magics/README.md).
+
+- **Kernel:** For technical details about the MATLAB kernel, see [MATLAB Kernel for Jupyter](https://github.com/mathworks/jupyter-matlab-proxy/blob/main/src/jupyter_matlab_kernel/README.md).
 
 ## Open MATLAB in a Browser
 
@@ -189,8 +194,6 @@ This opens an untitled `.m` file where you can write MATLAB code with syntax hig
 * Currently, this package allows you to edit MATLAB `.m` files but not to execute them.
 * To open a new MATLAB `.m` file, you can also use the JupyterLab command palette. Press `CTRL+SHIFT+C`, then type `New MATLAB File` and press `Enter`.
 
-
-
 ## Limitations
 
 * This package has limitations. For example, it does not support certain MATLAB commands. For details, see [Limitations](https://github.com/mathworks/jupyter-matlab-proxy/blob/main/limitations.md).

src/jupyter_matlab_kernel/kernel.py

@@ -6,34 +6,26 @@
 import sys
 import time
 
-# Import Dependencies
+# Import Third-Party Dependencies
 import ipykernel.kernelbase
 import psutil
 import requests
-from matlab_proxy import settings as mwi_settings
-from matlab_proxy import util as mwi_util
 from requests.exceptions import HTTPError
 
+# Import Dependencies
 from jupyter_matlab_kernel import mwi_comm_helpers, mwi_logger
+from jupyter_matlab_kernel.magic_execution_engine import (
+    MagicExecutionEngine,
+    get_completion_result_for_magics,
+)
+from jupyter_matlab_kernel.mwi_exceptions import MATLABConnectionError
+from matlab_proxy import settings as mwi_settings
+from matlab_proxy import util as mwi_util
 
 _MATLAB_STARTUP_TIMEOUT = mwi_settings.get_process_startup_timeout()
 _logger = mwi_logger.get()
 
 
-class MATLABConnectionError(Exception):
-    """
-    A connection error occurred while connecting to MATLAB.
-
-    Args:
-        message (string): Error message to be displayed
-    """
-
-    def __init__(self, message=None):
-        if message is None:
-            message = 'Error connecting to MATLAB. Check the status of MATLAB by clicking the "Open MATLAB" button. Retry after ensuring MATLAB is running successfully'
-        super().__init__(message)
-
-
 def is_jupyter_testing_enabled():
     """
     Checks if testing mode is enabled
@@ -174,7 +166,7 @@ def start_matlab_proxy(logger=_logger):
             break
 
     # Error out if the server is not found!
-    if found_nb_server == False:
+    if not found_nb_server:
         logger.error("Jupyter server associated with this MATLABKernel not found.")
         raise MATLABConnectionError(
             """
@@ -224,7 +216,7 @@ def start_matlab_proxy(logger=_logger):
             return matlab_proxy_url, nb_server["base_url"], headers
 
     logger.error(
-        f"MATLABKernel could not communicate with matlab-proxy through Jupyter server"
+        "MATLABKernel could not communicate with matlab-proxy through Jupyter server"
     )
     logger.error(f"Jupyter server:\n{nb_server}")
     raise MATLABConnectionError(
@@ -267,6 +259,8 @@ def __init__(self, *args, **kwargs):
         # multiple kernels which are running simultaneously
         self.log.debug(f"Initializing kernel with id: {self.ident}")
         self.log = self.log.getChild(f"{self.ident}")
+        # Initialize the Magic Execution Engine.
+        self.magic_engine = MagicExecutionEngine(self.log)
 
         try:
             # Start matlab-proxy using the jupyter-matlab-proxy registered endpoint.
@@ -312,6 +306,29 @@ async def interrupt_request(self, stream, ident, parent):
 
         self.session.send(stream, "interrupt_reply", content, parent, ident=ident)
 
+    def modify_kernel(self, states_to_modify):
+        """
+        Used to modify MATLAB Kernel state
+        Args:
+            states_to_modify (dict): A key value pair of all the states to be modified.
+
+        """
+        self.log.debug(f"Modifying the kernel with {states_to_modify}")
+        for key, value in states_to_modify.items():
+            if hasattr(self, key):
+                self.log.debug(f"set the value of {key} to {value}")
+                setattr(self, key, value)
+
+    def handle_magic_output(self, output, outputs=None):
+        if output["type"] == "modify_kernel":
+            self.modify_kernel(output)
+        else:
+            self.display_output(output)
+            if outputs is not None and not self.startup_checks_completed:
+                # Outputs are cleared after startup_check.
+                # Storing the magic outputs to display them after startup_check completes.
+                outputs.append(output)
+
     def do_execute(
         self,
         code,
@@ -328,44 +345,71 @@ def do_execute(
         """
         self.log.debug(f"Received execution request from Jupyter with code:\n{code}")
         try:
+            accumulated_magic_outputs = []
+            performed_startup_checks = False
+
+            for output in self.magic_engine.process_before_cell_execution(
+                code, self.execution_count
+            ):
+                self.handle_magic_output(output, accumulated_magic_outputs)
+
+            skip_cell_execution = self.magic_engine.skip_cell_execution()
+            self.log.debug(f"Skipping cell execution is set to {skip_cell_execution}")
+
             # Complete one-time startup checks before sending request to MATLAB.
             # Blocking call, returns after MATLAB is started.
-            if not self.startup_checks_completed:
-                self.perform_startup_checks()
-                self.display_output(
-                    {
-                        "type": "stream",
-                        "content": {
-                            "name": "stdout",
-                            "text": "Executing ...",
-                        },
-                    }
+            if not skip_cell_execution:
+                if not self.startup_checks_completed:
+                    self.perform_startup_checks()
+                    self.display_output(
+                        {
+                            "type": "stream",
+                            "content": {
+                                "name": "stdout",
+                                "text": "Executing ...",
+                            },
+                        }
+                    )
+                    if accumulated_magic_outputs:
+                        self.display_output(
+                            {"type": "clear_output", "content": {"wait": False}}
+                        )
+                    performed_startup_checks = True
+                    self.startup_checks_completed = True
+
+                if performed_startup_checks and accumulated_magic_outputs:
+                    for output in accumulated_magic_outputs:
+                        self.display_output(output)
+
+                # Perform execution and categorization of outputs in MATLAB. Blocks
+                # until execution results are received from MATLAB.
+                outputs = mwi_comm_helpers.send_execution_request_to_matlab(
+                    self.murl, self.headers, code, self.ident, self.log
                 )
-                self.startup_checks_completed = True
 
-            # Perform execution and categorization of outputs in MATLAB. Blocks
-            # until execution results are received from MATLAB.
-            outputs = mwi_comm_helpers.send_execution_request_to_matlab(
-                self.murl, self.headers, code, self.ident, self.log
-            )
+                if performed_startup_checks and not accumulated_magic_outputs:
+                    self.display_output(
+                        {"type": "clear_output", "content": {"wait": False}}
+                    )
 
-            self.log.debug(
-                "Received outputs after execution in MATLAB. Clearing output area"
-            )
+                self.log.debug(
+                    "Received outputs after execution in MATLAB. Clearing output area"
+                )
+
+                # Display all the outputs produced during the execution of code.
+                for idx in range(len(outputs)):
+                    data = outputs[idx]
+                    self.log.debug(f"Displaying output {idx+1}:\n{data}")
 
-            # Clear the output area of the current cell. This removes any previous
-            # outputs before publishing new outputs.
-            self.display_output({"type": "clear_output", "content": {"wait": False}})
+                    # Ignore empty values returned from MATLAB.
+                    if not data:
+                        continue
+                    self.display_output(data)
 
-            # Display all the outputs produced during the execution of code.
-            for idx in range(len(outputs)):
-                data = outputs[idx]
-                self.log.debug(f"Displaying output {idx+1}:\n{data}")
+            # Execute post execution of MAGICs
+            for output in self.magic_engine.process_after_cell_execution():
+                self.handle_magic_output(output)
 
-                # Ignore empty values returned from MATLAB.
-                if not data:
-                    continue
-                self.display_output(data)
         except Exception as e:
             self.log.error(
                 f"Exception occurred while processing execution request:\n{e}"
@@ -380,8 +424,12 @@ def do_execute(
                 # checks for subsequent execution requests
                 self.startup_checks_completed = False
 
+            # Clearing lingering message "Executing..." before displaying the error message
+            if performed_startup_checks and not accumulated_magic_outputs:
+                self.display_output(
+                    {"type": "clear_output", "content": {"wait": False}}
+                )
             # Send the exception message to the user.
-            self.display_output({"type": "clear_output", "content": {"wait": False}})
             self.display_output(
                 {
                     "type": "stream",
@@ -421,19 +469,31 @@ def do_complete(self, code, cursor_pos):
 
         # Fetch tab completion results. Blocks untils either tab completion
         # results are received from MATLAB or communication with MATLAB fails.
-        try:
-            completion_results = mwi_comm_helpers.send_completion_request_to_matlab(
-                self.murl, self.headers, code, cursor_pos, self.log
-            )
-        except (MATLABConnectionError, HTTPError) as e:
-            self.log.error(
-                f"Exception occurred while sending shutdown request to MATLAB:\n{e}"
-            )
+
+        magic_completion_results = get_completion_result_for_magics(
+            code, cursor_pos, self.log
+        )
 
         self.log.debug(
-            f"Received completion results from MATLAB:\n{completion_results}"
+            f"Received Completion results from MAGIC:\n{magic_completion_results}"
         )
 
+        if magic_completion_results:
+            completion_results = magic_completion_results
+        else:
+            try:
+                completion_results = mwi_comm_helpers.send_completion_request_to_matlab(
+                    self.murl, self.headers, code, cursor_pos, self.log
+                )
+            except (MATLABConnectionError, HTTPError) as e:
+                self.log.error(
+                    f"Exception occurred while sending shutdown request to MATLAB:\n{e}"
+                )
+
+            self.log.debug(
+                f"Received completion results from MATLAB:\n{completion_results}"
+            )
+
         return {
             "status": "ok",
             "matches": completion_results["matches"],
@@ -551,7 +611,7 @@ def perform_startup_checks(self):
                             "type": "stream",
                             "content": {
                                 "name": "stdout",
-                                "text": f"Starting MATLAB ...\n",
+                                "text": "Starting MATLAB ...\n",
                             },
                         }
                     )