Debugging

Note

This is a quick reference, for an in-depth tutorial, click here.

Note

All this section assumes you’re familiar with the Python debugger. See the documentation here

Executing task in debugging mode

To jump to the first line of a task and start a debugging session:

ploomber interact

Then:

dag['task-name'].debug()

Note

.debug() only works with Python functions, scripts, and notebooks.

To get the list of task names: list(dag).

After running .debug(), you’ll start a debugging session. You can use the next command to jump to the next line. Type quit, and hit enter to exit the debugging session.

Post-mortem debugging

Run and start a debugging session as soon as a task raises an exception.

ploomber task {task-name} --debug
ploomber build --debug
changelog

New in version 0.20: Added support for post-mortem debugging in notebooks using --debug

New in version 0.20: Added --debug option to ploomber task

Post-mortem debugging (debug later)

Added in version 0.20

Run the pipeline and serialize errors from all failing tasks for later debugging. This is useful when running tasks in parallel or notebooks overnight.

ploomber task {task-name} --debuglater
ploomber build --debuglater

Then, to start a debugging session:

dltr {task-name}.dump

Once you’re done debugging, you can delete the {task-name}.dump file.

Note

Only built-in objects will be stored in the .dump file, for others, (e.g., pandas data frames, numpy arrays) only the string representation is stored. To serialize all: pip install 'debuglater[all]'

Important

Using --debuglater will serialize all the variables, ensure you have enough disk space, especially if running tasks in parallel.

changelog

New in version 0.20: Added --debuglater option to ploomber task

New in version 0.20: Added --debuglater option to ploomber build

Breakpoints

Note

This only work with Python functions, go to the next section to learn how to debug scripts/notebooks.

Breakpoints allow you to start a debugger at given line:

def my_task(product):
    # debugging session starts here...
    from ipdb import set_trace; set_trace()
    # code continues...

Then:

ploomber build --debug

Debugging in Jupyter/VSCode

If you’re using Jupyter or similar (e.g. notebooks in VSCode), you can debug there.

Post-portem

If your code raises an exception, execute the following in a new cell, and a debugging session will start:

%debug

Click here to see the %pdb documentation.

If you want a debugging session to start whenever your code raises an exception:

%pdb

Note

run %pdb again to turn it off.

Click here to see the %pdb documentation.

Breakpoints

Once you’re in Jupyter, you can add a breakpoint at the line you want to debug:

def some_code_called_from_the_notebook():
    # debugging session starts here...
    from ipdb import set_trace; set_trace()
    # code continues...

The breakpoint can be in a module (i.e., something that you imported using a import statement)

Visual debugger

JupyterLab recently incorporated a native debugger, click here to learn more.