Debugging¶
Note
This is a quick reference, for an in-depth tutorial, click here.
Note
All this section assumes you’re familiar with the Python debugger. See the documentation here
Executing task in debugging mode¶
To jump to the first line of a task and start a debugging session:
ploomber interact
Then:
dag['task-name'].debug()
Note
.debug()
only works with Python functions, scripts, and notebooks.
To get the list of task names: list(dag)
.
After running .debug()
, you’ll start a debugging session. You can use
the next
command to jump to the next line. Type quit
, and hit enter
to exit the debugging session.
Post-mortem debugging¶
Run and start a debugging session as soon as a task raises an exception.
ploomber task {task-name} --debug
ploomber build --debug
changelog
New in version 0.20: Added support for post-mortem debugging in notebooks using --debug
New in version 0.20: Added --debug
option to ploomber task
Post-mortem debugging (debug later)¶
Added in version 0.20
Run the pipeline and serialize errors from all failing tasks for later debugging. This is useful when running tasks in parallel or notebooks overnight.
ploomber task {task-name} --debuglater
ploomber build --debuglater
Then, to start a debugging session:
dltr {task-name}.dump
Once you’re done debugging, you can delete the {task-name}.dump
file.
Note
Only built-in objects will be stored in the .dump
file, for others,
(e.g., pandas data frames, numpy arrays) only the string representation
is stored. To serialize all: pip install 'debuglater[all]'
Important
Using --debuglater
will serialize all the variables, ensure you have
enough disk space, especially if running tasks in parallel.
changelog
New in version 0.20: Added --debuglater
option to ploomber task
New in version 0.20: Added --debuglater
option to ploomber build
Breakpoints¶
Note
This only work with Python functions, go to the next section to learn how to debug scripts/notebooks.
Breakpoints allow you to start a debugger at given line:
def my_task(product):
# debugging session starts here...
from ipdb import set_trace; set_trace()
# code continues...
Then:
ploomber build --debug
Debugging in Jupyter/VSCode¶
If you’re using Jupyter or similar (e.g. notebooks in VSCode), you can debug there.
Post-portem¶
If your code raises an exception, execute the following in a new cell, and a debugging session will start:
%debug
Click here to see the %pdb
documentation.
If you want a debugging session to start whenever your code raises an exception:
%pdb
Note
run %pdb
again to turn it off.
Click here to see the %pdb
documentation.
Breakpoints¶
Once you’re in Jupyter, you can add a breakpoint at the line you want to debug:
def some_code_called_from_the_notebook():
# debugging session starts here...
from ipdb import set_trace; set_trace()
# code continues...
The breakpoint can be in a module (i.e., something that you imported
using a import
statement)
Visual debugger¶
JupyterLab recently incorporated a native debugger, click here to learn more.