Products

class ploomber.products.EmptyProduct
delete(force=False)

Deletes the product

exists()

This method returns True if the product exists, it is not part of the metadata, so there is no cached status

class ploomber.products.File(identifier)

A file (or directory) in the local filesystem

Parameters

identifier (str or pathlib.Path) – The path to the file (or directory), can contain placeholders (e.g. {{placeholder}})

delete(force=False)

Deletes the product

exists()

This method returns True if the product exists, it is not part of the metadata, so there is no cached status

class ploomber.products.GenericProduct(identifier, client=None)

GenericProduct is used when there is no specific Product implementation. Sometimes it is technically possible to write a Product implementation but if you don’t want to do it you can use this one. Other times is is not possible to provide a concrete Product implementation (e.g. we cannot store arbitrary metadata in a Hive table). GenericProduct works as any other product but its metadata is stored not in the Product itself but in a different backend.

Parameters
  • identifier (str) – An identifier fot this product, can contain placeholders (e.g. {{placeholder}})

  • client (ploomber.clients.DBAPIClient or SQLAlchemyClient, optional) – The client used to store metadata for this product. Only required if no dag-level client has been declared using dag.clients[class]

Notes

exists does not check for product existence, just checks if metadata exists delete does not perform actual deletion, just deletes metadata

delete(force=False)

Deletes the product

exists()

This method returns True if the product exists, it is not part of the metadata, so there is no cached status

property name

Used as identifier in the database

class ploomber.products.GenericSQLRelation(identifier, client=None)

A GenericProduct whose identifier is a SQL relation, uses SQLite as metadata backend

Parameters
  • identifier (tuple of length 3) – A tuple with (schema, name, kind) where kind must be either ‘table’ or ‘view’

  • client (ploomber.clients.DBAPIClient or SQLAlchemyClient, optional) – The client used to store metadata for this product. Only required if no dag-level client has been declared using dag.clients[class]

property name

Used as identifier in the database

class ploomber.products.MetaProduct(products)

Exposes a Product-like API to allow Tasks to create more than one Product, it is automatically instantiated when a Task is initialized with a sequence or a mapping object in the product parameter. While it is recommended for Tasks to only have one Product (to keep them simple), in some cases it makes sense. For example, a Jupyter notebook (executed via NotebookRunner), for fitting a model might as well serialize the things such as the model and any data preprocessors

to_json_serializable()

Returns a JSON serializable version of this product

class ploomber.products.PostgresRelation(identifier, client=None)

A PostgreSQL relation

Parameters
  • identifier (tuple of length 3) – A tuple with (schema, name, kind) where kind must be either ‘table’ or ‘view’

  • client (ploomber.clients.DBAPIClient or SQLAlchemyClient, optional) – The client used to connect to the database. Only required if no dag-level client has been declared using dag.clients[class]

Examples

>>> from ploomber.products import PostgresRelation
>>> relation = PostgresRelation(('schema', 'some_table', 'table'))
>>> str(relation) # returns qualified name
delete(force=False)

Deletes the product

exists()

This method returns True if the product exists, it is not part of the metadata, so there is no cached status

class ploomber.products.Product(identifier)

Abstract class for all Products

prepare_metadata

A hook to execute before saving metadata, should include a “metadata” parameter and might include “product”. “metadata” will be a dictionary with the metadata to save, it is not recommended to change any of the existing keys but additional key-value pairs might be included

Type

callable

abstract delete(force=False)

Deletes the product

abstract exists()

This method returns True if the product exists, it is not part of the metadata, so there is no cached status

render(params, **kwargs)

Render Product - this will render contents of Templates used as identifier for this Product, if a regular string was passed, this method has no effect

to_json_serializable()

Returns a JSON serializable version of this product

class ploomber.products.SQLRelation(identifier)

A product that represents a SQL relation but has no metadata

Parameters

identifier (tuple of length 3) – A tuple with (schema, name, kind) where kind must be either ‘table’ or ‘view’

delete(force=False)

Deletes the product

exists()

This method returns True if the product exists, it is not part of the metadata, so there is no cached status

class ploomber.products.SQLiteRelation(identifier, client=None)

A SQLite relation

Parameters
  • identifier (tuple of length 3 or 2) – A tuple with (schema, name, kind) where kind must be either ‘table’ or ‘view’. If passed a tuple with length 2, schema is assumed None. Schemas in SQLite represent other databases when using the ATTACH command.

  • client (ploomber.clients.DBAPIClient or SQLAlchemyClient, optional) – The client used to connect to the database. Only required if no dag-level client has been declared using dag.clients[class]

Examples

>>> from ploomber.products import SQLiteRelation
>>> relation = SQLiteRelation(('schema', 'some_table', 'table'))
>>> str(relation) # returns qualified name
delete()

Deletes the product

exists()

This method returns True if the product exists, it is not part of the metadata, so there is no cached status

property name

Used as identifier in the database