Fields

pagemodel.fields

Fields package for pagemodel.

This package provides the building blocks for declarative HTML/XML extraction.

Core descriptors
  • BaseField – abstract base class for all field descriptors, implementing caching, pipeline support, and export registration.
  • Field – unified field descriptor that replaces all specialised types from earlier versions (TextField, AttrField, etc.). Supports CSS and XPath selectors, text/attribute extraction, JSON parsing with optional Pydantic validation, fragment wrapping, and multiple-value modes.
  • XpathField – specialised descriptor for raw XPath queries, returning the exact result of doc.xpath() without automatic text extraction.
Fragment support
  • fragment – class decorator that turns a plain class into an HtmlFragment and wraps it in a FragmentDescriptor for use on a page.
  • FragmentDescriptor – descriptor that extracts elements matching a CSS selector, instantiates the associated fragment class, and caches the result.
Utilities
  • export – decorator that marks a method for inclusion in export() output.

All public symbols are re‑exported from the package root for convenience.

BaseField

Base descriptor for all field types.

Provides caching, pipeline support, automatic registration for export, and low‑level CSS/XPath query execution. Concrete field classes must implement _get(instance, owner).

Methods:
__get__ 
__get__(instance, owner)

Extract the value on first access, apply the pipeline, and cache.

__or__ 
__or__(other)

Add a pipeline step (or pipe) to the field.

Field

Universal field descriptor.

Automatically detects CSS vs XPath selectors, pre‑compiles them, and generates a branch‑free extractor for maximum speed.

Parameters:
  • selector (str) –

    CSS selector, XPath expression (//h1), or xpath://h1.

  • attr (str, default: None ) –

    If set, extract an attribute instead of text.

  • json (bool, default: False ) –

    If True, parse text/attribute content as JSON.

  • model (type, default: None ) –

    Pydantic model class to validate JSON data (only used with json=True).

  • multiple (bool, default: False ) –

    If True, return a list of values (text/attr/JSON) instead of a single one.

  • fragment (type[BaseFragment], default: None ) –

    If set, wrap each selected element into the given fragment class.

  • index (int or None, default: 0 ) –

    Index of the element to pick (0-based). None returns all matched elements (as a list of fragments if fragment is set, otherwise list of strings). Ignored when multiple=True.

  • default (any, default: None ) –

    Value returned when the selector matches nothing or index is out of range.

  • namespaces (dict, default: None ) –

    Namespace prefix-to-URI mapping for XPath expressions.

  • converter (callable, Pipe, or None, default: None ) –

    Applied to the extracted value (or each value when multiple=True).

  • is_exported (bool, default: True ) –

    Whether to include this field in export(). Default True.

FragmentDescriptor

Descriptor for @fragment decorated classes. Caches the extracted fragment(s) on the page instance.

Parameters:
  • selector (str) –

    CSS selector (relative to the page document) that matches the elements to be wrapped.

  • fragment_cls (type[HtmlFragment]) –

    The fragment class (a subclass of HtmlFragment) to instantiate for each matched element.

  • multiple (bool, default: False ) –

    If True, return a list of fragments (one per matched element). Otherwise return a single fragment or default.

  • default (any, default: None ) –

    Value returned when no elements are matched (only relevant when multiple=False).

  • name (str or None, default: None ) –

    Explicit name to use in export() output. If None, the export name is derived from the class attribute name by converting it to snake_case.
fragment_cls property 
fragment_cls

The fragment class used to wrap matched elements.

Methods:
__get__ 
__get__(instance, owner)

Extract fragment(s) on first access and cache the result in the instance dictionary.

XpathField

Extract data using an XPath expression.

Returns the raw result of doc.xpath(selector), typically a list of strings or elements. Use index to pick a specific value.

Parameters:
  • selector (str) –

    XPath expression (e.g. //h1/text(), //div/@data-value).

  • index (int or None, default: 0 ) –

    If None, return the whole list. If an integer, return the element at that index (default 0). If out of range, return default.

  • default (any, default: None ) –

    Value returned when the XPath matches nothing or index is out of range.

  • namespaces (dict, default: None ) –

    Namespace prefix-to-URI mapping for the XPath expression.

  • converter (callable or Pipe, default: None ) –

    Applied to the final value (before returning).

  • is_exported (bool, default: True ) –

    Whether to include this field in export() output. Default True.

Functions:

export 

export(arg)

Decorator that marks a method for inclusion in export().

Can be used without arguments::

@export
def computed_value(self):
    return ...

or with a custom export name::

@export("custom_name")
def some_method(self):
    return ...

fragment 

fragment(
    selector, *, multiple=False, default=None, name=None
)

Class decorator that turns a plain class into an HtmlFragment and returns a FragmentDescriptor for use on a BasePage.

Usage::

class Page(BasePage):
    @fragment(".product", multiple=True)
    class Product:
        name: str = ".name"
        price: float = ".price @data-value"
Parameters:
  • selector (str) –

    CSS selector (relative to the page document) that matches the elements to be wrapped.

  • multiple (bool, default: False ) –

    If True, the descriptor will return a list of fragments (one per matched element). Otherwise it returns a single fragment or default.

  • default (any, default: None ) –

    Value returned when no elements are matched (only relevant when multiple=False).

  • name (str or None, default: None ) –

    Explicit name to use in export() output. If None, the export name is derived from the class attribute name by converting it to snake_case.