Integrazion of DataFrameModel with supported pydantic types

[zarch]

Dear all,

I would like to define a DataFrameModel that is using pydantic types.
From the documentation I did not see an example, but from my understanding of the documentation it should be possible.
However, when I execute the code I get an error that I don't know how to fix.

"""Test integration pandera and pydantic types."""
from io import StringIO

import pandas as pd
import pandera as pa
from pandera.typing import DataFrame, Index, Series
from pydantic import AnyHttpUrl, ByteSize


class DatasetLinks(pa.DataFrameModel):

    """Define a DataFrameModel."""

    Location: Series[str] = pa.Field(
        nullable=False,
        coerce=True,
    )
    QuadKey: Series[int] = pa.Field(
        nullable=False,
        coerce=True,
    )
    Url: Series[AnyHttpUrl] = pa.Field(
        nullable=False,
        coerce=True,
    )
    Size: Series[ByteSize] = pa.Field(
        nullable=False,
        coerce=True,
    )


@pa.check_types
def dataset_links() -> DataFrame[DatasetLinks]:
    """Return a validated dataframe of type DatasetLinks."""
    return pd.read_csv(
        StringIO(
            """Location,QuadKey,Url,Size
Abyei,122321003,https://minedbuildings.blob.core.windows.net/global-buildings/2023-10-16/global-buildings.geojsonl/RegionName%3DAbyei/quadkey%3D122321003/part-00198-1b124b55-19d6-4754-9116-1c5a66bed369.c000.csv.gz,4.8KB
Abyei,122321021,https://minedbuildings.blob.core.windows.net/global-buildings/2023-10-16/global-buildings.geojsonl/RegionName%3DAbyei/quadkey%3D122321021/part-00173-1b124b55-19d6-4754-9116-1c5a66bed369.c000.csv.gz,7.8KB
Afghanistan,123011320,https://minedbuildings.blob.core.windows.net/global-buildings/2023-10-16/global-buildings.geojsonl/RegionName%3DAfghanistan/quadkey%3D123011320/part-00103-1b124b55-19d6-4754-9116-1c5a66bed369.c000.csv.gz,69.3KB
Afghanistan,123011321,https://minedbuildings.blob.core.windows.net/global-buildings/2023-10-16/global-buildings.geojsonl/RegionName%3DAfghanistan/quadkey%3D123011321/part-00142-1b124b55-19d6-4754-9116-1c5a66bed369.c000.csv.gz,1.3MB
""",
        ),
    )

if __name__ == "__main__":
    print(dataset_links())

The return error is:

$ python src.py
Traceback (most recent call last):
  File "src.py", line 47, in <module>
    print(dataset_links())
          ^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/pandera/decorators.py", line 755, in _wrapper
    return _check_arg("return", out)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/pandera/decorators.py", line 639, in _check_arg
    schema = schema_model.to_schema()
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/pandera/api/pandas/model.py", line 256, in to_schema
    columns, index = cls._build_columns_index(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/pandera/api/pandas/model.py", line 357, in _build_columns_index
    dtype = annotation.arg(**dtype_kwargs)  # type: ignore
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for url
  URL input should be a string or URL [type=url_type, input_value=UrlConstraints(max_length...None, default_path=None), input_type=UrlConstraints]
    For further information visit https://errors.pydantic.dev/2.4/v/url_type

The dtype_kwargs variable is:

{
    'url': UrlConstraints(
        max_length=None, 
        allowed_schemes=['http', 'https'], 
        host_required=None, 
        default_host=None, 
        default_port=None, 
        default_path=None
    )
}

While the annotation.arg is: <class 'pydantic_core._pydantic_core.Url'>.

import pandas as pd
import pandera as pa
from pandera.typing import DataFrame, Index, Series
from pydantic import AnyHttpUrl, UrlConstraints

dtype_kwargs = {
    'url': UrlConstraints(
        max_length=None,
        allowed_schemes=['http', 'https'],
        host_required=None,
        default_host=None,
        default_port=None,
        default_path=None
    )
}

# raise the same error:
# ValidationError: 1 validation error for url
#   URL input should be a string or URL [type=url_type, input_value=UrlConstraints(max_length...None, default_path=None), input_type=UrlConstraints]
#     For further information visit https://errors.pydantic.dev/2.4/v/url_type
AnyHttpUrl(**dtype_kwargs)

Is it pandas that is not handling correctly the data type annotations?

[cosmicBboy]

Hi @zarch direct use of pydantic types is currently not supported. Today you can only do row-wise validation using a pydantic model: https://pandera.readthedocs.io/en/stable/pydantic_integration.html#using-pydantic-models-in-pandera-schemas

Feel free to open up an issue and create a PR for this!

Basically it boils down to implementing a pandera type in pandas_engine.py, see this guide for more info

[zarch]

Hi @cosmicBboy ,
sorry I've missed the documentation page :-P.

Regarding the possible PR, basically I should add for each PydanticType a class as done for instance for pd.Period:

@Engine.register_dtype
@immutable(init=True)
class Period(DataType):
    """Representation of pandas :class:`pd.Period`."""

    type: pd.PeriodDtype = dataclasses.field(default=None, init=False)  # type: ignore[assignment]  # noqa
    freq: Union[str, pd.tseries.offsets.DateOffset]

    def __post_init__(self):
        object.__setattr__(self, "type", pd.PeriodDtype(freq=self.freq))

    @classmethod
    def from_parametrized_dtype(cls, pd_dtype: pd.PeriodDtype):
        """Convert a :class:`pandas.PeriodDtype` to
        a Pandera :class:`pandera.engines.pandas_engine.Period`."""
        return cls(freq=pd_dtype.freq)  # type: ignore

Is it correct? If yes, I can play a bit to understand what I can do.