Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ibis check backend #1831

Merged
merged 6 commits into from
Dec 10, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions pandera/backends/ibis/checks.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
from pandera.api.checks import Check
from pandera.api.ibis.types import IbisData
from pandera.backends.base import BaseCheckBackend

from pandera.backends.ibis.utils import select_column
from pandera.constants import CHECK_OUTPUT_KEY

CHECK_OUTPUT_SUFFIX = f"__{CHECK_OUTPUT_KEY}__"
Expand Down Expand Up @@ -52,7 +52,9 @@ def apply(self, check_obj: IbisData):
"""Apply the check function to a check object."""
if self.check.element_wise:
selector = (
s.cols(check_obj.key) if check_obj.key is not None else s.all()
select_column(check_obj.key)
if check_obj.key is not None
else s.all()
)
out = check_obj.table.mutate(
s.across(
Expand Down Expand Up @@ -134,7 +136,8 @@ def postprocess(
"""Postprocesses the result of applying the check function."""
passed = check_output[CHECK_OUTPUT_KEY].all()
failure_cases = check_output.filter(~_[CHECK_OUTPUT_KEY]).drop(
s.endswith(f"__{CHECK_OUTPUT_KEY}__") | s.cols(CHECK_OUTPUT_KEY)
s.endswith(f"__{CHECK_OUTPUT_KEY}__")
| select_column(CHECK_OUTPUT_KEY)
)

if check_obj.key is not None:
Expand Down
10 changes: 10 additions & 0 deletions pandera/backends/ibis/utils.py
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I personally don't love that have to define a function abstraction like this, but not sure there's a better option. I guess patching in a cols before 9.5 is possible, but perhaps not any less confusing?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I think patching and the current approach are pretty much equivalent. In my mind the function abstraction is a little easier to reason about.

Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
"""Utility functions for the Ibis backend."""

from ibis import selectors as s


def select_column(*names):
"""Select a column from a table."""
if hasattr(s, "cols"):
return s.cols(*names)
return s.c(*names)
Loading