Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error when no column adheres to reg_ex with Pyspark SQL DataFrameModel: 'list' object has no attribute 'tolist' #1799

Open
2 of 3 tasks
PabloHervadaSDG opened this issue Sep 3, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@PabloHervadaSDG
Copy link

Describe the bug
When validating a pyspark dataframe using DataFrameModel with Pysaprk SQL, if there is a regular expression that is not met by any column of the dataframe, pandera can't validate the df, due to a bug handling the list of columns that match the reg_ex, it seems.

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of pandera.
  • (optional) I have confirmed this bug exists on the main branch of pandera.

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

from pyspark.sql import SparkSession
import pyspark.sql.types as T
import pandera.pyspark as pa

# Create a example dataframe
spark = SparkSession.builder.master("local").appName("pyspark-pandera").getOrCreate()

schema = T.StructType([
    T.StructField("user_name", T.StringType(), True),
    T.StructField("email", T.StringType(), True)
])

data = [
    ("Alice", "[email protected]"),
    ("Bob", "[email protected]"),
]

df = spark.createDataFrame(data, schema)

# Validate dataframe with Pandera DataFrameModel
class ExampleModel(pa.DataFrameModel):
    user_name: T.StringType() = pa.Field(nullable=False)
    # There is no field that matches this reg_ex
    age_field: T.IntegerType() = pa.Field(alias="(.+)age(.+)", regex=True)

df_out = ExampleModel.validate(df)

Expected behavior

I expect df to be validated, and df_out to contain all the errors with the dataframe, according to the dataframe model.
In contrast, ExampleModel.validate(df) leads to an error handling the reg_ex columns:

AttributeError: 'list' object has no attribute 'tolist'

Desktop (please complete the following information):

  • OS: Windows 11
  • Browser: Chrome
  • Version: Pandera version 0.20.3

Screenshots

image

@PabloHervadaSDG PabloHervadaSDG added the bug Something isn't working label Sep 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant