@vianmixtkz Great writeup. This is something that would be great for Pandera to support.

Thanks @vianmixtkz this is an interesting use case: the way pandas handles mixed-type columns is to represent the data in an object dtype column.

One thing we should clarify in the semantics of this feature is the following: we can interpret Union[str, float] either as:

1. the column is either a str column or a float column
2. the column is an object column that contains either str or float values

Do we need special syntax to differentiate between these two cases, or is that something that we leave to the pandera type engine to handle? I.e.:

• if the column is str dtype, then pass
• if the column is float dtype, then pass
• if the column is object data type, check that values are str or float. If so, then pass.
• fail if none of the above conditions are met.

Here what I described is matching case 2. That's is in a given column, I'll have for example str on some rows and floats on other rows.
But it would be nice to support both cases anyway.

With something like:

Case 1

class InputSchema(pa.DataFrameModel):
    year: Series[int] = pa.Field(gt=2000, coerce=True)
    month: Series[int] = pa.Field(ge=1, le=12, coerce=True)
    day: Series[int] = pa.Field(ge=0, le=365, coerce=True)
    comment : Union[Series[str], Series[float]] = pa.Field() # comment is either only str or only float in a given DataFrame

Case 2

class InputSchema(pa.DataFrameModel):
    year: Series[int] = pa.Field(gt=2000, coerce=True)
    month: Series[int] = pa.Field(ge=1, le=12, coerce=True)
    day: Series[int] = pa.Field(ge=0, le=365, coerce=True)
    comment : Series[Union[str,float]] = pa.Field() # comment is a column containing str on some rows and float on other rows

And yeah, I think the behavior you are describing is what users would expect

Do we need special syntax to differentiate between these two cases, or is that something that we leave to the pandera type engine to handle? I.e.:

if the column is str dtype, then pass # passes in case 1 and 2
if the column is float dtype, then pass # passes in case 1 and 2
if the column is object data type, check that values are str or float. If so, then pass. # passes only in case 2
fail if none of the above conditions are met.

Just bumping this thread.

Any consensus how to proceed? Seem like the #1227 is stale.

Revisiting this issue and thinking about it a little bit, here's another proposal for this issue:

from pandera.engines.pandas_engine import Object
from typing import Annotated

class Model(pa.DataFrameModel):
    union_column : Union[str, float]  # the column data type must be either a str or float

    object_column: Object = pa.Field(dtype_kwargs={"allowable_types": [str, float]})
    # or use the annotated types
    object_column: Annotated[Object, [str, float]]

This syntax is less ambiguous as to what the actual type of the column is vs. the values within it are. However, it does require importing a special Object type.

I'm still open to the more ambiguous behavior where Union[str, float] would cover all of these cases though. Open to further discussion on this!

Re: this proposal: #1152 (comment)

Unfortunately col: Series[TYPE] and col: TYPE in a DataFrameModel are equivalent so Union[Series[str], Series[float]] and Series[Union[str,float]] would effectively be equivalent, and would also introduce more complexity to the handling of types in DataFrameModel, which I don't think would be worth it.

I'm not a fan of this case Union[Series[str], Series[float]] from this comment, where the series would consists of only string or only float. Its very ambiguous, the output would sorta change depending on what data you pass. These could be very well their own distinct schema.

Series[Union[str, float]] or Union[str, float] or str | float # python 3.10 , where the output could be either string or float. This case is more consistent.

I am pretty sure I need this as well. I'm trying to create a DataFrameModel that expects a timestamp in the index. My ideal would be to validate that it has a timezone, but not specify which timezone. Another acceptable type hint would be to accept any timestamp, tz-naive or tz-aware, and then add custom checks around timezone manually. But right now, if I use Index[pd.DatetimeTZDtype], a specific timezone must be provided, and pd.Timestamp fails if any timezone is provided. I'd love to hint a union of the two.

Propose something that isnt exactly just check valid type(s): Would be if it doesnt fall into the set datatype there is a set of values that is acceptable.

float but field can contain "NA"

Cant think of a personal use case to allow for all strings but where maybe data wasnt provided or invalid. Still want to check all the values in the column but dont want to have to edit all the strings to null or 0.