Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regression: generated Python functions return a list instead of a single value #3084

Open
hvbtup opened this issue Dec 6, 2024 · 9 comments

Comments

@hvbtup
Copy link

hvbtup commented Dec 6, 2024

I use swig 4.3.0 to generate Python wrappers for a C library.

For example, one of the C functions is defined as

EXPORT PTError API PBMGetLevel(BookmarkHandle h, int* level);

PTError is defined as

typedef enum PTError {
  PTSuccess = 0,
  PTNotPDF = 1001, ...
} PTError;

And I am using the following in the *.i file:

void create_pdf_exception(const char* api, PTError rc);

%typemap(in,numinputs=0,noblock=1) PTError* err (PTError temp) {
    temp = 0;
    $1 = &temp;
}

%typemap(in,numinputs=0,noblock=1) PTError* pErr (PTError temp) {
    temp = 0;
    $1 = &temp;
}

%typemap(out) PTError {
    if ($1 == 0 || $1 == PTAlreadyWritten) {
        $result = SWIG_Py_Void();
    } else {
        create_pdf_exception("$symname", $1);
        goto fail;
    }
}

%typemap(argout) (PTError* err) {
    if (*$1 == 0 || *$1 == PTAlreadyWritten) {
        ;
    } else {
        Py_XDECREF(resultobj);
        create_pdf_exception("$symname", *$1);
        goto fail;
    }
}

%typemap(argout) (PTError* pErr) {
    if (*$1 == 0 || *$1 == PTAlreadyWritten) {
        ;
    } else {
        Py_XDECREF(resultobj);
        create_pdf_exception("$symname", *$1);
        goto fail;
    }
}

With SWIG 4.1.0, calling the function PBMGetLevel(bmh) in Python returned a single integer -1.
With SWIG 4.3.0, this now returns a list with two elements: [None, -1].

The same happens for all functions in the C lib:
Instead of just the expected, they return a list containing None and the expected value.

I think that this issue is a consequence of the fix for #2905.

Comparing the C code generated with SWIG 4.1.0 and SWIG 4.3.0, it looks like this:

if (SWIG_IsTmpObj(res2)) {
  resultobj = SWIG_Python_AppendOutput(resultobj, SWIG_From_int((*arg2)), 0);
} else {
  int new_flags = SWIG_IsNewObj(res2) ? (SWIG_POINTER_OWN |  0 ) :  0 ;
  resultobj = SWIG_Python_AppendOutput(resultobj, SWIG_NewPointerObj((void*)(arg2), SWIGTYPE_p_int, new_flags), 0);
}

The additional argument 0 is given to the SWIG_Python_AppendOutput function.

Is this a bug in SWIG?
If not, what can I do in the *.i file to get the same behavior as in 4.1.0?

@vadz
Copy link
Member

vadz commented Dec 6, 2024

It looks like the detection of whether the function is void or not doesn't work in this case, and I'm not sure if this can be fixed automatically. Being able to mark the function as being isvoid would definitely help here...

BTW, do argout typemaps actually do anything in this case (and why are there 2 of them?)?

@hvbtup
Copy link
Author

hvbtup commented Dec 9, 2024

BTW, do argout typemaps actually do anything in this case ...

There are other functions in the C lib which use a pointer to a PTError struct as an argument.

...(and why are there 2 of them?)?

This is obviously a copy/paste error. It's a duplicate.

@hvbtup
Copy link
Author

hvbtup commented Dec 9, 2024

Hmm.
So it seems there is nothing I can do to make the C lib work with SWIG including error handling?

In this case, I'd have to move the the error handling logic into a higher level Python API which looks at the returned list, pop the first element from it and raise an exception if it's not None, with the exception details extracted from the PTError struct.

Or remove SWIG from the build completely, and find a different solution using ctypes and ctypesgen.

@vadz
Copy link
Member

vadz commented Dec 9, 2024

In this case, I'd have to move the the error handling logic into a higher level Python API which looks at the returned list, pop the first element from it and raise an exception if it's not None, with the exception details extracted from the PTError struct.

FWIW you should be do this at SWIG level with %pythoncode.

@hvbtup
Copy link
Author

hvbtup commented Dec 9, 2024

@vadz Thanks, so at least the workaround code can be saved where it belongs.

@hvbtup
Copy link
Author

hvbtup commented Dec 9, 2024

Just for information, my workaround code looks like this:


%pythoncode %{
"""
The following code is used as a workaround für SWIG issue #3084:
Regression: generated Python functions return a list instead of a single value.

Instead of at the C level, we take care of the PTError return value
at the Python level by using a wrapper function.

"""

def _wrap_error_return(func):
    def wrapped(*args, **kwargs):
        res = func(*args, **kwargs)
        # print(f"{func}({args}, {kwargs}) {res=}")
        if isinstance(res, list):
            if len(res) == 2:
                [err, res] = res
                if err:
                    raise RuntimeError(f"PDF-Error rc={err}")
        elif isinstance(res, int):
                err = res
                if err:
                    raise RuntimeError(f"PDF-Error rc={err}")
                res = None
        # print(f"Returning {res}")
        return res
    return wrapped
    
AddFileAttachment = _wrap_error_return(AddFileAttachment)
AddFileAttachmentW = _wrap_error_return(AddFileAttachmentW)
... many lines like these
%}

@jun66j5
Copy link

jun66j5 commented Dec 12, 2024

I've encountered the same issue while building Python bindings of Subversion 1.14.4 with SWIG 4.3.0.

Example:

svn_error_t *
svn_fs_commit_txn(const char **conflict_p, svn_revnum_t *new_rev,
                  svn_fs_txn_t *txn, apr_pool_t *pool)

with the following typemaps:

%typemap(out) svn_error_t * { ... }
%typemap(argout) const char ** { ... }
%typemap(argout) svn_revnum_t * { ... }

As a workaround, the following changed:

  • Always initializes the $result with an new list in %typemap(out).
  • Adjust the $result by the count of the list in %typemap(ret).
  • Ruby bindings have the same issue, so that the same changes are applied.
%typemap(out) svn_error_t * {
    if ($1 != NULL) {
        if ($1->apr_err != SVN_ERR_SWIG_PY_EXCEPTION_SET)
            svn_swig_py_svn_exception($1);
        else
            svn_error_clear($1);
        SWIG_fail;
    }
    Py_XDECREF($result);
    $result = PyList_New(0);
}

%typemap(ret) svn_error_t * {
    if ($result == NULL) {
      $result = Py_None;
      Py_INCREF($result);
    }
    else {
      switch (PyList_Size($result)) {
        case 0:
          $result = Py_None;
          Py_INCREF($result);
          break;
        case 1:
          {
            PyObject *tmp = $result;
            $result = PyList_GetItem(tmp, 0);
            Py_INCREF($result);
            Py_DECREF(tmp);
          }
          break;
      }
    }
}

See https://github.com/apache/subversion/blob/1.14.5/subversion/bindings/swig/include/svn_types.swg#L426-L464

@wsfulton
Copy link
Member

@hvbtup, some guess work is required with what you've provided, as how int * is being handled is not clear to me. You're probably setting something that uses argout typemaps for int * which is the area affected by #2905. You probably need to use a custom typemap to revert to the previous behaviour as documented in the CHANGES.current/CHANGES file:

https://github.com/swig/swig/blob/8fd4df01bd4111e5f7322fe4f9d4817ffd6f0c6d/CHANGES#L377-L397

so call SWIG_Python_AppendOutput and set 1 for the is_void parameter.

@hvbtup
Copy link
Author

hvbtup commented Dec 23, 2024

@wsfulton I don't understand your answer. In the description I showed the relevant code. As you can see, I did not use SWIG_Python_AppendOutput in there, and I simply don't know how I could change my code to call that function now.
Please keep in mind that I wrote the code ~ 20 years ago and I'm not an expert in writing SWIG typemaps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants