I always read SAS files with encoding='iso-8859-1' (see here). Does that help in your case?

I always read SAS files with encoding='iso-8859-1' (see here). Does that help in your case?

Thanks, but unfortunately no.
latin1 is the encoding that was used for the original sas7bdat so this is in fact the correct encoding to be used when reading the file to pandas.
(also, latin1 is just an alias for iso-8859-1 😉 )

Nevertheless, before submitting the issue I ran a script to check if the issue persists if using any other enconding for reading the data. (It does.)

Not sure if it is the case if another encoding is used when writing the original sas7bdat file.

(also, latin1 is just an alias for iso-8859-1 😉 )

thanks for that info :)

Hi,
I can confirm this is a problem with decoding the RLE compression, not related to encoding.
Looking at the docs for sas7bdat (actually the docs for the R package that sas7bdat is based on) SAS has an interesting compression with differing (and overlapping) options for how single and consecutive fields are compressed. It looks like pandas correctly decompresses some of the options, but not all.

I've tested a bit and found that if there are preceeding fields, then a field with 18 consecutive zeros that is OK, but preceeding fields then 19 zeros is NOT decompressed correctly.

Stepping through the code for sas7bdat, for the case that works the control byte for the 18 zeros is hex C, whereas for 19 zeros it is hex 4. The meaning of C is documented https://cran.r-project.org/web/packages/sas7bdat/vignettes/sas7bdat.pdf but hex 4 is not.
Control byte hex 4 is implemented in both sas7bdat and in pandas, but the implementation appears different.

sas7bdat (pure python):

            elif control_byte == 0x40:
                copy_counter = (
                    end_of_first_byte * 16  
                    (b(page[offset   i   1]) & 0xFF)
                )
                for _ in xrange(copy_counter   18):
                    result.append(c(page[offset   i   2]))
                    current_result_array_index  = 1
                i  = 2

pandas (cython):

        elif control_byte == 0x40:
            # not documented
            nbytes = end_of_first_byte * 16
            nbytes  = <int>(inbuff[ipos])
            ipos  = 1
            for _ in range(nbytes):
                result[rpos] = inbuff[ipos]
                rpos  = 1
            ipos  = 1

Looking around at other R/C implementations for reading SAS files, the Readstat C library by Evan Miller https://github.com/WizardMac/ReadStat/blob/master/src/sas/readstat_sas_rle.c could be a useful source of info and appears to be actively maintained. This also has code for control byte 0x50 (missing from both sas7bdat and pandas).

#define SAS_RLE_COMMAND_INSERT_BYTE18   4
#define SAS_RLE_COMMAND_INSERT_AT17     5
...
            case SAS_RLE_COMMAND_INSERT_BYTE18:
                insert_len = (*input  )   18   length * 256;
                insert_byte = *input  ;
                break;
            case SAS_RLE_COMMAND_INSERT_AT17:
                insert_len = (*input  )   17   length * 256;
                insert_byte = '@';
                break;

There is also a python wrapper https://github.com/Roche/pyreadstat around Evan Miller's library.

I can try to put together a PR to alter the decompression for control code 0x40 (and add code for 0x50), or (since we have other issues related to SAS files) should we look to use Evan Miller's library or pyreadstat rather than duplicating others' efforts within pandas? @jbrockmendel, @mroeschke - what do you think?

cc @bashtage

pyreadstat (it is a python wrapper around Evan Miller's Readstat) is already used in pandas.read_spss so I guess it would be an easy one to also provide it as an backend option for read_sas (see #5768, #26537).

This move would also solve other issues such as #37088, #35545, #22720, #18198

@ofajardo I agree pyreadstat is also significantly faster than pandas cython implementation

It sounds like a no brainer to move to pyreadstat given it is already a soft dep. The only question is what will the path look like. It would probably require adding optional support, deprecating pandas build-in, and then eventually removing the native reader.

Sounds like a plan!