Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate windows builds #109

Closed
6 of 7 tasks
Cimbali opened this issue Aug 6, 2019 · 7 comments
Closed
6 of 7 tasks

Automate windows builds #109

Cimbali opened this issue Aug 6, 2019 · 7 comments

Comments

@Cimbali
Copy link
Owner

Cimbali commented Aug 6, 2019

This a major pain that has been brought up in #107 and #108 most recently, but earliest attempts date have existed. Basically building on windows is an immensely painful process. I believed to have documented this already somewhere but it appears not, so I’m creating an issue to track progress on this. I haven’t looked at this in 6 months so I’m taking a look at things to see if they got un-stuck.

The core issues are:

  • Pympress depends on gtk3 and poppler, which we need to either build or obtain in binary form, and redistribute.
  • Distributing a python package as binary requires freezing it, from which we use cx_Freeze (though other alternatives

Basically the 2 main paths to build pympress for windows are either using Linux toolchains on Windows (i.e. MinGW/Msys and the related) or Microsoft toolchains (i.e. MSVC). Those are not compatible. So a system’s Python for Windows is the latter, while a python installed inside msys2 for example is the former.

The current/legacy technique relies on PyGObjectWin32, which uses the Microsoft way, with all of the packages pre-compiled, so it’s a rather pain-free mechanism. However this seems to be unmaintained and rather outdated by now. Also I do not particularly wish to boot an old machine up or have to keep a 32bit and 64bit python 3.4 on a Windows VM, this really does not scale.

Testing all of this via appveyor scripts or RDP is very slow, however this is as far as I’ve gotten along both paths:

  1. Going the msys2 way gives a functional pympress, so msys2 is currently the best way to install pympress on a windows system, but we can’t use it (yet?) to build installers automatically.

    • most (all?) dependencies are pre-compiled and distributed as “wheels” over pip. All install without problems.
    • The cx_Freeze package (and any other freezing package I tried) requires compilation, as no wheel is (was?) available, and this breaks. The cx_Freeze package’s current master (marcelotduarte/cx_Freeze@29ed764) can be installed from source using pip successfully.
    • The correct list of things (typically DLLs) to be included in the frozen package needs to be adjusted.
      The following warning message is also slightly worrying:

      *** WARNING *** unable to create version resource
      install pywin32 extensions first

  2. Going the MSVC way, we have to compile everything. Gtk 3 and cx_Freeze both work after some tweaking, but I can’t manage to get or install Poppler.

    • Gtk can be obtained from the gvsbuild project
    • From this you can then build pygobject and pycairo and test applications using Gtk seem to work
    • I’ve managed to install poppler from conda-forge, however the typelib (the bindings needed for the introspection) seem to be missing. So from gi.repository import Poppler breaks. Maybe by getting the recipe and building poppler ourselves with that recipe we could add the typelib?
    • cx_Freeze’s current master (marcelotduarte/cx_Freeze@29ed764) can be installed, but I have not tested its functionality
@Cimbali
Copy link
Owner Author

Cimbali commented Aug 7, 2019

Builds using msys2 are here: https://ci.appveyor.com/project/Cimbali/pympress build status badge

I’ve managed to test the 64-bit no-vlc version successfully. Now waiting for all the different settings to build (~10 minutes each), and then we can test the other ones.

@ComFreek
Copy link
Contributor

ComFreek commented Aug 8, 2019

Awesome! Where can I find the new binaries, though? They don't seem to be at https://github.com/Cimbali/pympress/releases/tag/v1.4.0.

@FriedrichFroebel
Copy link
Contributor

@ComFreek They seem to be available as build artifacts at AppVeyor only at the moment, for example at https://ci.appveyor.com/project/Cimbali/pympress/build/job/3fyiql1t0rqcy0nc/artifacts.

@Cimbali
Copy link
Owner Author

Cimbali commented Aug 8, 2019

There still are some problems. The VLC binaries are 1GB for example.

@Cimbali Cimbali closed this as completed Aug 8, 2019
@Cimbali
Copy link
Owner Author

Cimbali commented Aug 8, 2019

Sticking with non-vlc binaries as the system’s VLC is detected without problems. Not the windows way I suppose, but it also has its advantages in disk space for the user and so on.

@ComFreek
Copy link
Contributor

ComFreek commented Aug 8, 2019

@FriedrichFroebel Ah!

Can I count on the binaries being shipped with every GitHub release? As in https://github.com/Cimbali/pympress/releases. I do not intend to update my Choco package manually. Instead, I've got an automated solution querying the GitHub API for the releases and inferring the download links from the filenames of the uploaded assets. I've seen that the asset names for v1.4.0 and v1.2.0 differ. Would it be possible to stick to one naming convention so that my automation does not break?

This would be not set in stone. Changing the naming convention again would just require shortly pinging me before hand and adjusting my automation.

Not the windows way I suppose

As a Windows user, I find this totally fine. (Yes, I am probably biased by being a programmer and fan of automation 😄)

@Cimbali
Copy link
Owner Author

Cimbali commented Aug 8, 2019

The automated build & deployment to github releases and current naming schemes are here to stay. It’spympress-v{version}-{architecture}.msi and architecture will be one of the msys names, either x86_64 for 64 bits or i686 for 32 bits.

Maybe if some day we have non-mingw builds there’ll be a need to differentiate those, but I have no plans to work on that in the future. I’ve tested the mingw binaries, they work, and that’s good enough for me. Building for windows is really a most painful process.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants