Python 2 -> Python 3 IIAB strategic planning to prepare for 2020 #1810

holta · 2019-07-03T11:19:10Z

Python 2.7 will not be maintained past 2020. Originally, there was no official date. Recently, that date has been updated to January 1, 2020.

If the code you care about is still on Python 2, that's totally understandable. Most of PyPI's popular packages now work on Python 2 and 3, and more are being added every day. Additionally, a number of critical Python projects have pledged to stop supporting Python 2 soon. To ease the transition, the official porting guide has advice for running Python 2 code in Python 3.

tim-moody · 2019-07-08T18:00:33Z

biggest issue is switch from str and unicode types to bytes and str types.

traverseda · 2019-07-11T19:26:09Z

Do we have a list of code that needs to be updated?

holta · 2019-07-12T07:26:11Z

Do we have a list of code that needs to be updated?

These are the 4 8=12 that matter, if you have time to assess any quick?

(1) Arising from grep -rl '#!.*/python' /opt/iiab/iiab we have:

(2) Arising from grep -rl '#!.*/python' /opt/iiab/iiab-admin-console we have:

P.S. @tim-moody has done a preliminary conversion of cmdsrv. As mentioned above (4 days ago) the next step there needs a knowledge of bytes vs str and ZeroMQ.

traverseda · 2019-07-12T15:13:17Z

Well section 1 looks like it shouldn't be a problem, those can probably just be switched to python3 directly without any errors. The one I'd be most worried about is captive-portal.py.j2

I've been thinking about putting together an IIAB for myself for a while, so when I stand that up I should be able to confirm.

Cmdsrv does look pretty complicated, I'd rather avoid touching it if there's someone who's actually familiar with the code base working on it.

Section 2 is a lot more/denser code, and ~~while nothing jumps out at me as python2 only it would need to be gone over with a finer comb.~~ there's definitely some python2 only code there.

Now that I know what files are relevant when I build my IIAB node I can work on that.

If I change those files from #!/usr/bin/python to #!/usr/bin/python3 they will actually run under python 3? Do you know where they'd log to?

holta · 2019-07-12T15:23:18Z

The one I'd be most worried about is captive-portal.py.j2

@georgejhunt do you have any recommendations above?

I've been thinking about putting together an IIAB for myself for a while, so when I stand that up I should be able to confirm.

We highly recommend the Raspberry Pi 4 (if those are available in Nova Scotia ;) But a VM or old PC running Debian 10 or Ubuntu 18.04 should also work — even a pre-release daily build of Ubuntu 19.10 if you're daring: #1714

If I change those files from #!/usr/bin/python to #!/usr/bin/python3 they will actually run under python 3? Do you know where they'd log to?

Let me ask others like @tim-moody & @jvonau to see if they know-

traverseda · 2019-07-12T15:25:45Z

Probably going to throw it on an odroid-xu4 I have kicking around.

holta · 2019-07-13T00:43:34Z

Probably going to throw it on an odroid-xu4 I have kicking around.

What OS likely?

Keep an eye on these HW/OS options that stand the best chance of working!
https://github.com/iiab/iiab/wiki/IIAB-Platforms

traverseda · 2019-07-13T00:45:34Z

Armbian Sid if I can, armbian buster if I can't.

floydianslips · 2019-07-16T17:48:11Z

Hey @traverseda, I have had some luck with DietPi as well on Orange Pi's. You could give that a shot on your Odroid. I know they have an image for it.

holta · 2019-09-26T15:36:58Z

FYI @tim-moody is now working on converting Admin Console (and its CMDSRV & ZeroMQ) from Python 2 to Python 3.

holta · 2019-09-26T15:46:23Z

@tim-moody is reviewing our options under "3. Python 3 Migration" under "September 26, 2019" at http://minutes.iiab.io

Excerpt:

a. It’s later than I realized
b. Start by only writing any new scripts in 3
c. Gain experience with 2to3
d. Gain experience with byte/string

i. https://blog.feabhas.com/2019/02/python-3-unicode-and-byte-strings/
ii. https://docs.python.org/3/howto/unicode.html

e. Create /usr/local/lib/python3.x/dist-packages/iiab containing iiab_lib.py (all functions needed only by iiab) and iiab_admin_lib.py (containing functions needed by adm cons)
f. How can we avoid the version in the path?
g. Should we add docstrings?
h. Should we use conda?
i. Should we always use virtualenv?

tim-moody · 2019-09-27T15:50:56Z

turns out that /usr/lib/python3/dist-packages exists and is in python path

tim-moody · 2019-09-27T18:48:53Z

George prefers #! /usr/bin/env python which is not caught by the above grep
grep -rl '#!.*python' /opt/iiab/iiab yields:

/opt/iiab/iiab/roles/kiwix/templates/iiab-make-apache-config.py
/opt/iiab/iiab/roles/kiwix/templates/iiab-make-kiwix-lib.py
/opt/iiab/iiab/roles/network/templates/captive-portal/captive-portal.py.j2
/opt/iiab/iiab/roles/captive-portal/templates/capture-wsgi.py
/opt/iiab/iiab/roles/1-prep/templates/iiab_env.py.j2
/opt/iiab/iiab/roles/activity-server/files/bin/xs-check-activities
/opt/iiab/iiab/roles/activity-server/files/bin/xs-regenerate-activities
/opt/iiab/iiab/roles/osm-vector-maps/templates/iiab-update-map
/opt/iiab/iiab/scripts/roles-needing-docs.py

grep -rl '#!.*python' /opt/iiab/iiab-admin-console yields:
/opt/iiab/iiab-admin-console/roles/js-menu/utilities/menu-def-converter
/opt/iiab/iiab-admin-console/roles/cmdsrv/files/iiab-cmdsrv
/opt/iiab/iiab-admin-console/roles/cmdsrv/files/scripts/get_inet_speed
/opt/iiab/iiab-admin-console/roles/cmdsrv/files/scripts/get_kiwix_lib
/opt/iiab/iiab-admin-console/roles/cmdsrv/templates/iiab-cmdsrv-ctl.j2
/opt/iiab/iiab-admin-console/roles/cmdsrv/templates/scripts/iiab_update_menus.py
/opt/iiab/iiab-admin-console/roles/cmdsrv/templates/scripts/get_oer2go_catalog
/opt/iiab/iiab-admin-console/roles/cmdsrv/templates/scripts/get_kiwix_catalog

grep -rl '#!.*python' /opt/iiab/iiab-factory/
/opt/iiab/iiab-factory/content/kiwix/zim-list.txt
/opt/iiab/iiab-factory/content/kiwix/cp-zim
/opt/iiab/iiab-factory/content/kiwix/zim-concatenator.py
/opt/iiab/iiab-factory/content/osm-tiles/import.py
/opt/iiab/iiab-factory/content/subset-osm/mk-subset.py
/opt/iiab/iiab-factory/content/khan/ka-find
/opt/iiab/iiab-factory/content/khan/ka-vids
/opt/iiab/iiab-factory/content/khan/ka-scan
/opt/iiab/iiab-factory/content/rachel/gen-rachel-menus
/opt/iiab/iiab-factory/content/rachel/gen-rachel-menus-html
/opt/iiab/iiab-factory/content/rachel/get-rachel-menu-blocks
/opt/iiab/iiab-factory/box/generic/merge_local_vars.sh

holta · 2019-10-20T16:45:12Z

Below are 3 serious Python 3 issues we're now facing, with Python 2.7 being end-of-life'd in just over 2 months: (https://pythonclock.org)

MySQL database password-changing failure during fresh install of IIAB [using "sudo iiab"] on Ubuntu 19.04 [and now 18.04 too...whereas 19.10 & Debian 10.x work!] -- Ansible Python 3 needs python3-distutils #1714 MySQL database password-changing failure during fresh install of IIAB [using "sudo iiab"] on Ubuntu 19.04 [and 19.10 pre-releases? Debian ok??] -- Ansible Python 3 needs python3-distutils
CMDSRV requires more than python-systemd -> python3-systemd to run on Ubuntu 19.10 #2003 CMDSRV requires more than python-systemd -> python3-systemd to run on Ubuntu 19.10
Can python3-geojson be made to work, as required by OS's like Ubuntu 19.10 ? [osm-vector-maps] #2013 Can python3-geojson be made to work, as required by OS's like Ubuntu 19.10 ? [osm-vector-maps]

@jvonau @tim-moody @georgejhunt @traverseda do you know of others that should be added to the list?

holta · 2019-10-23T14:55:57Z

@jvonau recommends these 2 Ansible guides w.r.t. transitioning to Python 3:

The key to determining what Python Ansible's using under the covers seems to be...

ansible -m setup -i ansible_hosts localhost --connection=local | grep python
The output of which should include a line like...
- "ansible_python_version": "3.7.5rc1",

Or better yet:

ansible --version | grep "python version"
Which returns...
- python version = 3.7.5rc1 (default, Oct 8 2019, 16:47:45) [GCC 9.2.1 20191008]

e.g. above is the result after running apt install ansible to install Ansible 2.8.3 on a vanilla Ubuntu 19.10:

root@u19:~# apt install ansible
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  ieee-data python3-crypto python3-jinja2 python3-jmespath python3-kerberos
  python3-libcloud python3-lockfile python3-markupsafe python3-netaddr
  python3-ntlm-auth python3-requests-kerberos python3-requests-ntlm
  python3-selinux python3-winrm python3-xmltodict
Suggested packages:
  cowsay sshpass python-jinja2-doc python-lockfile-doc ipython3
  python-netaddr-docs
The following NEW packages will be installed:
  ansible ieee-data python3-crypto python3-jinja2 python3-jmespath
  python3-kerberos python3-libcloud python3-lockfile python3-markupsafe
  python3-netaddr python3-ntlm-auth python3-requests-kerberos
  python3-requests-ntlm python3-selinux python3-winrm python3-xmltodict
0 upgraded, 16 newly installed, 0 to remove and 0 not upgraded.
Need to get 8,850 kB of archives.
After this operation, 78.4 MB of additional disk space will be used.
Do you want to continue? [Y/n]

(i hit 'y' installing Ansible 2.8.3)

Strangely / FWIW running grep -rni python /etc/ansible does not yield/any similar details as to which Python is upholding Ansible:

root@u19:~# grep -rni python /etc/ansible
/etc/ansible/ansible.cfg:229:#       in python does not support them.
/etc/ansible/ansible.cfg:289:# support compiled into both the controller's python and the client's python.
/etc/ansible/ansible.cfg:290:# The names should match with the python Zipfile compression types:

holta · 2019-10-23T22:19:12Z

FYI @jvonau even the very latest Ansible (2.9.0rc5) currently being installed as part of IIAB's master ends up using Python 2.7 not Python 3 -- see the notes just above for background (e.g. serious bugs in Ansible 2.8.3 & 2.8.6 mentioned at #1737

In any case, here is the warning shown when running ./iiab-install on Ubuntu 19.10:

[DEPRECATION WARNING]: Distribution Ubuntu 19.10 on host 127.0.0.1 should use
/usr/bin/python3, but is using /usr/bin/python for backward compatibility with
prior Ansible releases. A future Ansible release will default to using the
discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html
for more information. This feature will be removed in version 2.12. Deprecation
warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

holta · 2019-10-24T12:53:00Z

See the last 2 lines of PR #2019 where @jvonau is prototyping an ./iiab-install command that uses Ansible with Python 3:

ansible -m setup -i $INVENTORY localhost --connection=local -e ansible_python_interpreter=/usr/bin/python3 | grep python
ansible-playbook -i $INVENTORY $PLAYBOOK --connection=local -e ansible_python_interpreter=/usr/bin/python3

@jvonau also mentions:

so the -e would go into iiab-install, runrole, ...

we would need to install some python3 support debs that we have identified in the issues, in the 3 ansible install files

maybe not

sudo apt list python3-distutils*
Listing... Done
python3-distutils-extra/stable 2.42 all
python3-distutils/stable,now 3.7.3-1 all [installed,automatic]

maybe just

sudo apt list python3-pymysql
Listing... Done
python3-pymysql/stable 0.9.3-1 all

yup

TASK [mysql-enable : Update MySQL root password for localhost root accounts, if mysql_enabled] **
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "The PyMySQL (Python 2.7 and Python 3.X) or MySQL-python (Python 2.X) module is required."}

PLAY RECAP **
127.0.0.1 : ok=25 changed=6 unreachable=0 failed=1 skipped=13 rescued=0 ignored=0

works now on rpi

and runs to completion

jvonau · 2019-10-28T05:44:54Z

FYI @jvonau even the very latest Ansible (2.9.0rc5) currently being installed as part of IIAB's master ends up using Python 2.7 not Python 3 -- see the notes just above for background (e.g. serious bugs in Ansible 2.8.3 & 2.8.6 mentioned at #1737

In any case, here is the warning shown when running ./iiab-install on Ubuntu 19.10:

[DEPRECATION WARNING]: Distribution Ubuntu 19.10 on host 127.0.0.1 should use
/usr/bin/python3, but is using /usr/bin/python for backward compatibility with
prior Ansible releases. A future Ansible release will default to using the
discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html
for more information. This feature will be removed in version 2.12. Deprecation
warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

This is a result of having installed python2.7 prior to the mysql crash (due to the missing package python3-pymysql) on the first run of anisble. Now when run a second time ansible sees the newly installed python2.7 and fallsback to using that, resulting in the mysql crash mysteriously going away running python2.7. #2019

tim-moody · 2019-10-30T17:23:00Z

mod_wsgi for apache can only be python 2 OR 3, not both. it is used by
captive portal
cmdsrv
pathagar

holta · 2019-10-30T17:46:55Z

mod_wsgi for apache can only be python 2 OR 3, not both. it is used by
captive portal
cmdsrv
pathagar

Good to know. This relates to Captive Portal work at #1182 that @georgejhunt is looking into, and #2003 "CMDSRV python-systemd -> python3-systemd isn't quite enough to run on Ubuntu 19.10" etc.

jvonau · 2019-10-30T18:04:58Z

mod_wsgi for apache can only be python 2 OR 3, not both. it is used by
captive portal
cmdsrv
pathagar

In the short term we get python2 out of the box on Raspbian and apache is the default webserver. However on Ubuntu what is needed on a pure python3 distro is to change the pip call to use python3 and tell pip to use python2 in the venv. Now python2.7 is not installed but we install what is needed in packages.yml for kalite and others' venv that are installed later, those four packages should be changed to python3. What program needs python-passlib? Perhaps cmdsrv could use a venv to contain the python2 dependencies?

The nginx changes could make mod_wsgi python3 only as uwsgi is used with nginx, which I see has new python dependency. Think pathagar and all other unused/unmaintained roles that require python2 be moved out of the main line code.

jvonau · 2019-11-10T06:11:18Z

#2034 addresses a short term need for vector-maps to install on Ubuntu-19.10 and provide backwards support for older python2 scripts till porting is complete.

iiab/iiab#1810 iiab/iiab#2034

holta · 2019-11-11T13:45:39Z

"Python 2 removal in sid/bullseye: Progress and next steps"
https://lists.debian.org/debian-devel-announce/2019/11/msg00000.html

holta · 2019-11-20T05:05:44Z

PR #2043 merged to facilitate this effort.

See @tim-moody's clarification of @jvonau's related work, at #2043 (comment)

holta · 2019-11-20T15:49:24Z

See also IIAB Admin Console's ongoing conversion to Python 3:

0.4 python3 Convert Admin Console to Python 3 - WIP iiab-admin-console#263 "0.4 python3 Convert Admin Console to Python 3 - WIP"
0.4 Transition of Scripts to Python3 iiab-admin-console#266 "0.4 Transition of Scripts to Python3"

holta · 2019-11-20T18:52:59Z

@jvonau has posted a request to IBM/Red Hat's Ansible developers here:

ansible/ansible#57342 (comment) "Please provide packages for python3 in PPA"

holta · 2019-12-21T01:39:36Z

With things like PR #2052 now merged, at what stage do we declare victory?

holta · 2020-01-09T16:50:38Z

@jvonau suggests leaving this tkt open pending Ansible & Nodesource.com full transition to Python 3.

holta · 2020-01-09T16:51:10Z

Related: #2105 "Update /opt/iiab/iiab/.travis.yml Python version? distro version? (integrate Ansible Lint?)"

holta · 2020-02-03T15:12:15Z

Related: #2231 "Should Calibre-Web use Python 3 instead of 2.7 ?"

jvonau · 2020-02-03T15:34:56Z

Better question is what about kalite, also python2.

holta · 2020-02-03T15:54:53Z

Better question is what about kalite, also python2.

@benjaoming what's the summary status here do you know?

benjaoming · 2020-02-03T17:47:34Z

@holta KA Lite will not support Python 3 in this current millennium :)

holta · 2020-04-08T16:02:15Z

Moving this ticket from IIAB 7.1 milestone to 7.2, to keep organized around any residual Python 2 -> 3 issues.

holta · 2020-09-12T10:59:17Z

Validate Ansible 2.10 expected 2020-08-13 [with simplified auto-installation of "Collections" ETA 2020-09-22 ?] #2479 'Validate Ansible 2.10 expected 2020-08-13 [with simplified auto-installation of "Collections" ETA 2020-09-22 ?]'
PR revamp ansible for the future #2512 'Revamp Ansible for the future'

holta · 2021-08-11T18:59:47Z

Good enough !!

KA Lite remains on Python 2, but there's nothing we can do about that ;)

PR kalite content script - How should kalite be integrated/automated into everyday workflows? #2928

holta added help wanted question labels Jul 3, 2019

holta added this to the 7.1 milestone Jul 3, 2019

holta mentioned this issue Oct 21, 2019

MySQL database password-changing failure during fresh install of IIAB [using "sudo iiab"] on Ubuntu 19.04 [and now 18.04 too...whereas 19.10 & Debian 10.x work!] -- Ansible Python 3 needs python3-distutils #1714

Closed

jvonau mentioned this issue Oct 24, 2019

python3 [Ansible] #2019

Merged

holta mentioned this issue Oct 30, 2019

Captive Portal improvements checklist #1182

Open

44 tasks

This was referenced Oct 30, 2019

CMDSRV requires more than python-systemd -> python3-systemd to run on Ubuntu 19.10 #2003

Closed

Node-RED fails to start on Ubuntu 19.10 #2015

Closed

holta mentioned this issue Nov 10, 2019

provide python2.7 virtual environment for iiab apps #2034

Closed

jvonau added a commit to jvonau/iiab-factory that referenced this issue Nov 10, 2019

use provided venv for 2.7 python code

c345a91

iiab/iiab#1810 iiab/iiab#2034

jvonau mentioned this issue Nov 10, 2019

use provided venv for 2.7 python code iiab/iiab-factory#86

Closed

holta mentioned this issue Nov 20, 2019

Add py_lib [Python 3] #2043

Merged

This was referenced Nov 20, 2019

GH NGINX 3 introduce /etc/iiab/iiab_state.yml begin breaking apart playbooks into (1) install.yml (2) setup.yml (3) enable.yml #2042

Closed

Temporary Fix [Python 3 for iiab-make-kiwix-lib, iiab-update-map etc] #2048

Closed

holta mentioned this issue Jan 4, 2020

NGINX & ./runrole & ./iiab-network [Dual WiFi] & IIAB Maps need approachable docs for newbie/legacy implementers #2096

Closed

holta mentioned this issue Feb 3, 2020

Should Calibre-Web use Python 3 instead of 2.7 ? #2231

Closed

holta modified the milestones: 7.1, 7.2 Apr 8, 2020

holta modified the milestones: 7.2, 8.0 Sep 12, 2020

jvonau mentioned this issue Aug 10, 2021

Notes: Stalled progress on nginx front #2762

Open

holta closed this as completed Aug 11, 2021

holta modified the milestones: 8.0, 7.2 Aug 11, 2021

Python 2 -> Python 3 IIAB strategic planning to prepare for 2020 #1810

Python 2 -> Python 3 IIAB strategic planning to prepare for 2020 #1810

Comments

holta commented Jul 3, 2019

tim-moody commented Jul 8, 2019

traverseda commented Jul 11, 2019

holta commented Jul 12, 2019

traverseda commented Jul 12, 2019 • edited Loading

holta commented Jul 12, 2019

traverseda commented Jul 12, 2019

holta commented Jul 13, 2019

traverseda commented Jul 13, 2019

floydianslips commented Jul 16, 2019

holta commented Sep 26, 2019 • edited Loading

holta commented Sep 26, 2019 • edited Loading

tim-moody commented Sep 27, 2019

tim-moody commented Sep 27, 2019

holta commented Oct 20, 2019 • edited Loading

holta commented Oct 23, 2019 • edited Loading

holta commented Oct 23, 2019 • edited Loading

holta commented Oct 24, 2019

jvonau commented Oct 28, 2019 • edited Loading

tim-moody commented Oct 30, 2019

holta commented Oct 30, 2019 • edited Loading

jvonau commented Oct 30, 2019

jvonau commented Nov 10, 2019

holta commented Nov 11, 2019

holta commented Nov 20, 2019

holta commented Nov 20, 2019 • edited Loading

holta commented Nov 20, 2019 • edited Loading

holta commented Dec 21, 2019 • edited Loading

holta commented Jan 9, 2020

holta commented Jan 9, 2020 • edited Loading

holta commented Feb 3, 2020

jvonau commented Feb 3, 2020

holta commented Feb 3, 2020

benjaoming commented Feb 3, 2020

holta commented Apr 8, 2020

holta commented Sep 12, 2020 • edited Loading

holta commented Aug 11, 2021 • edited Loading

traverseda commented Jul 12, 2019 •

edited

Loading

holta commented Sep 26, 2019 •

edited

Loading

holta commented Sep 26, 2019 •

edited

Loading

holta commented Oct 20, 2019 •

edited

Loading

holta commented Oct 23, 2019 •

edited

Loading

holta commented Oct 23, 2019 •

edited

Loading

jvonau commented Oct 28, 2019 •

edited

Loading

holta commented Oct 30, 2019 •

edited

Loading

holta commented Nov 20, 2019 •

edited

Loading

holta commented Nov 20, 2019 •

edited

Loading

holta commented Dec 21, 2019 •

edited

Loading

holta commented Jan 9, 2020 •

edited

Loading

holta commented Sep 12, 2020 •

edited

Loading

holta commented Aug 11, 2021 •

edited

Loading