The thumbnails / files are served by an unpuppetized instance deployment-upload which run a nginx proxy and uses scripts in /data/project/scripts.
We should create a Swift cluster in beta, import the files from /data/project/upoad7 and points MediaWiki to the Swift cluster.
| Status | Subtype | Assigned | Task | |
|---|---|---|---|---|
| · · · | ||||
| Open | None | T53494 Use Beta cluster as a true canary for code deployments (epic) | ||
| Resolved | Krenair | T84950 Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking) | ||
| Resolved | • AlexMonk-WMF | T102953 Completely remove Beta Cluster dependency on NFS | ||
| Resolved | • AlexMonk-WMF | T64835 Setup a Swift cluster on beta-cluster to match production | ||
| Resolved | fgiunchedi | T123512 beta swift labs instances requirements | ||
| Resolved | • AlexMonk-WMF | T140220 "UnderflowException from ... FancyCaptcha.class.php: Ran out of captcha images" at Special:CreateAccount | ||
| · · · |
I have created a root tracking task for object storage on labs: T114998. Continuous-Integration-Infrastructure definitely use a Swift store.
It isn't blocked on T114998 since that's for labs generally and would be setup totally different from production, while this should be set up *on VMs* and configured exactly the same as production
I've setup a swift cluster with backends deployment-ms-be01 / deployment-ms-be02 and the frontend at deployment-ms-fe01. Each backend is a xlarge instance and replication factor of 2.
filippo@deployment-puppetmaster:~$ swift list test-upload filippo@deployment-puppetmaster:~$ swift list test-upload | head -10 var/cache/apt/archives/arcconf_7.31.18856-1_amd64.deb var/cache/apt/archives/augeas-lenses_1.2.0-0ubuntu1.1_all.deb var/cache/apt/archives/bind9-host_1:9.9.5.dfsg-3ubuntu0.7_amd64.deb var/cache/apt/archives/binutils_2.24-5ubuntu3.1_amd64.deb var/cache/apt/archives/conntrack_1:1.4.1-1ubuntu1_amd64.deb var/cache/apt/archives/daemon_0.6.4-1_amd64.deb var/cache/apt/archives/debdeploy-common_0.0.9-1~trusty1_all.deb var/cache/apt/archives/debdeploy-minion_0.0.9-1~trusty1_all.deb var/cache/apt/archives/diamond_3.5-5 trusty1_all.deb var/cache/apt/archives/dnsutils_1:9.9.5.dfsg-3ubuntu0.7_amd64.deb
filippo@deployment-ms-fe01:~$ sudo swift-recon -d --human-readable =============================================================================== --> Starting reconnaissance on 2 hosts =============================================================================== [2016-01-20 14:24:15] Checking disk usage now Distribution Graph: 0% 2 ********************************************************************* Disk usage: space used: 213 MB of 214 GB Disk usage: space free: 214 GB of 214 GB Disk usage: lowest: 0.1%, highest: 0.1%, avg: 0.0995588055691% ===============================================================================
next steps:
Do we know how difficult it would be to run the import from NFS to Swift? Does it need some sort of scripting, or...?
@Krenair yeah first adding the right filebackend to filebackend-labs.php and then via maintenance/copyFileBackend.php from core likely will do the trick.
Also, this task completely fell off my radar, sorry about that! Happy to help with what's left tho.
From some past discussions (and maybe it is recorded on a task): we will want to clean up mass of crap that is in upload7. Specially temp directory, most probably thumbs and anything that got magically imported via instantCommons. That a large chunk of data that are not worth migrating.
There is a few hints at me deleting random files from time to time on https://tools.wmflabs.org/sal/releng?p=0&q=upload7&d= eg:
sudo find /data/project/upload7/*/*/temp -type f -delete
sudo rm /data/project/upload7/*/*/thumb/*
krenair@deployment-upload:~$ du -csh /data/project/upload7/math 16M /data/project/upload7/math 16M total krenair@deployment-upload:~$ du -csh /data/project/upload7/*/*/temp 18M /data/project/upload7/wikipedia/commons/temp 40K /data/project/upload7/wikipedia/de/temp 8.0K /data/project/upload7/wikipedia/en/temp 18M total
And all these old files from June/July 2012:
krenair@deployment-upload:~$ find /data/project/upload7/*/*/lockdir -type f /data/project/upload7/wikipedia/commons/lockdir/.nfs000000000010c1820000294d /data/project/upload7/wikipedia/commons/lockdir/.nfs000000000010c0ef0000294c /data/project/upload7/wikipedia/commons/lockdir/.nfs000000000010cdc950024fa2 /data/project/upload7/wikipedia/commons/lockdir/.nfs000000000010c2310000295a /data/project/upload7/wikipedia/commons/lockdir/.nfs000000000010c18f00002951 /data/project/upload7/wikipedia/de/lockdir/.nfs000000000011c1eb00029253 /data/project/upload7/wikipedia/en/lockdir/.nfs000000000011a5450002b2c1 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0f000026c07 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c2a500023d3c /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011cf0600024f99 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0e800026c02 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c00800023978 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0fb0000281c /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0ec00026c05 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c3f60000472c /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0f200026c0a /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c2df00024e17 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0e100026bf2 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0f7000027df /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000010025950021cf5 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011ca4600024f94 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0d300026b90 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c08200000de2 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c105000027f3 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c10b00026c30 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c27500029439 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c10300002827 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c11c0000003c /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c03c00025aa6 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c4ff000008eb /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c14d00029ec8 /data/project/upload7/wikipedia/simple/lockdir/.nfs0000000000090d6400016bbf /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000010029f000243f0 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0f9500027cf /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0f5000027bd /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0fd000027d2 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c10d00026c2f /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0df00002784 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0cb00026a8a /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000010025c00021d1d /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c1e950029436 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c4b600023d86 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0ff00026c3b /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c36100023d40 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000010025300021cc1 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c3fc0002b296 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c12300029432 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0ea00026c04 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0ee00026c0f /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c3ff0002b201 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c1070000283b /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c1010000282c /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011cf5a00000906 /data/project/upload7/wikipedia/simple/lockdir/.nfs0000000000090d6c00016c4c /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0e300026bfa /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c4030002b2b3 /data/project/upload7/wikipedia/simple/lockdir/.nfs00000000001002cf000007b3 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0e500026bf7 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000010025800021cfa /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c27950023d3a /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0db00002765 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c2d50002b1f4 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0dd00026b93 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c96200021a55 /data/project/upload7/wikipedia/simple/lockdir/.nfs00000000001002d000024e8b /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0d700026b59 /data/project/upload7/wikipedia/simple/lockdir/.nfs00000000001002ce000007a9 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000010025a00021d00 /data/project/upload7/wikipedia/simple/lockdir/.nfs000000000011c0d950026b5d
Given that we currently run all text through one single Varnish instance and all upload through another, I don't think we care about resiliency in beta much.
So, we need to:
Is that right @fgiunchedi ?
sounds right to me @AlexMonk-WMF, also mediawiki will need to be switched to deployment-ms-fe01 of course together with varnish
Mentioned in SAL [2016-07-08T10:31:40Z] <hashar> beta: mass delete http://commons.wikimedia.beta.wmflabs.org/wiki/Category:GWToolset_Batch_Upload files T64835
I have been purging a few files with pywikibot.
http://commons.wikimedia.beta.wmflabs.org/wiki/Special:MediaStatistics is also quite useful. 87% of files (or 26GB) are .tiff / .tif pages.
I have cleaned up temp files that @AlexMonk-WMF mentioned and we are down to roughly 3Gbytes down from roughly 25GBytes.
$ du -c -m -d1 /data/project/upload7 1 ./wiktionary 2 ./score 544 ./private 9 ./wikisource 1 ./wikiversity 2671 ./wikipedia 1 ./scripts 16 ./math 1 ./wikibooks 1 ./wikinews 1 ./wikivoyage 1 ./wikiquote 3242 . 3242 total $
Looks more manageable!
Mentioned in SAL [2016-07-09T00:46:13Z] <Krenair> T64835: Live-hacked some temporary swift config in
Mentioned in SAL [2016-07-09T00:46:24Z] <Krenair> T64835: foreachwikiindblist "% all-labs.dblist - private.dblist" extensions/WikimediaMaintenance/filebackend/setZoneAccess.php --backend=local-multiwrite
Mentioned in SAL [2016-07-09T00:46:28Z] <Krenair> T64835: mwscript extensions/WikimediaMaintenance/filebackend/setZoneAccess.php zerowiki --backend=local-multiwrite --private
I'm attempting to follow https://www.mediawiki.org/wiki/Thread:User_talk:Aaron_Schulz/Using_copyFileBackend.php_to_copy_from_old_file_repo_to_swift_cloud/reply but it only ever copies 0 files:
krenair@deployment-tin:/srv/mediawiki-staging$ mwscript copyFileBackend.php commonswiki --src local-backend --dst local-multiwrite --containers local-public --subdir 0 Doing container 'local-public', directory '0'... Copying file(s)... Copied 0 file(s). Finished container 'local-public', directory '0'. Done.
Same for all other subdirs
I found out deployment-tin does not have the NFS share /data/project/upload7 (on purpose apparently). So I guess the script can not find any.
Try from one of the app server such as deployment-mediawiki01 ?
Everything's in kind of a mess while I try this - I've got live hacks to config code on deployment-tin, a live hack (and so puppet is disabled) to make deployment-ms-fe01:/usr/local/lib/python2.7/dist-packages/wmf/rewrite.py rewrite to beta URLs instead of production ones - and I'm stuck on this HTTP 401 issue: http://upload.beta.wmflabs.org/wikipedia/commons/8/81/Icon_tools.png
https://wikitech.wikimedia.org/wiki/Swift/How_To seems hopelessly outdated
I still haven't got to figuring out the temp URL thing, cirrus key (?), or commonswiki's ForeignFileRepo, or working out whether we need to keep that GWToolset config in filebackend-labs.php
I've briefly looked at it, and it seems now the commons container is accessed in its non-sharded fashion? from /var/log/swift/proxy-access.log:
Jul 11 09:36:19 deployment-ms-fe01 proxy-server: x.x.x.x 10.68.18.109 11/Jul/2016/09/36/19 GET /v1/AUTH_mw/wikipedia-commons-local-public/8/81/Icon_tools.png HTTP/1.0 499 - Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36 - - 131 - tx1050a5de31584461b9086-0057836893 - 0.0317 - - 1468229779.385159016 1468229779.416816950
anyways let me know when you are online, I can help debug further too
Change 298297 had a related patch set uploaded (by Alex Monk):
deployment-prep: Point upload cache at swift, fix rewrite.py to use beta.wmflabs.org domains
Change 298299 had a related patch set uploaded (by Alex Monk):
[labs/deployment-prep] Switch file backends to swift
krenair@deployment-upload:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 12.04.5 LTS Release: 12.04 Codename: precise krenair@deployment-upload:~$ uname -a Linux deployment-upload 3.2.0-101-virtual #141-Ubuntu SMP Thu Mar 10 22:12:23 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux krenair@deployment-upload:~$ uptime 21:00:22 up 100 days, 23:50, 1 user, load average: 0.00, 0.01, 0.05 krenair@deployment-upload:~$ sudo halt W: molly-guard: SSH session detected! Please type in hostname of the machine to halt: deployment-upload Broadcast message from krenair@deployment-upload (/dev/pts/2) at 21:01 ... The system is going down for halt NOW!
Change 298299 merged by jenkins-bot:
[labs/deployment-prep] Switch file backends to swift
My NFS import ended up working a bit like this:
subdirs=`echo "0 1 2 3 4 5 6 7 8 9 a b c d e f archive/0 archive/1 archive/2 archive/3 archive/4 archive/5 archive/6 archive/7 archive/8 archive/9 archive/a archive/b archive/c archive/d archive/e archive/f" | tr -s " " "\012"` for subdir in $subdirs; do foreachwikiindblist "all-labs" copyFileBackend.php --src local-backend --dst local-multiwrite --containers local-public --subdir $subdir; done subdirs=`echo "0 1 2 3 4 5 6 7 8 9 a b c d e f g h i j k l m n o p q r s t u v w x y z" | tr -s " " "\012"` for subdir in $subdirs; do foreachwikiindblist "all-labs" copyFileBackend.php --src local-backend --dst local-multiwrite --containers local-deleted --subdir $subdir; done
@fgiunchedi: Okay, so I think I need your help to finish this off completely.
Change 298393 had a related patch set uploaded (by Alex Monk):
[labs/deployment-prep] Point RedisLockManager to actual redis servers
Change 298393 merged by jenkins-bot:
[labs/deployment-prep] Point RedisLockManager to actual redis servers
Mentioned in SAL [2016-07-11T23:24:36Z] <Krenair> Unmounted /data/project (NFS) on all active hosts (mediawiki0[1-3], jobrunner01, tmh01), leaving just deployment-upload (shutoff, to schedule for deletion soon) - T64835
Change 298404 had a related patch set uploaded (by Alex Monk):
Remove old pre-Swift directory variables referencing upload7
While dealing with the commit above for T129586 I noticed we probably need to rescue these from NFS:
$wgCaptchaDirectory = '/data/project/upload7/private/captcha/random'; $wgCaptchaDirectory = '/data/project/upload7/private/captcha'; $wgMathDirectory = '/data/project/upload7/math'; $wgScoreDirectory = '/data/project/upload7/score';
Along with this image that probably broke due to my messing around with the config while it was being uploaded: http://commons.wikimedia.beta.wmflabs.org/wiki/File:Test-1468270938.124573.png
(This change makes https://gerrit.wikimedia.org/r/#/c/298397/ into something that seems sane to do now by the way :))
the temp url key is set in PrivateSettings.php, and will need to be set on the swift account at runtime as outlined in http://docs.openstack.org/liberty/config-reference/content/object-storage-tempurl.html
cirrus key can live in labs' private (or secret?) git repo
- Can I put that puppet change in puppet SWAT?
sure, which change?
root@deployment-ms-fe01:/etc/swift# . account_AUTH_mw.env root@deployment-ms-fe01:/etc/swift# swift post -m "Temp-URL-Key:redacted" root@deployment-ms-fe01:/etc/swift#
Think I also sorted out the Cirrus key, hopefully.
https://gerrit.wikimedia.org/r/298297 - I've already scheduled it
Change 298297 merged by Filippo Giunchedi:
deployment-prep: Point upload cache at swift, fix rewrite.py to use beta.wmflabs.org domains
I hard-rebooted deployment-upload (turns out nova didn't like me running halt instead of powering off via nova?) to regain NFS access.
scp deployment-upload:/data/project/upload7/wikipedia/commons/4/44/Test-1468270938.124573.png Test-1468270938.124573.png
scp Test-1468270938.124573.png deployment-ms-fe01:Test-1468270938.124573.png
logged into deployment-ms-fe01, sudo -i; cd /etc/swift; . account_AUTH_mw.env; cd ~krenair; swift upload wikipedia-commons-local-public.44 Test-1468270938.124573.png --object-name 4/44/Test-1468270938.124573.png and the file was fixed.
This one is more difficult. These are all using the global-multiwrite backend, which uses the global-data wiki ID which of course my foreachwiki setZoneAccess etc. etc. run wouldn't've created anything for.
Compare P3375 with this:
root@deployment-ms-fe01:/etc/swift# swift list | grep -v local | grep -v timeline-render test-upload root@deployment-ms-fe01:/etc/swift#
@aaron, please could you comment on how those containers listed in P3375 are supposed to be created and configured?
Change 298812 had a related patch set uploaded (by Alex Monk):
[labs/deployment-prep] Remove old pre-Swift directory variables referencing /data/project/upload7
Change 298812 merged by jenkins-bot:
[labs/deployment-prep] Remove old pre-Swift directory variables referencing /data/project/upload7
@fgiunchedi got me more information about those swift containers (P3419) and I managed to recreate them in mwscript eval.php commonswiki:
$backend = FileBackendGroup::singleton()->get( 'global-multiwrite' ); $container = 'captcha-render'; $dir = $backend->getContainerStoragePath( $container ); $status = $backend->prepare( array( 'dir' => $dir ) ); $status->merge( $backend->publish( array( 'dir' => $dir, 'access' => true ) ) ); var_dump( $status->isOK() ); $container = 'math-render'; $dir = $backend->getContainerStoragePath( $container ); $status = $backend->prepare( array( 'dir' => $dir ) ); $status->merge( $backend->publish( array( 'dir' => $dir, 'access' => true ) ) ); var_dump( $status->isOK() ); $container = 'score-render'; $dir = $backend->getContainerStoragePath( $container ); $status = $backend->prepare( array( 'dir' => $dir ) ); $status->merge( $backend->publish( array( 'dir' => $dir, 'access' => true ) ) ); var_dump( $status->isOK() ); $backend = FileBackendGroup::singleton()->get( 'local-multiwrite' ); $container = 'gwtoolset-metadata'; $dir = $backend->getContainerStoragePath( $container ); $status = $backend->prepare( array( 'dir' => $dir, 'noAccess' => true, 'noListing' => true ) ); $status->merge( $backend->secure( array( 'dir' => $dir, 'noAccess' => true, 'noListing' => true ) ) ); var_dump( $status->isOK() );
That leaves us with these:
I also set up NFS on deployment-ms-fe01 and started importing from there directly:
root container is actually easy, assuming the file list in T130709 is still complete:
root@deployment-ms-fe01:~# swift post root -r '.r:*'
root@deployment-ms-fe01:~# swift stat root
Account: AUTH_mw
Container: root
Objects: 0
Bytes: 0
Read ACL: .r:*
Write ACL:
Sync To:
Sync Key:
Accept-Ranges: bytes
X-Timestamp: 1468437003.52441
X-Trans-Id: tx605a6cc76ffa466aa788c-0057869213
Content-Type: text/plain; charset=utf-8
root@deployment-ms-fe01:~# mkdir root
root@deployment-ms-fe01:~# cd root
root@deployment-ms-fe01:~/root# wget https://upload.wikimedia.org/{crossdomain.xml,favicon.ico,index.html,robots.txt}
[...]
Downloaded: 4 files, 10K in 0s (22.0 MB/s)
root@deployment-ms-fe01:~/root# ls
crossdomain.xml favicon.ico index.html robots.txt
root@deployment-ms-fe01:~/root# swift upload root *
robots.txt
index.html
favicon.ico
crossdomain.xml
root@deployment-ms-fe01:~/root# swift stat root
Account: AUTH_mw
Container: root
Objects: 4
Bytes: 10421
Read ACL: .r:*
Write ACL:
Sync To:
Sync Key:
Accept-Ranges: bytes
X-Timestamp: 1468437003.52441
X-Trans-Id: txb63b3fac4e7d4c5a90cde-0057869352
Content-Type: text/plain; charset=utf-8$wgFileBackends[] = [ 'name' => 'math-nfs', 'class' => 'FSFileBackend', 'lockManager' => 'nullLockManager', 'fileMode' => 0777, 'containerPaths' => [ 'math-render' => '/data/project/upload7/math' ], ];
and did this:
subdirs=`echo "0 1 2 3 4 5 6 7 8 9 a b c d e f" | tr -s " " "\012"` for subdir in $subdirs; do mwscript copyFileBackend.php commonswiki --src math-nfs --dst global-multiwrite --containers math-render --subdir $subdir; done
I think that's everything rescued from NFS upload7 now...
Thanks to @fgiunchedi for setting up the instances and to @ArielGlenn for getting a few last pieces of information about the production setup for me. I'm going to call this task done.
root@deployment-ms-fe01:~# swift-recon -d --human-readable =============================================================================== --> Starting reconnaissance on 2 hosts =============================================================================== [2016-07-13 20:36:28] Checking disk usage now Distribution Graph: 4% 2 ********************************************************************* Disk usage: space used: 9 GB of 214 GB Disk usage: space free: 205 GB of 214 GB Disk usage: lowest: 4.26%, highest: 4.26%, avg: 4.25607787311% ===============================================================================
Change 299123 had a related patch set uploaded (by Yuvipanda):
labs: Remove nfs for deployment-prep \o/
Change 298404 abandoned by Alex Monk:
Remove old pre-Swift directory variables referencing upload7
Reason:
I322fdd50