Page MenuHomePhabricator

Copy contents of https://svn.toolserver.org/ to Wikimedia Diffusion
Closed, ResolvedPublic

Description

A lot of people have their code stored in the Toolserver svn. A copy should be made to the wikimedia git server so that no code or history is lost.

An archive of the toolserver SVN repo can be found at https://archive.org/details/toolserver-svn

Details

Reference
bz58801

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 2:16 AM
bzimport added a project: Toolforge.
bzimport set Reference to bz58801.

How about a read only copy of the svn repo hosted as a tool in tool labs? It seems premature to copy all the code into gerrit repos.

(In reply to comment #1)

How about a read only copy of the svn repo hosted as a tool in tool labs? It
seems premature to copy all the code into gerrit repos.

ACK. There are also open requests to remove repos from svn.toolserver.org because they have moved to other locations (cf. https://jira.toolserver.org/browse/TS-1445, https://jira.toolserver.org/browse/TS-1573).

valhallasw also pointed out some time ago on toolserver-l (cf. http://article.gmane.org/gmane.org.wikimedia.toolserver/6370) that *any* user can copy their or other public repos off the Toolserver.

Can we agree upon Ryan's proposal to add a tool for this in Tool Labs? Who would be responsible for this tool?

That seems indeed like a reasonable approach. Given that this is "one shot", any of the current admins could create and set that tool up. I'm going to be travelling for the next week or so, but if nobody has had the opportunity to make that project by then I'll put it on my list for my return.

Proposal (from discussion at the Zürich Hackathon with Marc-André Pelletier):

Labs will host a backup of the svn repos on Toolserver. If somebody wants to create a live project out of it, it's possible. Labs staff won't do this though. Volunteers are welcome!

We'll ask Nosy/Marlen to create this backup and copy it to Labs.

(In reply to Silke Meyer (WMDE) from comment #5)

Proposal (from discussion at the Zürich Hackathon with Marc-André Pelletier):

Labs will host a backup of the svn repos on Toolserver. If somebody wants to
create a live project out of it, it's possible. Labs staff won't do this
though. Volunteers are welcome!

We'll ask Nosy/Marlen to create this backup and copy it to Labs.

Nosy has write access to /public/backups/toolserver.

What would you think of this approach:

ccing Merlijn. I hope he wants to help with the actual svn -> git import :-)

svn->git migration is not completely trivial, due to the free-form nature of SVN repos: some of the might be in the typical 'branches/', 'trunk/', 'tags/' format, but others might not.

In that aspect, just providing a read-only svn server would be the easiest option. Maybe we can run svnserve as webgrid task? I'm not sure about its security track record, though.

The list of used repositories (rev > 0) is here: http://tools.wmflabs.org/jira-bugimport/svn/000_USED_REPOSITORIES/ , including their root directory contents.

Indeed some of them are using the standard trunk/branch/tag layout, but many of them are not.

(In reply to Merlijn van Deen from comment #8)

[...]

In that aspect, just providing a read-only svn server would be the easiest
option. Maybe we can run svnserve as webgrid task? I'm not sure about its
security track record, though.

AFAIUI, that would require opening a fixed port from the outside with masking the IP for privacy reasons, etc. Setting up a lighttpd -> Apache -> mod_dav_svn chain isn't that easy either (one more advantage of Git: You can just dump a repository onto a webserver and everybody can clone it -- that's all).

However I'm still very certain that noone actually needs that :-). If the code is in active use, its authors will have moved it. If not, it's probably unusable by now. So I think having a read-only copy ( backup) in a shared directory will satisfy the demand.

Three of them have had commits in the last month (not counting nosy); 16 have had commits in the last year:

9984 May 7 2013 magnus.txt
1241 May 11 2013 geohack.txt

889 May 25  2013 multichill.txt
884 May 29  2013 qicvic.txt

2440 Jul 16 2013 mazder.txt

882 Aug 13  2013 p_wppb.txt
950 Aug 25  2013 drtrigon.txt
839 Aug 29  2013 jitse.txt

1491 Sep 2 2013 chris.txt
1086 Oct 10 2013 alexz.txt
1294 Oct 14 2013 p_erfgoed.txt

884 Oct 22  2013 p_dewpmp.txt
851 Feb 11 21:50 p_locator.txt

1365 Mar 2 16:07 gribeco.txt

830 Apr 16 07:04 nosy.txt
868 Apr 20 10:34 seth.txt
834 May  1 15:31 eccenux.txt
835 May  2 14:16 phe.txt

I think it would be a good plan if someone contacts those maintainers to see if they need any help moving. Silke, could you maybe do that?

Tried to copy the svn to /public/backups/toolserver - the mount is read-only.

Ok the stuff is still sitting in my home - where should I put it next?

I guess I will put all the stuff I backup to Labs first in my home. If this gets unhandy just let me know.

The backup can be found in tools-login:/home/nosy/svn

This has been moved to /public/backups/Toolserver/svn/

Restricted Application added a subscriber: StudiesWorld. · View Herald Transcript
bd808 added subscribers: demon, bd808.

I talked to @demon on irc and confirmed that diffusion can host SVN repos. He said it can and that he didn't see any reason we couldn't import the dump as a read-only repo.

bd808 renamed this task from Copy contents of https://svn.toolserver.org/ to Wikimedia git to Copy contents of https://svn.toolserver.org/ to Wikimedia Diffusion.Jan 21 2016, 10:34 PM
bd808 removed coren as the assignee of this task.
bd808 added a project: Diffusion.
bd808 updated the task description. (Show Details)
bd808 set Security to None.
bd808 added a subscriber: coren.

it seems that phabricator only supports accessing svn repos over ssh. I'm not sure if that will be a problem?

rTSVN repo created, downloading the svn dump now.

ok I downloaded and extracted the toolserver-svn archive. It contains 582 repositories, not just one. I'm not sure what to do about that - importing 582 more repos into phabricator isn't appealing at all.

ok I downloaded and extracted the toolserver-svn archive. It contains 582 repositories, not just one. I'm not sure what to do about that - importing 582 more repos into phabricator isn't appealing at all.

Can we just squish them into 1 big repo? SVN allows subtree checkouts.

looks like that's going to work. Currently importing the dumps into a new empty repo.

All but 4 of the repositories imported successfully into subdirectories on rTSVN. I'm not sure what to do about the 4 that had errors.

All but 4 of the repositories imported successfully into subdirectories on rTSVN. I'm not sure what to do about the 4 that had errors.

which four had errors...?

valhallasw assigned this task to mmodell.

From the list of repositories, all of them seem to have been imported. If something is missing at some point, there's always the dump at archive.org to come back to.

Aklapper closed subtask Restricted Task as Declined.Nov 12 2023, 10:50 AM

The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!