You can trash everything! Sorry for the delayed response.

Hi @Aklapper, Thanks for asking! I think this got stuck in code review. I'm happy to step in and move it forward once folks have time to code review it.

In T217922#5032775, @Halfak wrote:

@Shilad ^ any opinions on this database?

I've been lurking on this issue, and just wanted chime in with one bit of information I learned through experience. I'm not sure what model of Dell server you are using, but there are some subtle issues with fitting GPUs in a 2U server (and maybe others). In my case, a Nvidia GeForce card (which I know you aren't considering) technically fit in my 2U server, but the power location for the card meant it extruded beyond the enclosure.

That makes sense. There are plenty of other avenues I can explore without a GPU.

I have availability this week and next, but I think @Ottomata is right. It will be tough to do this work if it's attached to a production machine. How feasible is it to move it to one?

@dr0ptp4kt Thanks for chiming in, and I hope the leave was restful!

Hi all, I’m in a position to put a GPU to use now and am happy to help if I can. I want to make sure I understand where things stand: is the issue with the encumbered kernel driver or the open cl library? If it’s the former the situation is much more difficult.

Hi all, I am reopening this. Hooray :)

Yes! I think this is all set. I'm currently working on T174796 to create data needed for these instances.

Indeed, I now have Yarn access! Thanks @elukey!

@JAllemandou, thanks for the pointers! I think there's a little confusion on this, though. I volunteered to productionize Navigation Vectors (see T174796). I'm happy to also work on clickstream once this is done, but I think it will take several months to wrap up Navigation Vectors because of my teaching commitments.

Everything looks good now! Thanks for your quick help, @Ottomata! I'm going to close this ticket and get to work :)

One follow-up: The Navigation Vectors project uses Hive queries, so I think I also need the analytics-privatedata-users role. Is this correct? If so, should I start a new ticket, or can that also be added to this ticket?

Yes! I updated Help:SSH to indicate that DSA is being phased out.

@herron I am having some trouble logging in. I can get to bastion but not beyond. I'm suspicious that the key I gave you is a DSS key, not a DSA key. I requested a DSA key, but the public key starts with ssh-dss instead.

@RobH, I've signed the L3. My wikitech username is "Shilad Sen" and my preferred shell username is shiladsen. I created a new public SSH key for the production environment, and it is below I think that should be everything!

Also, I'll probably be using Docker images (we have a WikiBrain docker image). I presume that it's better to run the Docker image in a VM rather than on the host, but please let me know if that's not correct.

Thanks @Halfak and @Andrew This is exciting!

I am happy to help with engineering on this if we can find a way to make that work. I've set up navigation-based word2vec pipelines in similar environments (PySpark, Oozie, etc.) in the past.

Great! 24GB of memory and 4 cores would be great if that works for you.

Good questions! The big files are statistical models. So they take a while to build (a day or two), but they can be easily recreated. I think your suggestion of swapping the VMs over time seems reasonable. My only thought is if we could have a little more wiggle room... perhaps 300GB.. that would substantially reduce the rate at which we had to turn over the images.

I think Aaron was saying that although 200GB would probably work right now, it would't hold Wikipedia for very long. 500GB would definitely last for 5 years. Somewhere in between those sizes would work for a few years. Two of the large stoarage VMs would be plenty initially.

Just to follow up on this. Aaron's estimates are pretty accurate. The disk cached data structures require about 200GB for larger language editions right now. We would likely expand to 500GB over time (or if we require "more advanced" WikiBrain features). Is this possible?

I have spent quite a bit of time on this over the past few years. I do have a service that I could make available as an endpoint. HOWEVER, from what I've seen in my projects a much better approach is combining the work of Ellery Wulczyn on navigation vectors (https://meta.wikimedia.org/wiki/Research:Wikipedia_Navigation_Vectors) with the "standard" content-based approaches from Wikipedia.

Perfect. Thanks!

Hello! I believe I listed the wrong Wikitech username. It should be "Shilad Sen" instead of just "Shilad". Are you able to change this? Sorry for the mistake!

Than you!

As Aaron said, this software will push the limits of your largest VM (16GB). I'd feel much safer if I knew our system was sandboxed and had no possibility of affecting other software running on tools.wmflabs.org.