Page MenuHomePhabricator
Create Task
Maniphest T362678

Package request: install elixir and erlang-otp to the analytics clients
Closed, ResolvedPublic
Actions

Description

My team has written a tool in elixir and we've become quite fond of it. After some experimentation with different execution contexts, it turns out that the analytics cluster is far and away the best place for us to run, mostly because we need fast access to the Enterprise HTML dumps which are mounted there. Cloud VPS disk access was at best 10% as fast.

In order to run our application, we're compiling cmake, erlang-otp and elixir manually for each user which is pretty much not ideal, but survivable. We would be happy if the following software could be installed by puppet:

Compiling from source is easy and gives the safest binaries; I would be happy to help write or review deb packaging scripts. There are also precompiled packages available.

Outcome

We'll use Conda to install erlang and elixir, this is simple and newer software versions are available.

Event Timeline

awight created this task.Apr 16 2024, 3:18 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 16 2024, 3:18 PM
awight updated the task description. (Show Details)Apr 16 2024, 3:29 PM
awight added a comment.EditedApr 16 2024, 3:34 PM

Some of these packages already appear in debmonitor:

• lbowmaker added a project: Data-Platform-SRE.Apr 16 2024, 5:22 PM
MoritzMuehlenhoff subscribed.Apr 17 2024, 7:22 AM

It's worth nothing that the stat hosts are on Bullseye/Debian 11, which being provides the following versions:

  • Erlang 23.2.6
  • Elixir 1.10.3
  • cmake 3.18.4
BTullis subscribed.Apr 17 2024, 1:52 PM

Hi @awight - I'm happy to try to help here, but as @MoritzMuehlenhoff points out, trying to get packages from the Debian repositories that match your requirements may be quite tricky.
Some of our stat hosts (stat1004-8) are also still running buster, so their versions will be even further behind, although we're currently working to bring them up to date.

Have you thought about using conda environments for this requirement? See: https://wikitech.wikimedia.org/wiki/Data_Engineering/Systems/Conda#Creating_a_new_environment for more information.
You will be able to install your own packages, principally from conda-forge and get more up-to-date packages in this way, without the need to build the toolchain from scratch for each user.

e.g.
https://anaconda.org/conda-forge/erlang (currently 26.2.4)
https://anaconda.org/conda-forge/elixir (currently 1.16.2)
https://anaconda.org/conda-forge/cmake (currently 3.29.2)

Will this approach be suitable for you?

awight closed this task as Resolved.EditedApr 18 2024, 12:29 PM
awight claimed this task.

@BTullis Thanks for highlighting this possibility! I tried the Conda environment as you suggested and it works perfectly for our needs. I was able to compile and run my app, and even at high concurrency the performance seems to be the same as with the bare metal environment I had cobbled together previously.

Closing the task :-)

awight updated the task description. (Show Details)Tue, Nov 12, 6:30 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits