News & Info
Discussion forums have been enabled for this project, feel free to share your thoughts and ideas there.
v2.7.1
- In v2.7.0 dumping of configuration at goslmailer/gobler startup was removed.
- Now it can be selectively enabled with
debugconfig: true
parameter in config filev2.7.0
v2.5.0
New: In addition to json, now supports TOML configuration files (.toml file suffix)
BREAKING CHANGE:
qosmap
changed its format from map[int]string to map[string]int. Adopt the goslmailer config file accordingly."qosmap": { # map of your sites configured QoSes, with their timelimits (seconds), used for hint generation - "3600": "RAPID", - "28800": "SHORT", - "172800": "MEDIUM", - "1209600": "LONG" "qosmap": { # map of your sites configured QoSes, with their timelimits (seconds), used for hint generation "RAPID": 3600, "SHORT": 28800, "MEDIUM": 172800, "LONG": 1209600 }
- message delivery to:
- gathering of job statistics
- generating hints for users on how to tune their job scripts (see examples below)
- templateable messages (readme)
- message spooling and throttling, details here...
- easy to develop extensions for other protocols, more here...
Goslmailer (GoSlurmMailer) is a drop-in replacement MailProg for slurm.
With goslmailer configured as as the slurm mailer,
MailProg = /usr/bin/goslmailer
it provides users with the ability to specify a comma-separated list of receivers [connector:]target
in the sbatch --mail-user
switch to select where the messages will be sent out (similar to URI scheme).
e.g.
sbatch --mail-type=ALL --mail-user="mailto:useremailA,msteams:usernameB,telegram:NNNNNNN,usernameC"
To support future additional receiver schemes, a connector package has to be developed and its configuration block present in configuration file.
- discord bot
--mail-user=discord:<channelId>
- matrix bot
--mail-user=matrix:<roomId>
- telegram bot
--mail-user=telegram:<chatId>
- mailto
--mail-user=mailto:<email-addr>
- msteams webhook
--mail-user=msteams:<userId>
- slack web API
--mail-user=slack:<userId>
OR--mail-user=slack:<channelId>
- mattermost --mail-user=
mattermost
:channelId
See each connector details below...
If you would like to contribute to this project by developing a new connector, here you can find a heavily annotated connector boilerplate (fully functional) to help you get started.
Option 1. Download latest precompiled binaries here
Unpack, follow instructions
git clone https://github.com/CLIP-HPC/goslmailer.git
make test
make build
make install
Prerequisites:
- generated RSA keypair (passwordless) (
ssh-keygen -t rsa
) ssh $USER@localhost
must work without password
Known caveats:
- redhat/centos: must have lsb_release binary installed, package:
redhat-lsb-core
- ubuntu 22:
set enable-bracketed-paste off
present in~/.inputrc
- maybe/maybe not, depends if you see failed tests:
export TERM=dumb
in~/.bashrc
:) - in case of the following error:
run_goslmailer: validation *exec.RunRequest failed: target was empty at exec.run at workflow.run
run_loop_over_tests_casestest_00: run_goslmailer: validation *exec.RunRequest failed: target was empty at exec.run at workflow.run
try:
mkdir $HOME/.secret
ssh-keygen -b 1024 -t rsa -f id_rsa -P "" -f $HOME/.secret/id_rsa
touch ~/.ssh/authorized_keys
cat $HOME/.secret/id_rsa.pub >> ~/.ssh/authorized_keys
# downloads endly binary and runs endly tests
make
- place binary in a path to your liking
- place goslmailer.conf here:
/etc/slurm/goslmailer.conf
(default path)- OR: anywhere else, but then run the binary with
GOSLMAILER_CONF=/path/to/gosl.conf
in environment
- OR: anywhere else, but then run the binary with
- point slurm
MailProg
to the binary
- place binary in a path to your liking
- place gobler.conf in a path to your liking
- start the service (with -c switch pointing to config file)
- place binary in a path to your liking
- place tgslurmbot.conf in a path to your liking
- config file has the same format as goslmailer , so you can use the same one (other connectors configs are not needed)
- start the service (with -c switch pointing to config file)
- place binary in a path to your liking
- place matrixslurmbot.conf in a path to your liking
- config file has the same format as goslmailer, so you can use the same one (other connectors configs are not needed)
- start the service (with -c switch pointing to config file)
- place binary in a path to your liking
- place discoslurmbot.conf in a path to your liking
- config file has the same format as goslmailer, so you can use the same one (other connectors configs are not needed)
- start the service (with -c switch pointing to config file)
In high-throughput clusters or in situations where job/message spikes are common, it might not be advisable to try to send all of the incoming messages as they arrive. For these environments goslmailer can be configured to spool messages from certain connectors on disk, to be later processed by the gobler service.
gobler is a daemon program that can be configured to monitor specific spool directories for messages, process them and send out using the same connectors as goslmailer.
On startup, gobler reads its config file and spins-up a connector monitor
for each configured spool directory.
connector monitor
in turn spins up 3 goroutines: monitor
, picker
and numSenders
x sender
.
-
monitor :
- every
monitorT
seconds (or milliseconds) scans thespoolDir
for new messages and sends them to the picker
- every
-
picker :
- on receipt of new messages performs trimming of excessive messages, limiting the number of users messages in the system to
maxMsgPU
- every
pickerT
seconds (or milliseconds) picks the next message to be delivered and sends it to the sender (ordering by time of arrival)
- on receipt of new messages performs trimming of excessive messages, limiting the number of users messages in the system to
-
sender :
numSenders
goroutines are waiting for messages from the picker and try to deliver them. In case of failure, messages are returned to the picker for a later retry
connector | spooling/throttling capable (gobler) |
---|---|
mattermost | yes |
discord | yes |
matrix | no |
telegram | yes |
msteams | yes |
mailto | no |
slack | yes |
Specifies which receiver scheme is the default one, in case when user didn't specify --mail-user
and slurm sent a bare username.
"defaultconnector": "msteams"
Mailto covers for original slurm e-mail sending functionality, plus a little bit more. With connector parameters, you can:
- specify your e-mail client (ex slurm:
MailProg
, e.g. /usr/bin/mutt) - template mail client command line (e.g. custom subject line)
- template message body
- allowList the recipients
- HTML e-mail and mutt
- to make sure mutt sends HTML email with proper content-type header, add following lines to
/etc/Muttrc.local
- to make sure mutt sends HTML email with proper content-type header, add following lines to
# Local configuration for Mutt.
set content_type="text/html"
See annotated configuration example
Prerequisites for the mattermost connector:
- Create mattermost bot, instructions here.
- OPTIONAL/CONVENIENT: Start the bot daemon service mattermostbot (example config file).
- This is used to deliver
--mail-user=mattermost:chatId
messages, can also be looked up by the user in the Info window
- This is used to deliver
- If the bot is running, it will wake up on the configured
triggerString
and send the user a private message with slurm job submission instructions - Put
mattermost
block in goslmailer.conf (and optionally gobler.conf if spooling is used)
Additional helpful sources:
Prerequisites for the discord connector:
- a discord bot must be created and
- the bot daemon service discoslurmbot must be running (example config file).
- once the bot is running, it will wake up on the configured
triggerString
and send the user a private message with slurm job submission instructions
- User settings -> Advanced -> Developer mode ON
- Discord developer portal -> New Application -> Fill out BotName
- Once the application is saved, select Bot from left menu -> Add Bot -> message: "A wild bot has appeared!"
- Left menu: OAuth2 -> Copy Client ID
- Modify this url with the Client ID from 4. and open in browser:
https://discord.com/api/oauth2/authorize?client_id=<CLIENT-ID>&permissions=8&scope=bot
- "An external application BotName wants to access your Discord Account" message -> Select server -> Continue
- Grant Administrator permissions -> yes/no/maybe ? -> Authorize
- Discord developer portal -> Select BotName -> Bot menu -> Reset Token -> Copy and Save, to be used in discoslurmbot.conf
Or follow this tutorial
- discord bot and packaged developed using discordgo and with the help of Discord Gophers
Sends 1on1 or group chat messages about jobs via telegram messenger app
Prerequisites for the telegram connector:
- a telegram bot must be created and
- the bot daemon service tgslumbot must be running.
Site admins can create a telegram bot by messaging botfather.
Once the bot is created, you will receive a bot token
. Place the bot token
in the goslmailer/gobler config file in the telegram
connector section (see example below).
Start the tgslurmbot binary that serves as the bot.
When the chat/group chat with the bot is initiated and/or the bot receives a /start
command, he will reply with a chat-specific --mail-user=telegram:nnn
message which the user can use in his slurm job scripts to get the job messages.
See annotated configuration example
Sends messages to user-defined Matrix rooms.
The Matrix connector can send messages through a pre-defined user that you manually join into channels. In addition, you can setup matrixslurmbot to control this pre-defined user and have him join rooms upon invitation.
The first thing you need is to add the proper parameters to the connectors/matrix section of the config file.
To obtain the access token, you can run this command using curl:
curl -XPOST -d '{"type":"m.login.password", "user":"@myuser:matrix.org", "password":"mypassword"}' "https://matrix.org/_matrix/client/r0/login"
Which should get you a response akin to this one, containing the token:
{
"user_id": "@myuser:matrix.org",
"access_token": "syt_dGRpZG9ib3QXXXXXXXEyQMBEmvOVp_10Jm93",
"home_server": "matrix.org",
"device_id": "DMIIOMEYBK",
"well_known": {
"m.homeserver": {
"base_url": "https://matrix-client.matrix.org/"
}
}
}
Once you've done this, goslmailer will be able to send messages to any channel the configured user is a member of with a command like this:
sbatch --mail-type=ALL --mail-user='matrix:!VEbreVXXXkmLWjeGPi:matrix.org' --wrap "ls"
Where !VEbreVXXXkmLWjeGPi:matrix.org
is the ID of the channel you want to send
the message to. You can obtain this ID from the Settings>Advanced
section of
the room.
Instead of manually having the pre-defined user join certain rooms, you can use
matrixslurmbot
to control this user and have it join rooms automatically upon invitation.
For this you just need to start the matrixslurmbot
, pointing it to a config file identical to the one before using
(you could strip it down to contain only the 'matrix' section).
While the bot is running, any user can invite it to join a channel (e.g. /invite @mybotuser:matrix.org
). The bot will join the channel and post the
--mail-user:...
argument to be used when submitting jobs in order to receive a message in that
channel.
In the event that the pre-defined user is alone in a channel (i.e. all normal users left), the bot will make the pre-defined user leave and forget the channel.
Sends a message to a preconfigured ms teams channel webhook.
Since MS Teams does not provide with the option to send messages to users directly, only to channel webhooks, we have devised a way using MS Power Automate framework to pick up messages from this one configured sink channel and deliver them via private 1on1 chats to the recipient user.
Users listed in the --mail-user=msteams:userA,msteams:userB
will be sent as adaptive card mention entity.
A MS Power Automate workflow monitors the configured sink channel, parses the received adaptive card jsons, locates the mention
entity and delivers to it the copy of the message via private chat.
See annotated configuration example
Sends messages to direct messages or channels in Slack using its Web API and slack-go.
Prerequisite for the slack connector:
- A Slack App must be created
- Go to the Slack App portal. Give your app a name (such as "SlurmBot"), and choose the workspace you want to use the app in. You must be allowed to add apps to this workspace.
- Click on OAuth & Permissions on the sidebar. Scroll down to Scopes and click on Add an OAuth Scope. Add
chat:write
andim:write
. - Scroll back up to OAuth Tokens for Your Workspace and copy the Bot User OAuth Token. It should look something like
xoxb-5555555555555-5555555555555-XXXXXXXXXXXXXXXXXXXXXXX
. This will be used in your goslmailer config (goslmailer.conf
orgoslmailer.toml
), and gobler config if used.
The bot will be able to send to any channel it's in. Make sure it is not added to channels it shouldn't send messages in (such as an announcments channel). You can add the bot to a channel by pinging it (send a message with just @SlurmBot, for example) and then click on Add to Channel.
- slack-go used in this connector.
- using adaptive card schema version 1.5 doesn't work with our adaptive card, check if some element changed in designer
- tested: 1.0, 1.2 - work