This is the replacement for Pulsus which has been steadily serving up to 100M push notifications. But, given that it was still using the binary APNS protocol it was due for an upgrade.
Design:
- Asynchronous: a push client can just fire & forget.
- Multiple workers per push service.
- Less moving parts: when using Redis, you can push directly to the queue, bypassing the need for the Shove server to be up and running.
Supported push services:
- APNS
- Email: supports automatic creation of email digests in case the rate limit is exceeded
- FCM
- Telegram: supports squashing multiple messages into one in case the rate limit is exceeded
- Webhook: issue arbitrary webhook posts
- Web Push
Features:
- Feedback: asynchronously receive information on invalid device tokens.
- Queueing: both in-memory and persistent via Redis.
- Exponential back-off in case of failure.
- Prometheus support.
- Squashing of messages in case rate limits are exceeded.
Usage:
$ shove -h
Usage of ./shove:
-api-addr string
API address to listen to (default ":8322")
-apns-certificate-path string
APNS certificate path
-apns-sandbox-certificate-path string
APNS sandbox certificate path
-apns-workers int
The number of workers pushing APNS messages (default 4)
-email-host string
Email host
-email-port int
Email port (default 25)
-email-rate-amount int
Email max. rate (amount)
-email-rate-per int
Email max. rate (per seconds)
-email-tls
Use TLS
-email-tls-insecure
Skip TLS verification
-fcm-credentials-file string
Path to FCM service account JSON file
-fcm-workers int
The number of workers pushing FCM messages (default 4)
-queue-redis string
Use Redis queue (Redis URL)
-telegram-bot-token string
Telegram bot token
-telegram-rate-amount int
Telegram max. rate (amount)
-telegram-rate-per int
Telegram max. rate (per seconds)
-telegram-workers int
The number of workers pushing Telegram messages (default 2)
-webhook-workers int
The number of workers pushing Webhook messages
-webpush-vapid-private-key string
VAPID public key
-webpush-vapid-public-key string
VAPID public key
-webpush-workers int
The number of workers pushing Web messages (default 8)
Start the server:
$ shove \
-api-addr localhost:8322 \
-queue-redis redis://redis:6379 \
-fcm-credentials-file /etc/shove/fcm/credentials.json \
-apns-certificate-path /etc/shove/apns/production/bundle.pem -apns-sandbox-certificate-path /etc/shove/apns/sandbox/bundle.pem \
-webpush-vapid-public-key=$VAPID_PUBLIC_KEY -webpush-vapid-private-key=$VAPID_PRIVATE_KEY \
-telegram-bot-token $TELEGRAM_BOT_TOKEN
Push an APNS notification:
$ curl -i --data '{"service": "apns", "headers": {"apns-priority": 10, "apns-topic": "com.shove.app"}, "payload": {"aps": { "alert": "hi"}}, "token": "81b8ecff8cb6d22154404d43b9aeaaf6219dfbef2abb2fe313f3725f4505cb47"}' http://localhost:8322/api/push/apns
A successful push results in:
HTTP/1.1 202 Accepted
Date: Tue, 07 May 2019 19:00:15 GMT
Content-Length: 2
Content-Type: text/plain; charset=utf-8
OK
Push an FCM notification:
$ curl -i --data '{"message": {"notification": {"body": "Hello world!", "title": "Test"}, "token": "c7VmdNNHQaGTLkmi....15CmMs"}}' http://localhost:8322/api/push/fcm
Push a Webhook call, containing arbitrary body content:
$ curl -i --data '{"url": "http://localhost:8000/api/webhook", "headers": {"foo": "bar"}, "body": "Hello world!"}' http://localhost:8322/api/push/webhook
Or, post JSON:
$ curl -i --data '{"url": "http://localhost:8000/api/webhook", "headers": {"foo": "bar"}, "data": {"hello": "world!"}}' http://localhost:8322/api/push/webhook
Push a WebPush notification:
$ curl -i --data '{"subscription": {"endpoint":"https://updates.push.services.mozilla.com/wpush/v2/gAAAAAc4BA....UrjGlg","keys":{"auth":"Hbj3ap...al9ew","p256dh":"BeKdTC3...KLGBJlgF"}}, "headers": {"ttl": 3600, "urgency": "high"}, "token": "use-this-for-feedback-instead-of-subscription", "payload": {"hello":"world"}}' http://localhost:8322/api/push/webpush
The subscription (serialized as a JSON string) is used for receiving
feedback. Alternatively, you can specify an optional token parameter as done
in the example above.
Push a Telegram notification:
$ curl -i --data '{"method": "sendMessage", "payload": {"chat_id": "12345678", "text": "Hello!"}}' http://localhost:8322/api/push/telegram
Note that the Telegram Bot API documents chat_id as "Integer or String" --
Shove requires strings to be passed. For users that disconnected from your bot
the chat ID will be communicated back through the feedback mechanism. Here, the
token will equal the unreachable chat ID.
Outdated/invalid tokens are communicated back. To receive those, you can periodically query the feedback channel to receive token feedback, and remove those from your database:
$ curl -X POST 'http://localhost:8322/api/feedback'
{
"feedback": [
{"service":"apns-sandbox",
"token":"881becff86cbd221544044d3b9aeaaf6314dfbef2abb2fe313f3725f4505cb47",
"reason":"invalid"}
]
}
In order to keep your SMTP server safe from being blacklisted, the email service supports rate limitting. When the rate is exceeded, multiple mails are automatically digested.
$ shove \
-email-host localhost \
-email-port 1025 \
-api-addr localhost:8322 \
-email-rate-amount 3 \
-email-rate-per 10 \
-queue-redis redis://localhost:6379
Push an email:
$ curl -i -X POST --data @./scripts/email.json http://localhost:8322/api/push/email
If you send too many emails, you'll notice that they are digested, and at a later time, one digest mail is being sent:
2021/03/23 21:15:57 Using Redis queue at redis://localhost:6379
2021/03/23 21:15:57 Initializing Email service
2021/03/23 21:15:57 Serving on localhost:8322
2021/03/23 21:15:57 Shove server started
2021/03/23 21:15:57 email: Worker started
2021/03/23 21:15:57 email: Digester started
2021/03/23 21:15:58 email: Sending email
2021/03/23 21:15:59 email: Sending email
2021/03/23 21:15:59 email: Sending email
2021/03/23 21:16:00 email: Rate to [email protected] exceeded, email digested
2021/03/23 21:16:12 email: Rate to [email protected] exceeded, email digested
2021/03/23 21:16:18 email: Sending digest email
Shove is being used to push a high volume of notifications in a production environment, consisting of various microservices interacting together. In such a scenario, it is important that the various services are not too tightly coupled to one another. For that purpose, Shove offers the ability to post notifications directly to a Redis queue.
Posting directly to the Redis queue, instead of using the HTTP service endpoints, has the advantage that you can take Shove offline without disturbing the operation of the clients pushing the notifications.
Shove intentionally tries to make as little assumptions on the notification payloads being pushed, as they are mostly handed over as is to the upstream services. So, when using Shove this way, the client is responsible for handing over a raw payload. Here's an example:
package main
import (
"encoding/json"
"codeberg.org/pennersr/shove/pkg/shove"
"log"
"os"
)
type FCMNotification struct {
To string `json:"to"`
Data map[string]string `json:"data,omitempty"`
}
func main() {
redisURL := os.Getenv("REDIS_URL")
if redisURL == "" {
redis_URL = "redis://localhost:6379"
}
client := shove.NewRedisClient(redisURL)
notification := FCMNotification{
To: "token....",
Data: map[string]string{},
}
raw, err := json.Marshal(notification)
if err != nil {
log.Fatal(err)
}
err = client.PushRaw("fcm", raw)
if err != nil {
log.Fatal(err)
}
}
Used in production, over at:
- Drakdoo: Indicator based signals & alerts: 365.251.428 alerts fired and counting.