kyoo is the phonetic transcription of the word queue. It provides a job queue that can hold as much jobs as resources are available on the running system.
The queue has the following characteristics:
- No limit of jobs to be queued (only limited by system resources = memory)
- Concurrent processing of jobs using worker pools
- When stopping queue, pending jobs are still processed
The library contains a simple Job interface and a simple FuncExecutorJob that
just executes a given function and implements that interface. With that nearly all
kinds of workloads should be processable already but of course it is possible to add
custom implementations of the Job interface.
Possible use cases for the library are:
- Consumers for message queues like RabbitMQ or Amazon SQS
- Processing web server requests offloading time extensive work into background jobs
- All kinds of backend processing jobs like image optimization, etc.
The following example shows a simple http server offloading jobs to the jobqueue that is constantly processed in the background.
package main
import (
"fmt"
"log"
"net/http"
"runtime"
"time"
jobqueue "github.com/dirkaholic/kyoo"
"github.com/dirkaholic/kyoo/job"
)
var queue *jobqueue.JobQueue
func handler(w http.ResponseWriter, r *http.Request) {
queue.Submit(&job.FuncExecutorJob{Func: func() error {
return doTheHeavyBackgroundWork(r.URL.Path)
}})
fmt.Printf("%s - submitted %s !!\n", time.Now().String(), r.URL.Path)
fmt.Fprint(w, "Job added to queue.")
}
func main() {
queue = jobqueue.NewJobQueue(runtime.NumCPU() * 2)
queue.Start()
http.HandleFunc("/", handler)
log.Fatal(http.ListenAndServe(":8080", nil))
}
func doTheHeavyBackgroundWork(path string) error {
time.Sleep(2 * time.Second)
fmt.Printf("%s - processed %s !!\n", time.Now().String(), path)
return nil
}Test the offloading by sending a bunch of http requests to the server
$ for i in {1..10}; do http http://127.0.0.1:8080/test/$i; doneThe output on http server side should be similar like this
2020-01-09 21:36:36.156277 0100 CET m= 5.733617272 - submitted /test/1 !!
2020-01-09 21:36:36.443521 0100 CET m= 6.020861136 - submitted /test/2 !!
2020-01-09 21:36:36.730535 0100 CET m= 6.307874793 - submitted /test/3 !!
2020-01-09 21:36:37.021405 0100 CET m= 6.598744533 - submitted /test/4 !!
2020-01-09 21:36:37.311973 0100 CET m= 6.889312431 - submitted /test/5 !!
2020-01-09 21:36:37.609868 0100 CET m= 7.187208115 - submitted /test/6 !!
2020-01-09 21:36:37.895222 0100 CET m= 7.472561850 - submitted /test/7 !!
2020-01-09 21:36:38.160524 0100 CET m= 7.737863891 - processed /test/1 !!
2020-01-09 21:36:38.171491 0100 CET m= 7.748830724 - submitted /test/8 !!
2020-01-09 21:36:38.445832 0100 CET m= 8.023171514 - processed /test/2 !!
2020-01-09 21:36:38.448423 0100 CET m= 8.025762679 - submitted /test/9 !!
2020-01-09 21:36:38.730541 0100 CET m= 8.307880933 - submitted /test/10 !!
2020-01-09 21:36:38.735158 0100 CET m= 8.312497505 - processed /test/3 !!
2020-01-09 21:36:39.024788 0100 CET m= 8.602128093 - processed /test/4 !!
2020-01-09 21:36:39.315991 0100 CET m= 8.893331115 - processed /test/5 !!
2020-01-09 21:36:39.614848 0100 CET m= 9.192187633 - processed /test/6 !!
2020-01-09 21:36:39.896692 0100 CET m= 9.474031970 - processed /test/7 !!
2020-01-09 21:36:40.175952 0100 CET m= 9.753291345 - processed /test/8 !!
2020-01-09 21:36:40.451877 0100 CET m= 10.029216847 - processed /test/9 !!
2020-01-09 21:36:40.734289 0100 CET m= 10.311628415 - processed /test/10 !!