Hot Reloading: SelfHosted Component hot reloading (dapr#7239)

* Hot Reloading: SelfHosted Componenthot reloading Part of dapr#1172 Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature. If enabled, Daprd in Selfhosted mode will watch for resource updates from watching the resource directory(s). If a resource is detected as changed (spec is different to what is currently loaded), then Dapr will close/create/update the resource. Supports Component _except_ HTTP middlware which requires further changes to the HTTP server handler machinery. A warning is thrown if hot reloading is enabled and a HTTP middleware is updated. The hot reloader reconciler is made generic to enable other resource types to be added in future. --- When running in standalone mode, the disk loader watches the directories passed by --resources-path for updates to yaml files containing Components. When an event occurs, the reconciler and differ are responsible for determining whether any resources have been created, updated, or deleted, by comparing the local store specs with that of the remote. If any have changed, the resource is closed (if it exists), and then re-initialized (if it exists). A resource will only be closed if it has been deleted, only initialized if it has been created, and closed & initialized if it has been updated. We consider a resource to have changed generally if anything apart from its Kubernetes metadata or type meta object meta has changed, and therefore needs some kind of reloading. --- Currently, if a reloading component errors and `spec.ignoreErrors=false` then Daprd will gracefully close, like a component loaded at startup today. It is intended that in future the component will be re-inited on a backoff queue in future in the case of errors, even if `spec.ignoreErros=true`. HTTP middleware component hot reloading is not supported as it requires further changes to the HTTP server handler machinery, and I didn't want to grow this PR further. To use the HotReloading feature, which is currently only available as an alpha feature in Selfhosted mode, users need to add the following Configuration to the target Daprd; ```yaml apiVersion: dapr.io/v1alpha1 kind: Configuration metadata: name: hotreloading spec: features: - name: HotReload enabled: true ``` Signed-off-by: joshvanl <[email protected]> * Adds app ready checks for subscribed input bindings and pubsub Signed-off-by: joshvanl <[email protected]> * Linting Signed-off-by: joshvanl <[email protected]> * Linting Signed-off-by: joshvanl <[email protected]> * Linting Signed-off-by: joshvanl <[email protected]> * Linting Signed-off-by: joshvanl <[email protected]> * Linting Signed-off-by: joshvanl <[email protected]> * Updates reconciler to wait for processor queue to be empty after each event Signed-off-by: joshvanl <[email protected]> * Revert processor pending resource queue to 0 channel Signed-off-by: joshvanl <[email protected]> * Speed up hotreloading integration tests Signed-off-by: joshvanl <[email protected]> * Review comments Signed-off-by: joshvanl <[email protected]> * Move `tempDir` hot reload state test `TempDir()` to outer test scope to prevent `RemoveAll` on busy sqlite .db files. Signed-off-by: joshvanl <[email protected]> * Linitng Signed-off-by: joshvanl <[email protected]> * Linting Signed-off-by: joshvanl <[email protected]> * Adds review comments Signed-off-by: joshvanl <[email protected]> * Fix order of close and check bool in runtime_test.go Signed-off-by: joshvanl <[email protected]> * Updates state features in integration tests Signed-off-by: joshvanl <[email protected]> --------- Signed-off-by: joshvanl <[email protected]> Co-authored-by: Dapr Bot <56698301 [email protected]>
skyao · Dec 22, 2023 · 1c87383 · 1c87383
1 parent ace5b64
commit 1c87383
Show file tree

Hide file tree

Showing 70 changed files with 6,303 additions and 435 deletions.
diff --git a/dapr/proto/operator/v1/operator.proto b/dapr/proto/operator/v1/operator.proto
@@ -40,6  40,21 @@ service Operator {
   rpc HTTPEndpointUpdate (HTTPEndpointUpdateRequest) returns (stream HTTPEndpointUpdateEvent) {}
 }
 
 // ResourceEventType is the type of event to a resource.
 enum ResourceEventType {
   // UNKNOWN indicates that the event type is unknown.
   UNKNOWN = 0;
 
   // CREATED indicates that the resource has been created.
   CREATED = 1;
 
   // UPDATED indicates that the resource has been updated.
   UPDATED = 2;
 
   // DELETED indicates that the resource has been deleted.
   DELETED = 3;
 }
 
 // ListComponentsRequest is the request to get components for a sidecar in namespace.
 message ListComponentsRequest {
   string namespace = 1;
@@ -55,6  70,9 @@ message ComponentUpdateRequest {
 // ComponentUpdateEvent includes the updated component event.
 message ComponentUpdateEvent {
   bytes component = 1;
 
   // type is the type of event.
   ResourceEventType type = 2;
 }
 
 // ListComponentResponse includes the list of available components.
@@ -134,4  152,4 @@ message HTTPEndpointUpdateRequest {
 // HTTPEndpointsUpdateEvent includes the updated http endpoint event.
 message HTTPEndpointUpdateEvent {
   bytes http_endpoints = 1;
-}
 }
diff --git a/pkg/apis/components/v1alpha1/register.go b/pkg/apis/components/v1alpha1/register.go
@@ -24,8  24,8 @@ import (
 // SchemeGroupVersion is group version used to register these objects.
 var SchemeGroupVersion = schema.GroupVersion{Group: components.GroupName, Version: "v1alpha1"}
 
-// Kind takes an unqualified kind and returns back a Group qualified GroupKind.
-func Kind(kind string) schema.GroupKind {
 // GroupKind takes an unqualified kind and returns back a Group qualified GroupKind.
 func GroupKind(kind string) schema.GroupKind {
 	return SchemeGroupVersion.WithKind(kind).GroupKind()
 }
 

diff --git a/pkg/apis/components/v1alpha1/types.go b/pkg/apis/components/v1alpha1/types.go
@@ -17,9  17,15 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 
 	"github.com/dapr/dapr/pkg/apis/common"
 	"github.com/dapr/dapr/pkg/apis/components"
 	"github.com/dapr/dapr/utils"
 )
 
 const (
 	Kind    = "Component"
 	Version = "v1alpha1"
 )
 
 // genclient
 // genclient:noStatus
 // kubebuilder:object:root=true
@@ -41,6  47,16 @@ func (Component) Kind() string {
 	return "Component"
 }
 
 // GetName returns the component name.
 func (c Component) GetName() string {
 	return c.Name
 }
 
 // GetNamespace returns the component namespace.
 func (c Component) GetNamespace() string {
 	return c.Namespace
 }
 
 // LogName returns the name of the component that can be used in logging.
 func (c Component) LogName() string {
 	return utils.ComponentLogName(c.ObjectMeta.Name, c.Spec.Type, c.Spec.Version)
@@ -56,6  72,18 @@ func (c Component) NameValuePairs() []common.NameValuePair {
 	return c.Spec.Metadata
 }
 
 // EmptyMetaDeepCopy returns a new instance of the component type with the
 // TypeMeta's Kind and APIVersion fields set.
 func (c Component) EmptyMetaDeepCopy() metav1.Object {
 	n := c.DeepCopy()
 	n.TypeMeta = metav1.TypeMeta{
 		Kind:       Kind,
 		APIVersion: components.GroupName   "/"   Version,
 	}
 	n.ObjectMeta = metav1.ObjectMeta{Name: c.Name}
 	return n
 }
 
 // ComponentSpec is the spec for a component.
 type ComponentSpec struct {
 	Type    string `json:"type"`

diff --git a/pkg/apis/httpEndpoint/v1alpha1/register.go b/pkg/apis/httpEndpoint/v1alpha1/register.go
@@ -24,8  24,9 @@ import (
 // SchemeGroupVersion is group version used to register these objects.
 var SchemeGroupVersion = schema.GroupVersion{Group: httpendpoint.GroupName, Version: "v1alpha1"}
 
-// Kind takes an unqualified kind and returns back a Group qualified GroupKind.
-func Kind(kind string) schema.GroupKind {
 // GroupKind takes an unqualified kind and returns back a Group
 // qualified GroupKind.
 func GroupKind(kind string) schema.GroupKind {
 	return SchemeGroupVersion.WithKind(kind).GroupKind()
 }
 

diff --git a/pkg/apis/httpEndpoint/v1alpha1/types.go b/pkg/apis/httpEndpoint/v1alpha1/types.go
@@ -17,6  17,12 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 
 	"github.com/dapr/dapr/pkg/apis/common"
 	httpendpoint "github.com/dapr/dapr/pkg/apis/httpEndpoint"
 )
 
 const (
 	Kind    = "HTTPEndpoint"
 	Version = "v1alpha1"
 )
 
 // genclient
@@ -43,11  49,26 @@ func (HTTPEndpoint) Kind() string {
 	return kind
 }
 
 // GetName returns the component name.
 func (h HTTPEndpoint) GetName() string {
 	return h.Name
 }
 
 // GetNamespace returns the component namespace.
 func (h HTTPEndpoint) GetNamespace() string {
 	return h.Namespace
 }
 
 // GetSecretStore returns the name of the secret store.
 func (h HTTPEndpoint) GetSecretStore() string {
 	return h.Auth.SecretStore
 }
 
 // LogName returns the name of the component that can be used in logging.
 func (h HTTPEndpoint) LogName() string {
 	return h.Name   " ("   h.Spec.BaseURL   ")"
 }
 
 // NameValuePairs returns the component's headers as name/value pairs
 func (h HTTPEndpoint) NameValuePairs() []common.NameValuePair {
 	return h.Spec.Headers
@@ -83,6  104,18 @@ func (h HTTPEndpoint) HasTLSPrivateKey() bool {
 	return h.Spec.ClientTLS != nil && h.Spec.ClientTLS.PrivateKey != nil && h.Spec.ClientTLS.PrivateKey.Value != nil
 }
 
 // EmptyMetaDeepCopy returns a new instance of the component type with the
 // TypeMeta's Kind and APIVersion fields set.
 func (h HTTPEndpoint) EmptyMetaDeepCopy() metav1.Object {
 	n := h.DeepCopy()
 	n.TypeMeta = metav1.TypeMeta{
 		Kind:       Kind,
 		APIVersion: httpendpoint.GroupName   "/"   Version,
 	}
 	n.ObjectMeta = metav1.ObjectMeta{Name: h.Name}
 	return n
 }
 
 // HTTPEndpointSpec describes an access specification for allowing external service invocations.
 type HTTPEndpointSpec struct {
 	BaseURL string `json:"baseUrl" validate:"required"`

diff --git a/pkg/config/configuration.go b/pkg/config/configuration.go
@@ -43,6  43,8 @@ type Feature string
 const (
 	// Enables support for setting TTL on Actor state keys.
 	ActorStateTTL Feature = "ActorStateTTL"
 	// Enables support for hot reloading of Daprd Components and HTTPEndpoints.
 	HotReload Feature = "HotReload"
 )
 
 // end feature flags section

diff --git a/pkg/internal/apis/namevaluepair.go b/pkg/internal/apis/namevaluepair.go
@@ -13,7  13,13 @@ limitations under the License.
 
 package apis
 
-import "github.com/dapr/dapr/pkg/apis/common"
 import (
 	"fmt"
 
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 
 	"github.com/dapr/dapr/pkg/apis/common"
 )
 
 type GenericNameValueResource struct {
 	Name         string
@@ -42,3  48,11 @@ func (g GenericNameValueResource) GetSecretStore() string {
 func (g GenericNameValueResource) NameValuePairs() []common.NameValuePair {
 	return g.Pairs
 }
 
 func (g GenericNameValueResource) LogName() string {
 	return fmt.Sprintf("%s (%s)", g.Name, g.ResourceKind)
 }
 
 func (g GenericNameValueResource) EmptyMetaDeepCopy() metav1.Object {
 	return &metav1.ObjectMeta{Name: g.Name}
 }