Keeping cross service calculated/aggregated data up-to-date in realtime app #2464
Replies: 9 comments
-
@kshitizshankar This feature can be done in a simpler way. |
Beta Was this translation helpful? Give feedback.
-
@kshitizshankar Have some similar need. I'm also using feathers-vuex and feathers-graph-populate. Can you share some code or pseudo-code of your inbox-stats service? I can't figure out how you refresh your data. Thanks. |
Beta Was this translation helpful? Give feedback.
-
@DaddyWarbucks I am not that familiar with React - been using Vue and Feathers Vuex (which has direct integration to feathers API). The problem for me was not regarding joining two or more records - but more about computed properties (aggregated data). Consider the Gmail app - you have a count that shows the number of unread emails in your inbox. Whenever you mark an email(s) as read or unread, the count updates. To be able to do that, some piece of code needs to know that whenever an email's read field is changed - the count for that inbox needs to be updated. For this - my approach was - to create a separate service called "inbox-stats" - that listens to the "patched, removed" events from "emails" service and calculates and sends the updated stats if needed. For populating data and relationships - I am using https://github.com/marshallswain/feathers-graph-populate which hooks to the Feathers Vuex. Populating on the client as you described works as well and might even be better depending on the use case (feathersjs-ecosystem/feathers-vuex#397 (comment)) |
Beta Was this translation helpful? Give feedback.
-
I am interested in this as well. I have done some little experiments over at https://github.com/DaddyWarbucks/test-feathers-client-joins. I found the client side joins to have pretty impressive performance, even on REST. I am not sure if you are using react, but I have been interested in https://github.com/tannerlinsley/react-query lately. I think it will pair really well with a feathers client in general. Furthermore, I think you could write some custom hooks that do all of the "aggregation" for you there at the view level (instead of the service level). Maybe something like const listId = 1;
const { loading, data: list } = useFeathers(['lists', listId, { include: ['cards', ...] }]) // Get the list
const { data: list } = useQuery([listId], app.service('lists').get(listId));
// Rather than doing this next bit manually, you would read from the parent hook query `include`
// and loop over those properties to know what joins/aggregations to include. This would
// be pretty performant with a batchLoader and along with react-query's caching and such
// it should be pretty decent. You would also need to delete this `include` prop from the query
// before going to the actual service because its not a real query, just a way we are informing
// this next logic to aggregate/join our data the same way we might use it to inform fastJoin
// to do the joins at the service level. So we essentially decouple the joining from the service
// level and couple it at the view level...view level fastJoin basically.
// Then get the lists's cards
const { isIdle, data: cards } = useQuery(
['cards', list?.id],
app.service('lists').find({ query: { listId: list.id } }),
{
// `list` would be `null` at first (falsy),
// so the query will not execute until the list exists
enabled: list
}
);
// Once all joins/aggregations are completed, returned the final loading state and data Then you just bust the cache of individual service caches and the aggregated views will rerender. app.service('cards').on('created', data => {
// I am not sure the actual syntax here, but I know you can bust the cards service cache.
// So by busting this cache, our aggregated view will refresh its cards piece and rerender
reactQueryCache.bust(['cards'])
}) I haven't test this idea at all. But I have been thinking about it and I stumbled across this issue and thought I'd share. |
Beta Was this translation helpful? Give feedback.
-
@J3m5 Awesome! Makes me feel a lot better about this approach lol The only thing that came in the way of implementing this was - since we don't get the full hook context in custom events, I had to "hack" around it by sending additional data that is needed by the publishing logic to figure out the right channels (the user id and/or another identifier). Not sure if it breaks anything from the feathers perspective - will continue testing it out and see if I run into any issues. |
Beta Was this translation helpful? Give feedback.
-
I really like the way you think about it @kshitizshankar , this should to bee explored further. |
Beta Was this translation helpful? Give feedback.
-
@miguelrk I am exploring an events-based approach. I realized that the problem can be looked at from a different perspective. Basically, the core thing I was struggling with was - depending on an action that was executed on a service - I want to execute actions on other services. Currently, with the hooks based approach - I had to create hooks in the cards services itself that would handle what other services need to be updated when cards get updated. This results in too much coupling and becomes harder and harder to manage. To solve that, I am looking into an events-based approach. Instead of hooks that do cross-service actions, I am listening for events on other services that want to "subscribe" to changes and perform actions. So for the example, I mentioned, when the For the reactive aggregates - The same thing can be extended and the custom service can listen to events from the related services, figure out what needs to be changed and emit the changed values to the client. I am still exploring this - but so far it has been working quite nicely. |
Beta Was this translation helpful? Give feedback.
-
@miguelrk Sorry I missed your comment. I haven't looked more into it yet but I am doing it on the front-end for now (which will stop being viable for me soon). I guess the underlying problem to solve is how to make a reactive aggregate pipeline. I will look deeper into it soon - will add my findings here. |
Beta Was this translation helpful? Give feedback.
-
Hey @kshitizshankar! Somewhat late I guess but I also have a similar use case. Did you happen to solve this? I was thinking about creating a custom dataframe service, which would work sort of like a window above any other feathers service (e.g. readings, which are already stored). The dataframe service could then expose functionality to aggregate for instance. Else I have also thought about doing aggregations entirely client side, above any normal feathers query (e.g. GET /readings). However, doing it on the client might slow down performance and would miss several benefits of doing it on the server. Anyways, I would be grateful if you could share some light regarding this |
Beta Was this translation helpful? Give feedback.
-
Context: I've been trying to find examples/guides that might go over creating realtime charts and dashboards with FeathersJS to figure out how to make aggregated properties update in realtime with the rest of the app.
Consider a Trello board - each board has a bunch of lists and each list has a bunch of cards.
So, a simple setup with 3 services Boards, Lists, and Cards with a DB schema as follows
In my lists, I want to show a count of the number of cards present in that list - I added a fastJoin hook to calculate those counts but how do we keep these counts reactive? Right now, I have to write a bunch of hooks in cards service which detects if the listId is being changed and trigger an update on the old listid and new listid to recalculate the counts and broadcast them to the channels.
Is that how it's supposed to be done? - Feels like I might be doing something wrong here...
Building up from that, say I want to show realtime charts with aggregated values across a bunch of services - I can write a custom service to calculate those values and query that service from the client - BUT how do I "recalculate" them when one of the underlying values change? If I try to do it with the above approach - It just blows up with so much coupling between services that it becomes a nightmare to build and maintain.
What am I missing here?
Thanks!!
Beta Was this translation helpful? Give feedback.
All reactions