Gufran Mirza

In 2021, Elastic changed Elasticsearch's and Kibana's licenses away from open source. Like MongoDB, they had suffered from Amazon's taking the open source software developed primarily by Elastic's employees and selling it without contributing anything back to the open source projects. Amazon's success has been limited in that area while Elastic has succeeded offering its products as services on AWS As a result, Elastic has an open source option again. All of that said, some damage has occured in the interim. Read more here: https://bit.ly/4gm3fxg

4

Pub/Sub and queue look the same when you hear it for the first time.😱 Here is a simple explanation of what they are and the differences between them. In the illustration depicted below, 🔴 Messaging Queue: People come to the mall to shop. Before they check out, they must wait in the queue to complete the payment. Here, the mall is like a producer, producing people ( who have shopped and are ready to complete payment) who are sent to the queue to complete the payment. One by one, each person completes payment at the checkout counter. The checkout counter is like a consumer. As the queue grows ( more people join the line), more than one checkout ( one consumer ) is needed, and we will need to add more checkout counters ( more consumers). We have Producers, Queues, and Consumers. 🔴 Pub / Sub - Extending on the above example, after the checkout is completed, this customer data might be required by various services. For instance, in the image below, we have a real-time service that calculates how many people have completed the checkout. Hence, the checkout service has subscribed to receive this data. Likewise, marketing and inventory services might subscribe to receive this data. We have Producers, Topics, and Subscribers. 𝗣𝘂𝗯/𝘀𝘂𝗯:⏹️ The subscribers don’t have to reveal anything to the publishers. Messages are in the topic even after multiple subscribers consume it. Messages are sent to multiple subscribers simultaneously and asynchronously Message ordering and sending to various consumers is not guaranteed Producers and consumers have loose coupling 𝗤𝘂𝗲𝘂𝗲: ⏹️ Queues need to reveal who they are sending messages to Producers would put the messages in the queue, storing them until one of the consumers consumes them Producers put the messages in the queue, typically sent using some ordering and sequentially to a single receiver Message ordering and sending to consumers is guaranteed Producers and consumers have some sort of relationship #engineering #systemdesign #interview #productdevelopment #eventdrivenarchitecture

66

16 Comments

Postgres as a cache?? 🤔 I had never thought or heard of using Postgres as a cache on top of a primary Postgres DB until I came across this fascinating talk from Zerodha about their back-office Postgres deep dive. Among various interesting use cases, the talk highlights some essential principles that are helpful for both beginners and advanced users: 1) Know your data and query patterns – Understanding your data is critical for choosing the right database, partitioning strategy, and indexing. 2) Use indexes wisely – Indexes are great for speeding up queries but can slow down writes and consume space. 3) Master query plans! 4) Avoid nested joins on heavy tables – These are difficult to optimize and can drag down performance. 5) Use a caching layer – For high-traffic services, a proper caching layer is must! 6) Materialized views for read-heavy use cases – In scenarios where data doesn’t change frequently, materialized views boost performance and lower DB stress.

16

Open Policy Agent (OPA) and Rego are powerful tools for implementing policy-as-code in Kubernetes and other environments. OPA is an open-source, general-purpose policy engine that enables unified, context-aware policy enforcement across the entire stack. In Kubernetes, it's often used for admission control and authorization. Rego is the high-level declarative language used to specify policy as code in OPA. It allows you to express complex policies in a concise and readable format. Here's how OPA and Rego work together in Kubernetes: 1. Integration with Kubernetes: OPA can be deployed as a admission controller webhook in Kubernetes. This allows it to intercept and evaluate requests to the Kubernetes API server before they are persisted. 2. Writing Policies: Policies are written in Rego and typically stored in ConfigMaps or as files in the OPA container. 3. Policy Enforcement: When a request is made to the Kubernetes API, it's sent to OPA for evaluation. OPA applies the Rego policies and returns an allow/deny decision. Gatekeeper is an open-source project that provides a more Kubernetes-native way to use OPA (Open Policy Agent) for policy enforcement. It extends Kubernetes with Custom Resource Definitions (CRDs) for defining and managing policies using Rego. Reference for implementing gatekeeper for enforcing labels:

6

Use Flux and SpinKube together! Check out this latest blog post based on the conversation Sohan Maheshwar had with Flux expert Kingdon Barrett https://lnkd.in/gbGwsddq #spin #wasm #flux #cicd

11

Lunar.dev is for Lambdas too. API calls in AWS Lambda can be a traffic jam, impacting reliability, speed, and cost. Discover how Lunar.dev smooths the ride by both providing visibility and offering proactive remediations for external API errors. 🔗 Read the full blog for a smoother API Consumption journey!

30