Metis

𝗕𝗲𝗶𝗻𝗴 𝗮 𝗽𝗿𝗼𝗱𝘂𝗰𝘁 𝗺𝗮𝗻𝗮𝗴𝗲𝗿 𝗶𝘀 𝗰𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗶𝗻𝗴 when developers struggle to deliver tasks from start to finish. Whether it's due to domain complexity or technological obstacles, finding better solutions is a constant priority. Thankfully, we can easily address database-related issues. Read on to learn how. 𝗧𝗲𝗮𝗺𝘀 𝗮𝗶𝗺 𝗳𝗼𝗿 𝘀𝗲𝗮𝗺𝗹𝗲𝘀𝘀 𝗱𝗮𝘁𝗮𝗯𝗮𝘀𝗲 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 at scale with secure deployments. Their goals include not only correct design but also scalable solutions through thorough testing, such as load tests, code reviews, automated CI/CD pipelines, and real-time monitoring to ensure high-quality software delivery. Despite these efforts, challenges persist - especially when it comes to database reliability. Insufficient test coverage often fails to capture performance under real-world load conditions, leading to unsatisfactory results in production and negatively affecting user satisfaction. 𝗔𝗰𝗰𝘂𝗿𝗮𝘁𝗲 𝗹𝗼𝗮𝗱 𝘁𝗲𝘀𝘁𝗶𝗻𝗴 𝗶𝘀 𝗮𝗹𝘀𝗼 𝗮 𝗵𝘂𝗿𝗱𝗹𝗲, as it's time-consuming, expensive, and often occurs too late in the development cycle to influence code changes or prevent integration issues that arise after code review merges. A lack of ownership over databases only worsens these problems. Siloed work and poor communication prevent teams from becoming fully engaged with the entire process—from code evolution to deployment, post-deployment monitoring, and proactive troubleshooting. 𝗢𝗻 𝘁𝗼𝗽 𝗼𝗳 𝘁𝗵𝗮𝘁, 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿𝘀 𝗼𝗳𝘁𝗲𝗻 𝗹𝗮𝗰𝗸 𝘁𝗵𝗲 𝗿𝗶𝗴𝗵𝘁 𝘁𝗼𝗼𝗹𝘀, forcing them to manually identify and resolve issues by sifting through logs or untangling complex interactions across various stages of the Software Development Life Cycle (SDLC), which increases debugging times and stress. However, with better tools that automatically address database issues, developers can eliminate these hurdles. This would simplify their work and lighten the load on product management, allowing teams to take full responsibility for their databases. 𝗗𝗶𝘀𝗰𝗼𝘃𝗲𝗿 𝗵𝗼𝘄 𝗠𝗲𝘁𝗶𝘀 𝗺𝗮𝗸𝗲𝘀 𝘁𝗵𝗶𝘀 𝗽𝗼𝘀𝘀𝗶𝗯𝗹𝗲, 𝘀𝘁𝗿𝗲𝗮𝗺𝗹𝗶𝗻𝗶𝗻𝗴 𝗱𝗮𝘁𝗮𝗯𝗮𝘀𝗲 𝗿𝗲𝗹𝗶𝗮𝗯𝗶𝗹𝗶𝘁𝘆, 𝗿𝗲𝗱𝘂𝗰𝗶𝗻𝗴 𝗰𝗼𝗺𝗺𝘂𝗻𝗶𝗰𝗮𝘁𝗶𝗼𝗻 𝗼𝘃𝗲𝗿𝗵𝗲𝗮𝗱, 𝗮𝗻𝗱 𝗲𝗻𝗵𝗮𝗻𝗰𝗶𝗻𝗴 𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆 by identifying issues early in the development process.

Postgres is everywhere! See why. #postgres #sql #database #monitoring #observability

Let’s see what’s new in PostgreSQL 17. 𝗤𝘂𝗲𝗿𝘆 𝗔𝗻𝗱 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝗮𝗹 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗜𝗺𝗽𝗿𝗼𝘃𝗲𝗺𝗲𝗻𝘁𝘀 PostgreSQL 17 builds on recent releases and continues to enhance performance throughout the system. The vacuum process, responsible for reclaiming storage, now uses a new internal data structure that reduces memory usage by up to 20x, while also improving the time it takes to complete its tasks. Additionally, the previous 1GB memory limit for the vacuum process (controlled by `maintenance_work_mem`) has been removed, allowing you to allocate more resources to vacuuming. This release introduces a new interface for stream I/O, leading to performance improvements in sequential scans and when running ANALYZE. PostgreSQL 17 also offers new configuration parameters that manage the scalability of transaction, subtransaction, and multixact buffers. The optimizer in PostgreSQL 17 can now use planner statistics and the sort order of common table expressions (WITH queries) to speed up query execution. The release also improves the performance of queries using the IN clause with a B-tree index. Additionally, PostgreSQL 17 can now eliminate redundant IS NOT NULL checks for columns with a NOT NULL constraint, and it skips processing queries with an IS NULL clause on NOT NULL columns. Finally, parallel index builds are now supported for BRIN indexes in this release. 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻𝗲𝗱 𝗔𝗻𝗱 𝗗𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗲𝗱 𝗪𝗼𝗿𝗸𝗹𝗼𝗮𝗱𝘀 𝗘𝗻𝗵𝗮𝗻𝗰𝗲𝗺𝗲𝗻𝘁𝘀 PostgreSQL 17 introduces greater flexibility in partition management by enabling both partition splitting and merging. It also adds support for identity columns and exclusion constraints in partitioned tables. Additionally, the PostgreSQL foreign data wrapper (postgres_fdw) now delivers performance improvements for queries with EXISTS and IN subqueries, as these can be pushed down to the remote server for execution. The release enhances logical replication with features that simplify its use in high-availability workloads and upgrades. Starting with PostgreSQL 17, logical replication slots no longer need to be dropped during upgrades with pg_upgrade, eliminating the need to resynchronize data afterward. It also introduces failover control for logical replication, offering more management options in high-availability setups. Furthermore, logical replication subscribers can now utilize hash indexes for lookups, and the new pg_createsubscriber command-line tool simplifies adding logical replication to replicas using physical replication. 𝗔𝗻𝗱 𝗠𝘂𝗰𝗵 𝗠𝗼𝗿𝗲! PostgreSQL remains the best choice for cloud-native applications. No surprise Postgres is everywhere. Read our new article (in the comments) to learn what else has changed.

Product managers often encounter numerous challenges along the way. Ideally, they want to focus on their products without being constrained by technology. However, this isn't always as simple as it seems. If you want to avoid being slowed down by your database, keep reading. 𝗔𝘃𝗼𝗶𝗱 𝗥𝗼𝗹𝗹𝗯𝗮𝗰𝗸𝘀 One of the biggest obstacles to progress is having to halt and reverse changes. This often occurs when incorrect changes are deployed to the database, especially during slow schema migrations or due to misconfigurations. 𝗘𝗻𝗮𝗯𝗹𝗲 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿𝘀 𝘁𝗼 𝗪𝗼𝗿𝗸 𝗜𝗻𝗱𝗲𝗽𝗲𝗻𝗱𝗲𝗻𝘁𝗹𝘆 Developers can face delays when they need to rely on other teams for access to logs or to troubleshoot database issues. It's hard to move quickly when they're hindered by such dependencies. Frustration builds when developers are not empowered to take full ownership of their work. 𝗣𝗹𝗮𝗻 𝗬𝗼𝘂𝗿 𝗣𝗿𝗼𝗱𝘂𝗰𝘁 𝗥𝗲𝗹𝗶𝗮𝗯𝗹𝘆 Technical issues, especially a slow database, can disrupt any product and have a widespread impact across the organization, affecting critical business metrics. Developers then need to troubleshoot and optimize database queries, often derailing planned sprint work and making it difficult to look ahead. The right tools can streamline this process. Developers need visibility into the database's inner workings to effectively analyze queries. Having access to execution plans, statistics, configurations, and other performance-related factors is key to keeping systems optimized. 𝗨𝘀𝗲 𝗠𝗲𝘁𝗶𝘀 𝗧𝗼𝗱𝗮𝘆 Empowering developers with the right tools fosters faster progress. As a product manager, it’s crucial to avoid rollbacks, outages, data loss, and production downtime. Metis helps by analyzing changes during development, testing, and deployment, allowing you to move faster, improve velocity, and enhance your DORA metrics. With Metis, your developers can focus on their work with greater confidence and efficiency.

See our new infographic about user-defined functions in SQL! #sql #database #udf

Let’s see what’s new in MySQL 9. 𝗝𝗮𝘃𝗮𝗦𝗰𝗿𝗶𝗽𝘁 𝗦𝘁𝗼𝗿𝗲𝗱 𝗣𝗿𝗼𝗴𝗿𝗮𝗺𝘀 MySQL Enterprise Edition now supports stored programs written in JavaScript. This is supported by stored procedures and stored functions and is provided by the Multilingual Engine Component (MLE). JS support conforms to the ECMAScript 2023 specification and runs in strict mode. 𝗩𝗘𝗖𝗧𝗢𝗥 𝗧𝘆𝗽𝗲 𝗦𝘂𝗽𝗽𝗼𝗿𝘁 MySQL 9.0 introduces support for the VECTOR column type, a data structure consisting of a list of entries (4-byte floating-point values). These entries can be represented as either a binary string or a list-formatted string. A VECTOR column is defined with a specified maximum length or number of entries (in parentheses); the default is 2048, with a maximum of 16,383 entries. 𝗜𝗻𝗹𝗶𝗻𝗲 𝗔𝗻𝗱 𝗜𝗺𝗽𝗹𝗶𝗰𝗶𝘁 𝗙𝗼𝗿𝗲𝗶𝗴𝗻 𝗞𝗲𝘆 𝗖𝗼𝗻𝘀𝘁𝗿𝗮𝗶𝗻𝘁𝘀 MySQL now enforces inline foreign key specifications, which were previously recognized by the parser but ignored. Additionally, MySQL 9.0 supports implicit references to primary key columns of the parent table. 𝗘𝗫𝗣𝗟𝗔𝗜𝗡 𝗔𝗡𝗔𝗟𝗬𝗭𝗘 𝗜𝗡𝗧𝗢 𝗪𝗶𝘁𝗵 𝗝𝗦𝗢𝗡 Support has been added for saving JSON output from EXPLAIN ANALYZE into a user-defined variable. The variable can later be used as a JSON argument for any of MySQL's JSON functions. 𝗘𝘃𝗲𝗻𝘁 𝗗𝗗𝗟 𝗜𝗻 𝗣𝗿𝗲𝗽𝗮𝗿𝗲𝗱 𝗦𝘁𝗮𝘁𝗲𝗺𝗲𝗻𝘁𝘀 Starting with MySQL 9.0.0, we can now use CREATE EVENT with prepared statement. 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗦𝗰𝗵𝗲𝗺𝗮 𝗦𝘆𝘀𝘁𝗲𝗺 𝗩𝗮𝗿𝗶𝗮𝗯𝗹𝗲 𝗧𝗮𝗯𝗹𝗲𝘀 MySQL 9.0 introduces two new tables in the Performance Schema that offer details about server system variables. These are variables_metadata and global_variable_attributes. 𝗖𝗼𝗿𝗿𝗲𝗹𝗮𝘁𝗲𝗱 𝗦𝘂𝗯𝗾𝘂𝗲𝗿𝘆 𝗪𝗶𝘁𝗵 𝗟𝗜𝗠𝗜𝗧 𝟭 Previously, a subquery had to exclude a LIMIT clause to be eligible for transformation into an outer left join with a derived table. In MySQL 9.0, this restriction has been relaxed slightly, allowing subqueries with a LIMIT 1 clause to undergo this transformation. MySQL has adopted a three-month release cycle, with major Long-Term Support (LTS) releases every two years. According to Oracle, in October we can expect the release of MySQL 8.4.2 LTS and MySQL 8.0.39, which will include bug fixes and security updates, as well as the MySQL 9.1 Innovation release, introducing new features along with bug and security fixes. This makes MySQL even more suitable for modern cloud-native projects.

𝗣𝗹𝗮𝘁𝗳𝗼𝗿𝗺 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝘀 𝘁𝗵𝗿𝗶𝘃𝗲 𝗼𝗻 𝗮𝘂𝘁𝗼𝗺𝗮𝘁𝗶𝗼𝗻. Our goal is to have everything scripted and operating seamlessly, without the need for manual intervention. Unfortunately, databases can be slow, unreliable, or prone to failure after deployments, impacting both business operations and customer experience. It's no wonder we're eager to resolve these issues quickly. But what if we could avoid these problems entirely? Imagine if our databases could self-heal and resolve issues automatically - or even better, prevent them from occurring in the first place. Read on to discover how Metis can help ensure your databases never fail and handle issues automatically. 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝟮.𝟬 𝗔𝘀 𝗮 𝗦𝗼𝗹𝘂𝘁𝗶𝗼𝗻 Database monitoring alone isn't sufficient for troubleshooting issues. Overloading developers with data points and indecipherable charts doesn't help them get any closer to finding solutions. The problem with raw data is that it shows us symptoms, not the causes. We might see that CPU usage has spiked or memory is running low, but it doesn't tell us what triggered the issue or how to resolve it. Platform Engineers need to protect their developers from these issues. What we need is observability 2.0. We must turn metrics into actionable insights and meaningful correlations. It’s not enough to simply see a spike in CPU usage; we need to understand what was deployed recently, what changes were made, which queries were affected, how the database configuration was altered, and more. All these pieces need to be presented together to make troubleshooting clear and straightforward. 𝗚𝗲𝘁 𝗔𝗻𝘀𝘄𝗲𝗿𝘀 𝗜𝗻𝘀𝘁𝗲𝗮𝗱 𝗼𝗳 𝗗𝗮𝘁𝗮 Capturing data points and facts is only the beginning. The real challenge is transforming those data points into a coherent narrative that explains the root cause of the issues. Is the spike in CPU usage due to an inefficient extension? Perhaps it’s a result of recent code changes, or maybe it's due to a shift in data distribution flowing into the application. There could be numerous factors contributing to system overload. To uncover the causes, we need a deep understanding of how databases operate. We need systems capable of analyzing database internals and drawing insightful conclusions. This means understanding query execution plans, statistics, index structures, data organization, and routine database operations. This should be automatically analyzed and presented to developers in a clear, actionable format. 𝗟𝗲𝘁 𝗬𝗼𝘂𝗿 𝗜𝘀𝘀𝘂𝗲𝘀 𝗙𝗶𝘅 𝗧𝗵𝗲𝗺𝘀𝗲𝗹𝘃𝗲𝘀 Once we have identified the solutions, there's no need for manual intervention anymore. Much like with CI/CD, the goal is to automate as much as possible to reduce the risk of errors and disruptions. Manual processes are often prone to mistakes, so it's best to eliminate them entirely. Plus, automation is essential for optimizing performance.

See our new infographics about #sql #partitions!

𝗕𝗲𝘆𝗼𝗻𝗱 𝗧𝗿𝗮𝗱𝗶𝘁𝗶𝗼𝗻𝗮𝗹 𝗠𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴! In today's software development landscape, creating cutting-edge applications requires an acute focus on the developers who craft these digital experiences from start to finish; henceforth, contemporary tools are not just expected but demanded to be inherently developer-centric - offering environments that promote efficiency and creativity. Observability 2.0 goes beyond traditional monitoring paradigms by embedding a continuous feedback mechanism directly into the development lifecycle itself rather than as an afterthought or separate process, demanding transparency across every phase of software production to maintain system health at all times while ensuring that code quality and application performance adhere strictly to enterprise standards. This modern approach mandates developers work within a cohesive platform ecosystem where debugging tools are tightly integrated with the IDE - this immediate access allows for quick identification, analysis, and resolution of issues directly from their coding space without extraneous context switching or external dependency on legacy systems often associated with traditional observability setups. 𝗧𝗲𝗹𝗲𝗺𝗲𝘁𝗿𝘆 𝗜𝘀 𝗧𝗵𝗲 𝗞𝗲𝘆 Observability 2.0 demands a sophisticated layer of telemetry capture, where metrics related to performance (response times), throughput (transactions per second), resource utilization (CPU and memory usage) as well as log files from each unit test or integration phase are all recorded with precision - this data is then processed using advanced analytics tools that provide a granular view of system health, enabling developers to proactively pinpoint the root cause before problems escalate into critical failures. 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 𝗪𝗶𝘁𝗵 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿𝘀’ 𝗧𝗼𝗼𝗹𝘀 Integration of developer-centric observability tools into the daily workflow requires careful planning and thoughtful architecture that supports various testing environments - this may range from unit tests to endurance runs in production replicas, ensuring that monitoring is not just an afterthought but a pervasive element throughout development. Embracing developer tools with Observability 2.0 in mind is no longer optional but rather an imperative element - developers today require these advanced features as part and parcel of their everyday coding practice just like they would rely on any other essential toolkit such as version control systems or build automation. Modern solutions must evolve beyond conventional boundaries and only Observability 2.0 can meet the requirements.

𝗢𝗿𝗴𝗮𝗻𝗶𝘇𝗶𝗻𝗴 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁 𝘁𝗲𝗮𝗺𝘀 𝗽𝗼𝘀𝗲𝘀 𝗰𝗼𝗻𝘀𝗶𝗱𝗲𝗿𝗮𝗯𝗹𝗲 𝗱𝗶𝗳𝗳𝗶𝗰𝘂𝗹𝘁𝗶𝗲𝘀 𝗳𝗼𝗿 𝗽𝗹𝗮𝘁𝗳𝗼𝗿𝗺 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝘀 due to developers' desire for component ownership, which provides several benefits such as task prioritization and workflow optimization through the adoption of new tools or processes without collaborating with other groups working on different aspects of a system. However, when it comes to databases — components that are often outside their control — these advantages can be hindered by inefficiencies resulting from cross-team dependencies. The issue becomes more pronounced during periods of high workload where developers find themselves unable to manage the database effectively without assistance due to a lack of ownership and direct access, which leads us to explore potential solutions that enable independent control over databases within development teams in subsequent sections. 𝗚𝗶𝘃𝗲 𝗧𝗵𝗲𝗺 𝗧𝗼𝗼𝗹𝘀 In your role as a platform engineer responsible for furnishing the appropriate toolsets for developer teams, one primary objective is facilitating their ability to take comprehensive ownership over both codebase management and database interaction — an endeavor that necessitates not just autonomy but also access to tools empowering them with knowledge in query optimization. These tools should serve as catalysts for developers' ability to independently evaluate the execution of queries in real-time and assess schema modifications with confidence concerning data integrity while minimizing risks such as loss or corruption, all through an interface that provides actionable metrics and automates vital testing processes without requiring continuous developer oversight. 𝗟𝗲𝘁 𝗧𝗲𝗮𝗺𝘀 𝗕𝗲 𝗜𝗻𝗱𝗲𝗽𝗲𝗻𝗱𝗲𝗻𝘁 One persistent impediment for developers is excessive reliance on meetings coupled with real-time dialogue, both known to be substantial consumers of valuable working hours that could otherwise contribute towards meaningful progress in development tasks. To alleviate this issue and facilitate greater autonomy amongst the team members specifically regarding database management — a domain often left outsourced due to complex interdependencies on other components—providing tools is pivotal so they can independently handle all facets of their databases. By granting them such capabilities, developers are empowered to self-manage and optimize this critical system component without the need for continuous external input or assistance from others whose work might not directly impact database functionality but still requires coordination due to its integral role in overall project success. Platform engineers should empower developers to take full ownership of their databases. Embrace Metis and unlock the full potential of your development teams.