{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":772490032,"defaultBranch":"main","name":"zed","ownerLogin":"osbre","currentUserCanPush":false,"isFork":true,"isEmpty":false,"createdAt":"2024-03-15T09:45:09.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/23292709?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1710495916.136745","currentOid":""},"activityList":{"items":[{"before":"3e6a9f68907154009f6741bb8a2ec037c00b7bd8","after":"0533923f91c4a7ce2bb4fb74b82cdc881a193aa2","ref":"refs/heads/main","pushedAt":"2024-04-09T16:56:49.000Z","pushType":"push","commitsCount":37,"pusher":{"login":"osbre","name":"Ostap Brehin","path":"/osbre","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/23292709?s=80&v=4"},"commit":{"message":"Reduce memory usage to represent buffers by up to 50% (#10321)\n\nThis should help with some of the memory problems reported in\r\nhttps://github.com/zed-industries/zed/issues/8436, especially the ones\r\nrelated to large files (see:\r\nhttps://github.com/zed-industries/zed/issues/8436#issuecomment2037442695),\r\nby **reducing the memory required to represent a buffer in Zed by\r\n~50%.**\r\n\r\n### How?\r\n\r\nZed's memory consumption is dominated by the in-memory representation of\r\nbuffer contents.\r\n\r\nOn the lowest level, the buffer is represented as a\r\n[Rope](https://en.wikipedia.org/wiki/Rope_(data_structure)) and that's\r\nwhere the most memory is used. The layers above — buffer, syntax map,\r\nfold map, display map, ... — basically use \"no memory\" compared to the\r\nRope.\r\n\r\nZed's `Rope` data structure is itself implemented as [a `SumTree` of\r\n`Chunks`](https://github.com/zed-industries/zed/blob/8205c52d2bc204b8234f9306562d9000b1691857/crates/rope/src/rope.rs#L35-L38).\r\n\r\nAn important constant at play here is `CHUNK_BASE`:\r\n\r\n`CHUNK_BASE` is the maximum length of a single text `Chunk` in the\r\n`SumTree` underlying the `Rope`. In other words: It determines into how\r\nmany pieces a given buffer is split up.\r\n\r\nBy changing `CHUNK_BASE` we can adjust the level of granularity\r\nwithwhich we index a given piece of text. Theoretical maximum is the\r\nlength of the text, theoretical minimum is 1. Sweet spot is somewhere\r\ninbetween, where memory use and performance of write & read access are\r\noptimal.\r\n\r\nWe started with `16` as the `CHUNK_BASE`, but that wasn't the result of\r\nextensive benchmarks, more the first reasonable number that came to\r\nmind.\r\n\r\n### What\r\n\r\nThis changes `CHUNK_BASE` from `16` to `64`. That reduces the memory\r\nusage, trading it in for slight reduction in performance in certain\r\nbenchmarks.\r\n\r\n### Benchmarks\r\n\r\nI added a benchmark suite for `Rope` to determine whether we'd regress\r\nin performance as `CHUNK_BASE` goes up. I went from `16` to `32` and\r\nthen to `64`. While `32` increased performance and reduced memory usage,\r\n`64` had one slight drop in performance, increases in other benchmarks\r\nand substantial memory savings.\r\n\r\n| `CHUNK_BASE` from `16` to `32` | `CHUNK_BASE` from `16` to `64` |\r\n|-------------------|--------------------|\r\n|\r\n![chunk_base_16_to_32](https://github.com/zed-industries/zed/assets/1185253/fcf1f9c6-4f43-4e44-8ef5-29c1e5d8e2b9)\r\n|\r\n![chunk_base_16_to_64](https://github.com/zed-industries/zed/assets/1185253/d82a0478-eeef-43d0-9240-e0aa9df8d946)\r\n|\r\n\r\n### Real World Results\r\n\r\nWe tested this by loading a 138 MB `*.tex` file (parsed as plain text)\r\ninto Zed and measuring in `Instruments.app` the allocation.\r\n\r\n#### standard allocator\r\nBefore, with `CHUNK_BASE: 16`, the memory usage was ~827MB after loading\r\nthe buffer.\r\n\r\n| `CHUNK_BASE: 16` |\r\n|---------------------|\r\n|\r\n![memory_consumption_chunk_base_16_std_alloc](https://github.com/zed-industries/zed/assets/1185253/c1e04c34-7d1a-49fa-bb3c-6ad10aec6e26)\r\n|\r\n\r\n\r\nAfter, with `CHUNK_BASE: 64`, the memory usage was ~396MB after loading\r\nthe buffer.\r\n\r\n| `CHUNK_BASE: 64` |\r\n|---------------------|\r\n|\r\n![memory_consumption_chunk_base_64_std_alloc](https://github.com/zed-industries/zed/assets/1185253/c728e134-1846-467f-b20f-114a582c7b5a)\r\n|\r\n\r\n\r\n#### `mimalloc`\r\n\r\n`MiMalloc` by default and that seems to be pretty aggressive when it\r\ncomes to growing memory. Whereas the std allocator would go up to\r\n~800mb, MiMalloc would jump straight to 1024MB.\r\n\r\nI also can't get `MiMalloc` to work properly with `Instruments.app` (it\r\nalways shows 15MB of memory usage) so I had to use these `Activity\r\nMonitor` screenshots:\r\n\r\n| `CHUNK_BASE: 16` |\r\n|---------------------|\r\n|\r\n![memory_consumption_chunk_base_16_mimalloc](https://github.com/zed-industries/zed/assets/1185253/1e6e05e9-80c2-4ec7-9b0e-8a6fa78836eb)\r\n|\r\n\r\n| `CHUNK_BASE: 64` |\r\n|---------------------|\r\n|\r\n![memory_consumption_chunk_base_64_mimalloc](https://github.com/zed-industries/zed/assets/1185253/8a47e982-a675-4db0-b690-d60f1ff9acc8)\r\n|\r\n\r\n### Release Notes\r\n\r\nRelease Notes:\r\n\r\n- Reduced memory usage for files by up to 50%.\r\n\r\n---------\r\n\r\nCo-authored-by: Antonio ","shortMessageHtmlLink":"Reduce memory usage to represent buffers by up to 50% (zed-industries…"}},{"before":"0325bda89a0b52de465cf01c1d36030fcb071d3b","after":"3e6a9f68907154009f6741bb8a2ec037c00b7bd8","ref":"refs/heads/main","pushedAt":"2024-04-08T02:27:23.000Z","pushType":"push","commitsCount":4,"pusher":{"login":"osbre","name":"Ostap Brehin","path":"/osbre","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/23292709?s=80&v=4"},"commit":{"message":"Bump PyGithub","shortMessageHtmlLink":"Bump PyGithub"}},{"before":"b9f7a37f5dbde0a38841c1dbf6da9e0f06032dac","after":"0325bda89a0b52de465cf01c1d36030fcb071d3b","ref":"refs/heads/main","pushedAt":"2024-04-06T17:55:23.000Z","pushType":"push","commitsCount":364,"pusher":{"login":"osbre","name":"Ostap Brehin","path":"/osbre","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/23292709?s=80&v=4"},"commit":{"message":"Improve lsp notifications (#10220)\n\n1. They now will not go off-screen\r\n2. You can scroll long messages.\r\n3. Only one notification per language server is shown at a time\r\n4. The title/text are now distinguished visually\r\n5. You can copy the error message to the clipboard\r\n\r\nFixes: #10217\r\nFixes: #10190\r\nFixes: #10090\r\n\r\nRelease Notes:\r\n\r\n- Fixed language server notifications being too large\r\n([#10090](https://github.com/zed-industries/zed/issues/10090)).","shortMessageHtmlLink":"Improve lsp notifications (zed-industries#10220)"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAELDwkHwA","startCursor":null,"endCursor":null}},"title":"Activity · osbre/zed"}