UPGRAGDE NOTICE

UPGRAGDE NOTICE
If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps.
if you CubeFS version is v3.3.2 or before, and need to upgrade to v3.3.*,you must follow these upgrade steps:

1. When upgrading the metanode node, you need to add "raftSyncSnapFormatVersion": 0 to the configuration file.
2. After all metanodes are upgraded, remove this configuration item so that raftSyncSnapFormatVersion defaults to 1.
3. Restart all metanodes.
4. Please upgrade the client at the end, the master has enhanced the check of the client, otherwise it will cause the client to mount abnormally

If your Blobstore version is v1.1.0 or before which built with cubefs-blobstore (https://github.com/cubefs/cubefs-blobstore) , please refer to UPGRADE to v3.2.0 following these steps #1556.

Note:
1.Provides basic capabilities of the snapshot version in v3.4.0(beta version, disabled by default)
2.During the upgrade to version 3.4.0, there might be temporary leader unavailability in the replicas due to inconsistencies in the Raft protocol among the replicas. This issue arises from the changes made to the Raft library in version 3.4.0. Once all replicas are upgraded to version 3.4.0, this inconsistency will be resolved.

Main Feature

• master: Auto decommission for bad disk. （#3494，@bboyCH4）
• master: Meta for automatic repair of data replicas. (#3495, @bboyCH4)
• meta: Synchronize accessTime between replicas for meta. (#3496, @bboyCH4)
• cli: Cli support configuration and querying for decommission operations. (#3497, @NaturalSelect)
• master/datanode/cli: Datanode bad disk report to master adds IoErrPartititionCnt and TotalPartititionCnt. (#2679, @true1064)
• master,datanode:Retain the data of replicas that have been deleted by RaftForce for a period of time. (#3499, @bboyCH4)

Enhance

• master: Atomicity of data partition decommission operation. (#3500, @bboyCH4)
• master: Cli display the repair progress of data replica. (#3502, @bboyCH4)
• master: Querying the decommission progress of datanode is too slow. (#3504, @bboyCH4)
• master: Simplify the return results of the datanode/disk query decomission status interface. (#3507, @bboyCH4)
• master: Add a new interface to query the status of decommission tokens. (#3509, @bboyCH4)
• master: The decommission progress can display data partitions have not undergone the decommission operation. (#3512, @bboyCH4)
• master/datanode: Audit log for dp decommission (#3513, @NaturalSelect)
• master/data/meta/client: Add version information to the metrics reported by each subsystem of CubeFS. (#3516, @NaturalSelect)
• data: The ReloadSnapshot process of the data partition is taking too long. (#3517, @Victor1319)
• meta: Under heavy pressure from deleting a large number of extents, the deletion efficiency is too low. (#3518, @NaturalSelect)
• master: Simplify the return results of the datanode/disk query decomission status interface. (#3507, @bboyCH4)
• client: When volume is deleted, cfs-client need to exit. (#3512, @longerfly)
• master: Add config to control http pool size. (#3567, @Victor1319)
• metanode: Persist access time for inode in meta node . (#3565, @bboyCH4)
• client: Support libsdk to determine whether it is a file or directory based on mode.(#3566 , @longerfly)

Bugfix

• master: Decommission both replicas of the dp results in both replicas failing to decommission.. (#3498, @bboyCH4）
• master/data:After the cluster performs decommission operations, many data partitions are backed up to old directories with dates on the datanode. (#3501, @bboyCH4)
• master/data:The replica repair process is continuously retried . (#3503, @bboyCH4)
• data: Decommission operations often fail due to nodes being inactive. (#3506, @bboyCH4)
• master: Using RaftForce to decommission two replicas is stuck. (#3510, @bboyCH4)
• master: After executing the cancellation of the decommission operation, the disk remains in a disabled state. (#3511, @bboyCH4)
• master/data:The interface for recovering bad disk is timing out. (#3514, @bboyCH4)
• data: Encountering an I/O error while writing WAL logs can lead to a panic in the data service.(#3515, @bboyCH4)
• master: Only choose from the specific zone in function canWriteForNode. (#3519, @true1064)
• client: client update extents from meta and drop extents in cache leadto ek conflict. (#3520, @longerfly)
• client: if volume name not match regexp when mounted, return failed immediately. (#3522, @longerfly)
• master: The data node decommission has no response. (#3528, @bboyCH4)

UPGRAGDE NOTICE

UPGRAGDE NOTICE
If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps.
if you CubeFS version is v3.2.1 or before, and need to upgrade to v3.3.*,you must follow these upgrade steps:

1. When upgrading the metanode node, you need to add "raftSyncSnapFormatVersion": 0 to the configuration file.
2. After all metanodes are upgraded, remove this configuration item so that raftSyncSnapFormatVersion defaults to 1.
3. Restart all metanodes.
4. Please upgrade the client at the end, the master has enhanced the check of the client, otherwise it will cause the client to mount abnormally

If your Blobstore version is v1.1.0 or before which built with cubefs-blobstore (https://github.com/cubefs/cubefs-blobstore) , please refer to UPGRADE to v3.2.0 following these steps #1556.

Main Feature

• objectnode: ObjectNode support post object. （#2695，@yhjiango）
• objectnode: S3 sts and signature auth. (#2488, @yhjiango)
• libsdk/meta: Support dir lock. (#3344, @longerfly)
• master/data/meta: Support write disable option for volume. (#2537, @NaturalSelect)
• master: Support freeze volume and delayed deletion of volume. (#2922, @shuqiang-zheng)
• master: Support compressing client/partitions api response data to save bandwidth。 (#3314, @lily-lee)

Enhance

• objectnode: Concurrent multipart upload security mechanism (#2928, @tangdeyi)
• master: enable rename atomic operation by default (#2956, @Victor1319)
• master: Optimize the display of capacity when "used" is greater than "total". (#3315, @true1064)
• libsdk: Support more api function for libsdk. (#3316, @longerfly)
• data: Little io error only set current datapartition as broken. (#3317, @true1064)
• client: Write to tiny extent when file size less than 1M. (#3318, @leonrayang)
• master: Optimize CPU cost when invoking the volStat api (#3319, @true1064)
• master: Optimize the speed of applying raft snapshot for master module. (#3320, @NaturalSelect)
• meta: Support rotating and deleting inode files to avoid excessive size. (#3321, @NaturalSelect)
• data: Speed up the startup speed of the datanode. (#3322, @NaturalSelect)
• master: Optimize the volume deletion process. (#3323, @Victor1319)
• master: Add a rdonly flag to the client/partitions api to indicate whether a volume is read-only. (#3324, @true1064)
• data: Support asynchronous deletion of expired datapartitions. (#3325, @Victor1319)
• data: Prioritize listening to external service port when starting the datanode. (#3326, @Victor1319)
• data: Synchronously persist dp meta information when creating dp.(#3335, @bboyCH4)
• client: Support retrying requests for failed extent writes when the client encounters limit io error. ( #3338, @NaturalSelect)

Bugfix

• objectnode: PartNumber of multipart must be greater than 0. (#2885, @yhjiango）
• objectnode: Listobjectv1 with delimiter may lead to panic. (#2931, @tangdeyi)
• client: When the recycle bin is enabled, avoid deleting non-empty directories. (#3328, @bboyCH4)
• libsdk: Fix some issues encountered during the usage of the libsdk. (#3329, @longerfly)
• data: Optimize the blocking issue during data partition migration in snapshot synchronization. (#3330, @bboyCH4)
• meta: Fix the issue of possible duplication caused by the unpersisted meta unique id. (#3331, @true1064)
• master: Master follower replicas only request the leader replica to obtain the partition list.(#3332, @NaturalSelect)
• client: Fix the issue that interruption causes failure to write data to ec.(#3333, @Victor1319)
• data: Fix the issue of memory leaks caused by blocking during data transmission between datanode leader and follower. (#3336, @Victor1319)
• data: During the process of reusing network connections, there is a possibility of using incorrect network response messages. (#3337, @Victor1319)

UPGRAGDE NOTICE

UPGRAGDE NOTICE
If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps.
if you CubeFS version is v3.2.1 or before, and need to upgrade to v3.3.*,you must follow these upgrade steps:

1. When upgrading the metanode node, you need to add "raftSyncSnapFormatVersion": 0 to the configuration file.
2. After all metanodes are upgraded, remove this configuration item so that raftSyncSnapFormatVersion defaults to 1.
3. Restart all metanodes.
4. Please upgrade the client at the end, the master has enhanced the check of the client, otherwise it will cause the client to mount abnormally

If your Blobstore version is v1.1.0 or before which built with cubefs-blobstore (https://github.com/cubefs/cubefs-blobstore) , please refer to UPGRADE to v3.2.0 following these steps #1556.

Main Feature

• client: Add a trash feature to client (#2291, @bboyCH4)

Enhance

• datanode: Support cleaning up garbage data on the datanode (#2880, @Victor1319)
• metanode: Audit log in metanode (#2899, @NaturalSelect)
• datanode: limit datanode's disk flow and concurrent iops (#2900, @sejust)
• client: Too many waring errors log when use trash (#2915, @bboyCH4)

Bugfix

• client: Audit's writer maybe nil leadto error (#2901, @longerfly)
• gapi: fix read body entirely into memory (#2691, @tangdeyi)
• objectnode: Timing attack can leak user passwords and CubeFS leaks users key in logs from cncf security audit (#2781, @leonrayang)
• client: Optimize insecure random number generation in function util/string.go:RandomString (#2698, @true1064)
• docs: Add Security Best practice (#2853, @leonrayang)
• gapi: Timing attack can leak user passwords and CubeFS leaks users key in logs from cncf security audit (#2781, @leonrayang)
• objectnode: fix limiter lib deadlock (#2794, @tangdeyi)
• master\objectnode: some commits related with security audit (#2824, @leonrayang)
• master: qos.Lock of assignClientsNewQos forget release and trigger deadlock(#2861, @leonrayang)
• bcache: fix possible deadlock of bacahe(#2819, @longerfly)
• ClusterMgr: fix volume manager deadlock(#2793, @tangdeyi)
• blobnode: fix put shard deadlock (#2790, @mawei029)
• client: inode leak when use trash(#2911, @bboyCH4)
• client: trash encounters IO error when dealing with log file names(#2912, @bboyCH4)
• client: trash delete interval do not work(#2914, @bboyCH4)
• client: Failed to delete a large directory with trash enable(#2917, @bboyCH4)

Release v3.3.0 - 2023/08/16

UPGRAGDE NOTICE

UPGRAGDE NOTICE
If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps.
if you CubeFS version is v3.2.1 or before, and need to upgrade to v3.3.*,you must follow these upgrade steps:

1. When upgrading the metanode node, you need to add "raftSyncSnapFormatVersion": 0 to the configuration file.
2. After all metanodes are upgraded, remove this configuration item so that raftSyncSnapFormatVersion defaults to 1.
3. Restart all metanodes.
4. Please upgrade the client at the end, the master has enhanced the check of the client, otherwise it will cause the client to mount abnormally

If your Blobstore version is v1.1.0 or before which built with cubefs-blobstore (https://github.com/cubefs/cubefs-blobstore) , please refer to UPGRADE to v3.2.0 following these steps #1556.

Main Feature

• objectnode: ObjectNode Bucket Policy (#1761, @tangdeyi),CORS(#1988, @tangdeyi),S3 ACl (#1990, @yhjiango)
• master\metanode: quota added union directory feature (#2087, @baijiaruo)
• master: add mp step size configuration and dynamic adjustment (#2486, @baijiaruo)
• master\metanode: meta operations guarantee atomicity (#2487, @wuchunhuan @Victor1319)
• master\metanode\datanode:enhance system stability(#1981, @leonrayang @true1064 @Victor1319 @baijiaruo @bboyCH4 @sjp00556 @JasonHu520 )
• master\metanode: support uid space limitation (#1980, @leonrayang)
• cfsauto: cubefs surpport autofs mount (#1977, @lleiielll，@leonrayang)

Enhance

• master: broken disk check is wrong in some case (#1547, @Victor1319)
• master: Don't use global variable when implementing maxDpCntLimit func (#1897, @NaturalSelect)
• master\cli: The cli tool supports creating 1, 2 copies, and changing the number of copies (#1978, @leonrayang)
• master\cli: Support 2replicas volume down to 1 replica and recycling resources (#1979, @leonrayang)
• client: Optimize client/partitions interface implementation to improve system stability (#1981, @leonrayang)
• master: follower support metrics report (#1982, @leonrayang)
• master: balance leader meta partitions on metanodes (#1985, @xiaocai1215)
• objectnode: ObjectNode supports large file copy (#1989, @tangdeyi)
• datanode: need a quick way to handle client retries too long when reading fault datapartitions which can't be repaired, such as mark these datapartitons as discarded and trigger client fast fail (#1993, @true1064)
• master: needs a settable swich to allow or forbid metaPartition decommission in cluster (#2139, @true1064)
• objectnode: Objnode cold volume read and write buffer optimization (#2309, @tangdeyi)
• master\metanode: add file statistics function(#2489, @liubingxing)
• master: manage access iplist to support client-side IP validity check (#2490, @leonrayang)
• datanode: Strengthen the availability of datanode, and it can still start successfully when a certain dp is abnormal (#2492, @baijiaruo)

Bugfix

• objectnode: overwrite the existing mulitpart (#1700, @tangdeyi)
• unittest: Unit Test Supplement(#1903, @NaturalSelect @jjyaoao)
• metanode: Fix the problem which snapshot was interrupted but saved the metadata (#1986, @xiaocai1215)
• master: master's metrics missing dataparition and missing metaparition not deleted after the fault replica is decommissioned (#2008, @true1064)
• objectnode: ObjectNode UploadPart Interface doesn't compare the Content-Md5 with the body checksum (#2077, @tangdeyi)
• master: A response was sent but not returned (#2084, @NaturalSelect)
• objectnode: Bucket metadata synchronization (#2106, @tangdeyi)
• metanode: not use stream conn pool when delete extents in metanode (#2134, @Victor1319)
• metanode: meta partition raft follower may apply wrong cursor when apply snapshot from leader (#2149, @true1064)
• metanode: metanode create metapartition raft panic (#2230, @Victor1319)
• objectnode: fix bucket policy and location processing (#2239, @yhjiango)

Release v3.2.1 - 2023/03/16

UPGRAGDE NOTICE

UPGRAGDE NOTICE
If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

If your Blobstore version is v1.1.0 or before which built with cubefs-blobstore (https://github.com/cubefs/cubefs-blobstore) , please refer to UPGRADE to v3.2.0 following these steps #1556.

Main Feature

• meta\client: add quota of children dentries for directory (#1763, @wuchunhuan )
• master: add qps limiter for individual API on master (#1766, @wuchunhuan )
• client: add audit log for fuse client (#1764, @wuchunhuan )
• objectnode\sdk: objectnode supports docking erasure-code subsytem(blobstore) (#1765, @wuchunhuan)
• master\datanode: Add speed control of dp decommission for disk or datanode (#1778, @bboyCH4)

Enhance

• master: master support leader switching by external trigger (#1772, @leonrayang )
• metanode\master: metanode support follower read (#1775, @leonrayang)
• master: Add metrics for meta-partition consistency detection (#1776, @leonrayang)
• master: add API statistics log for master (#1767, @wuchunhuan)
• datanode: Datanode startup should not be influnced by missing data partitions (#1773, @leonrayang)
• master: volume's capacity must be bigger than 120% used size (#1723, @Victor1319)
• metanode: metanode support config memRatio of the machine. (#1725, @Victor1319)
• datanode: datanode support config diskPathPrefix to manage disk path list. (#1727, @Victor1319)
• metanode: reduce extent alloc memory when metanode snapshot (#1728, @Victor1319)
• metanode: random write may cause metanode memory grow (#1729, @Victor1319)
• datanode: bad disk space is calculated in datanode's total space (#1601, @Victor1319)
• datanode: no premission disk can't be detected (#1602, @Victor1319)
• master\cli: add display of unavailable replica for cfs-cli datapartition check (#1771 ,@true1064)
• datanode: datanode should compute datapartition used size right after loading extents from disk (#1782 ,@true1064)
• client:enhance support nfs access operation (#1798, @leonrayang )
• master: add some metircs (#1785, @liubingxing @litao)
• master\datanode\client\cli: fix the typos and format code (#1787, @liubingxing @litao)
• datanode: add lack datapartitions monitor (#1789, @guojunhao)
• master: create data partition exclude decommissioned disk (#1790 , @guojunhao)
• datanode: Optimize the qos for random write (#1791, @litao)
• metanode: speed up metanode startup (#1792, @liubingxing)
• cli: add more info to cli/fsck cmd (#1793, @liubingxing @litao)

Bugfix

• master: master snapshot recover not reset local rocksdb info (#1522, @wuchunhuan )
• master:Memory cost too fast during restart in case of data partition‘s count is magnity (#1774 , @leonrayang)
• client: Readonly dp can still accept write request from client (#1779, @bboyCH4)
• metanode: Metanode should not establish connection to blobstore for cold volume (#1781, @bboyCH4)
• client: blockcache service may be oom when the client caches many large files concurrently (#1783, @zhangtianjiong)
• datanode: too may delete requests may cause a lot of tcp connections. (#1724, @Victor1319)
• master: when master creating data partition, should use vol.dataPartitionSize as datapartition.total (#1777 ,@true1064)
• master: In addition to all replica status is readwrtie, still need to consider volume used space before check set datapartition status as readwrite (#1780 ,@true1064)
• master: master auto creates dataparitions when volume goes from full to not full (#1784 ,@true1064)
• unit test: Fix the pipeline cubefs-ci error problem (#1795, @baijiaruo)
• metanode\client:do rename on the file which is soft link not worked well (#1797 , @leonrayang)
• meta\master: fix some bug to improve cluster stability (#1786, @liubingxing @litao)

Release v3.2.0 - 2022/10/14

UPGRAGDE NOTICE

If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

If your Blobstore version is v1.1.0 or before which built with cubefs-blobstore (https://github.com/cubefs/cubefs-blobstore) , please refer to UPGRADE to v3.2.0 following these steps #1556.

Main Feature

• Merge data subsystem blobstore(ec) to main branch.
• blobstore: Merge some service modules #1563

Enhance

• blobstore: ClusterMgr support simple key-value persistent storage #1566
• blobstore: Remove blobstore's dependency on monogdb #1493
• blobstore: Remove blobstore's dependency on consul #1507
• blobstore: Support audit log filtering #1506

Bugfix

• blobstore: Blobnode use the same erasure encoder and buff-pool libraries as Access #1498
• blobstore: Separate flow control for repair and migration traffic #1508
• blobstore: Retry http body under rpc request #1567

UPGRAGDE NOTICE

If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Bugfix

• master: add leaderSize param in create partition req #1456

Enhance

• master: support decommission broken replica for 2-replica no leader dp #1456
• datanode: in case disk full, repair size is also thought as used size #1456
• raft: raftstore.monitor can be optimized #1543

UPGRAGDE NOTICE

If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Bugfix

• metanode: Show metapartition id correctly #1554
• metanode: remove redundant memcopy when reading raft snapshot #1552

Enhance

• master: update metanode or datanode id need drop old one before #1555
• metanode: add metric mpDentryCount #1553
• datanode: fallocate return interrupt err try again #1551

UPGRAGDE NOTICE

If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Main Feature

• master\client\datanode: ：Provide QoS services to enhance multi-tenant isolation #1447
• client: Caching acceleration #1446
• master\datanode: Support two replicas data storage #1389
• master: Add vol label in cfs_metanode_mpInodeCount #1482
• client: Support config posixAcl for volmue #1485
• master: Add datapartition count limit for datanode. #1481

Bugfix

• datanode: TinyDeleteRecord file will be very big when one replica is down #1433
• client: EnablePosixACL is not vaild in centos system #1393
• blockcache:Blockcache memory overflow and be killed #1422
• yum source: Yum install script fails due to missing packages #1410
• unit test: S3test random path busy to produce data and timeout #1515
• master: Zone name check not strict enough while crossZone enabled #1479
• datanode\master: Avoid writing operation failed because of disk full #1519

Enhance

• master: Enable volume update from 3 to 2 replics #1516

UPGRAGDE NOTICE
If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Main Feature

• master: support to delete data replica with force #1258

Bugfix

• master: revert the deletion process in the tiny extent to avoid deletion failure caused by different logical and physical sizes #1439
• master: add metapartition replica check to fix replicas status still writeable while metanode unavalible #1440