AWS Storage Blog

Optimizing storage management with Amazon FSx for Lustre storage quotas

Data-intensive use cases across industries are getting larger each year. Use cases that require multi-user file systems, such as user shares for data science or computational engineering, can benefit from cloud solutions to help manage file storage consumption.

In this blog post, I walk you through the Amazon FSx for Lustre storage quota feature, discussing quota parameters and quota administration commands. This feature enables customers to limit the amount of disk space and the number of files that a user or a group can use. It can set quotas on both users, and groups consuming several files and disk space resources.

Storage quota administration

You can use Amazon FSx for Lustre storage quotas to manage and regulate the storage usage of set of users and/or groups in your organization. The storage quotas are useful for organizations that use Amazon FSx for Lustre across multiple sets of users running own separate workloads. After mounting the FSx for Lustre file system, you can administer storage quotas using Lustre File System (LFS) quota control commands on Lustre clients. Root users can set quotas on the lustre client for individual users and groups, though it is not possible to enforce storage quotas on the root user. If the group quota exists on the file system, Amazon FSx for Lustre checks the quota of the creator’s group and the file size. Those factors then count towards the group’s quota before writing the file to the partition. If no group quota exists, the storage quota checks the user quota before writing the file to the blocks (Object Storage Targets – OSTs). The system administrator can control the over-uses of the users and groups allocated space by setting up the blocks, or inode (Metadata Targets – MDTs) quotas.

Three quota parameters control storage quota administration: soft limits, grace periods, and hard limits.

Soft limits

Soft limits are a numerical value set by the administrator to track user and group level usage. The user or group can still allocate blocks or inode even after exceeding the soft limit, but once it exceeds the grace period, the timer starts. When the grace period time expires, the user limits are checked. If it is still higher than the soft limit, the soft limit then becomes a hard limit and stops the user or group from allocating any new blocks or inodes. The only way to keep under the soft limit is to delete the files by user or group, or configure a larger soft limit that is smaller than the hard limit.

Grace periods

Storage quotas use grace periods in the process of setting up soft limits for users and groups. Grace periods enable users and groups to go beyond their assigned soft limits for a certain period of time, which is set in seconds. Grace periods can be set at the user metadata and data levels.

Hard limits

Hard limits are the absolute limit set. The administrator can modify the hard limit to a higher quota limit for users. When a user has reached the hard limit, all future writes on the block or inode prompts them with an error, “quota exceeded” (EDQUOT).

Storage quotas solution

The system administrators can use lfs commands on Amazon EC2 Lustre client systems to work with and configure storage quotas.

lfs quota: Use this command to display general quota information, including disk usage and limits for users and groups.

lfs quota [-q] [-v] [-h] [-o obd_uuid] [-u|-g|-p uname|uid|gname|gid|projid] /mount_point

lfs quota -t {-u|-g|-p} /mount_point

lfs setquota:  Use this command to set the storage quota limits and fine-tune the three quota parameters (grace periods, soft limits, and hard limits).

lfs setquota {-u|--user|-g|--group|-p|--project} username|groupname [-b block-softlimit] \
             [-B block_hardlimit] [-i inode_softlimit] \
             [-I inode_hardlimit] /mount_point

Note: As of the publishing of this post, Amazon FSx for Lustre does not support default quotas or project quotas.

lfs Storage quota command examples

Ensure that you have created and mounted the Amazon FSx for Lustre file system to the Amazon EC2 instance, as per these instructions. You can connect to the Amazon EC2 Lustre client instance using EC2InstanceConnect.

Use the following command to display general quota information, including disk usage and limits for the user running the command and the primary group:

$ sudo lfs quota /mnt/fsx

Disk quotas for usr root (uid 0):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
       /mnt/fsx 2419350       0       0       -  416058       0       0       -
Disk quotas for grp root (gid 0):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
       /mnt/fsx 2419350       0       0       -  416058       0       0       -

You can execute the command to display general quota information for a specific user:

$ sudo lfs quota -u ec2-user /mnt/fsx
Disk quotas for usr ec2-user (uid 1000):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
       /mnt/fsx      25*      1       2       -       1       1       2       -

You can use the following command to list the detailed quota statistics for each MDT and OST for a specific user. You can see the metadata usage on the MDTs and storage usage on OSTs.

$ sudo lfs quota -u ec2-user -v /mnt/fsx
Disk quotas for usr ec2-user (uid 1000):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
       /mnt/fsx      25*      1       2       -       1       1       2       -
fsx-MDT0000_UUID
                     25       -       0       -       1*      -       1       -
fsx-OST0000_UUID
                      0       -       0       -       -       -       -       -
fsx-OST0001_UUID
                      0       -       0       -       -       -       -       -
fsx-OST0002_UUID
                      0       -       0       -       -       -       -       -
fsx-OST0003_UUID
                      0       -       0       -       -       -       -       -
fsx-OST0004_UUID
                      0       -       0       -       -       -       -       -
fsx-OST0005_UUID
                      0       -       0       -       -       -       -       -

You can use the following command to display general quota information for a specific group:

$ sudo lfs quota -g testgroup /mnt/fsx
Disk quotas for grp testgroup (gid 1002):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
       /mnt/fsx       0       0       0       -       0       0       0       -

You can use the following command to display block and inode grace times for user quotas (the default block and inode grace time is one week):

$ sudo lfs quota -t -u /mnt/fsx
Block grace time: 1w; Inode grace time: 1w

lfs commands for setting limits

The setquota command is used to set file system quotas for user or group levels. A system administrator can change user and group grace time setting using this command.

Setting user quotas

In this example, the quota for user “ec2-user” is set to 2044 kilobytes, and the inode hard limit is 2001 files:

$ sudo lfs setquota -u ec2-user -b 2044 -B 2048 -i 2001 -I 2001 /mnt/fsx

To retrieve the quota allocated for “ec2-user,” You can execute the following command on the Lustre client:

$ sudo lfs quota -u ec2-user -v /mnt/fsx
Disk quotas for usr ec2-user (uid 1000):
Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
/mnt/fsx      25    2044    2048       -       1    2001    2001       -
fsx-MDT0000_UUID
25       -       0       -       1*      -       1       -
fsx-OST0000_UUID
0       -       0       -       -       -       -       -
fsx-OST0001_UUID
0       -       0       -       -       -       -       -
fsx-OST0002_UUID
0       -       0       -       -       -       -       -
fsx-OST0003_UUID
0       -       0       -       -       -       -       -
fsx-OST0004_UUID
0       -       0       -       -       -       -       -
fsx-OST0005_UUID
0       -       0       -       -       -       -       -

Setting group quotas

In this example, the quota for group “testgroup” is set to 2044 kilobytes, and the inode hard limit is 2001 files.

$ sudo lfs setquota -g testgroup -b 2044 -B 2048 -i 2001 -I 2001 /mnt/fsx

The following command displays quota allocated by the group “testgroup.”

$ sudo lfs quota -g testgroup -v /mnt/fsx
Disk quotas for grp testgroup (gid 1002):
Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
/mnt/fsx       0    2044    2048       -       0    2001    2001       -
fsx-MDT0000_UUID
0       -       0       -       0       -       0       -
fsx-OST0000_UUID
0       -       0       -       -       -       -       -
fsx-OST0001_UUID
0       -       0       -       -       -       -       -
fsx-OST0002_UUID
0       -       0       -       -       -       -       -
fsx-OST0003_UUID
0       -       0       -       -       -       -       -
fsx-OST0004_UUID
0       -       0       -       -       -       -       -
fsx-OST0005_UUID
0       -       0       -       -       -       -       -
Total allocated inode limit: 0, total allocated block limit: 0

Changing a user’s grace time

User has to log in and configure the grace time for blocks and inodes.

Use the following command to set a grace time for all users. In this example, I use a block grace time of 1000 seconds and an inode grace time of 0 weeks 4 days.

$ sudo lfs setquota -t -u -b 1000 -i 0w4d /mnt/fsx

To validate and view the quota that was just set for all users, run the following quota command to report the quotas usage:

$ sudo lfs quota -t -u /mnt/fsx
lock grace time: 16m40s; Inode grace time: 4d

Changing a group’s grace time

Use the following command to set a grace time for all groups. In this example, I use a block grace time of 1000 seconds and an inode grace time of 0 weeks 4 days.

$ sudo lfs setquota -t -g -b 1000 -i 0w4d /mnt/fsx

To validate and view the quota that was just set for all groups, run the quota command to report the quotas usage:

$ sudo lfs quota -t -g /mnt/fsx

Block grace time: 16m40s; Inode grace time: 4d

Testing the quota limits and grace periods

As a reminder, the storage quota is in place, with 2044 kilobytes for block and 2001 files for inode for the user “ec2-user,” and the grace period is set it for 1 week.

To validate and view the quota that was just set for a particular user “ec2-user,” run the following quota command to report the usage of the quota:

$ sudo lfs quota -u ec2-user -v /mnt/fsx
Disk quotas for usr ec2-user (uid 1000):
Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
/mnt/fsx      25    2044    2048       -       1    2001    2001       -
fsx-MDT0000_UUID
25       -       0       -       1       -    1024       -
fsx-OST0000_UUID
0       -       0       -       -       -       -       -
fsx-OST0001_UUID
0       -       0       -       -       -       -       -
fsx-OST0002_UUID
0       -       0       -       -       -       -       -
fsx-OST0003_UUID
0       -       0       -       -       -       -       -
fsx-OST0004_UUID
0       -       0       -       -       -       -       -
fsx-OST0005_UUID
0       -       0       -       -       -       -       -
Total allocated inode limit: 1024, total allocated block limit: 0

You can execute the following command to display the block grace time:

$ sudo lfs quota -t -u   /mnt/fsx
Block grace time: 1w; Inode grace time: 1w

You can execute the following script to create a temporary “test_storage_quota_file” of 2 MB to test the storage quota limit:

$ cd /mnt/fsx
$ dd if=/dev/zero of=test_storage_quota_file bs=2 count=100000

You can check the storage quota for the “ec2-user,” and verify the grace period timer once after the creation of temporary “test_storage_file”:

$ sudo lfs quota -u ec2-user -v /mnt/fsx
Disk quotas for usr ec2-user (uid 1000):
Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
/mnt/fsx    1973*   2044    2048 6d23h25m21s      13    2001    2001       -
fsx-MDT0000_UUID
31       -       0       -      13       -    1024       -
fsx-OST0000_UUID
25*      -      25       -       -       -       -       -
fsx-OST0001_UUID
25*      -      25       -       -       -       -       -
fsx-OST0002_UUID
1821       -    2048       -       -       -       -       -
fsx-OST0003_UUID
25*      -      25       -       -       -       -       -
fsx-OST0004_UUID
25*      -      25       -       -       -       -       -
fsx-OST0005_UUID
21*      -      21       -       -       -       -       -
Total allocated inode limit: 1024, total allocated block limit: 2169

The * shows that the user has exceeded the quota limit.

Testing block and inode quota limit

The following is an example for creating 10,000 files of 10 bytes each from “test_storage_quota_file,” which had a limit of 2 MB. The use case is to test the inode quota limit of 2001 files and 2044 kilobytes for the block by splitting the temporary file “test_storage_quota_file.” It creates random files starting with “xaaaaaa.” When the quota limit exceeds, it prompts with an error “Disk quota exceeded,” and creates files with 0 bytes in the Amazon FSx for Lustre mounted file system.

$ split -b 10 -a 10 test_storage_quota_file
split: xaaaaaaaaaa: Disk quota exceeded

The total allocated block limit is the sum of all the block limits allocated to OSTs, and the inode quota per MDTs. When a user or group has fully consumed their storage quota, from one OST or MDT, the user or group may not be able to create new files. Their ability to create new files is lost regardless of the quota available on other OSTs or MDTs.

Amazon FSx for Lustre file systems allocates storage quota space in fixed sizes called qunits, the minimum qunit size is 1-M bytes for block space and 1-K inodes for inode space. It is not possible to set the quota limit below the minimal qunit size.

The following error appears when trying to set the quota limit below the minimal qunit size:

$ sudo lfs setquota -u ec2-user -b 1024 -B 1028 -i 1 -I 2 /mnt/fsx
warning: block softlimit is smaller than the miminal qunit size, please see the help of setquota or Lustre manual for details.
warning: inode softlimit is smaller than the miminal qunit size, please see the help of setquota or Lustre manual for details.
warning: inode hardlimit is smaller than the miminal qunit size, please see the help of setquota or Lustre manual for details.

To avoid overruns of quota size, it is advisable to check the workloads of users and groups before designing the storage quota limits and grace periods.

Performance impact of using storage quotas on your file systems

You may be wondering about the potential performance impact of using storage quotas with Amazon FSx for Lustre. Even when a storage quota is set for a user or group, or the quotas are close to the limits, there are no noticeable performance impacts identified on an Amazon FSx file system.

Cleaning up

Once the storage management test using storage quotas is finished, be sure to remove the deployed resources that facilitated the test. Delete the Amazon EC2 instance and Amazon FSx for Lustre file system in the account to avoid incurring future costs on EC2 and Amazon FSx. Unless you are going to reuse the resources later.

To clean up your Amazon FSx for Lustre file system, you can run the following command in the AWS CloudShell console:

$ aws fsx delete-file-system \  --file-system-id <<insert your file system id>>

To clean up your Amazon EC2 instance, you can run the following command in the AWS CloudShell console:

$ aws ec2 terminate-instances \   --instance-ids <<insert your ec2 instance id>>

Conclusion

In this blog, you learned how to use Storage quotas to monitor and control user and group level storage consumption on the lustre file systems. You created an Amazon FSx for Lustre file system as part of the testing and validating storage quotas. Then, you used lfs commands to set the quota limits and validated by listing the quota values.  You can use Storage quotas for managing multiuser file systems such as user shares for data researchers, data scientists, computational engineers, and genomics researchers.

You can now use Amazon FSx for Lustre storage quotas on all Amazon FSx for Lustre file systems, and are available in all public AWS Regions.

If you would like to read more, check out the Amazon FSx for Lustre product page, user guide, and pricing page.

Thanks for reading this blog post on storage quotas for Amazon FSx for Lustre file systems. If you have any comments or questions, please do not hesitate to leave them in the comments section.

Arun Km

Arun Km

Arun is a Global HCLS Senior Technical Account Manager in Enterprise Support at Amazon Web Services. He works with some of the largest AWS genomics customers in the world, assisting them in their adoption of AWS services. In his free time, Arun enjoys spending time with his family, watching spy series, and reading.