Note: This topic is not relevant to SmartStore indexes. See Configure data retention for SmartStore indexes.
You can configure maximum index size in a number of ways:
- On a per-index basis
- For hot/warm and cold buckets separately
- Across indexes, using volumes
To configure index storage size, you set attributes in indexes.conf. For more information on the attributes mentioned in this topic, read "Configure index storage".
Caution: While processing indexes, the indexer might occasionally exceed the configured maximums for short periods of time. When setting limits, be sure to factor in some buffer space. Also, note that certain systems, such as most Unix systems, maintain a configurable reserve space on their partitions. You must take that reserve space, if any, into account when determining how large your indexes can grow.
Configure index size for each index
To set the maximum index size on a per-index basis, use the maxTotalDataSizeMB attribute. When this limit is reached, buckets begin rolling to frozen.
Configure index size according to bucket type
To set the maximum size for homePath (hot/warm bucket storage) or coldPath (cold bucket storage), use the maxDataSizeMB settings:
# set hot/warm storage to 10,000MBhomePath.maxDataSizeMB = 10000 # set cold storage to 5,000MBcoldPath.maxDataSizeMB = 5000
The maxDataSizeMB attributes can be set globally or for each index. An index-level setting will override a global setting. To control bucket storage across groups of indexes, use the maxVolumeDataSizeMB attribute, described below.
When the size of the homePath directory exceeds homePath.maxDataSizeMB, the indexer rolls the oldest warm bucket to cold, moving the bucket to the coldPath directory.
When the size of the coldPath directory exceeds coldPath.maxDataSizeMB, the indexer rolls the oldest cold bucket to frozen.
Configure index size with volumes
You can manage disk usage across multiple indexes by creating volumes and specifying maximum data size for them. A volume represents a directory on the file system where indexed data resides.
Volumes can store data from multiple indexes. You would typically use separate volumes for hot/warm and cold buckets. For instance, you can set up one volume to contain the hot/warm buckets for all your indexes, and another volume to contain the cold buckets.
You can use volumes to define homePath and coldPath. You cannot use them to define thawedPath.
In addition, you must use volumes if you explicitly define bloomHomePath.
Configure a volume
To set up a volume, use this syntax:
[volume:<volume_name>]path = <pathname_for_volume>
You can also optionally include a maxVolumeDataSizeMB attribute, which specfies the maximum size for the volume.
For example:
[volume:hot1]path = /mnt/fast_diskmaxVolumeDataSizeMB = 100000
The example defines a volume called "hot1", located at /mnt/fast_disk, with a maximum size of 100,000MB.
Similarly, this stanza defines a volume called "cold1" that uses a maximum of 150,000MB:
[volume:cold1]path = /mnt/big_diskmaxVolumeDataSizeMB = 150000
Use a volume
Once you configure volumes, you can use them to define an index's homePath and coldPath. For example, using the volumes configured above, you can define two indexes:
[idx1]homePath = volume:hot1/idx1coldPath = volume:cold1/idx1[idx2]homePath = volume:hot1/idx2coldPath = volume:cold1/idx2
You can use volumes to manage index storage space in any way that makes sense to you. Usually, however, volumes correlate to hot/warm and cold buckets, because of the different storage requirements typical when dealing with different bucket types. So, you will probably use some volumes exclusively for designating homePath (hot/warm buckets) and others for coldPath (cold buckets).
When a volume containing warm buckets reaches its maxVolumeDataSizeMB, it starts rolling buckets to cold. When a volume containing cold buckets reaches its maxVolumeDataSizeMB, it starts rolling buckets to frozen. If a volume contains both warm and cold buckets (which will happen if an index's homePath and coldPath are both set to the same volume), the oldest bucket will be rolled to frozen.
Put it all together
This example shows how to use the per-index homePath.maxDataSizeMB and coldPath.maxDataSizeMB attributes in combination with volumes to maintain fine-grained control over index storage. In particular, it shows how to use those attributes to prevent bursts of data into one index from triggering massive bucket moves from other indexes. You can use these per-index settings to ensure that no index will ever occupy more than a specified size, thereby alleviating that concern.
# global settings# Inheritable by all indexes: No hot/warm directory (homePath) can exceed 1 TB.# Individual indexes can override this setting.homePath.maxDataSizeMB = 1000000# volumes[volume:caliente]path = /mnt/fast_diskmaxVolumeDataSizeMB = 100000[volume:frio]path = /mnt/big_diskmaxVolumeDataSizeMB = 1000000# indexes[i1]homePath = volume:caliente/i1# homePath.maxDataSizeMB is inherited from the global settingcoldPath = volume:frio/i1# coldPath.maxDataSizeMB not specified anywhere: # This results in no size limit - old-style behavior[i2]homePath = volume:caliente/i2homePath.maxDataSizeMB = 1000 # overrides the global defaultcoldPath = volume:frio/i2coldPath.maxDataSizeMB = 10000 # limits the size of cold buckets[i3]homePath = /old/style/pathhomePath.maxDataSizeMB = 1000coldPath = volume:frio/i3coldPath.maxDataSizeMB = 10000
