We are rebuilding our Weaviate classes overnight from scratch with batch inserts. Weaviate stores its data on a 60Gb network drive. Once each class is rebuilt, we remove its original version.
But during this operation of rebuilding of these Weaviate classes (total disk size of which is about 40Gb), our monitoring shows that about 4Tb read and write traffic occurred against that network drive!!!
This is quite a high ratio of disk read/writes compared to the total amount of data the classes represents (4Tb vs 40Gb).
Is there any way to decrease the amount of disk read/write traffic during the batch loading of new classes?
Please keep in mind, that these classes are not used while they are loaded, only once the loading is completed, which might allow for some optimizations of disk activities.