Code Example of Java IO Stream Splitting a File into Multiple Subfiles

File splitting and merging is a common requirement, for example: when uploading a large file, you can first split it into small blocks, upload it to the server, and then merge it. Many high-end distributed file systems (such as Google's GFS, Taobao's TFS) also split or merge files by block.

Let's look at the basic idea:

If there is a large file, specify the split size after (for example: according to1M cut)

step 1:

First, calculate the final number of small files N based on the size of the original file and the split size

step 2:

Create these N small files on the disk

step 3:

Open multiple threads (thread count = number of split files), in each thread, use the seek function of RandomAccessFile to position the read pointer to the start of each segment in the original file, then read forward by a specified size (i.e., the block size), and finally write to the corresponding split file. Since multiple threads process in parallel, each writes to its own small file, which is relatively fast in speed.

The following code splits a file into multiple sub-files, each with a size of100K

package testIO;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.util.Arrays;
public class subStream {
	public static void main(String[] args) {
		//First read the source file into memory 
		int eachSize=100*1024;
		File srcFile =new File("F:",/test/test.txt");
		//Create a file object 
		splitFile(srcFile,eachSize);
	}
	public static void splitFile(File srcFile,int eachSize){
		//Determine if the file meets the split requirements 
		if(srcFile.length()==0){
			throw new RuntimeException("The file does not meet the split requirements");
		}
		byte[] fileContent= new byte[(int) srcFile.length()];
		try {
			//Read the file content into memory 
			FileInputStream fis=new FileInputStream(srcFile);
			fis.read(fileContent);
			fis.close();
		}
		catch (Exception e) {
			e.printStackTrace();
		}
		//Calculate how many parts need to be split 
		int fileNumber;
		if(fileContent.length%eachSize==0){
			fileNumber = fileContent.length/eachSize;
		}
			fileNumber = fileContent.length/eachSize+1;
		}
		for (int i=0;i<fileNumber;i++{
			String fileName = srcFile.getName()+"-"+i+".txt";
			File fi = new File(srcFile.getParent(), fileName);
			//Create the split file under the current file path 
			byte[] eachContent;
			//Copy the content of the source file to the split file 
			if(i!=fileNumber-1{
				eachContent = Arrays.copyOfRange(fileContent, eachSize*i, eachSize*(i+1));
			}
				eachContent = Arrays.copyOfRange(fileContent, eachSize*i, fileContent.length);
			}
			try {
				FileOutputStream fos = new FileOutputStream(fi);
				fos.write(eachContent);
				fos.close();
				System.out.printf("Output subfile %s, its size is %d, each size is %d\n", fi.getAbsoluteFile(), fi.length(), eachContent.length);
			}
			catch (Exception e) {
				// TODO: handle exception 
				e.printStackTrace();
			}
		}
	}
}

Summary

This is the complete content of the code example of splitting a file into multiple subfiles using java IO stream in this article. I hope it will be helpful to everyone. Those who are interested can continue to read other related topics on this site, and welcome comments to point out deficiencies. Thank you for your support to this site!

Statement: The content of this article is from the Internet, owned by the original author. The content is contributed and uploaded by Internet users spontaneously. This website does not own the copyright, has not been edited by humans, and does not assume relevant legal liability. If you find any content suspected of copyright infringement, please send an email to: notice#w3Please report via email to codebox.com (replace # with @ when sending email) and provide relevant evidence. Once verified, this site will immediately delete the content suspected of infringement.