java - 将 3GB 文件转换为字节数组

Question

我正在尝试将 3GB 文件转换为字节数组但得到 OOM( OutOfMemoryError)。

我们尝试了

RandomAccessFile randomAccessFile = new RandomAccessFile(sourceLocation.getAbsolutePath(), "r");
MappedByteBuffer mappedByteBuffer = randomAccessFile.getChannel().map(FileChannel.MapMode.READ_ONLY, 0, randomAccessFile.length()); //while it is large file, it threw 'mmap failed: ENOMEM (Out of memory)' exception.

byte[] data = new byte[1024 * 1024];  // 1MB read at time
while (mappedByteBuffer.hasRemaining()) {
    int remaining = data.length;

    if (mappedByteBuffer.remaining() < remaining)
        remaining = mappedByteBuffer.remaining();
        mappedByteBuffer.get(data, 0, remaining);
    }

    mappedByteBuffer.rewind();
    byte fileContent[] = mappedByteBuffer.array(); //while it is small file, it threw 'java.nio.ReadOnlyBufferException' exception.
    randomAccessFile.close();
}

我的自定义请求正文：我的请求正文准备的自定义请求正文类

import android.os.Looper;

import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;

import okhttp3.MediaType;
import okhttp3.RequestBody;
import okio.BufferedSink;

public class ProgressRequestBodyTemp extends RequestBody
{
    private static final int DEFAULT_BUFFER_SIZE = 2048;
    private File mFile;
    private String mPath;
    private String mFileType;
    private int mItemIndex;
    private UploadCallbacks mListener;

    private byte[] encryptedData;
    private ByteArrayInputStream bis;

    public ProgressRequestBodyTemp(final String fileType, final File file, byte[] encryptedData, final int itemIndex, final UploadCallbacks listener)
    {
        this.mFile = file;
        this.mFileType = fileType;
        this.mItemIndex = itemIndex;
        this.mListener = listener;

        this.encryptedData = encryptedData;

        try
        {
            bis = new ByteArrayInputStream(encryptedData); // Convert byte array into input stream for send data to server
        }
        catch (Exception ex)
        {
            ex.printStackTrace();
        }
    }

    @Override
    public MediaType contentType()
    {
        try
        {
            return MediaType.parse(mFileType);
        }
        catch (Exception ex)
        {
            return MediaType.parse("");
        }
    }

    @Override
    public long contentLength() throws IOException
    {
        return encryptedData.length;
    }

    @Override
    public void writeTo(BufferedSink sink) throws IOException
    {
        long fileLength = mFile.length();
        byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];

        long uploaded = 0;
        try
        {
            int read;
            android.os.Handler handler = new android.os.Handler(Looper.getMainLooper());
            while ((read = bis.read(buffer)) != -1)
            {

                // update progress on UI thread
                handler.post(new ProgressUpdater(uploaded, encryptedData.length));

                uploaded += read;
                sink.write(buffer, 0, read);
            }
        }
        finally
        {
            bis.close();
        }
    }

    public interface UploadCallbacks
    {
        void onProgressUpdate(int itemIndex, int percentage);
    }

    private class ProgressUpdater implements Runnable
    {
        private long mUploaded;
        private long mTotal;

        public ProgressUpdater(long uploaded, long total)
        {
            mUploaded = uploaded;
            mTotal = total;
        }

        @Override
        public void run()
        {
            if (mListener != null)
            {
                mListener.onProgressUpdate(mItemIndex, (int) (100 * mUploaded / mTotal));
            }
        }
    }
}

我的请求：上传文件的请求

File sourceLocation = new File(".....");
byte fileContent[] = .... // byte array

ProgressRequestBody requestFile  = new ProgressRequestBody(fileType, sourceLocation, fileContent, itemIndex, this);
MultipartBody.Part body = MultipartBody.Part.createFormData("file", sourceLocation.getName(), requestFile); // MultipartBody.Part is used to send also the actual file name
Call<String> mediaCall = getRetrofit().addMedia("SOME STRING DATA", body);

它是给OutOfMemoryError的，请建议将大文件转换为字节数组的最佳方法。

任何帮助都非常感谢。提前致谢。

标签： javaandroidout-of-memory

Answer 1

使用InputStream和OutputStream。

它们适用于大量数据，例如，当您使用 3GB 数据并且无法将其加载到内存中时。

如果您尝试上传文件，请使用FileInputStream. 创建一个File对象，将其传递给FileOutputStream构造函数并开始从其读取字节InputStream到byte[]缓冲区，然后将字节发送OutputStream到服务器。

这种方法不会导致，OutOfMemoryError因为您只读取足够的字节来填满缓冲区，缓冲区的大小应该约为2KB - 8KB。缓冲区满后，您将字节写入服务器。缓冲区写入服务器后，您再次读取缓冲区，该过程继续进行，直到整个文件上传。

使用 FileInputStream 的示例

        File file = new File("yourfile.txt");
        FileInputStream fis = null;
        OutputStream outputStream = null;

        url = new URL(desiredUrl);
        HttpURLConnection connection = (HttpURLConnection) url.openConnection();


        try {
            fis = new FileInputStream(file);
            connection.setDoOutput(true);
            outputStream = connection.getOutputStream();

            int actuallyRead;
            byte[] buffer = new byte[2048];
            while ((actuallyRead = fis.read(buffer)) != -1) {
                //do something with bytes, for example, write to the server
            outputStream.write(buffer);

            }

        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
                if (fis != null)
                    fis.close();
            } catch (IOException ex) {
                ex.printStackTrace();
            }
            try {
                if (outputStream != null)
                    outputStream.close();
            } catch (IOException ex) {
                ex.printStackTrace();
            }

        }

注意：这种方法并不意味着每次填充缓冲区时都需要重新连接到服务器。它将不断地写入服务器，直到完成对文件的处理。这将在同一个单一连接下发生。