opencv - Camerax ImageCapture 传递给 Chaquopy 太慢了

Question

我用 Chaquopy 制作了 Camerax 应用程序：

• ImageCapture 用例捕获图像（jpg）
• 将其转换为位图
• 将位图转换为字符串
• 将字符串传递给 Python
• Python OpenCV 获取图像分辨率并将其返回给 Android（如字符串）
• 在 Textview 上显示

应用程序正在运行，但问题是：

1. 应用程序太慢：15-20 秒（字符串转换？）
2. 它显示错误的分辨率（分辨率在 ImageCapture 配置中设置）

• 我怎样才能加快我的应用程序？
• 除了base64字符串之外，还有另一种将图像传递给python的方法吗？

Android CameraAtivity 代码：

'''

public class CameraActivity extends AppCompatActivity {
    private PreviewView previewView;
    private ImageCapture imageCapture;
    private ListenableFuture<ProcessCameraProvider> cameraProviderFuture;
    private TextView textView;
    private Button button;



    @Override
    protected void onCreate(@Nullable Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_camera);
        previewView = findViewById(R.id.previewView);
        cameraProviderFuture = ProcessCameraProvider.getInstance(this);
        textView = findViewById(R.id.length);
        button = findViewById(R.id.button);


        cameraProviderFuture.addListener(new Runnable() {
            @Override
            public void run() {
                try {
                    ProcessCameraProvider cameraProvider = cameraProviderFuture.get();
                    bindImageAnalysis(cameraProvider);
                } catch (ExecutionException | InterruptedException e) {
                    e.printStackTrace();
                }
            }
        }, ContextCompat.getMainExecutor(this));
    }


    private void bindImageAnalysis(@NonNull ProcessCameraProvider cameraProvider) {

        Preview preview = new Preview.Builder().build();
        CameraSelector cameraSelector = new CameraSelector.Builder()
                .requireLensFacing(CameraSelector.LENS_FACING_BACK).build();
        preview.setSurfaceProvider(previewView.createSurfaceProvider());

        Executor cameraexecutor = ContextCompat.getMainExecutor(this);
        ImageCapture imageCapture = new ImageCapture.Builder().setTargetResolution(new Size(1200, 1600)).build();
        button.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View view) {
                imageCapture.takePicture(cameraexecutor, new ImageCapture.OnImageCapturedCallback() {
                    @Override
                    public void onCaptureSuccess(@NonNull ImageProxy image){
                        ByteBuffer byteBuffer = image.getPlanes()[0].getBuffer();

                        byte[] bytes = new byte[byteBuffer.capacity()];
                        byteBuffer.get(bytes);

                        Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
                        ByteArrayOutputStream baos = new ByteArrayOutputStream();
                        bitmap.compress(Bitmap.CompressFormat.PNG, 100, baos);
                        byte[] imageBytes = baos.toByteArray();
                        String imgString = Base64.encodeToString(imageBytes, Base64.DEFAULT);


                        PyObject obj = pythonn(imgString);

                        textView.setText(obj.toString());

                    }

                    @Override
                    public void onError(@NonNull ImageCaptureException error) {
                        error.printStackTrace();
                    }
                });
            }

        });


        cameraProvider.bindToLifecycle((LifecycleOwner)this, cameraSelector, imageCapture,
                preview);
    }

    private PyObject pythonn(String imgString) {
        if(!Python.isStarted())
            Python.start(new AndroidPlatform(this ));
        Python py = Python.getInstance();

        final PyObject pyobj = py.getModule("shape");
        PyObject obj = pyobj.callAttr("main",imgString);
        return obj;
    }
}

'''

Python代码；

import cv2
import numpy as np
import base64

def main(imgString):

    decoded_data = base64.b64decode(imgString)
    np_data = np.fromstring(decoded_data,np.uint8)
    img = cv2.imdecode(np_data,cv2.IMREAD_UNCHANGED)

    if img.shape[0] > img.shape[1]:
        img = cv2.transpose(img)
    
    return str(str(img.shape[0]) + "_"  + str(img.shape[1])) 

所有代码都在 Github 上： https ://github.com/kintipu/Camerax_ImageCapture_Chaquopy_OpenCV_ImageResolution/tree/master

2021 年 3 月 29 日更新：我将尝试使用 bytearray，就像 mhsmith 建议的那样。与此同时，我尝试了另一件事：

• 捕获图像并将其保存为 jpg

• 然后直接从python中读取图像 App在模拟器中工作，但在真正的手机Android上代码：'''

  公共类 CameraActivity 扩展 AppCompatActivity { private PreviewView previewView; 私有图像捕获图像捕获；私有 ListenableFuture cameraProviderFuture；私有文本视图文本视图；私人按钮按钮；私有上下文上下文；

    @Override
    protected void onCreate(@Nullable Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_camera);
        previewView = findViewById(R.id.previewView);
        cameraProviderFuture = ProcessCameraProvider.getInstance(this);
        textView = findViewById(R.id.length);
        button = findViewById(R.id.button);
  
  
        cameraProviderFuture.addListener(new Runnable() {
            @Override
            public void run() {
                try {
                    ProcessCameraProvider cameraProvider = cameraProviderFuture.get();
                    bindImageAnalysis(cameraProvider);
                } catch (ExecutionException | InterruptedException e) {
                    e.printStackTrace();
                }
            }
        }, ContextCompat.getMainExecutor(this));
    }
  
  
    private void bindImageAnalysis(@NonNull ProcessCameraProvider cameraProvider) {
  
        Preview preview = new Preview.Builder().build();
        CameraSelector cameraSelector = new CameraSelector.Builder()
                .requireLensFacing(CameraSelector.LENS_FACING_BACK).build();
        preview.setSurfaceProvider(previewView.createSurfaceProvider());
  
        Executor cameraexecutor = ContextCompat.getMainExecutor(this);
        ImageCapture imageCapture = new ImageCapture.Builder().setTargetResolution(new Size(1200, 1600)).build();
  
  
        button.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View v) {
  
  
                File file = new File(getBatchDirectoryName(), "photo"+ ".jpg");
  
                ImageCapture.OutputFileOptions outputFileOptions = new ImageCapture.OutputFileOptions.Builder(file).build();
                imageCapture.takePicture(outputFileOptions, cameraexecutor, new ImageCapture.OnImageSavedCallback () {
                    @Override
                    public void onImageSaved(@NonNull ImageCapture.OutputFileResults outputFileResults) {
                        new Handler().post(new Runnable() {
                            @Override
                            public void run() {
                                Toast.makeText(CameraActivity.this, "Image Saved successfully" + file.getAbsolutePath(), Toast.LENGTH_LONG).show();
  
                                PyObject obj = pythonn();
                                textView.setText(obj.toString());
                            }
  
  
                        });
  
                    }
                    @Override
                    public void onError(@NonNull ImageCaptureException error) {
                        error.printStackTrace();
                    }
                });
            }
        });
  
  
        cameraProvider.bindToLifecycle((LifecycleOwner)this, cameraSelector, imageCapture,
                preview);
    }
  
  
  
  
    public String getBatchDirectoryName() {
  
        String app_folder_path = "";
        app_folder_path = Environment.getExternalStorageDirectory().toString() + "/images";
        File dir = new File(app_folder_path);
        if (!dir.exists() && !dir.mkdirs()) {
  
        }
  
        return app_folder_path;
    }
  
  
  
  
  
    private PyObject pythonn() {
        if(!Python.isStarted())
            Python.start(new AndroidPlatform(this ));
        Python py = Python.getInstance();
  
        final PyObject pyobj = py.getModule("shape");
        PyObject obj = pyobj.callAttr("main");
  
        return obj;
    }
  

  }

'''

Python代码：

'''

import cv2
import numpy as np
from android.os import Environment


path = str(str(Environment.getExternalStorageDirectory()) + "/images" + "/photo.jpg")


def main():
    

    img = cv2.imread(path)
    
    return str(str(img.shape[0]) + "_"  + str(img.shape[1])) 

'''

2021 年 4 月 4 日更新：

通过一些修改，我让它变得更快：

• 将 bytearray 之类的图像传递给 python（而不是字符串和 base64 转换）-根据 Malcom Smith 的建议
• 位图压缩设置为 JPEG（而不是 PNG）
• Python 从 Create 开始

安卓代码；'''

public class CameraActivity extends AppCompatActivity {
    private PreviewView previewView;
    private ImageCapture imageCapture;
    private ListenableFuture<ProcessCameraProvider> cameraProviderFuture;
    private TextView textView;
    private Button button;



    @Override
    protected void onCreate(@Nullable Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_camera);
        previewView = findViewById(R.id.previewView);
        cameraProviderFuture = ProcessCameraProvider.getInstance(this);
        textView = findViewById(R.id.length);
        button = findViewById(R.id.button);
        
        if(!Python.isStarted())
              Python.start(new AndroidPlatform(this ));


        cameraProviderFuture.addListener(new Runnable() {
            @Override
            public void run() {
                try {
                    ProcessCameraProvider cameraProvider = cameraProviderFuture.get();
                    bindImageAnalysis(cameraProvider);
                } catch (ExecutionException | InterruptedException e) {
                    e.printStackTrace();
                }
            }
        }, ContextCompat.getMainExecutor(this));
    }


    private void bindImageAnalysis(@NonNull ProcessCameraProvider cameraProvider) {

        Preview preview = new Preview.Builder().build();
        CameraSelector cameraSelector = new CameraSelector.Builder()
                .requireLensFacing(CameraSelector.LENS_FACING_BACK).build();
        preview.setSurfaceProvider(previewView.createSurfaceProvider());

        Executor cameraexecutor = ContextCompat.getMainExecutor(this);
        ImageCapture imageCapture = new ImageCapture.Builder().setTargetResolution(new Size(1200, 1600)).build();
        button.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View view) {
                imageCapture.takePicture(cameraexecutor, new ImageCapture.OnImageCapturedCallback() {
                    @Override
                    public void onCaptureSuccess(@NonNull ImageProxy image){
                        ByteBuffer byteBuffer = image.getPlanes()[0].getBuffer();

                        byte[] bytes = new byte[byteBuffer.capacity()];
                        byteBuffer.get(bytes);

                        Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
                        ByteArrayOutputStream baos = new ByteArrayOutputStream();
                        bitmap.compress(Bitmap.CompressFormat.JPEG, 100, baos);
                        byte[] imageBytes = baos.toByteArray();

//                        String length = String.valueOf((imgString.length()));
//                        textView.setText(length);

                        PyObject obj = pythonn(imageBytes);

                        textView.setText(obj.toString());

                    }

                    @Override
                    public void onError(@NonNull ImageCaptureException error) {
                        error.printStackTrace();
                    }
                });
            }

        });


        cameraProvider.bindToLifecycle((LifecycleOwner)this, cameraSelector, imageCapture,
                preview);
    }

    private PyObject pythonn(byte[] imageBytes) {

        Python py = Python.getInstance();

        final PyObject pyobj = py.getModule("shape");
        PyObject obj = pyobj.callAttr("main",imageBytes);
        return obj;
    }
}

Python： '''

import cv2
import numpy as np



def main(imageBytes):
         np_data = np.asarray(imageBytes,np.uint8)
         img = cv2.imdecode(np_data,cv2.IMREAD_UNCHANGED)
         if img.shape[0] > img.shape[1]:
                img = cv2.transpose(img)

         img = cv2.resize(img, (1600, 1200))
    
         return str(str(img.shape[0]) + "_"  + str(img.shape[1]))

更新了 github：https ://github.com/kintipu/Camerax_ImageCapture_Chaquopy_OpenCV_ImageResolution/blob/master/README.md

标签： opencvandroid-cameraxchaquopy

Answer 1

不需要使用 base64，您只需发送 PNG 编码的byte[]数组，它将在 Python 中作为字节类对象接收。我不确定这个对象是否会被 直接接受cv2.imdecode，但肯定会被np.arrayor接受np.frombuffer。（不推荐将二进制数据传递给np.fromstring，正如其文档页面所解释的那样。）

如果您不确定哪行代码导致延迟，您可以在行Log之间添加调用（或printPython 中的调用），然后检查Logcat中的时间戳。