时间戳

首页 > 解决方案 > 在 AR-session Android 中使用 WebRTC 摄像头

android - 在 AR-session Android 中使用 WebRTC 摄像头

问题描述

我正在尝试用来自 webrtc 的流式摄像头的帧替换来自设备摄像头的帧（通常在 AR 会话中使用）。为了渲染我正在使用的 webrtc 流并渲染我正在使用的webrtc.SurfaceViewRendererAR 会话，这两个查看器按照他们应该单独执行的方式工作，但现在我想将它们组合起来。问题是我不知道如何从 webrtc 流中提取帧。我发现的 closets 函数是捕获像素，但它总是返回 null。opengl.GLSurfaceViewactivity_main.xmlBitmap bmp = surfaceViewRenderer.getDrawingCache();

如果我可以从 surfaceViewRenderer 获取像素，我的想法是将其绑定到纹理，然后将此纹理渲染为 AR 场景中的背景

我一直在关注的代码可以在https://github.com/google-ar/arcore-android-sdk/blob/master/samples/hello_ar_java/app/src/main/java/com/google/ar/找到核心/examples/java/helloar/HelloArActivity.java

这是使用设备摄像头渲染 AR 场景的代码：

public void onDrawFrame(GL10 gl10) {

    GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT | GLES20.GL_DEPTH_BUFFER_BIT);

    try {
            session.setCameraTextureName(backgroundRenderer.getTextureId());

            //replace this frame with the frames that is rendered in webrtc's SurfaceViewRenderer
            Frame frame = session.update();
            Camera camera = frame.getCamera();

            backgroundRenderer.draw(frame);
    .
    .
    .

这就是我的activity_main.xml样子。最后我将删除 SurfaceViewRenderer 部分

<LinearLayout
    <org.webrtc.SurfaceViewRenderer
     android:id="@+id/local_gl_surface_view"
     android:layout_width="match_parent"
     android:layout_height="248dp"
     android:layout_gravity="bottom|end" />

    <android.opengl.GLSurfaceView
     android:id="@+id/surfaceview"
     android:layout_width="match_parent"
     android:layout_height="195dp"
     android:layout_gravity="top" />

</LinearLayout>

标签： androidarcorewebrtc-android

解决方案

我知道这是一个老问题，但如果有人需要答案，就在这里。

因此，经过几天的头脑风暴和大量的 RnD，我找到了一种解决方法。你不需要额外添加任何东西。所以这里的故事是同时使用 ArCore 和 WebRtc 并将您在 ArCore 会话中看到的内容分享到 WebRtc 以便远程用户看到它们。正确的？

好吧，诀窍是将您的相机交给 ArCore，而不是与 webrtc 共享相机，而是创建一个屏幕共享 Video Capturer。它非常简单，WebRtc 已经原生支持它。（如果您也需要源代码，请告诉我）。

通过将所有这些标志设置到您的窗口，以全屏方式打开您的 ArCore 活动。并正常启动您的 ArCore 会话，然后初始化您的 webrtc 调用并提供屏幕共享逻辑而不是视频源和瞧！ArCore + WebRTC。

我毫无问题地实现了这一点，延迟也很好。~100 毫秒

编辑

我认为共享屏幕不是一个好主意（远程用户将能够看到所有敏感数据，例如通知等）。而我所做的是

扩展 WebRTC 的 ViewCapturer 类。

使用 PixelCopy 类创建位图。

并将其与正在播放 ArCore 会话的SurfaceView一起提供，然后将其转换为位图，将这些位图转换为帧，然后将其发送到 WebRTC 的onFrameCapruted函数。

这样，您可以只共享 SurfaceView 而不是共享整个屏幕，而且您不需要任何额外的权限，例如屏幕录制，这看起来比实际的实时视频共享要好得多。

编辑2（最终解决方案）

步骤1：

创建一个自定义视频捕获器，它将扩展 webrtc 的 VideoCapturer

import android.content.Context;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.graphics.Matrix;
import android.opengl.GLES20;
import android.opengl.GLUtils;
import android.os.Build;
import android.os.Handler;
import android.os.HandlerThread;
import android.os.Looper;
import android.os.SystemClock;
import android.util.Log;
import android.view.PixelCopy;
import android.view.SurfaceView;

import androidx.annotation.RequiresApi;

import com.jaswant.webRTCIsAwesome.ArSessionActivity;

import org.webrtc.CapturerObserver;
import org.webrtc.JavaI420Buffer;
import org.webrtc.SurfaceTextureHelper;
import org.webrtc.TextureBufferImpl;
import org.webrtc.VideoCapturer;
import org.webrtc.VideoFrame;
import org.webrtc.VideoSink;
import org.webrtc.YuvConverter;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.nio.ByteBuffer;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;

@RequiresApi(api = Build.VERSION_CODES.N)
public class CustomVideoCapturer implements VideoCapturer, VideoSink {

private static int VIEW_CAPTURER_FRAMERATE_MS = 10;
private int width;
private int height;
private SurfaceView view;
private Context context;
private CapturerObserver capturerObserver;
private SurfaceTextureHelper surfaceTextureHelper;
private boolean isDisposed;
private Bitmap viewBitmap;
private Handler handlerPixelCopy = new Handler(Looper.getMainLooper());
private Handler handler = new Handler(Looper.getMainLooper());
private AtomicBoolean started = new AtomicBoolean(false);
private long numCapturedFrames;
private YuvConverter yuvConverter = new YuvConverter();
private TextureBufferImpl buffer;
private long start = System.nanoTime();
private final Runnable viewCapturer = new Runnable() {
    @RequiresApi(api = Build.VERSION_CODES.N)
    @Override
    public void run() {
        boolean dropFrame = view.getWidth() == 0 || view.getHeight() == 0;

        // Only capture the view if the dimensions have been established
        if (!dropFrame) {
            // Draw view into bitmap backed canvas
            final Bitmap bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
            final HandlerThread handlerThread = new HandlerThread(ArSessionActivity.class.getSimpleName());
            handlerThread.start();
            try {
                if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) {
                    PixelCopy.request(view, bitmap, copyResult -> {
                        if (copyResult == PixelCopy.SUCCESS) {
                            viewBitmap = getResizedBitmap(bitmap, 500);
                            if (viewBitmap != null) {
                                Log.d("BITMAP--->", viewBitmap.toString());
                                sendToServer(viewBitmap, yuvConverter, start);
                            }
                        } else {
                            Log.e("Pixel_copy-->", "Couldn't create bitmap of the SurfaceView");
                        }
                        handlerThread.quitSafely();
                    }, new Handler(handlerThread.getLooper()));
                } else {
                    Log.i("Pixel_copy-->", "Saving an image of a SurfaceView is only supported from API 24");
                }
            } catch (Exception ignored) {
            }
        }
    }
};
private Thread captureThread;

public CustomVideoCapturer(SurfaceView view, int framePerSecond) {
    if (framePerSecond <= 0)
        throw new IllegalArgumentException("framePersecond must be greater than 0");
    this.view = view;
    float tmp = (1f / framePerSecond) * 1000;
    VIEW_CAPTURER_FRAMERATE_MS = Math.round(tmp);
}

private static void bitmapToI420(Bitmap src, JavaI420Buffer dest) {
    int width = src.getWidth();
    int height = src.getHeight();
    if (width != dest.getWidth() || height != dest.getHeight())
        return;
    int strideY = dest.getStrideY();
    int strideU = dest.getStrideU();
    int strideV = dest.getStrideV();
    ByteBuffer dataY = dest.getDataY();
    ByteBuffer dataU = dest.getDataU();
    ByteBuffer dataV = dest.getDataV();
    for (int line = 0; line < height; line++) {
        if (line % 2 == 0) {
            for (int x = 0; x < width; x += 2) {
                int px = src.getPixel(x, line);
                byte r = (byte) ((px >> 16) & 0xff);
                byte g = (byte) ((px >> 8) & 0xff);
                byte b = (byte) (px & 0xff);
                dataY.put(line * strideY + x, (byte) (((66 * r + 129 * g + 25 * b) >> 8) + 16));
                dataU.put(line / 2 * strideU + x / 2, (byte) (((-38 * r + -74 * g + 112 * b) >> 8) + 128));
                dataV.put(line / 2 * strideV + x / 2, (byte) (((112 * r + -94 * g + -18 * b) >> 8) + 128));
                px = src.getPixel(x + 1, line);
                r = (byte) ((px >> 16) & 0xff);
                g = (byte) ((px >> 8) & 0xff);
                b = (byte) (px & 0xff);
                dataY.put(line * strideY + x, (byte) (((66 * r + 129 * g + 25 * b) >> 8) + 16));
            }
        } else {
            for (int x = 0; x < width; x += 1) {
                int px = src.getPixel(x, line);
                byte r = (byte) ((px >> 16) & 0xff);
                byte g = (byte) ((px >> 8) & 0xff);
                byte b = (byte) (px & 0xff);
                dataY.put(line * strideY + x, (byte) (((66 * r + 129 * g + 25 * b) >> 8) + 16));
            }
        }
    }
}

public static Bitmap createFlippedBitmap(Bitmap source, boolean xFlip, boolean yFlip) {
    try {
        Matrix matrix = new Matrix();
        matrix.postScale(xFlip ? -1 : 1, yFlip ? -1 : 1, source.getWidth() / 2f, source.getHeight() / 2f);
        return Bitmap.createBitmap(source, 0, 0, source.getWidth(), source.getHeight(), matrix, true);
    } catch (Exception e) {
        return null;
    }
}

private void checkNotDisposed() {
    if (this.isDisposed) {
        throw new RuntimeException("capturer is disposed.");
    }
}

@Override
public synchronized void initialize(SurfaceTextureHelper surfaceTextureHelper, Context context, CapturerObserver capturerObserver) {
    this.checkNotDisposed();
    if (capturerObserver == null) {
        throw new RuntimeException("capturerObserver not set.");
    } else {
        this.context = context;
        this.capturerObserver = capturerObserver;
        if (surfaceTextureHelper == null) {
            throw new RuntimeException("surfaceTextureHelper not set.");
        } else {
            this.surfaceTextureHelper = surfaceTextureHelper;
        }
    }
}

@Override
public void startCapture(int width, int height, int fps) {
    this.checkNotDisposed();
    this.started.set(true);
    this.width = width;
    this.height = height;
    this.capturerObserver.onCapturerStarted(true);
    this.surfaceTextureHelper.startListening(this);
    handler.postDelayed(viewCapturer, VIEW_CAPTURER_FRAMERATE_MS);


    /*try {
        Bitmap bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
        HandlerThread handlerThread = new HandlerThread(CustomVideoCapturer.class.getSimpleName());
        capturerObserver.onCapturerStarted(true);
        int[] textures = new int[1];
        GLES20.glGenTextures(1, textures, 0);
        YuvConverter yuvConverter = new YuvConverter();
        TextureBufferImpl buffer = new TextureBufferImpl(width, height, VideoFrame.TextureBuffer.Type.RGB, textures[0], new Matrix(), surfaceTextureHelper.getHandler(), yuvConverter, null);
     //   handlerThread.start();
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) {
            new Thread(() -> {
                while (true) {
                    PixelCopy.request(view, bitmap, copyResult -> {
                        if (copyResult == PixelCopy.SUCCESS) {
                            viewBitmap = getResizedBitmap(bitmap, 500);
                            long start = System.nanoTime();
                            Log.d("BITMAP--->", viewBitmap.toString());
                            sendToServer(viewBitmap, yuvConverter, buffer, start);
                        } else {
                            Log.e("Pixel_copy-->", "Couldn't create bitmap of the SurfaceView");
                        }
                        handlerThread.quitSafely();
                    }, new Handler(Looper.getMainLooper()));
                }
            }).start();
        }
    } catch (Exception ignored) {
    }*/
}

private void sendToServer(Bitmap bitmap, YuvConverter yuvConverter, long start) {
    try {
        int[] textures = new int[1];
        GLES20.glGenTextures(0, textures, 0);
        buffer = new TextureBufferImpl(width, height, VideoFrame.TextureBuffer.Type.RGB, textures[0], new Matrix(), surfaceTextureHelper.getHandler(), yuvConverter, null);
        Bitmap flippedBitmap = createFlippedBitmap(bitmap, true, false);
        surfaceTextureHelper.getHandler().post(() -> {
            if (flippedBitmap != null) {
                GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_NEAREST);
                GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_NEAREST);
                GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, 0, flippedBitmap, 0);

                VideoFrame.I420Buffer i420Buf = yuvConverter.convert(buffer);

                long frameTime = System.nanoTime() - start;
                VideoFrame videoFrame = new VideoFrame(i420Buf, 180, frameTime);
                capturerObserver.onFrameCaptured(videoFrame);
                videoFrame.release();
                try {
                    viewBitmap.recycle();
                } catch (Exception e) {

                }
                handler.postDelayed(viewCapturer, VIEW_CAPTURER_FRAMERATE_MS);
            }
        });
    } catch (Exception ignored) {

    }
}

@Override
public void stopCapture() throws InterruptedException {
    this.checkNotDisposed();
    CustomVideoCapturer.this.surfaceTextureHelper.stopListening();
    CustomVideoCapturer.this.capturerObserver.onCapturerStopped();
    started.set(false);
    handler.removeCallbacksAndMessages(null);
    handlerPixelCopy.removeCallbacksAndMessages(null);
}

@Override
public void changeCaptureFormat(int width, int height, int framerate) {
    this.checkNotDisposed();
    this.width = width;
    this.height = height;
}

@Override
public void dispose() {
    this.isDisposed = true;
}

@Override
public boolean isScreencast() {
    return true;
}

private void sendFrame() {
    final long captureTimeNs = TimeUnit.MILLISECONDS.toNanos(SystemClock.elapsedRealtime());

    /*surfaceTextureHelper.setTextureSize(width, height);

    int[] textures = new int[1];
    GLES20.glGenTextures(1, textures, 0);
    GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, textures[0]);

    Matrix matrix = new Matrix();
    matrix.preTranslate(0.5f, 0.5f);
    matrix.preScale(1f, -1f);
    matrix.preTranslate(-0.5f, -0.5f);


    YuvConverter yuvConverter = new YuvConverter();
    TextureBufferImpl buffer = new TextureBufferImpl(width, height,
            VideoFrame.TextureBuffer.Type.RGB, textures[0],  matrix,
            surfaceTextureHelper.getHandler(), yuvConverter, null);

    GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_NEAREST);
    GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_NEAREST);
    GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, 0, viewBitmap, 0);

    long frameTime = System.nanoTime() - captureTimeNs;
    VideoFrame videoFrame = new VideoFrame(buffer.toI420(), 0, frameTime);
    capturerObserver.onFrameCaptured(videoFrame);
    videoFrame.release();

    handler.postDelayed(viewCapturer, VIEW_CAPTURER_FRAMERATE_MS);*/

    // Create video frame
    JavaI420Buffer buffer = JavaI420Buffer.allocate(viewBitmap.getWidth(), viewBitmap.getHeight());
    bitmapToI420(viewBitmap, buffer);
    VideoFrame videoFrame = new VideoFrame(buffer,
            0, captureTimeNs);

    // Notify the listener
    if (started.get()) {
        ++this.numCapturedFrames;
        this.capturerObserver.onFrameCaptured(videoFrame);
    }
    if (started.get()) {
        handler.postDelayed(viewCapturer, VIEW_CAPTURER_FRAMERATE_MS);
    }
}

public long getNumCapturedFrames() {
    return this.numCapturedFrames;
}

/**
 * reduces the size of the image
 *
 * @param image
 * @param maxSize
 * @return
 */
public Bitmap getResizedBitmap(Bitmap image, int maxSize) {
    int width = image.getWidth();
    int height = image.getHeight();

    try {
        Bitmap bitmap =  Bitmap.createScaledBitmap(image, width, height, true);
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        bitmap.compress(Bitmap.CompressFormat.JPEG, 50, out);
        return BitmapFactory.decodeStream(new ByteArrayInputStream(out.toByteArray()));

    } catch (Exception e) {
        return null;
    }
}

@Override
public void onFrame(VideoFrame videoFrame) {

}


}

第 2 步（使用）：

CustomVideoCapturer videoCapturer = new CustomVideoCapturer(arSceneView, 20);
videoSource = peerConnectionFactory.createVideoSource(videoCapturer.isScreencast());
videoCapturer.initialize(surfaceTextureHelper, context, videoSource.getCapturerObserver());
videoCapturer.startCapture(resolutionHeight, resolutionWidth, 30);

在此之后，您应该能够将您的 AR 帧流式传输给远程用户。

第 3 步（可选）：

您可以使用getResizedBitmap()来更改帧的分辨率和大小。记住：位图操作可能是过程密集型的。

我可以随意在此代码中提出任何类型的建议或优化。这是我经过数周的忙碌后想出的。

推荐阅读