android - 使用 RenderScript 旋转肖像模式的 YUV 图像数据
问题描述
对于视频图像处理项目,我必须旋转传入的 YUV 图像数据,以便数据不是水平显示而是垂直显示。我使用了这个项目,它让我对如何将 YUV 图像数据转换为 ARGB 以进行实时处理有了深刻的了解。该项目的唯一缺点是它只在景观中。纵向模式没有选项(我不知道为什么 Google 的人们提供了一个仅处理横向的示例示例)。我想改变它。
因此,我决定使用自定义 YUV 到 RGB 脚本来旋转数据,使其以纵向模式显示。下面的 GIF 演示了应用程序在我应用任何旋转之前如何显示数据。
要知道,在Android中,即使设备处于纵向模式,YUV图像数据也以横向呈现(在我开始这个项目之前我不知道。再次,我不明白为什么没有可用的方法可用于一次调用旋转帧)。这意味着即使设备处于纵向模式,起点也位于左下角。但在纵向模式下,每一帧的起点应该在左上角。我对字段使用矩阵表示法(例如(0,0)、(0,1)等)。注意:我从这里拿了草图:
要旋转横向框架,我们必须重新组织字段。这是我对草图(见上文)所做的映射,它显示了横向模式下的单帧yuv_420
。映射应将框架旋转 90 度:
first column starting from the bottom-left corner and going upwards:
(0,0) -> (0,5) // (0,0) should be at (0,5)
(0,1) -> (1,5) // (0,1) should be at (1,5)
(0,2) -> (2,5) // and so on ..
(0,3) -> (3,5)
(0,4) -> (4,5)
(0,5) -> (5,5)
2nd column starting at (1,0) and going upwards:
(1,0) -> (0,4)
(1,1) -> (1,4)
(1,2) -> (2,4)
(1,3) -> (3,4)
(1,4) -> (4,4)
(1,5) -> (5,4)
and so on...
实际上,发生的情况是第一列成为新的第一行,第二列成为新的第二行,依此类推。从映射中可以看出,我们可以进行以下观察:
- 结果的
x
坐标始终等于y
左侧的坐标。所以,我们可以这么说x = y
。 - 我们总能观察到的是,对于结果的 y 坐标,以下等式必须成立:
y = width - 1 - x
。(我测试了草图中的所有坐标,它总是正确的)。
因此,我编写了以下渲染脚本内核函数:
#pragma version(1)
#pragma rs java_package_name(com.jon.condino.testing.renderscript)
#pragma rs_fp_relaxed
rs_allocation gCurrentFrame;
int width;
uchar4 __attribute__((kernel)) yuv2rgbFrames(uint32_t x,uint32_t y)
{
uint32_t inX = y; // 1st observation: set x=y
uint32_t inY = width - 1 - x; // 2nd observation: the equation mentioned above
// the remaining lines are just methods to retrieve the YUV pixel elements, converting them to RGB and outputting them as result
// Read in pixel values from latest frame - YUV color space
// The functions rsGetElementAtYuv_uchar_? require API 18
uchar4 curPixel;
curPixel.r = rsGetElementAtYuv_uchar_Y(gCurrentFrame, inX, inY);
curPixel.g = rsGetElementAtYuv_uchar_U(gCurrentFrame, inX, inY);
curPixel.b = rsGetElementAtYuv_uchar_V(gCurrentFrame, inX, inY);
// uchar4 rsYuvToRGBA_uchar4(uchar y, uchar u, uchar v);
// This function uses the NTSC formulae to convert YUV to RBG
uchar4 out = rsYuvToRGBA_uchar4(curPixel.r, curPixel.g, curPixel.b);
return out;
}
该方法似乎有效,但它有一个小错误,如下图所示。如我们所见,相机预览处于纵向模式。但是我的相机预览左侧有一条非常奇怪的颜色线。为什么会这样?(请注意,我使用后置摄像头):
任何解决问题的建议都会很棒。自 2 周以来,我一直在处理这个问题(YUV 从横向到纵向的旋转),这是迄今为止我自己能得到的最好的解决方案。我希望有人可以帮助改进代码,以便左侧奇怪的颜色线也消失。
更新:
我在代码中所做的分配如下:
// yuvInAlloc will be the Allocation that will get the YUV image data
// from the camera
yuvInAlloc = createYuvIoInputAlloc(rs, x, y, ImageFormat.YUV_420_888);
yuvInAlloc.setOnBufferAvailableListener(this);
// here the createYuvIoInputAlloc() method
public Allocation createYuvIoInputAlloc(RenderScript rs, int x, int y, int yuvFormat) {
return Allocation.createTyped(rs, createYuvType(rs, x, y, yuvFormat),
Allocation.USAGE_IO_INPUT | Allocation.USAGE_SCRIPT);
}
// the custom script will convert the YUV to RGBA and put it to this Allocation
rgbInAlloc = RsUtil.createRgbAlloc(rs, x, y);
// here the createRgbAlloc() method
public Allocation createRgbAlloc(RenderScript rs, int x, int y) {
return Allocation.createTyped(rs, createType(rs, Element.RGBA_8888(rs), x, y));
}
// the allocation to which we put all the processed image data
rgbOutAlloc = RsUtil.createRgbIoOutputAlloc(rs, x, y);
// here the createRgbIoOutputAlloc() method
public Allocation createRgbIoOutputAlloc(RenderScript rs, int x, int y) {
return Allocation.createTyped(rs, createType(rs, Element.RGBA_8888(rs), x, y),
Allocation.USAGE_IO_OUTPUT | Allocation.USAGE_SCRIPT);
}
其他一些辅助功能:
public Type createType(RenderScript rs, Element e, int x, int y) {
if (Build.VERSION.SDK_INT >= 21) {
return Type.createXY(rs, e, x, y);
} else {
return new Type.Builder(rs, e).setX(x).setY(y).create();
}
}
@RequiresApi(18)
public Type createYuvType(RenderScript rs, int x, int y, int yuvFormat) {
boolean supported = yuvFormat == ImageFormat.NV21 || yuvFormat == ImageFormat.YV12;
if (Build.VERSION.SDK_INT >= 19) {
supported |= yuvFormat == ImageFormat.YUV_420_888;
}
if (!supported) {
throw new IllegalArgumentException("invalid yuv format: " + yuvFormat);
}
return new Type.Builder(rs, createYuvElement(rs)).setX(x).setY(y).setYuvFormat(yuvFormat)
.create();
}
public Element createYuvElement(RenderScript rs) {
if (Build.VERSION.SDK_INT >= 19) {
return Element.YUV(rs);
} else {
return Element.createPixel(rs, Element.DataType.UNSIGNED_8, Element.DataKind.PIXEL_YUV);
}
}
调用自定义渲染脚本和分配:
// see below how the input size is determined
customYUVToRGBAConverter.invoke_setInputImageSize(x, y);
customYUVToRGBAConverter.set_inputAllocation(yuvInAlloc);
// receive some frames
yuvInAlloc.ioReceive();
// performs the conversion from the YUV to RGB
customYUVToRGBAConverter.forEach_convert(rgbInAlloc);
// this just do the frame manipulation , e.g. applying a particular filter
renderer.renderFrame(mRs, rgbInAlloc, rgbOutAlloc);
// send manipulated data to output stream
rgbOutAlloc.ioSend();
最后但最不重要的是,输入图像的大小。您在上面看到的方法的 x 和 y 坐标基于此处表示为 mPreviewSize 的预览大小:
int deviceOrientation = getWindowManager().getDefaultDisplay().getRotation();
int totalRotation = sensorToDeviceRotation(cameraCharacteristics, deviceOrientation);
// determine if we are in portrait mode
boolean swapRotation = totalRotation == 90 || totalRotation == 270;
int rotatedWidth = width;
int rotatedHeigth = height;
// are we in portrait mode? If yes, then swap the values
if(swapRotation){
rotatedWidth = height;
rotatedHeigth = width;
}
// determine the preview size
mPreviewSize = chooseOptimalSize(
map.getOutputSizes(SurfaceTexture.class),
rotatedWidth,
rotatedHeigth);
因此,基于此,在我的情况下x
将会是mPreviewSize.getWidth()
并且y
将会是。mPreviewSize.getHeight()
解决方案
见我的YuvConverter。它的灵感来自android-Renderscript 将 NV12 yuv 转换为 RGB。
它的rs
部分非常简单:
#pragma version(1)
#pragma rs java_package_name(whatever)
#pragma rs_fp_relaxed
rs_allocation Yplane;
uint32_t Yline;
uint32_t UVline;
rs_allocation Uplane;
rs_allocation Vplane;
rs_allocation NV21;
uint32_t Width;
uint32_t Height;
uchar4 __attribute__((kernel)) YUV420toRGB(uint32_t x, uint32_t y)
{
uchar Y = rsGetElementAt_uchar(Yplane, x + y * Yline);
uchar V = rsGetElementAt_uchar(Vplane, (x & ~1) + y/2 * UVline);
uchar U = rsGetElementAt_uchar(Uplane, (x & ~1) + y/2 * UVline);
// https://en.wikipedia.org/wiki/YCbCr#JPEG_conversion
short R = Y + (512 + 1436 * V) / 1024; // 1.402
short G = Y + (512 - 352 * U - 731 * V) / 1024; // -0.344136 -0.714136
short B = Y + (512 + 1815 * U ) / 1024; // 1.772
if (R < 0) R == 0; else if (R > 255) R == 255;
if (G < 0) G == 0; else if (G > 255) G == 255;
if (B < 0) B == 0; else if (B > 255) B == 255;
return (uchar4){R, G, B, 255};
}
uchar4 __attribute__((kernel)) YUV420toRGB_180(uint32_t x, uint32_t y)
{
return YUV420toRGB(Width - 1 - x, Height - 1 - y);
}
uchar4 __attribute__((kernel)) YUV420toRGB_90(uint32_t x, uint32_t y)
{
return YUV420toRGB(y, Width - x - 1);
}
uchar4 __attribute__((kernel)) YUV420toRGB_270(uint32_t x, uint32_t y)
{
return YUV420toRGB(Height - 1 - y, x);
}
Flutter 中使用了我的 Java 包装器,但这并不重要:
public class YuvConverter implements AutoCloseable {
private RenderScript rs;
private ScriptC_yuv2rgb scriptC_yuv2rgb;
private Bitmap bmp;
YuvConverter(Context ctx, int ySize, int uvSize, int width, int height) {
rs = RenderScript.create(ctx);
scriptC_yuv2rgb = new ScriptC_yuv2rgb(rs);
init(ySize, uvSize, width, height);
}
private Allocation allocY, allocU, allocV, allocOut;
@Override
public void close() {
if (allocY != null) allocY.destroy();
if (allocU != null) allocU.destroy();
if (allocV != null) allocV.destroy();
if (allocOut != null) allocOut.destroy();
bmp = null;
allocY = null;
allocU = null;
allocV = null;
allocOut = null;
scriptC_yuv2rgb.destroy();
scriptC_yuv2rgb = null;
rs = null;
}
private void init(int ySize, int uvSize, int width, int height) {
if (bmp == null || bmp.getWidth() != width || bmp.getHeight() != height) {
bmp = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
if (allocOut != null) allocOut.destroy();
allocOut = null;
}
if (allocY == null || allocY.getBytesSize() != ySize) {
if (allocY != null) allocY.destroy();
Type.Builder yBuilder = new Type.Builder(rs, Element.U8(rs)).setX(ySize);
allocY = Allocation.createTyped(rs, yBuilder.create(), Allocation.USAGE_SCRIPT);
}
if (allocU == null || allocU.getBytesSize() != uvSize || allocV == null || allocV.getBytesSize() != uvSize ) {
if (allocU != null) allocU.destroy();
if (allocV != null) allocV.destroy();
Type.Builder uvBuilder = new Type.Builder(rs, Element.U8(rs)).setX(uvSize);
allocU = Allocation.createTyped(rs, uvBuilder.create(), Allocation.USAGE_SCRIPT);
allocV = Allocation.createTyped(rs, uvBuilder.create(), Allocation.USAGE_SCRIPT);
}
if (allocOut == null || allocOut.getBytesSize() != width*height*4) {
Type rgbType = Type.createXY(rs, Element.RGBA_8888(rs), width, height);
if (allocOut != null) allocOut.destroy();
allocOut = Allocation.createTyped(rs, rgbType, Allocation.USAGE_SCRIPT);
}
}
@Retention(RetentionPolicy.SOURCE)
// Enumerate valid values for this interface
@IntDef({Surface.ROTATION_0, Surface.ROTATION_90, Surface.ROTATION_180, Surface.ROTATION_270})
// Create an interface for validating int types
public @interface Rotation {}
/**
* Converts an YUV_420 image into Bitmap.
* @param yPlane byte[] of Y, with pixel stride 1
* @param uPlane byte[] of U, with pixel stride 2
* @param vPlane byte[] of V, with pixel stride 2
* @param yLine line stride of Y
* @param uvLine line stride of U and V
* @param width width of the output image (note that it is swapped with height for portrait rotation)
* @param height height of the output image
* @param rotation rotation to apply. ROTATION_90 is for portrait back-facing camera.
* @return RGBA_8888 Bitmap image.
*/
public Bitmap YUV420toRGB(byte[] yPlane, byte[] uPlane, byte[] vPlane,
int yLine, int uvLine, int width, int height,
@Rotation int rotation) {
init(yPlane.length, uPlane.length, width, height);
allocY.copyFrom(yPlane);
allocU.copyFrom(uPlane);
allocV.copyFrom(vPlane);
scriptC_yuv2rgb.set_Width(width);
scriptC_yuv2rgb.set_Height(height);
scriptC_yuv2rgb.set_Yline(yLine);
scriptC_yuv2rgb.set_UVline(uvLine);
scriptC_yuv2rgb.set_Yplane(allocY);
scriptC_yuv2rgb.set_Uplane(allocU);
scriptC_yuv2rgb.set_Vplane(allocV);
switch (rotation) {
case Surface.ROTATION_0:
scriptC_yuv2rgb.forEach_YUV420toRGB(allocOut);
break;
case Surface.ROTATION_90:
scriptC_yuv2rgb.forEach_YUV420toRGB_90(allocOut);
break;
case Surface.ROTATION_180:
scriptC_yuv2rgb.forEach_YUV420toRGB_180(allocOut);
break;
case Surface.ROTATION_270:
scriptC_yuv2rgb.forEach_YUV420toRGB_270(allocOut);
break;
}
allocOut.copyTo(bmp);
return bmp;
}
}
性能的关键是 renderscript 可以被初始化一次(这就是为什么YuvConverter.init()
是public)并且下面的调用非常快。
推荐阅读
- node.js - 在猫鼬中递归查找
- applescript - 在 AppleScript 中过滤记录
- javascript - Photoshop CC Javascript - 删除/剪切选区
- javascript - 在 onbeforeunload 事件期间执行调用
- php - 无法将 PHP7 安装到 AWS EC2 实例?
- python - 如何将参数传递给夹具并在测试方法中使用此夹具?
- ruby-on-rails-4 - 升级到 Rails 5.2 后 nil:NilClass 的未定义方法“expr”
- mysql - 使用 percona xtrabackup 将 Mysql 数据库迁移到 AroraDB
- python - Python-camelot(安装时出现错误:GhostscriptNotFound)
- scenebuilder - 场景生成器按钮 ID