Efficient Image Processing - Nicolas Roard
-
Upload
paris-android-user-group -
Category
Technology
-
view
1.108 -
download
4
Transcript of Efficient Image Processing - Nicolas Roard
![Page 1: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/1.jpg)
EFFICIENT IMAGE PROCESSING ON ANDROID
Nicolas Roard
![Page 2: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/2.jpg)
EFFICIENT IMAGE PROCESSING
• Works well on all hardware
• Fast, ideally realtime interaction
• Handles complex and flexible processing
• Handles large images
• Minimize memory usage
![Page 3: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/3.jpg)
WE WANT TO HAVE OUR CAKE AND EAT IT TOO
![Page 4: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/4.jpg)
ANDROID KITKATPHOTO EDITOR
• Non-destructive edits
• Full-size images processing
• Combine effects freely
• Easy to use, yet powerful: grow with the user
![Page 5: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/5.jpg)
NON-DESTRUCTIVE EDITS
• Effects are modifiable or reversible without quality loss
• Allow re-edits of processed images
![Page 6: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/6.jpg)
RENDERSCRIPT (RS)
• Cool thingy that let you do fast image processing
![Page 7: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/7.jpg)
TIMELINE3 VERSIONS, 10 MONTHS
![Page 8: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/8.jpg)
4.2 - NOVEMBER 2012
• Color FX (9 looks)
• 11 Borders
• Geometry: Straighten, Rotate, Crop, Mirror
• Filters & Tools
• Autocolor, Exposure, Vignette, Contrast, Shadows, Vibrance, Sharpness (RS-based), Curves, Hue, Saturation, BW Filter
• Non-destructive edits (in the editor -- save create a copy)
• Exposed history
![Page 9: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/9.jpg)
G+ EDITOR - MAY 2013
• RenderScript implementations of Snapseed filters
• Frames, Film, Drama, Retrolux
• Non-destructive
• Cloud-based (local processing only used for caching and UI interactions)
![Page 10: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/10.jpg)
4.3 - JULY 2013
• Move to RenderScript
• New 16 image-based borders (ported from Snapseed, RS-based)
• Filters & Tools
• Highlights, Improved Vignette
• Local adjustment (ported from Snapseed, RS-based)
• New Tablet UI, refined UI, introduction of the state panel instead of the history panel
![Page 11: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/11.jpg)
4.4 - SEPTEMBER 2013
• Filters & Tools
• Custom borders, Drawing tool, negative, posterize
• RS filters: Graduated filter, Vignette, per channel saturation, sharpness/structure
• Refined UI (animations, etc.)
• Pinch to zoom enabled (full-res zoom)
• Re-edits enabled
• Background save service, export, print support
![Page 12: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/12.jpg)
DEMO
![Page 13: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/13.jpg)
![Page 14: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/14.jpg)
![Page 15: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/15.jpg)
SOME ADDITIONAL INFOS
• Phone and Tablet UI
• Filters in C & RenderScript
• Works on Full Size images -- largest tried was a 278MP image on a Nexus 7 2nd gen. Limited by available RAM.
• Nearly all of the editor is in AOSP!
![Page 16: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/16.jpg)
IMAGE PROCESSING
![Page 17: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/17.jpg)
PIPELINE
OriginalImage
![Page 18: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/18.jpg)
PIPELINE
OriginalImage Filter
![Page 19: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/19.jpg)
PIPELINE
OriginalImage Filter Processed
Image
![Page 20: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/20.jpg)
IMAGE PROCESSING
• In Java
• In native code (JNI calls to C/C++)
• In OpenGLES2
• RenderScript
![Page 21: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/21.jpg)
JAVA
• Use getPixel()
• Use getPixels()
• Use copyPixelsToBuffer() [premultiplied]
• GC calls. GC calls everywhere.
![Page 22: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/22.jpg)
NATIVE CODE
• Pass a Bitmap through JNI to C/C++
• Quite fast & pretty easy to work with (pointer to the bitmap -- and no GC!)
• JNI / Native can be fastidious
• Handling different CPU architectures can be an issue
• Optimizations can be complicated
• JNI management
![Page 23: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/23.jpg)
OPENGL ES 2.0
• Fast -- can write interactive processing
• Hard to ensure the shaders will perform well on all devices
• Limited in size (max texture size...)
• Needs adhoc shaders, i.e. fixed pipelines.
• Expensive to retrieve processed image
![Page 24: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/24.jpg)
RENDERSCRIPT
![Page 25: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/25.jpg)
“RENDERSCRIPT IS A FRAMEWORK FOR RUNNING COMPUTATIONALLY INTENSIVE TASKS AT HIGH PERFORMANCE ON ANDROID. RENDERSCRIPT IS PRIMARILY ORIENTED FOR USE WITH
DATA-PARALLEL COMPUTATION, ALTHOUGH SERIAL COMPUTATIONALLY INTENSIVE WORKLOADS CAN BENEFIT AS WELL.”
![Page 26: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/26.jpg)
• Write “kernels” in a C99-like language with vector extensions and useful intrinsics
• RenderScript executes them in parallel, on the GPU or CPU
• Java used to manage lifetime of objects/allocations and control of execution
• Portability
![Page 27: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/27.jpg)
RENDERSCRIPT
• Fast -- through LLVM optimizations and Parallelization
• Supports CPU / GPU
• Compatibility Library
• Easy to offload to the background
• Pretty easy to write
![Page 28: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/28.jpg)
RENDERSCRIPT
• Cannot allocate memory from kernels, need to do it from outside
• RenderScript can be called from Java or from Native
• Compatibility library!
![Page 29: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/29.jpg)
![Page 30: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/30.jpg)
![Page 31: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/31.jpg)
![Page 32: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/32.jpg)
HOW?
• Optimized Math library
• Optimizations on the device
• Easier to read & maintain (vector math library helps)
![Page 33: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/33.jpg)
ALLOCATIONS
• Bound to Scripts
• Can be bound to SurfaceTexture (producer & consumer)
• Can share memory between Allocation and Bitmap
![Page 34: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/34.jpg)
HOW TO USE IT
![Page 35: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/35.jpg)
#pragma version(1)#pragma rs java_package_name(com.example.rsdemo)
uchar4 __attribute__((kernel)) color(uchar4 in) { return in;}
1. SCRIPT
![Page 36: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/36.jpg)
RenderScript mRS = RenderScript.create(mContext);
ScriptC_filter filter = new ScriptC_filter(mRS,mResources, R.raw.filter);
2. CREATE CONTEXT
![Page 37: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/37.jpg)
Bitmap bitmapIn = BitmapFactory.decodeResource(getResources(), R.drawable.monumentvalley);
Bitmap bitmapOut = bitmapIn.copy(bitmapIn.getConfig(), true);
3. LOAD BITMAP
![Page 38: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/38.jpg)
Allocation in = Allocation.createFromBitmap(mRS, bitmapIn, Allocation.MipmapControl.MIPMAP_NONE, Allocation.USAGE_SCRIPT
| Allocation.USAGE_SHARED);
Allocation out = Allocation.createTyped(mRS, in.getType());
4. CREATE ALLOCATIONS
![Page 39: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/39.jpg)
filter.forEach_color(in, out); out.copyTo(bitmapOut);
5. APPLY THE SCRIPT
![Page 40: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/40.jpg)
![Page 41: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/41.jpg)
ScriptIntrinsicBlur blur = ScriptIntrinsicBlur.create(mRS, Element.U8_4(mRS));
blur.setRadius(25.f);blur.setInput(in);blur.forEach(in);
SCRIPT INTRINSICS
![Page 42: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/42.jpg)
![Page 43: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/43.jpg)
![Page 44: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/44.jpg)
READY TO USE• ScriptIntrinsic3DLUT
• ScriptIntrinsicBlend
• ScriptIntrinsicBlur
• ScriptIntrinsicColorMatrix
• ScriptIntrinsicConvolve3x3
• ScriptIntrinsicConvolve5x5
• ScriptIntrinsicLUT
• ScriptIntrinsicYuvToRGB
• ScriptGroup
![Page 45: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/45.jpg)
PAINT IT BLACK (OR GRAY)
![Page 46: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/46.jpg)
uchar4 __attribute__((kernel)) color(uchar4 in) { return in;}
SCRIPT
![Page 47: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/47.jpg)
uchar4 __attribute__((kernel)) grey(uchar4 in) { in.g = in.r; in.b = in.r; return in;}
SCRIPT
![Page 48: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/48.jpg)
uchar4 __attribute__((kernel)) grey(uchar4 in) { in.gb = in.r; return in;}
SCRIPT
![Page 49: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/49.jpg)
![Page 50: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/50.jpg)
void Java_com_example_rsdemo_ProcessImage_processBitmap(JNIEnv* env, jobject this, jobject bitmap, jint width, jint height) {
unsigned char* rgb = 0; AndroidBitmap_lockPixels(env, bitmap, (void**) &rgb); int len = width * height * 4; int i; for (i = 0; i < len; i+=4) { int red = rgb[i]; rgb[i+1] = red; rgb[i+2] = red; } AndroidBitmap_unlockPixels(env, bitmap);}
NDK
![Page 51: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/51.jpg)
LOCAL EFFECT
![Page 52: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/52.jpg)
![Page 53: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/53.jpg)
uchar4 __attribute__((kernel)) grey(uchar4 in) { in.g = in.r; in.b = in.r; return in;}
SCRIPT
![Page 54: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/54.jpg)
uchar4 __attribute__((kernel)) grey1(uchar4 in, uint32_t x, uint32_t y) {
in.g = in.r; in.b = in.r; return in;}
SCRIPT
![Page 55: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/55.jpg)
uchar4 __attribute__((kernel)) grey1(uchar4 in, uint32_t x, uint32_t y) {
float range = (float) x / width; uint32_t grey = (1 - range) * in.r; in.r = grey; in.g = grey; in.b = grey; return in;}
SCRIPT
![Page 56: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/56.jpg)
![Page 57: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/57.jpg)
uchar4 __attribute__((kernel)) grey1(uchar4 in, uint32_t x, uint32_t y) {
float range = (float) x / width; uint32_t grey = (1 - range) * in.r; in.r = grey; in.g = grey; in.b = grey; return in;}
SCRIPT
![Page 58: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/58.jpg)
uchar4 __attribute__((kernel)) grey2(uchar4 in, uint32_t x, uint32_t y) {
float range = (float) x / width; uint32_t grey = (1 - range) * in.r; in.r = (in.r * range) + grey; in.g = (in.g * range) + grey; in.b = (in.b * range) + grey; return in;}
SCRIPT
![Page 59: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/59.jpg)
uchar4 __attribute__((kernel)) grey3(uchar4 in, uint32_t x, uint32_t y) {
float range = (float) x / width; float4 pixel = rsUnpackColor8888(in); float grey = (1 - range) * pixel.r; pixel.r = pixel.r * range + grey; pixel.g = pixel.g * range + grey; pixel.b = pixel.b * range + grey; return rsPackColorTo8888(
clamp(pixel, 0.f, 1.0f));}
SCRIPT
![Page 60: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/60.jpg)
uchar4 __attribute__((kernel)) grey4(uchar4 in, uint32_t x, uint32_t y) {
float range = (float) x / width; float4 pixel = rsUnpackColor8888(in); float grey = (1 - range) * pixel.r; pixel.rgb = pixel.rgb * range + grey; return rsPackColorTo8888(
clamp(pixel, 0.f, 1.0f));}
SCRIPT
![Page 61: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/61.jpg)
EXAMPLE: BLOOM
• Select the bright pixels
• Blur the result
• Add the blurred bright pixels back to the image
![Page 62: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/62.jpg)
![Page 63: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/63.jpg)
![Page 64: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/64.jpg)
private void brightPass(int[] pixels, int width, int height) { int threshold = (int) (brightnessThreshold * 255); int r; int g; int b;
int luminance; int[] luminanceData = new int[3 * 256]; // pre-computations for conversion from RGB to YCC for (int i = 0; i < luminanceData.length; i += 3) { luminanceData[i ] = (int) (i * 0.2125f); luminanceData[i + 1] = (int) (i * 0.7154f); luminanceData[i + 2] = (int) (i * 0.0721f); }
WITH JAVA
![Page 65: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/65.jpg)
int index = 0; for (int y = 0; y < height; y++) { for (int x = 0; x < width; x++) { int pixel = pixels[index];
// unpack the pixel's components r = pixel >> 16 & 0xFF; g = pixel >> 8 & 0xFF; b = pixel & 0xFF; // compute the luminance luminance = luminanceData[r * 3] + luminanceData[g * 3 + 1] + luminanceData[b * 3 + 2]; // apply the treshold to select the brightest pixels luminance = Math.max(0, luminance - threshold); int sign = (int) Math.signum(luminance);
// pack the components in a single pixel pixels[index] = 0xFF000000 | (r * sign) < < 16 | (g * sign) << 8 | (b * sign);
index++; } }}
![Page 66: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/66.jpg)
uniform sampler2D baseImage;uniform float brightPassThreshold;
void main(void) { vec3 luminanceVector = vec3(0.2125, 0.7154, 0.0721); vec4 sample = texture2D(baseImage, gl_TexCoord[0].st);
float luminance = dot(luminanceVector, sample.rgb); luminance = max(0.0, luminance - brightPassThreshold); sample.rgb *= sign(luminance); sample.a = 1.0;
gl_FragColor = sample;}
WITH GL SHADER
![Page 67: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/67.jpg)
float brightPassThreshold;
uchar4 __attribute__((kernel)) brightPass(uchar4 in) { float3 luminanceVector = { 0.2125, 0.7154, 0.0721 }; float4 pixel = rsUnpackColor8888(in); float luminance = dot(luminanceVector, pixel.rgb);
luminance = max(0.0f, luminance - brightPassThreshold); pixel.rgb *= sign(luminance); pixel.a = 1.0; return rsPackColorTo8888(clamp(pixel, 0.f, 1.0f));}
WITH RENDERSCRIPT
![Page 68: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/68.jpg)
ScriptIntrinsicBlur blur = ScriptIntrinsicBlur.create(mRS, Element.U8_4(mRS));ScriptIntrinsicBlend blend = ScriptIntrinsicBlend.create(mRS, Element.U8_4(mRS));
filter.set_brightPassThreshold(0.15f);filter.forEach_brightPass(in, out); blur.setRadius(25.f);blur.setInput(out);blur.forEach(out);blend.forEachAdd(in, out);
out.copyTo(bitmapOut);
JAVA-SIDE
![Page 69: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/69.jpg)
WORKING WELL EVERYWHERE
![Page 70: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/70.jpg)
WORKING WELL ON ALL HARDWARE
• Architect for the worst
• Scale with the device capabilities
• Screen size / dpi
• Available memory
• Available CPU / GPU
• Think about what is a downgraded experience
![Page 71: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/71.jpg)
LOADING• Load in the background
• AsyncTask, or use a background thread
• Bitmaps loading
• query the size
• inSampleSize
• reuseBitmap
• BitmapRegionDecoder
![Page 72: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/72.jpg)
QUERY THE SIZE
BitmapFactory.Options options =new BitmapFactory.Options();
options.inJustDecodeBounds = true;BitmapFactory.decodeResource(
getResources(), R.id.myimage, options);int imageHeight = options.outHeight;int imageWidth = options.outWidth;String imageType = options.outMimeType;
![Page 73: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/73.jpg)
INSAMPLESIZE
• Only load what you need
• needs to be a power of two, so for a 2048x2048 image,
• insamplesize=2 => 1024x1024 image
• insamplesize=4 => 512x512 image
![Page 74: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/74.jpg)
CALCULATE INSAMPLESIZE
if (bounds.width() > destination.width()) {int sampleSize = 1;int w = bounds.width();while (w > destination.width()) {
sampleSize *= 2;w /= sampleSize;
}options.inSampleSize = sampleSize;
}
![Page 75: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/75.jpg)
REUSE BITMAP
BitmapFactory.Options options;(...)Bitmap inBitmap = ...(...)options.inBitmap = inBitmap;
API level 11 (Android 3.0)Before API level 19 (Android 4.4) only same size
![Page 76: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/76.jpg)
BITMAP REGIONDECODER
InputStream is = ...BitmapRegionDecoder decoder =
BitmapRegionDecoder.newInstance(is, false);Rect imageBounds = ...Bitmap bitmap =
decoder.decodeRegion(imageBounds, options);
API level 11 (Android 3.0)
![Page 77: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/77.jpg)
PIPELINE
![Page 78: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/78.jpg)
PIPELINE
• Run in a background service (used when saving too)
• Mix C, RenderScript, java filtering (canvas draw)
• multiple pipelines in parallel (direct preview, highres, icons, full res, geometry, saving)
![Page 79: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/79.jpg)
FLEXIBLE PROCESSING
• Unbounded pipeline
• No fixed order
• Complex filters
• Geometry-based
• Global
• Local
![Page 80: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/80.jpg)
FILTER TYPES
Color Fx Geometry * Borders
Crop
Straighten
Rotate
Mirror
Contrast
Saturation
Local
Vignette
Image-based
Parametric
Color transforms
![Page 81: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/81.jpg)
COLOR FX - 3D LUTVintage
Instant
Washout
X-Process
![Page 82: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/82.jpg)
CACHING
• Cache RS scripts, allocations
• Cache original bitmaps
• Aggressively destroy/recycle resources in filters to keep memory low
• If possible, filters should process the input bitmap directly
• N-1 cache
![Page 83: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/83.jpg)
MEMORY USAGE
• Bitmap cache heavily reusing bitmaps
• LruCache class (available in support lib too)
• Pick image sizes depending on the device resolution / DPI
• Have a path ready for a downgraded experience
![Page 84: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/84.jpg)
DEVICE CAPABILITIES
• Ask the system
• Runtime.getRuntime().maxMemory()
• New isLowRamDevice() API
• Handles low-memory signals
• Handles java.lang.OutOfMemory exceptions
![Page 85: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/85.jpg)
PIPELINE CACHE
OriginalImage
![Page 86: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/86.jpg)
PIPELINE CACHE
OriginalImage Filter Processed
Image
![Page 87: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/87.jpg)
REALISTICALLY...
OriginalImage
![Page 88: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/88.jpg)
REALISTICALLY...
OriginalImage Filter Processed
ImageFilter Filter Filter
![Page 89: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/89.jpg)
PROCESSING
OriginalImage Filter Processed
ImageFilter Filter Filter
![Page 90: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/90.jpg)
PROCESSING
OriginalImage Filter Processed
ImageFilter Filter Filter
![Page 91: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/91.jpg)
PROCESSING
OriginalImage Filter Processed
ImageFilter Filter Filter
![Page 92: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/92.jpg)
N-1 CACHING
OriginalImage Filter Processed
ImageFilter Filter FilterFilter
![Page 93: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/93.jpg)
N-1 CACHING
OriginalImage Filter Processed
ImageFilter Filter FilterFilter
![Page 94: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/94.jpg)
N-1 CACHING
OriginalImage Filter Processed
ImageFilter Filter FilterFilter
![Page 95: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/95.jpg)
N-1 CACHING
OriginalImage Filter Processed
ImageFilter Filter FilterFilter
![Page 96: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/96.jpg)
N-1 CACHING
OriginalImage Filter Processed
ImageFilter Filter Filter
![Page 97: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/97.jpg)
N-1 CACHING
• No silver bullet
• Only really useful when the user manipulates the last filters of the pipeline...
• ...but this is after all the more common scenario!
![Page 98: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/98.jpg)
REALTIME INTERACTION
![Page 99: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/99.jpg)
REALTIME INTERACTION
• Background processing
• Minimize allocations -- Careful with Garbage Collection!
• Optimized filters, RenderScript helps
• Caching in the pipeline
• Low/High resolution preview
![Page 100: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/100.jpg)
PREVIEW SYSTEM
Bitmap
Preset
Bitmap
Preset
Bitmap
Preset
UI Thread Processing Thread
![Page 101: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/101.jpg)
PREVIEW SYSTEM
Bitmap
Preset
Bitmap
Preset
Bitmap
Preset
UI Thread Processing Thread
![Page 102: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/102.jpg)
PREVIEW
Continuous preview
High-res preview
Full resolution
![Page 103: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/103.jpg)
PREVIEW SYSTEM
Low-res preview
High-res preview
Full-res preview
If new request, delay more
Request
![Page 104: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/104.jpg)
HOW TO CHEAT
• The triple-buffer preview can have a low resolution
• The UI elements and controls are animated and manipulated at 60 fps on the UI thread
• The rendering pipeline can (and needs to) be interrupted
![Page 105: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/105.jpg)
INTERRUPTION IN RENDERSCRIPT
LaunchOptions options = new LaunchOptions();options.setX(xStart, xEnd);options.setY(yStart, yEnd);mScript.forEach_vignette(in, out, options);
![Page 106: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/106.jpg)
LaunchOptions options = new LaunchOptions(); boolean even = true; int tile = 128; int height = bitmapIn.getHeight(); int width = bitmapIn.getWidth(); for (int yStart = 0; yStart < height; yStart += tile) { for (int xStart = 0; xStart < width; xStart += tile) { int xEnd = xStart + tile; int yEnd = yStart + tile; options.setX(xStart, xEnd); options.setY(yStart, yEnd); if (even) { filter.forEach_grey(in, out, options); } else { filter.forEach_color(in, out, options); } even = !even; } }
![Page 107: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/107.jpg)
![Page 108: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/108.jpg)
FULL RESOLUTION PREVIEW
![Page 109: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/109.jpg)
FULL RESOLUTIONPREVIEW
• Ideally, we should use a tile-based rendering
• At the moment, we use BitmapFactory region decoder instead
• Filters need to be able to handle partial regions
• future payoff: streaming save
![Page 110: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/110.jpg)
FUTURE
![Page 111: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/111.jpg)
FUTURE -- IN THE PHOTO EDITOR
• Merging filters
• RenderScript
• At the pipeline level (color cubes...)
• Streaming saving
• Some code is there (AOSP), but not used
![Page 112: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/112.jpg)
FUTURE -- IN ANDROID FRAMEWORK
• TileView
• Adding filtering pipeline in android framework
• Image loader improvements
• RAW support? Color correction?
![Page 113: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/113.jpg)
13223x559870MP image
~60 tiles, ~15Mb
![Page 114: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/114.jpg)
13223x559870MP image
~60 tiles, ~15Mb
![Page 115: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/115.jpg)
TILEVIEW
TileView TileViewAdapter
ImageTileViewAdapterTileGrid
TileCache TestTileViewAdapterTile
![Page 116: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/116.jpg)
TILEVIEW ADAPTER
public interface TileViewAdapter { public int getTileSize(); public int getContentWidth(); public int getContentHeight(); public void onPaint(float scale,
float dx, float dy, Bitmap bitmap); public Bitmap getFullImage(int max); void setDebug(boolean debug);}
![Page 117: Efficient Image Processing - Nicolas Roard](https://reader033.fdocuments.net/reader033/viewer/2022052505/554f9302b4c905d25b8b52e9/html5/thumbnails/117.jpg)
QUESTIONS, SUGGESTIONS?
• RenderScript documentation:
• http://developer.android.com/guide/topics/renderscript/compute.html
• contact: [email protected]