SoMA Product Description

SoMA Product Description

Summary This document is the product description of the Sofica Multimedia Test Automation Solution (SoMA). SoMA is robot aided camera performance test system including software and hardware components. The main purpose of the SoMA is to automate execution of camera features validation process, which usually requires manual work. The document describes the system environment, components and principles of the test flow.

Contents

SUMMARY .............................................................................................................................................. 2

CONTENTS ............................................................................................................................................. 4

TERMS AND ABBREVIATIONS ............................................................................................................ 6

1 CONCEPT OF SOMA .......................................................................................................................... 7

2 VERIFICATION ENVIRONMENT ........................................................................................................ 8

2.1 Robot Arm ........................................................................................................................................ 9

2.2 Work Flow ...................................................................................................................................... 10

3 SOMA ARCHITECTURE ................................................................................................................... 11

3.1 Testing Different Software Layers ............................................................................................... 12

3.2 Platform Requirements of the SoMA ........................................................................................... 12

4 TEST EXECUTION SEQUENCE IN DUT .......................................................................................... 13

5 TESTING LEVELS ............................................................................................................................. 14

6 API TESTING ..................................................................................................................................... 15

6.1 Three steps of the API testing ...................................................................................................... 15

6.2 API Testing Targets ....................................................................................................................... 15

7 FUNCTIONAL TESTING OF THE CAMERA FEATURES ................................................................ 16

7.1 Result Reporting ............................................................................................................................ 16

7.2 Test Configuration ......................................................................................................................... 17

7.3 Functional Testing Targets .......................................................................................................... 17

7.4 Functional Tests ............................................................................................................................ 17

8 PERFORMANCE TESTING ............................................................................................................... 18

8.1 Performance testing targets ......................................................................................................... 18

8.2 Performance Tests ........................................................................................................................ 18

9 QUALITY TESTING ........................................................................................................................... 20

9.1 Quality Test Charts ........................................................................................................................ 20

9.2 Quality Tests .................................................................................................................................. 20

Terms and Abbreviations

API Application Program Interface

CPIQ Camera Phone Image Quality, currently part of the IEEE standardization organization, work group P1858

Dalvik Dalvik is the process virtual machine (VM) in Google's Android operating system. It is the software that runs the apps on Android devices.

DUT Device Under Test (i.e. target hardware)

Exif Exchangeable Image File Format

IPS display In-Plane Switching Display

ISP Image Signal Processing

SoMA Sofica Multimedia Test Automation Solution. The solution includes physical test environment building blocks and test automation enabler software as well as test cases used for automatic verification.

Test box Isolated box including the physical test environment

WinPRT Windows Phone Runtime library

Table 1: Terms and abbreviations

1 Concept of SoMA SoMA offers a new way of testing camera systems. SoMA, an robot aided automated testing system of camera features, ensures early problem detection during R&D and contributes to improved product quality. The core of the SoMA is the verification server, which contains services to validate different camera features. These services verifies the captured images and recorded video using mathematical algorithms to analyze content of data. The services are used by the device under test (DUT) using the SoMA API. Moreover, the test results are stored to the database and they can be investigated via the user interface as Figure 1 illustrates.

Figure 1: Concept of SoMA Even the SoMA offers complete test system, it can be integrated to the existing test system and used part by part. The SoMA DUT tests can be replaced by existing tests and the interface against the verification service is made using SoMA API which is easy to integrate. Onwards, the SoMA database can be replaced using existing database as well as the user interface of the SoMA. All in all, SoMA can be used as a stand-alone test system, which contains all the required parts of the automated test environment or it can be integrated to the existing test system, when the ready test mass can be used. In this case, the SoMA increases the existing test coverage and also automatically verifies the real output of the camera: captured images and recorded video.

2 Verification Environment The automated verification of the captured images and video stream requires stable and isolated environment to prevent the disturbances. For example, reflections of external light sources will affect to the testing. Therefore, the SoMA system offers an isolated test box, which ensures objective, reliable and repeatable test environment. The default SoMA test environment consist of two different reference image sources: a LCD –screen is used to the functional tests and transparent test charts are used for the quality regression testing. The usage of the LCD screen offers efficient way to use different reference data whereas the standard test charts allows to make very accurate quality tests. The quality test charts are illuminated using a light box, which is mounted inside the test environment. The environment supports up to 12 devices under tests (DUTs). The devices are located in mounting bracket inside the test box. The robot arm selects required device from the mounting bracket and adjust it towards the LCD screen or test chart depending on the executed test. The device swapping is done automatically by the SoMA.

Figure 2: SoMA verification environment The Verification Environment consists of

The test box that removes the interference and lightning reflections coming from the surrounding environment.

Device Under Test can be any device that has a camera attached.

LCD Display, the display uses an IPS-based panel and the native resolution is 3860x2140

Arduino based logic for audio/video synchronization testing

Light box to illuminate the quality test charts

A robot arm selects a DUT from the mounting bracket and adjusts the DUT against the LCD screen or test chart with correct orientation and distance.

2.1 Robot Arm The support of several testable devices requires a robot arm inside the test box. The robot has twofold functionality: it selects the required device from the slots of mounting bracket and calibrates the device position towards the LCD-screen or test chart. In the Figure 3 the robot arm has picked up a DUT from the first slot of mounting bracket and adjusted the device against Macbeth test chart to perform the color accuracy test.

Figure 3: Robot arm mounting in the test box The DUTs are mounted to the tool exchanger that is compatible towards the robot arm. Five connectors in the tool exchanger enable to connect DUT using USB. The details of the mounting bracket and the tool exchanger are shown in Figure 4.

Figure 4: DUTs in the mounting bracket

2.2 Work Flow SoMA solution can be integrated as a part of automated development and testing workflow. Figure 5 shows the Sofica’s automated tool chain, which executes SW building and testing activities automatically for the software directly from version control. This solution includes Test Automation and Test Results Generation parts of automated tool chain as well as SW Update Automation. Test Automation and SW Update can be initiated from continuous integration and dashboard user interface called Jenkins and as a result, it provides test results to the dashboard.

Figure 5: Automated Tool Chain

3 SoMA Architecture The SoMA architecture can be divided to four entities: Test server containing UI features, database, verification server and device under test (DUT). By default, the test server, database and verification server are located to same PC. As defined in Figure 6, Jenkins continuous integration server with SoMA plugins provides the user interface of SoMA. Onwards, testrunner-lite provides access to DUT and controls test execution based on XML files. The DUT contains the tests which are using the verification server services via the verification server API. The SoMA can be integrated to the existing test system by adding required method calls to the test cases. These methods will open the connection to the server, send the test data to the server and correspondingly receives the test results from the server. The interface against the server is not mandatory in all test cases. For example, the API tests of the SoMA do not require test data verification by the verification server. The verification server contains two main components: engine and test services. The engine handles the message sending between verification server and DUT and the service plugin administration. Each testable feature has own verification service which includes the verification algorithms of the specific testable feature. The database contains results of each test. Captured images, detailed results, function sequences and possible crash data are all stored in the results database (MongoDB).

Figure 6: Verification Server Architecture

A simplified test flow is introduced in Figure 6. The test execution steps as follows: 1. Test execution is started automatically or by user. 2. Jenkins plugin executes testrunner with given XML-file and other configurable

parameters. 3. Testrunner starts test in DUT. The connection between test server and DUT can be,

for example, adb connection. 4. The test uses SoMA API to query required services from the verification server. 5. The service verifies the captured image according to the test parameters and saves

the captured image and detailed results to results database. 6. When testrunner receives the result of the test execution, it saves the function

sequences and possible crash data to the database and informs Jenkins that test is executed.

7. When user queries the test result… 8. … SoMA Jenkins plugin opens a PHP-page which parses the specific results from the

database.

3.1 Testing Different Software Layers Different software layers can be tested using SoMA as the Figure 7 defines. The most usual scenario is to test the user application API and the adaptation software layer towards the hardware. Making this kind of double testing, the location of possible software errors can be resolved as well as the performance of each software layer can be measured.

Figure 7: Testing different software layers using SoMA

3.2 Platform Requirements of the SoMA The verification server requires Linux operating system. The server HW itself does not have any strict requirements; a normal workstation is enough to execute the test verifications. Verification server API is made with pure ANSI C++. This ensures the compatibility with most of the compilers. However, all the possible SW environments and platforms where the API is located are not known. Thus, a ready-compiled API library binary for all the possible combinations cannot be delivered.

4 Test execution Sequence in DUT Figure 8 shows an example test case sequence which is integrated to use the SoMA verification server. The yellow parts are the existing “capture image” test which is extended to support the verification of the data (green parts in the DUT). At the start-up, the existing test requests the reference image to be shown on the LCD display. The existing test code uses the camera of the DUT to take image from the display. Using the API, the existing test sends the captured data to the verification server and asks it to verify the correctness of the image. Finally, the test gets the result from the verification server and can set the verdict for the test accordingly.

Figure 8: Execution sequence

5 Testing levels SoMA contains several, separately selectable and usable testing levels as shown in the Figure 9. The testing levels are ensuring efficient testing at key phases of development. Depending on the existing test entity, different combination of these test levels can be selected to support the camera development phases in the most efficient way.

Figure 9: SoMA testing levels For example, the compatibility of the camera API can be ensured using the API testing. Onwards, when the features of the camera are implemented, the functional tests are effective. At the same time, performance tests can be started to detect the bottlenecks as early as possible. Stress and quality testing finalizes the testing entity by ensuring the robustness of the system and the final quality of the captured images. Even if tests are divided according to the development phases, all test levels can be used thorough the whole camera R&D process to avoid the regression. Following chapters defines the detailed information of the test levels. Also examples of detailed test descriptions are offered.

6 API Testing API testing of SoMA is more than a traditional API testing. Normally, the API testing concentrates to verify the compatibility of the API and test the parameter ranges of each method. The API testing of SoMA verifies also the dependencies between methods and makes robustness verification by testing illegal method scenarios. The API testing offers also possibility to compare the API compatibility and robustness between different devices. The API testing does not use reference images and the testing can be stand-alone test inside the DUT without the verification server and test environment.

6.1 Three steps of the API testing The API testing of the SoMA can be divided to the three steps: parameter tests, method dependency tests and method sequence tests. The first step contains the traditional API testing: the parameter space verification. The step verifies the functionality of the method by testing the smallest and greatest parameter value as well as random values between them. Also, a negative testing is done by given illegal parameter values to the method and verifying that the method returns correct error code. The second step of the API testing verifies the connectivity between methods. Normally, there are two kinds of dependences between methods: callbacks and set-get method pairs. The API testing test the set-get pairs by setting the parameters via set method and verifying, if corresponding get method returns correct value. The functionality of the get method is verified also in the situation, when illegal parameter value is set via the set method. Onwards, the callback logic also is ensured by detecting the required callback calls when corresponding methods are called. The third step of the API testing verifies the method sequences and the stability of the system when illegal method sequences are executed. This has revealed to be a very effective and stressful test to the camera systems and several robustness improvements are done after the sequence tests. The results of the API testing as well as exact method sequence are stored to the database and can be accessed via the user interface. Normally, the test result is simple pass/fail –status, however, the testing can also cause severe effects like crashes. In this case, also the detailed crash information is saved to the database and it is available via the user interface.

6.2 API Testing Targets The API testing can be implemented against every API in several platforms. Currently, the SoMA API testing has large reference implementation against Android CameraHardwareInterface. The API testing contains 200 different tests against this interface. CameraHardwareInterface is the device driver interface of Android camera system and it offers C++ API. However, same kind of tests can be done, for example, against Java-API. SoMA has also option to execute external tests, for example, Android Compatibility Test Suite (CTS) tests. When SoMA API testing verifies the device driver interface, CTS ensures the compatibility of the application layer. Together these features create a very comprehensive testing entity of the whole camera system.

7 Functional Testing of the Camera Features The individual functional testing cases are called as ‘services’. The term is used because they are located in the server and they offer different verification services to the DUT. The characteristic of the service is depending on the required verification demand and the content of the test data.

7.1 Result Reporting Each test gives a simple pass/fail result. In addition, tests provide more detailed information about the result as defined in Figure 10. This information is stored to the database and optionally, also to the file system. Moreover, the method sequence and all used parameters of the methods can be saved and investigated later. All these results can be accessed via the user interface.

Figure 10: Results of SoMA testing As in the API testing, functional testing can cause also severe effects like crashes. In this case, also the detailed crash information (e.g. logcat, core dump) is saved to the database and is available via the user interface.

7.2 Test Configuration Each test has configurable items which are read from the configuration file. The configuration file defines the used reference data for each case. Many of the tests have also configurable threshold values that are used to determine whether the test is a failure or not. The configuration file is structured so that each test has its own keywords, under which the configurable items are.

7.3 Functional Testing Targets The functional testing can be implemented against the camera API in several platforms. Currently, the SoMA functional testing has a large reference implementation against Android CameraHardwareInterface, Android Java Camera API and the Windows Phone Runtime Camera API of the Windows Phone.

7.4 Functional Tests Currently, SoMA supports following functional tests:

• Image resolution • Video resolution • Image content verification • Video content verification • Image EXIF data correctness • Auto Focus • Auto Exposure • Auto Exposure compensation • Auto White Balance • Dynamic Range • Flash • Zoom • Face Detection • Smile Detection • Blink Detection • Audio / Video synchronization • Color effects • IR hotspot

Upcoming features

• Image stabilization test • Panorama test

8 Performance Testing The performance testing reveals the bottlenecks of the camera system by measuring the execution times of different camera functionalities and combinations of the camera features. Very high resolution camera sensors may cause critical performance issues when megabyte sized images are processed in a very short time. The performance testing uses timestamps when the methods of the camera API are called and correspondingly when the result is received or callback method is executed. The execution time is simply the difference of these timestamps. The performance testing of SoMA measures and saves the execution time of each camera feature. For example, the compression time of the encoder can be measured as well as the shutter time with different auto focus settings. Even if this is quite straightforward measuring, different combinations of features may change the performance values significantly. SoMA enables to measure different combinations and recognize the most critical ones. Mainly, the performance tests do not require reference images. However, there are some features which affects to the performance and the image capturing circumstances should be static to ensure objective testing results. For example, the execution time of the auto focus and auto exposure depends on the environment. Therefore, part of the performance tests can optionally use the verification server. SoMA enables also to study performance trends between test executions. This is an essential feature to detect improvement of the development or, vice versa, detect the regression issues as early as possible. The performance testing enables also performance value comparison and benchmarking between different devices.

8.1 Performance testing targets The performance testing can be implemented against the camera API in several platforms. Currently, the SoMA performance testing has a large reference implementation against Android CameraHardwareInterface, Android Java Camera API and the Windows Phone Runtime Camera API of the Windows Phone.

8.2 Performance Tests Following list describes examples of SoMA performance tests:

OpenCameraHardware_time - Measures the time to initialize the camera.

AutoFocus_time - Measures the time that camera takes when focusing on to target.

Shutter_time - Measures the shutter lag.

CompressedImg_time - Measures the time that camera takes to capture compressed image.

FocusShutterCompressed_time_Default - Measures the time taken for Autofocus, shutter lag and compressed image capture with default resolution.

FiveShot_totalTime_Default - Measures the time to take 5 pictures in a row with default resolution.

ShotToShot_Burst_3_time_Default - Measures shot-to-shot time with burst mode image capture. Test case captures 3 images, and calculates the time for each received image.

PreviewFrame_time - Measures the time to receive first preview frame after startPreview is called.

RawImg_time - Measures the time that camera takes to capture raw image.

9 Quality Testing The quality testing is the next step from functional tests. Currently, four different camera characteristics are measured: noise of the system, color reproducing, true resolution and the lens specific issues like distortion and chromatic aberration. These tests give a good overall description of the camera quality. More ISP specific tests like texture blurring are currently in progress.

9.1 Quality Test Charts There are two different methods to show the test charts: Lens Distortion and Lateral Chromatic Aberration uses LCD-screen as a source of the test chart. Other quality tests use transparent test charts which are illuminated using a light box. All the test charts are standard test charts described by IEEE or CPIQ organizations.

9.2 Quality Tests Following list describes examples of SoMA quality tests:

Noise - Measures the amount of noise in the camera system. Uses combined ISO 14524 OECF, ISO 15739 test chart.

Color Accuracy – Measures the color reproducing of the camera system. Uses Macbeth color test chart.

Resolution – Measures the true resolution of the camera system. Uses Enhanced ISO 12233 Resolution test chart.

Lateral Chromatic Aberration – Measures the phenomenon, where the different colors have different refractive index in the lens system. Measured using a dot chart according to the CPIQ instructions.

Lens Distortion – Measures the geometric changes which are caused by the lens system. Measured using a dot chart according to the CPIQ instructions.