The Latest Achievements In The Area Of Artificial Intelligence And Ai Camera

Abstract

In computer science, artificial intelligence, sometimes called machine intelligence, is intelligence demonstrated by machines, in contrast to the natural intelligence displayed by humans. An AI accelerator is a class of microprocessor or computer system designed as hardware acceleration for artificial intelligence applications, especially artificial neural networks, machine vision and machine learning. AI cameras are simply cameras that use AI programs to wisely deal with images and videos. Computational photography is usually the core of an AI-powered camera. They are designed for the fast processing of rapidly changing image data, which would use more processor bandwidth and power in a conventional chip. These tweaks are particularly useful for camera-based AI, which tends to intersect with things like augmented reality and face recognition.

Introduction

Today, our life is relying on our smartphones and other mobile devices to do many of our daily tasks. From surfing the web, playing games and staying in touch with the world via instant messaging or social media, our smartphones are an integral part of everyday life. New generation of mobile devices is set to be smarter than anything we’ve seen previously, Unbelievable advanced AI capabilities to provide users with a smoother, more intelligent experience than ever before with a very small AI microprocessor with neural processing. Future of processing is to divert from CPU to NPU (Neural Processing Unit). Before we get into details, let's figure out if an AI chip is really all that different from existing CPUs. A term you'll hear a lot in the industry with reference to AI lately is 'heterogeneous computing.' It refers to systems that use multiple types of processors, each with specialized functions, to gain performance or save energy. The idea isn't new (plenty of existing chipsets use it) the three new offerings in question just employ the concept to varying degrees. Apple A12 Bionic SoC was released on 2018 based on octa core Neural Engine. Samsung's Exynos 9820 possess integrated Neural Processing Unit (NPU). This NPU allows the processor to perform AI-related functions seven times faster than its predecessor.

Smarter users are not only satisfied with high efficiency processors performance. Every user wants to capture their moments in more efficient way. So camera performance is also basic need in mobile processing field. Smart look and smarter outcome is the goal in this era. AI is about new kinds of software, initially to make up for smartphones’ lack of zoom lenses. Software is becoming more and more important for smartphones because they have a physical lack of optics, so we’ve seen the rise of computational photography that tries to replicate an optical zoom. “Top-end smartphones are increasingly featuring dual-lens cameras, but the Google Pixel 3 uses a single camera lens with computational photography which shows that software implementations are most important to achieve Smart goal. Automatic red-eye removal has been in DSLR cameras for years, as has face detection and, lately even smile detection, whereby a selfie is automatically taken when the subject cracks a grin Background Blur Panorama Bokeh and Portraits made DSLR Cams more special but now a day with AI Camera with smarter Software implementations these all tasks can be done with small lenses merged in slim smart mobile phones. Detailed imaging and capturing more than it looks is the target of AI Camera.

AI Processors

With the advancement in mobile system-on-chip (SoC) technologies, the performance of portable Android devices has increased by a multiple times within past few years. With their multi-core processors, dedicated GPUs, and gigabytes of RAM, the capabilities of current smartphones have already gone far beyond running the standard built-in phone applications or simple mobile games. Whereas their computational power already significantly exceeds the needs of most everyday use cases, artificial intelligence algorithms still remain challenging even for high-end smartphones and tablets. Despite the fact that many machine learning solutions are highly useful when deployed on end-user devices, running them on mobile platforms is associated with a huge computational over head on phone CPUs and a serious drain on battery power. Here while using AI processors mobiles and other processing devices have become more smarter. Smart (AI Supported) processor knows what to process. Priorities of processing, GPU clocking and CPU clocking where to perform with efficient capability. AI processor knows its frequency of usage. As we know heavy processing needs heat to exhaust, smart processor keeps device cool by keeping processor on minimal clocking as requires to process all tasks in queue.

Hardware Acceleration

Hardware acceleration is the process by which an app will use other hardware components on your system to perform certain tasks in order to work more efficiently. While the initial based on microprocessor computers were mostly equipped with a single, stand-alone CPU, it soon became clear that its computational performance is too limited for a number of multimedia applications. This led to the creation of special co-processors working in parallel with the main CPU. Their architecture was optimized for many signal processing tasks. As hardware manufacturers knew that single core as not sufficient to complete in future processing task. So manufacturers starting working on parallel processing. They introduce dual, quad, hexa and octa core processors etc for personal processing devices. And with passage of time technology of processor’s architecture was revolutionized. Microprocessors built architecture comes to nanometer technology. Now in 2019 most efficient AI based processor are working with 7 nanometer transistor technology.

Reducing size of transistor cause minimal clock or processing cycle, which means more processing in less time. Processor doesn’t only mean to process command given to it. Processor controls all the hardware’s and software implemented to the system. With the integration of cameras and many multimedia features like music and video playback in mobile devices. In other hand the integrated DSPs (Digital Signal Processing) started to be extensively used for image, video and sound processing. In contrast to what happened with desktop computers, DSPs were not displaced here by CPUs and GPUs because they often offered superior performance at lower power consumption, so critical for portable devices. In recent years, the computational power of mobile DSPs and other SoC components has grown drastically, and now, complemented by GPUs, NPUs and dedicated AI cores, they enable AI and deep learning-based computations.

QUALCOMM Chipsets

Qualcomm is an American semiconductor and wireless telecommunications company, founded in1985. Its first Snapdragon mobile SoC QSD8250 was released in 2007 and already featured a dedicated AMDZ430 GPU and the first commercial generation of QDSP6 Hexagon DSPs. In 2009, after the acquisition of AMD’s mobile graphics division, the corresponding GPU series was renamed to Adreno (anagram from Radeon), and its successors are present under this name in all current Snapdragon SoCs. The DSP architecture has also undergone significant changes from the first (2006) to the current seventh generation, and is now supporting wide vector extensions (HVX), dynamic multi-threading, The main Snapdragon CPU cores have an Arm-based architecture and usually feature Qualcomm’s own customized in-house design, often developed based on Arm Cortex cores. These three components (CPUs with the Arm NEON instruction set, GPUs and DSPs) form Snapdragon’s heterogeneous computing architecture well suitable for running various AI algorithms. The Qualcomm chipsets are now covering around 55% of the smartphone SoC market and are installed in many popular smartphones, tablets, and wearables.

Qualcomm first addressed the problem of on-device AI inference hardware acceleration in the Snapdragon 820 in May 2015 and also announced its proprietary Snapdragon Neural Processing Engine (SNPE) SDK in May 2016, which offers runtime acceleration across all Snapdragon’s processing components. The SDK supports common deep learning model frameworks, such as Caffe/Caffe2, TensorFlow, PyTorch, Chainer, MxNet, CNTK and PaddlePaddle via ONNX. It is designed to enable developers to run their own custom neural network models on various Qualcomm-powered devices. The SDK is supported on 17 Snapdragon mobile processors starting from premium (Snapdragon 845, 835, 820), high tier (Snapdragon 710, 670, 660, 652, 650, 653, 636, 632, 630, 626 and 625) as well as the mid-tier (Snapdragon 450, 439, 429). It also supports the Qualcomm Vision Intelligence Platform (QCS603 and QCS605), designed for efficient machine learning on IoT devices.

Huawei’s HiSilicon Chipsets

HiSilicon is a Chinese semiconductor company founded in 2004 as a subsidiary of Huawei. Its first mobile processor (K3V1) was introduced in 2008, but the first commercially successful product used in a number of Android devices was the next SoC generation (K3V2) released in 2012 and featuring four Arm Cortex-A9 CPU cores and a Vivante GPU. In 2014, a new Kirin SoC family consisting of mid-range (600 Series) and high-end (900 Series) chipsets was launched as a successor to the K3 series and is used in Huawei devices until now. Unlike Qualcomm, HiSilicon does not create customized CPU and GPU designs and all Kirin chipsets are based on off-the-shelf Arm Cortex CPU cores and various versions of Mali GPUs. A different approach was also developed for accelerating AI computations: instead of relying on GPUs and DSPs, HiSilicon introduced a specialized neural processing unit (NPU) aimed at fast vector and matrix-based computations widely used in AI and deep learning algorithms. According to Huawei, it delivers up to 25 times better performance and 50 times greater efficiency compared to the standard quad core Cortex-A73 CPU cluster. The NPU design was licensed from the Cambricon Technologies Company (Cambricon-1A chip) and is said to deliver a peak performance of about 1.92 TFLOPs, though this number mainly refers to quantized 8-bit computations. This NPU first appeared in the Kirin 970 SoC, and later two enhanced NPUs were also integrated into the subsequent Kirin 980 chipset. It should be noted that other SoCs apart from Kirin 970/980 do not contain this NPU module and are currently unable to provide acceleration for third party AI-based applications. The aforementioned chipsets can be found only inside Huawei devices as they are not sold to external OEM companies; the current total market share of HiSilicon SoCs is around 10%. To give external access to Kirin’s NPU, Huawei released in late 2017 the HiAI [38] Mobile Computing Platform SDK, providing APIs for executing deep learning models on hardware resources integrated within Kirin SoC. This SDK is now supporting only Caffe, Tensorflow Mobile and Lite frameworks, though in future releases it might also offer support for Caffe2 and ONNX.It provides acceleration for16-bit float, 8bit and 1-bit quantized models, and can additionally speed-up sparse models by skipping multiply-add operations containing zero variables. Apart from low-level APIs, the HiAI Engine also provides a ready-to-use implementation of several computer vision algorithms including image categorization, face and facial attribute detection, document detection and correction, image super-resolution, QR code detection, etc. Starting from Android 8.1(EMUI8.1), Huawei is including NNAPI drivers for its Kirin 970/980 chipsets that are generally based on the HiAI implementation. Currently, they are providing support only for 16-bit float models, quantized networks will be supported in the future releases. It should be mentioned that all Huawei devices that are based on other chipsets do not contain NNAPI drivers as they are lacking the abovementioned NPU module.

MediaTek Chipsets/Neuro Pilot SDK

MediaTek is a Taiwanese semiconductor company spun off from the United Microelectronics Corporation in 1997. Its mobile division was launched in 2004 and soon after this MediaTek released its first mobile chipsets that were used in many entry-level Chinese phones and smartphones produced at that time. It gained popularity on the global smartphone market in 2013 with the introduction of the MediaTek 657x/658x family of dual and quad-core SoCs with Mali or PowerVR graphics, and later with the release of 64-bit MediaTek MT67xx chipsets they became widely used in many Android devices from various OEMs, getting a market share of about 20%. Similarly to Huawei, MediaTek is integrating into its SoCs standard ArmCortex CPU cores and Mali or PowerVR GPUs. At the beginning of 2018, MediaTek addressed the problem of accelerating machine learning-based applications by launching their Helio P60 platform with embedded AI processing unit (APU). This APU can deliver the performance of up to 280GMAC/s for 8-bit computations and is primarily used for accelerating quantized neural networks, while float models are running on four Cortex-A53 CPU cores and Mali-G72 MP3 GPU clocked at 800MHz. Thus, MediaTek’s approach lies in between the solutions from Huawei and Qualcomm: a dedicated chip for quantized computations (as in Kirin’s SoC) and CPU/GPU for float ones (as in Snapdragon chipsets). The release of the Helio P60 was accompanied by the introduction of MediaTek’s NeuroPilot SDK constructed around TensorFlow Lite and Android NNAPI. The SDK is supporting purely MediaTek NeuroPilot-compatible chipsets (currently Helio P60 only). There also exists a corresponding stand-alone version of NNAPI drivers supporting float and quantized models. None the less, except for the P60 developer platform, only one

Arm Cortex CPUs/Mali GPUs/NN SDK

Currently, all CPU cores integrated into mobile SoCs are based on the Arm architecture, and in devices not supporting HA for machine learning applications these CPUs are responsible for running all AI algorithms. To speed-up the computations in this case, Arm has introduced a number of specific instruction sets aimed at fast vector- and matrix-based calculations. The most notable technology here is the Arm NEON an advanced SIMD (single instruction multiple data) architecture extension first introduced in Arm v7 processors. NEON basically implements DSP-like instructions for concurrent computations and allows the simultaneous execution of upto16x8-bit, 8x16-bit, 4x32-bit, 2x64-bit integer and 8x16-bit, 4x32-bit, 2x64-bit floating-point operations. Additionally, Arm has recently presented its new DynamIQ technology that is able to efficiently utilize all cores with in a single Arm CPU for parallel computations, and a specific instruction for calculating dot products in the Arm v8.4-A microarchitecture. Many of these optimized instructions are integrated in Google’s default NNAPI drivers, handling the CPU path when no other means for acceleration are available. Apart from that, Arm has also presented the Arm NN SDKto accelerate machine learning computations on mobile SoCs. It provides both the CPU and GPU paths for ML workloads, along with parsers for TensorFlow, Caffe, ONNX and TFLite. On the CPU side it is compatible with any platform with Armv7 and above CPUs (assuming NEON avail ability), with key low level optimizations for specific architectures.

AI Cameras

When hearing the words ‘AI’, ‘Machine Learning’ or ‘bot’ most people tend to visualize a walking, talking android robot which looks like something out of a Sci-Fi movie and immediately assume about a time far away in the future. AI has been around us for years now and is currently residing in your smartphone. With the advent of technology, it is becoming increasingly common to see visually appealing images with ultrahigh resolution. People no longer need to learn using tools like Photoshop and CorelDRAW to enhance and alter their images. AI is already being used in every aspect of image augmentation and manipulation in order to produce the best possible pictures. However, the latest idea to emerge is actually using AI to generate images, synthetically. Nearly every image that you might have seen would have been a captured photograph or manually created by a living, breathing person. There are possibly hundreds of tools for producing images manually but they do require a human presence to preside over the process. However, imagine a computer program that draws from scratch whatever you tell it to. Microsoft’s Drawing Bot might be one of the first and only such technologies that make this possible.

How AI is Changing Photography

If you’re wondering how good your next phone’s camera is going to be, it’d be wise to pay attention to what the manufacturer has to say about AI. Beyond the hype and bluster, the technology has enabled staggering advances in photography over the past couple of years. the most impressive recent advancements in photography have taken place at the software and silicon level rather than the sensor or lens — and that’s largely thanks to AI giving cameras a better understanding of what they’re looking at.

Ratio Between Hardware and Software to get AI Environment for Perfect Click

Last year in 2018 here started a trend of implementing multiple camera lenses to mobile phones in name of getting perfect click by using AI algorithm. It was a game changer step in mobile phones industry. Hardware and AI based Software were combined to get revolutionary images quality captured with a smart phone. Who could imagine this achievement when we were switching mobile phones from VGA to Megapixel based mobile phones that there will be a day when mobile phones will be capable of capturing images and videos better than Professional’s HDR cameras. Looking at Huawei Mate 20 Pro, flagship model by Huawei of 2018 it had 3 cameras on rear side ensuring of capturing video quality of qualifying Censor Board’s standard’s qualification. Suitable hardware and software implementation made Huawei to achieve this goal. At the other hand Google launched its Flagship model of Google pixel series i.e Google Pixel 3 which had only 1 camera lens at its rear side. Google achieved approximately same picture quality as Mate 20 Pro provides by implementing great software based AI algorithms to its device. Google surprised its audience by its revolutionary AI techniques.

AI Camera Applications

Face Recognition

The goal of this task is to retrieve the most similar face to a given one from an existing facial database. To do this, a neural network is first trained to produce a small feature vector for each facial image that encodes its visual features and is invariant to face scaling, shifts and rotations. After the network is trained, it is applied to a new facial image and produces its feature vector that is then used to retrieve the closest vector (and the respective identity) from the database. Face Recognition tool also helps to use it as biometrics for securing devices. Android Lollipop introduced the Trusted face feature, which allows you to unlock your tablet or phone using face recognition. That said, it's not as reliable as Apple's Face ID, and people can still access your Android device if they know your password.

Image Deblurring

This application is aimed at removing Gaussian blur from images, which is done using the SRCNN network one of the first CNNs proposed for the super resolution problem that is now widely used as a baseline for many image-to-image translation tasks. The architecture of this network is very shallow: three layers with 9×9 and 5×5 filters, in total 69,162 parameters and around 64B multiplyadd operations for HD-resolution image.

As a result, the size of the saved pre-trained network is only 278KB. Two artificial intelligence systems built by Google are able to transform a heavily pixellated, low quality, image into a clear photo or a person or object. Computer scientists from Google Brain, the central Google AI team, have shown it's not only possible to enhance a picture's resolution, they can fill in missing details in the process.

Image Enhancement

It improves the visual appearance of an image. Techniques are used for image enhancement using artificial neural network and fuzzy logic. It denoises and enhance an image when it is corrupted by different noises such as salt and pepper, gaussian and non-gaussian noises. In Image analysis, denoising and enhancing are most important pre-processing and post-processing steps. Several filters have been illustrated till date but have many limitations. In the proposed technique, Artificial neural network determines type of noises whereas Fuzzy logic used for denoising and enhancement purpose.

Face Beauty

If you own a phone which runs Android, the chances are it has a camera setting you don't know about. Popular smartphones like the Samsung Galaxy S8, the Galaxy S7, the Huawei P9, and many others often feature something called 'Face Beauty' or 'Beauty mode' when you turn on the front camera to take a selfie. Theoretically, it does what it says on the tin: airbrushing magic to make you look prettier in photos. Usually, it makes your skin look smoother and your eyes brighter.

Filters and Effects

In graphics and image-editing programs, image effects are predefined algorithms that enable you to add special effects to your images. The actual effects will depend on the software you use. You can usually choose different effects that will change the edges of your image, noise level, gradient and other aspects of your image.

With most programs you can simply select the name of the image effect and the program will produce the image with the effect, then you can save the changed image. You can also choose undo if you don't like it and select another one to try. Also, you may find your program offers an effects browser, which will open your image as a thumbnail in a browser window that allows you to quickly preview how each effect will look before performing the action. Image effects are used as a way to change your image to add an artistic look, make textured patterns, or produce an enhanced real-world view. Some graphics programs will offer a few predefined effects, others designed with effects in mind may offer hundreds of image effects to choose from.

Portrait or Bokeh

In photography and digital photography, portrait mode is a function of the digital camera that is used when you are taking photos of a single subject. When taking photos in portrait mode, the digital camera will automatically uses a large aperture to help keep the background out of focus by using a narrow depth of field so the subject being photographed is the only thing in focus. DSLR cameras are well known to get portraits which is expensive and heavy to carry and mobility in very low as compared to mobile. With help of AI and using efficient camera lese we are able to capture DSLR’s standards equivalent pictures with our cameras.

HDR+

HDR (high dynamic range) is virtually ubiquitous in smartphones now, but sometimes it’s an optional feature you can turn on and off yourself. With the Pixel 3, it’s on all the time, so you don’t need to worry about it.

High dynamic range brings the ability to capture bright highlights without blowing them out, but also pick up on details in the shadows. While in the past, camera phones typically took photos that suffered from noise and low dynamic range, Google found a way around the problem. HDR+ captures multiple shots and then combines them to produce a single image that is much less noisy and shows all the detail in both light and dark areas.

Conclusion

In this paper, we discussed the latest achievements in the area of machine learning, Artificial Intelligence and AI Camera. First, we presented an overview of all currently existing mobile chipsets that can be potentially used for accelerating the execution of neural networks on smartphones and other portable devices, and described popular mobile frameworks for running AI algorithms on mobile devices. We discussed DSP’s and SoC’s and touched most popular processor’s manufacturer and their short history. Then in AI camera chapter we discussed AI camera and its revolutionary hand in our lives. AI camera and its applications, and mportance of hardware and software to achieve AI Camera’s goals.

References

1. Krizhevsky, A.,Sutskever,I.,Hinton,G.E.: Imagenetclassificationwith deepconvolutionalneuralnetworks. In: Advancesinneuralinformation processing systems. (2012) 1097–1105
2. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2016) 2818–2826
3. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017) 1, 7
4. Ignatov, A., Kobyshev, N., Timofte, R., Vanhoey, K., Van Gool, L.: Dslr-quality photos on mobile devices with deep convolutional networks. In: the IEEE Int. Conf. on Computer Vision (ICCV). (2017)
5. Ignatov, A., Kobyshev, N., Timofte, R., Vanhoey, K., Van Gool, L.: Wespe: weakly supervised photo enhancer for digital cameras. arXiv preprint arXiv:1709.01118 (2017)
6. Ignatov, A., Timofte, R., et al.: Pirm challenge on perceptual image enhancement on smartphones: Report. In: European Conference on Computer Vision Workshops. (2018)
7. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deepconvolutionalnetworks. IEEEtransactionsonpatternanalysisand machine intelligence38(2) (2016) 295–307 1, 8
8. Ledig, C., Theis, L., Husz´ar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A.P., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image super-resolution
9. Timofte, R., Gu, S., Wu, J., Van Gool, L., et al.: Ntire 2018 challenge on single image super-resolution: Methods and results. In: The IEEEConferenceonComputerVisionandPatternRecognition(CVPR) Workshops. (June 2018) 1
10. Netzer,Y.,Wang,T.,Coates,A.,Bissacco,A.,Wu,B.,Ng,A.Y.: Reading digits in natural images with unsupervised feature learning. In: NIPS workshop on deep learning and unsupervised feature learning. Volume 2011. (2011) 5 1