At its core, Edge AI is the combination of Edge computing and Edge intelligence to run machine learning tasks directly on end devices which generally consists of an in-built microprocessor and sensors, while the data processing task is also completed locally and stored at the edge node end. The implementation of machine learning models in edge AI will decrease the latency rate and will improve the network bandwidth. Edge AI helps applications that rely on real-time data processing by assisting with data, learning models, and inference. The edge AI hardware market valued at USD 6.88 billion is expected to reach USD 39 billion by 2030 at a CAGR of 18.8% as per the report by Valuates Reports.

The advancement of IoT & adoption of smart technologies by consumer electronics and automotive among others are fuelling the AI hardware market forward. Edge AI processors with on-device analytics are going to enhance the opportunities for the AI hardware market. NVIDIA, Google, AMD, Lattice, Xilinx, and Intel are some of the edge computing platforms providers for such cognitive AI applications design. Furthermore, the advancement of emerging technologies such as deep learning, AI hardware accelerators, neural networks, computer vision, optical character recognition, natural language processing, etc. opens all-new horizons of opportunities. While businesses are rapidly moving towards a decentralized computer architecture, they are also discovering new ways to use this technology to increase productivity.

What is Edge Computing?

Edge computing brings the computing and storage of data closer to the devices that collect it, rather than relying on a primary site that might be far away. This ensures that data does not suffer from latency and redundancy issues that limit an application’s efficiency. The amalgamation of Machine Learning into edge computing gives rise to new, resilient, and scalable AI systems in a wide range of industries.

Myth: Will Edge Computing suppress Cloud Computing?

No, edge computing is not going to replace nor suppress cloud computing, instead, the edge will complement with cloud environment for better performance and leverage machine learning tasks to a greater extent.

Need for Edge AI Hardware Accelerators

Running complex machine learning tasks on edge devices requires specialized AI hardware accelerators, which boost speed & performance, offer greater scalability, maximum security, reliability & efficient data management.

VPU (Vision Processing Unit)

A vision processing unit is a sort of microprocessor aimed at accelerating machine learning and AI algorithms. It balances Edge AI workload with high efficiency and supports tasks like image processing, which is like a video processing unit used with neural networks. It works on low power and high-performance precision.

GPU (Graphical Processing Unit)

An electronic circuit capable of producing graphics for display on an electronic device is referred to as a GPU. It can process multiple data simultaneously, making them ideal for machine learning, video editing, and gaming applications. With their ability to perform complex machine learning tasks, they are being extensively used in mobiles, tablets, workstations, and gaming consoles nowadays.

TPU (Tensor Processing Unit)

Google introduced the Tensor Processing Unit (TPU), an ASIC for executing Machine Learning (ML) algorithms based on neural networks. It uses less energy and operates more efficiently. Google Cloud Platform with TPUs is a good choice for ML applications that don’t require a lot of cloud infrastructure.

Applications of Edge AI across industries

Smart Factories

Edge AI can be applied for predictive maintenance belonging to the equipment industry, by which edge devices can perform analysis on stored data to identify scenarios wherein a failure might occur before the actual failure happens.

Autonomous Vehicles

Self-driving vehicles are one of the best examples of incorporating edge AI technology into the automobile industry, where the integration helps detection and identification of objects thereby eliminating chances for accidents. It aids in avoiding collision with pedestrians/other vehicles, detecting roadblocks, which requires immediate real-time data processing, as plenty of lives are at stake.

Edge AI

Industrial IoT

With the enablement of Computer Vision to Industrial IoT, visual inspections can be done effortlessly without much human intervention thereby increasing operational efficiency and improving the productivity in assembly lines.

Smart Healthcare

Edge Artificial intelligence can help in the healthcare industry via wearables enhancing surveillance of a patient’s health and forecasting early disorders. These details can also be used to provide patients with effective treatments in real-time. Patient data can be secured with HIPAA compliance in place.

Benefits of using Machine Learning on Edge

Higher Scalability
As the demand for the number of interconnected IoT devices is on the rise across industries, Edge AI is becoming an absolute choice due to its efficient and timely data processing without relying heavily on a cloud-based centralized network.

Data Protection & Security
Since Edge devices are not completely dependent on cloud resources, attackers cannot bring down the whole cloud data center/server system to a standstill point.

Low Operational Risks
Since Edge AI is based on a distributed model, in case of failure it will not affect the entire system chain, as in the case of cloud, which is based on a centralized model. The failure of individual edge devices will not pose a huge threat to the entire system.

Reduced Latency Rate
With the implementation of Edge AI, the computation can be performed in milliseconds. This is possible as there is no need to send data to the cloud for initial processing, thereby saving time and reduction of latency in the data processing.

Cost-effectiveness
Edge AI saves a lot of bandwidth, as the transfer of data is minimized. This also reduces the capacity requirements for cloud services which makes Edge AI a cost-effective solution, when compared to cloud-based ML solutions.

In several instances, machine learning models are complex and quite big. In such situations, it becomes extremely difficult to shift these models to compact edge devices. Without proper precautions if efforts are put to reduce the complexity of the algorithms, the processing perfection will take a toll, and also, the computation power will become limited. Hence at the initial development stage, it’s crucial to evaluate all the failure points. Most priority should be given to testing the trained model perfectly on different types of devices and operating systems.

At Softnautics, we provide machine learning services and solutions with expertise on edge platforms (TPU, RPi), NN compiler for the edge, and tools like TensorFlow, TensorFlow Lite, Docker, GIT, AWS deepLens, Jetpack SDK, and many more targeted for domains like automotive, Multimedia, Industrial IoT, Healthcare, Consumer, and Security-Surveillance. Softnautics can help businesses to build high-performance edge ML solutions like object/lane detection, face/gesture recognition, human counting, key-phrase/voice command detection, and more across various platforms. Our team of experts has years of experience working on various edge platforms, cloud ML platforms, and ML tools/ technologies.

Read our success stories related to Machine Learning expertise to know more about our services for accelerated AI solutions.

Contact us at business@softnautics.com for any queries related to your solution or for consultancy.

[elementor-template id=”11388″]

Overview

Xilinx’s new heterogeneous compute platform, Versal Adaptive Compute Acceleration Platform (ACAP), efficiently combines the power of software and hardware programmability. Versal ACAP devices are used for a wide range of applications such as data center, Wireless 5G, AI/ML, A & D Radars, Automotive, and wired applications.

Hardware Architecture

Versal ACAP is powered by scalable, adaptable, and Intelligent engines. On-chip memory access for all the machines is enabled via network on chip (NoC).

Source: Xilinx

Scalar Engines

Scalar engines power platform computing, decision making, and control. For general-purpose computing, dual-core ARM cortex- A72 Application Processing Unit (APU) is used in versal. APU supports virtualization allowing multiple software stacks to run simultaneously. The dual core ARM cortex R5F Realtime Processing Unit (RPU) is available for real-time applications. RPU can be configured as a single/dual processor in lockstep mode. RPU can be used for variety of time-critical applications, e.g., safety in the automotive domain.

Platform Management Controller

Platform Management Controller (PMC) is responsible for boot, configuration, partial re-configuration, and general platform management tasks, including power, clock, pin control, reset management, and system monitoring. It is also responsible for device life cycle management, including security.

Adaptable Engines

The adaptable engines feature the classic FPGA technology – the programable silicon. Adaptable engines include DSP engines (Adaptable), configurable logic blocks (Intelligent), and two types of RAM (Block RAM and Ultra RAM (adaptable)). Using such a configurable structure, users can create any kind of accelerator for different kinds of applications.

Intelligent Engines

AI engines are software programable and hardware adaptable. They are an array of VLIW SIMD vector processors used for ML/AI inference and advanced signal processing. AI engine is tile-based architecture. Each tile is made of a vector processor, scaler processor, dedicated program and data memory, dedicated AXI data movement channels, DMA, and locks.

Network on Chip

Network on Chip (NoC) makes Versal ACAPs even more powerful by connecting all engines, memory hierarchy, and highspeed IOs. NoC makes each hardware component and soft IP modules accessible to each other and the software via a memory-mapped interface.

Software Support

Xilinx introduced Vitis – A unified software development platform that enables embedded software and accelerated applications on heterogeneous Xilinx platforms, including FPGAs, SoCs, and Versal ACAPs.

The Vitis unified software development platform provides sets of open-source libraries, enabling developers to build hardware-accelerated applications without hardware knowledge. It also provides Xilinx Runtime Library (XRT), including firmware, board utilities, kernel driver, user-space libraries, and APIs. Vitis also provides an AI development environment including a deep learning framework like TensorFlow, PyTorch, and Caffe, and offers comprehensive APIs to prune, quantize, optimize, debug, and compile trained networks to achieve the highest AI inference performance.

Source: Xilinx

Softnautics have a wide range of expertise on various platforms including vision and image processing on VLIW SIMD vector processor, FPGA design development, Linux kernel driver development, Platform and Power Management Multimedia development.

Softnautics is developing high-performance Vision & ML/AI solutions using Versal ACAP by utilizing high bandwidth and configurable NoC, AI engine tile array in tandem with DMA and interconnect with PL. Versal’s high bandwidth interfaces and high compute processors can improve performance. One such use-case that Softnautics is already developing Scene Text Detection solution using Vitis AI & DPU.

The Scene Text Detection use-case demands high power compute for LSTM operations. Our AI/ML engineers’ team evaluates their design to leverage the custom memory hierarchy and the multicast stream capability on AI interconnect and AI-optimized vector instructions to gain the best performance. With a powerful AI Engine DMA capability and ping-pong buffering of stream data onto local tile memory, the ability of parallel processing opens a plethora of optimized implementations. Direct memory access (DMA) in the AI Engine tile moves data from the incoming stream(s) to local memory and from local memory to outgoing stream(s). Configuration interconnect (through memory-mapped AXI4 interface) with a shared, transaction-based switched interconnect provides access from external masters to internal AI Engine tile.

Further, cascade streams across multiple AI Engine tiles allow for greater flexibility in design by accommodating multiple ML inference instances. Along with the deep understanding of Versal ACAP memory hierarchies, AI Engine Tiles, DMA, and parallel processing, Softnautics’ extensive experience in leading ML/AI frameworks TensorFlow, PyTorch, and Caffe aids in creating end to end accelerated ML/AI pipelines with a focus on pre/post-processing of streams and model customization.

Softnautics has also been an early major contributor in Versal ACAP Platform Management related developments. Some of the key contributions in this space involve developing software components on Versal, such as platform management library (xilpm), Arm Trusted Firmware, Linux device drivers, u-boot for Platform Management.

Through our hands-on experience on Versal ACAP for AI/ML, Machine Vision & Platform Management, Softnautics can help customers take their concepts to design & deployment in a seamless fashion.

Read our success stories related to Machine Learning expertise to know more about our services for accelerated AI solutions.

Contact us at business@softnautics.com for any queries related to your solution or for consultancy.

[elementor-template id=”12042″]

VITIS is a unified software platform for developing SW (BSP, OS, Drivers, Frameworks, and Applications) and HW (RTL, HLS, Ips, etc.) using Vivado and other components for Xilinx FPGA SoC platforms like ZynqMP UltraScale+ and Alveo cards. The key component of VITIS SDK, the VITIS AI runtime (VART), provides a unified interface for the deployment of end ML/AI applications on Edge and Cloud.

Vitis™ AI components:

• Optimized IP cores
• Tools
• Libraries
• Models
• Example Reference Designs

Inference in machine learning is computation-intensive and requires high memory bandwidth and high performance compute to meet the low-latency and high-throughput requirements of various end applications.

Vitis AI Workflow

Xilinx Vitis AI provides an innovative workflow to deploy deep learning inference applications on Xilinx Deep Learning Processing Unit (DPU) using a simple process:

Source: Xilinx

• The Deep Processing Unit (DPU) is a configurable computation engine optimized for convolution neural networks for deep learning inference applications and placed in programmable logic (PL). DPU contains efficient and scalable IP cores that can be customized to meet many different applications’ needs. The DPU defines its own instruction set, and the Vitis AI compiler generates instructions.
• VITIS AI compiler schedules the instructions in an optimized manner to get the maximum performance possible.
• Typical workflow to run any AI Application on Xilinx ZynqMP UltraScale+ SoC platform comprises:

1. Model Quantization
2. Model Compilation
3. Model Optimization (Optional)
4. Build DPU executable
5. Build software application
6. Integrate VITIS AI Unified APIs
7. Compile and link the hybrid DPU application
8. Deploy the hybrid DPU executable on FPGA

AI Quantizer

AI Quantizer is a compression tool for the quantization process by converting 32-bit floating-point weights and activations to fixed point INT8. It can reduce the computing complexity without losing accurate information for the model. The fixed point model needs less memory, thus providing faster execution and higher power efficiency than floating-point implementation.

AI Quantizer

AI Compiler

AI compiler maps a network model to a highly efficient instruction set and data flow. Input to the compiler is Quantized 8-bit neural network, and output is DPU kernel, the executable which will run on the DPU. Here, the unsupported layers need to be deployed in the CPU OR model can be customized to replace and remove those unsupported operations. It also performs sophisticated optimizations such as layer fusion, instruction scheduling and reuses on-chip memory as much as possible.

Once we get Executable for the DPU, one needs to use Vitis AI unified APIs to Initialize the data structure, initialize the DPU, implement the layers not supported by the DPU on CPU & Add the pre-processing and post-processing on a need basis on PL/PS.

AI Compiler

AI Optimiser

With its world-leading model compression technology, AI Optimizer can reduce model complexity by 5x to 50x with minimal impact on accuracy. This deep compression takes inference performance to the next level.

We can achieve desired sparsity and reduce runtime by 2.5x.

AI Optimizer

AI Profiler

AI Profiler can help profiling inference to find caveats causing a bottleneck in the end-to-end pipeline.

Profiler gives a designer a common timeline for DPU/CPU/Memory. This process doesn’t change any code and can also trace the functions and do profiling.

AI Profiler

AI Runtime

VITIS AI runtime (VART) enables applications to use unified high-level runtime APIs for both edge and cloud deployments, making it seamless and efficient. Some of the key features are:

• Asynchronous job submission
• Asynchronous job collection
• C++ and Python implementations
• Multi-threading and multi-process execution

Vitis AI also offers DSight, DExplorer, DDump, & DLet, etc., for various task execution.

DSight & DExplorer
DPU IP offers a number of configurations to specific cores to choose as per the network model. DSight tells us the percentage utilization of each DPU core. It also gives the efficiency of the scheduler so that we could tune user threads. One can also see performance numbers like MOPS, Runtime, memory bandwidth for each layer & each DPU node.

Softnautics have a wide range of expertise on various edge and cloud platforms, including vision and image processing on VLIW SIMD vector processor, FPGA, Linux kernel driver development, platform and power management multimedia development. We provide end-to-end ML/AI solutions from dataset preparation to application deployment on edge and cloud and including maintenance.

We chose the Xilinx ZynqMP UltraScale+ platform for high-performance to compute deployments. It provides the best application processing, highly configurable FPGA acceleration capabilities, and  VITIS SDK to accelerate high-performance ML/AI inferencing. One such application we targeted was face-mask detection for Covid-19 screening. The intention was to deploy multi-stream inferencing for Covid-19 screening of people wearing masks and identify non-compliance in real time, as mandated by various governments for Covid-19 precautions guidelines.

We prepared a dataset and selected pre-trained weights to design a model for mask detection and screening. We trained and pruned our custom models via the TensorFlow framework. It was a two-stage deployment of face detection followed by mask detection. The trained model thus obtained was passed through VITIS AI workflow covered in earlier sections. We observed 10x speed in inference time as compared to CPU. Xilinx provides different debugging tools and utilities that are very helpful during initial development and deployments. During our initial deployment stage, we were not getting detections for mask and Non-mask categories. We tried to match PC-based inference output with the output from one of the debug utilities called Dexplorer with debug mode & root-caused the issue to debug this further. Upon running the quantizer, we could tune the output with greater calibration images and iterations and get detections with approximation. 96% accuracy on the video feed. We also tried to identify the bottleneck in the pipeline using AI profiler and then taking corrective actions to remove the bottleneck by various means, like using HLS acceleration to compute bottleneck in post-processing.

Face Detection via AI

Read our success stories related to Machine Learning expertise to know more about our services for accelerated AI solutions.

Contact us at business@softnautics.com for any queries related to your solution or for consultancy.

[elementor-template id=”12063″]

The rich, precise high-level semantics embodied in the text helps understand the world around us and build autonomous-capable solutions that can be deployed in a live environment. Therefore, automatic text reading from natural environments, also known as scene text detection/recognition or PhotoOCR, has become an increasingly popular and important research topic in computer vision.

As the written form of human languages evolved, we developed thousands of unique font-families. When we add case (capitals/lower case/uni-case/small caps), skew (italic/roman), proportion (horizontal scale), weight, size-specific (display/text), swash, and serifization (serif/sans in super-families), the number grows in millions, and it makes text identification an exciting discipline for Machine Learning.

Xilinx as a choice for OCR solutions

Today, Xilinx powers 7 out of 10 new developments through its wide variety of powerful platforms and leads the FPGA-based system design trends. Softnautics chose Xilinx for implementing this solution because of the integrated Vitis™ AI stack and strong hardware capabilities.

Xilinx Vitis™ is a free and open-source development platform that packages hardware modules as software-callable functions and is compatible with standard development environments, tools, and open-source libraries. It automatically adapts software and algorithms to Xilinx hardware without the need for VHDL or Verilog expertise.

Selecting the right Xilinx Platform

The comprehensive and rich Xilinx toolset and ecosystem make prototyping a very predictable process expedites the development of the solutions to reduce overall development time by up to 70%.
Softnautics chose Xilinx Ultrascale+ platform as it offers the best of application processing and FPGA acceleration capabilities. It also provides impressive high-level synthesis capability resulting in 5x system-level performance per watt compared to earlier variants. It supports Xilinx Vitis AI that offers a wide range of capabilities to build AI inferencing using acceleration libraries.

Softnautics used Xilinx Vitis AI stack and acceleration utilizing the software to create a hybrid application and implemented LSTM functionality for effective sequence prediction by porting/migrating TensorFlow-lite to ARM. It is running on Processing Side (PS) using the N2Cube Software. Image pre- and post-processing was achieved using HLS through Vivado and Vitis was used for inferencing using CTPN (Connectionist Text Proposal Network). We eventually graduated the solution to real-time scene text detection with video pipeline and improved the model with a robust dataset.

Scene Text Detection

There are many implementations available, and new ones are being researched. Still, a series of grand challenges may still be encountered when detecting and recognizing text in the wild. The difficulties in natural scene mainly stem from three differences when compared to scripts in documents:

• Diversity and Variability are arising from languages, colors, fonts, sizes, orientations, etc.
• Vibrant background on which text is written
• The aspect ratios and layouts of scene text may vary significantly

This type of solution has extensive applicability in various fields requiring real-time text detection on a video stream with higher accuracy and quick recognition. Few of these application areas are:

• Parking validation — Cities and towns are using mobile OCR to validate if cars are parked according to city regulations automatically. Parking inspectors can use a mobile device with OCR to scan license plates of vehicles and check with an online database to see if they are permitted to park.
• Mobile document scanning — A variety of mobile applications allow users to take a photo of a document and convert it to text. This OCR task is more challenging than traditional document scanners because photos have unpredictable image angles, lighting conditions, and text quality.
• Digital asset management – The software helps organize rich media assets such as images, videos, and animations. A key aspect of DAM systems is the search-ability of rich media. By running OCR on uploaded images and video frames, DAM can make rich media searchable and enrich it with meaningful tags.

Softnautics team has been working on Xilinx FPGA based solutions that require design and software framework implementation. Our vast experience with Xilinx and understanding of intricacies ensured we took this solution from conceptualization to proof-of-concept within 4 weeks. Using our end-to-end solution building expertise, you can visualize your ideas with the fastest concept realization service on Xilinx Platforms and achieve greatly reduced time-to-market.

Read our success stories related to Machine Learning expertise to know more about our services for accelerated AI solutions.

Contact us at business@softnautics.com for any queries related to your solution or for consultancy.

Source: Xilinx