Project S05 - Dynamic Real-Time Material Map

Principal Investigators: Prof. Dr. Diana Göhringer, TUD / Prof. Dr. Thomas Kaiser, UDE

Achieved results and methods

To fully harness the potential of SAR technique in advancing the MARIE’s vision, exploration is conducted across the frequency spectrum even from 5 GHz to 1.5 THz. The study investigates the benefits and limitations of various frequency regions for targeted applications, encompassing both planar and circular SAR configurations. It specifically focuses on eight frequency bands: 5-10 GHz, 68-92 GHz, 75-110 GHz, 0.122-0.168 THz, 0.22-0.33 THz, 0.325-0.5 THz, 0.85-1.1 THz, and 1.1-1.5 THz, for a comparative analysis. The imaging targets under scrutiny include both metal and plastic materials. With metal, the focus is on surface analysis and the plastic object provides the opportunity for analysis of volume scattering. It is demonstrated that the SAR image quality enhances significantly at the THz spectrum (> 100 GHz) and the frequency spectrum of 0.22-0.5 THz has a vast potential for both high-resolution surface imaging and look-through imaging [1].

Following the comparative analysis, a frequency spectrum ranging from 0.325 THz to 0.5 THz has been selected for high-resolution 3D mapping of multi-object indoor environments [2, 3]. This selection aims to facilitate the expansion of the SAR map for comprehensive object recognition, encompassing detection, localization, and classification. Concealed and hidden object scenarios are also considered to validate the objectives of object recognition in both free-space and concealed cases. The high-resolution mapped environment is firstly processed for object detection, where Speeded Up Robust Features (SURF) are extracted. Further, features are clustered in groups based on the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm. As of the 3D-mapped environment, it becomes feasible to estimate the 3D positions of detected objects with an accuracy in the range of sub-mm (see S04). Finally, the classification of detected objects is addressed using machine learning techniques. The THz training dataset is scarce, and especially for indoor objects, currently no public-domain dataset is available to the best of our knowledge. Hence, a dataset is generated and a supervised machine learning-based Support Vector Machine (SVM) model is implemented. The findings are published in [2], with some of the key results depicted in Fig. 1. The camera picture of the mapping environment is shown on the left, and the estimated clusters 1-4 are represented in the center in Fig. 1, outlining detected objects in the corresponding SAR image. Finally, the recognized labeled objects (keyboard, calculator, USB stick and mobile) are shown on the right in Fig. 1. Moreover, the developed model robustness is also evaluated, where truncated (incomplete) or modified SAR images are considered. This evaluation is beneficial as in many cases such as time-critical applications, generating a precise focused SAR image might be complex. In addition, there could be cases that introduce unintentional artifacts. However, based on the proposed method, there is a possibility or opportunity to classify the object in images with artifacts. A prediction accuracy of 93% is achieved based on the showcased results in [2]. Moreover, with the philosophy of a fast material map supported by radar and camera sensors, a drone-based setup is replicated. It employs a side-looking 2D SAR configuration, and the mapped environment objects in the SAR image are recognized with a radar-camera fusion method proposed in [4].

Concentrating on the analysis of surface roughness, the assessment of the scattering characteristics between rough and smooth surfaces at 1.5 THz is emphasized. This evaluation is conducted using a SAR processing sequence as introduced in [5]. Although the roughness average differs in the µm range, the variance in the scattering behavior is observed in the acquired results. Furthermore, for drone-based THz SAR, MOCO is one of the primary concerns due to the high instability of the mobile platform and especially in the case of a flying drone. Therefore, an investigation of the effects of sub-mm translational deviations is undertaken in [6]. Additionally, a new threshold, framed in terms of the necessary accuracy, is established as a more lenient constraint, considering the trade-off with image quality. The findings reveal the different positioning accuracy requirements along the range and cross-range directions. It has been found that the impact of errors along the propagation direction, in the investigated geometry, is the most sensitive. On the other hand, the effects of errors in the perpendicular direction to the propagation and flying path is the least sensitive. Based on the analysis, an analytical model is developed to calculate the cut-off motion errors along different dimensions. The estimated cut-off limits are defined, where the image quality is not anymore acceptable or degraded adversely. The limits are well supported by the simulation and measurement results. For MOCO, the integration solution of localization and SAR is used. Firstly, the proposed solution is substantiated through simulation outcomes published in [6]. Moreover, the validity of the solution is confirmed through experimental results as SLAM presented in [7]. A version of a room equipped with a multi-tag system is considered for real-time applications, and a successful demonstration of indoor environment mapping for the frequency spectrum of 75-110 GHz is addressed. Additionally, the SAR-assisted sensing and clutter compression of harmonics [8, 9] and non-harmonic tags [10] are exploited.

Further, the penetration capabilities of the THz spectrum are investigated especially in the case of wall-scanning [11] and sensing behind fire and smoke. Considering the Non-Line-of-Sight (NLoS) THz sensing, multipath propagation is exploited up to the fifth order of reflection [12]. Many cases are investigated, where a metal reflector is used to excite the target of interest located in the same or different room. Also, the link budget analysis is conducted, taking into account the influence of the incidence angle (from 0° to 90°) on the reflector. Based on the analysis, 2D NLoS imaging is established [13]. Regarding the development and evaluation of 3D imaging algorithms, a theoretical derivation of the Fourier transform of the SAR data cube is performed. The obtained results are then verified and compared with the Range Migration Algorithm and Backprojection Algorithm [14].

In terms of material characterization, SAR assisted characterization model is developed and experimented with various materials such as glass, metal, wood, and plastic. In this approach, an environment enriched with multi-material objects is mapped and SAR image cube is generated. The cube is explored for the detection of multiple objects. Based on the estimation of the detected object's position and SAR aperture configuration, a specific set of SAR raw data is chosen for assessing the material properties of the object. The selected dataset undergoes filtering for compression of the impact from the environmental clutter and followed by the extraction of material-dependent signatures, targeting both the frequency and time domains. Further, focusing on off-board processing, accelerated imaging is investigated in consideration of multi-core Central Processing Unit (CPU) and Graphical Processing Unit (GPU) architectures. In motivation with the findings of different architectures, topologies, and methods, a hybrid processing scheme is proposed based on multi-core-multi-threaded architecture. The proposed scheme is evaluated on a High Performance Cluster (HPC) node featuring the Nvidia GPU RTX A600. In comparison to traditional sequential processing schemes, the benchmark showcased a maximum speedup of 132x in the case of 2D THz SAR image (with dimensions of 3000×3000 pixels) reconstruction and 64.5x for the reconstruction of 3D THz SAR image with 500³ voxels. Further, aiming for effective data compression from a signal processing perspective, interpolation methods are explored, enabling the efficient utilization of computational resources.  An extension of existing linear, cubic, and sinc interpolation algorithms to interpolate complex-valued SAR data with a phase control procedure is introduced [15]. Moreover, in focus on the reduction in the mapping time, a fast sampling approach is investigated [16].

To accelerate radar signal processing algorithms for THz SAR imaging and achieve maximum on-board processing, a tile-based platform called AGILER for heterogeneous many-core systems is proposed [17, 18, 19]. The platform is based on a scalable and modular tile-based architecture which features a high degree of heterogeneity supporting heterogeneous Instruction Set Architectures (ISAs) as well as custom hardware accelerators for general and domain specific workloads [20]. As reconfigurable hardware accelerators have shown high energy efficiency in ML and signal processing applications, a Domain-Aware Coarse-Grained Reconfigurable Architectures (DA-CGRA) can be integrated in the platform to support fast on-board computing of signal processing applications [21, 22]. It can run either as a standalone accelerator or in the AGILER tile. Due to the high demand of memory in THz SAR applications and inefficient performance of Dynamic Random-Access Memory (DRAM), we have explored new paradigms for memory subsystem [23, 24] in the platform. Therefore, a fast high-level machine learning framework called NDP-RANK is built to predict the most suitable memory subsystem per application. NDP-RANK can efficiently decide which Near Data Processing (NDP) system is suitable for an application.

The proposed AGILER platform supports self-adaptation to realize several many-core configurations and taxonomies at run-time. Moreover, it aims to ease the development and realization of heterogeneous many-core architectures by reducing the design time and the non-recurrent engineering costs. As shown in Fig. 2, the platform consists of four types of heterogeneous tiles: a main general purpose tile, two heterogeneous RISC-V ISA tiles, and a  tile for hardware accelerators like DA-CGRA. The tiles are connected using a Network-on-Chip (NoC). The degree of scalability for tile-based architectures relies on the inter-tile communication fabric which on recent many-core approaches depends on scalable NoC variant topologies. For self-adaptation, a custom internal reconfiguration management unit is implemented to manage and control the reconfiguration process at run-time to realize different heterogeneous many-core configurations.

The design is capable of residing domain-specific hardware accelerators in tiles to support a variety of application requirements and achieve higher performance. AGILER can reach a maximum of 685 MOPS performance and ~152 MOPS/W energy efficiency at 120 MHz on Xilinx Virtex Ultrascale+ FPGA. The architecture details of DA-CGRA are depicted in Fig. 3. DA-CGRA consists of heterogenous Processing Elements (PEs) which are connected through scalable switch boxes. The number of PEs and type of each PE is determined through application profiling. Due to heterogeneity, simple interconnect and word-level reconfigurability, DA-CGRA can achieve higher energy efficiency which is one of the concerns in mobile scenarios. DA-CGRA is synthesized using ASIC Nangate 45nm open cell library and can achieve the performance of 1300 MOPS and energy efficiency of 358 MOPS/mW while running signal processing applications at frequency of 100 MHz. Leveraging the advantage of domain-specific architectures, DA-CGRA can further increase the performance and energy-efficiency of AGILER by ~1.8x and 23x, respectively.

To further decrease the power consumption of custom hardware accelerators, we also explored Near-Threshold Computing (NTC) which is a powerful technique to reduce energy consumption of the circuit. NTC is an emerging approach that involves operating electronic devices at near threshold voltage levels. NTC is employed in a systolic array, due to the applicability of systolic arrays in accelerating ML and signal processing applications. Systolic arrays can be regarded as a subset of CGRAs which have demonstrated a high energy efficiency in accelerating ML applications targeting mobile devices. To detect faults caused by NTC in systolic array, we employed an Algorithm-Based Fault Tolerance (ABFT) technique. We have successfully reduced the dynamic power consumption by 48% while running matrix multiplication on AMD XILINX ZCU102 Ultrascale+ FPGA board. This approach can also be employed to convolution kernels in ML applications with negligible accuracy loss.

Selected project-related publications

1. Batra, J. Barowski, D. Damyanov, M. Wiemeler, I. Rolfes, T. Schultze, J. C. Balzer, D. Göhringer, and T. Kaiser, “Short-Range SAR Imaging From GHz to THz Waves,” IEEE Journal of Microwaves, vol. 1, no. 2, pp. 574-585, 2021. [DOI: 10.1109/JMW.2021.3063343]
2. Batra, F. Sheikh, M. Khaliel, M. Wiemeler, D. Göhringer, and T. Kaiser, “Object Recognition in High-Resolution Indoor THz SAR Mapped Environment,” Sensors, vol. 22, no. 10, article number 3762, 2022. [DOI: 10.3390/s22103762]
3. Sheikh, A. Prokscha, A. Batra, D. Lessy, B. Salah, B. Sievert, M. Degen, A. Rennings, M. Jalali, J. T. Svejda, P. Alibeigloo, C. Preuss, E. Mutlu, R. Kress, S. Clochiatti, K. Kolpatzeck, T. Kubiczek, I. Ullmann, K. Root, F. Brix, U. Krämer, M. Vossiek, J. C. Balzer, N. G. Weimann, T. Kaiser, and D. Erni, “Towards Continuous Real-Time Plant and Insect Monitoring by Miniaturized THz Systems,” IEEE Journal of Microwaves, vol. 3, no. 3, pp. 913-937, 2023. [DOI: 10.1109/JMW.2023.3278237]
4. Batra, M. El-Absi, M. Wiemeler, D. Göhringer and T. Kaiser, “Indoor THz SAR Trajectory Deviations Effects and Compensation With Passive Sub-mm Localization System,” in IEEE Access, vol. 8, pp. 177519-177533, 2020. [DOI: 10.1109/ACCESS.2020.3026884]
5. Batra, A. A. Abbas, J. Sánchez-Pastor, M. El-Absi, A. Jiménez-Sáez, M. Khaliel, J. Barowski, M. Wiemeler, D. Göhringer, I. Rolfes, R. Jakoby, and T. Kaiser, “Millimeter Wave Indoor SAR Sensing Assisted With Chipless Tags-Based Self-Localization System: Experimental Evaluation,” IEEE Sensors Journal, vol. 24, no. 1, pp. 844-857, 2024. [DOI: 10.1109/JSEN.2023.3332431]
6. T. Vu, M. I. Pettersson, A. Batra and T. Kaiser, “Fourier Transform of SAR Data Cube and 3-D Range Migration Algorithm,” in IEEE Transactions on Aerospace and Electronic Systems, vol. 58, no. 3, pp. 2584-2591, June 2022. [DOI: 10.1109/TAES.2021.3134724]
7. Ivanenko, V. T. Vu, A. Batra, T. Kaiser, and M. I. Pettersson, “Interpolation Methods with Phase Control for Backprojection of Complex-Valued SAR Data,” Sensors, vol. 22, no. 13, article number 4941, 2022. [DOI: 10.3390/s22134941]
8. Kamaleldin and D. Göhringer, “AGILER: An Adaptive Heterogeneous Tile-Based Many-Core Architecture for RISC-V Processors,” in IEEE Access, vol. 10, pp. 43895-43913, 2022. [DOI: 10.1109/ACCESS.2022.3168686]
9. Iskandar, M. A. Abd El Ghany, and D. Goehringer, “NDP-RANK: Prediction and ranking of NDP systems performance using machine learning,” Microprocessors and Microsystems, vol. 96, p. 104707, 2023. [DOI: 10.1016/j.micpro.2022.104707]
10. Iskandar, M. A. Abd El Ghany, and D. Göhringer, “Near-Memory Computing on FPGAs with 3D-Stacked Memories: Applications, Architectures, and Optimizations,” ACM Trans. Reconfigurable Technol. Syst., vol. 16, no. 1, Article 16, 32 pages, March 2023. [DOI: 10.1145/3547658]