【Deep Learning OCR Series·5】Principle and Implementation of Attention Mechanism

## Introduction The Attention Mechanism is an important innovation in the field of deep learning, which simulates selective attention in human cognitive processes. In OCR tasks, the attention mechanism can help the model dynamically focus on important areas in the image, significantly improving the accuracy and efficiency of text recognition. This article will delve into the theoretical foundations, mathematical principles, implementation methods, and specific applications of attention mechanisms in OCR, providing readers with comprehensive technical understanding and practical guidance. ## Biological Implications of Attention Mechanisms ### Human Visual Attention System The human visual system has a strong ability to selectively pay attention, which allows us to efficiently extract useful information in complex visual environments. When we read a piece of text, the eyes automatically focus on the character that is currently being recognized, with moderate suppression of the surrounding information. **Characteristics of Human Attention**: - Selectivity: Ability to select important sections from a large amount of information - Dynamic: Attention focuses dynamically adjust based on task demands - Hierarchicality: Attention can be distributed at different levels of abstraction - Parallelism: Multiple related regions can be focused on simultaneously - Context-Sensitivity: Attention allocation is influenced by contextual information **Neural Mechanisms of Visual Attention**: In neuroscience research, visual attention involves the coordinated work of multiple brain regions: - Parietal cortex: responsible for the control of spatial attention - Prefrontal cortex: responsible for goal-oriented attention control - Visual Cortex: Responsible for feature detection and representation - Thalamus: serves as a relay station for attention information ### Computational Model Requirements Traditional neural networks typically compress all input information into a fixed-length vector when processing sequence data. This approach has obvious information bottlenecks, especially when dealing with long sequences, where early information is easily overwritten by subsequent information. **Limitations of Traditional Methods**: - Information bottlenecks: Fixed-length encoded vectors struggle to hold all important information - Long-Distance Dependencies: Difficulty modeling relationships between elements that are far apart in an input sequence - Computational Efficiency: The entire sequence needs to be processed to get the final result - Explainability: Difficulty understanding the model's decision-making process - Flexibility: Unable to dynamically adjust information processing strategies based on task demands **Solutions to Attention Mechanisms**: The attention mechanism allows the model to selectively focus on different parts of the input while processing each output by introducing a dynamic weight allocation mechanism: - Dynamic Selection: Dynamically select relevant information based on current task requirements - Global Access: Direct access to any location of the input sequence - Parallel Computing: Supports parallel processing to improve computational efficiency - Explainability: Attention weights provide a visual explanation of the model's decisions ## Mathematical Principles of Attention Mechanisms ### Basic Attention Model The core idea of the attention mechanism is to assign a weight to each element of the input sequence, which reflects how important that element is to the task at hand. **Mathematical Representation**: Given the input sequence X = {x₁, x₂, ..., xn} and the query vector q, the attention mechanism calculates the attention weight for each input element: α_i = f(q, x_i) # Attention score function α̃_i = softmax(α_i) = exp(α_i) / Σj exp(αj) # Normalized weight The final context vector is obtained by weighted summing: c = Σᵢ α̃_i · x_i **Components of Attention Mechanisms**: 1. Query: Indicates the information that needs to be paid attention to at present 2. Key: The reference information used to calculate the attention weight 3. Value: Information that actually participates in the weighted sum 4. **Attention Function**: A function that calculates the similarity between queries and keys ### Detailed explanation of the attention score function The attention score function determines how the correlation between the query and the input is calculated. Different scoring functions are suitable for different application scenarios. **1. Dot-Product Attention**: α_i = q^T · x_i This is the simplest attention mechanism and is computationally efficient, but requires queries and inputs to have the same dimensions. **Pros**: - Simple calculations and high efficiency - Small number of parameters and no additional learnable parameters required - Effectively distinguish between similar and dissimilar vectors in high-dimensional space **Cons**: - Require queries and keys to have the same dimensions - Numerical instability can occur in high-dimensional space - Lack of learning ability to adapt to complex similarity relationships **2. Scaled Dot-Product Attention**: α_i = (q^T · x_i) / √d where d is the dimension of the vector. The scaling factor prevents the gradient disappearing problem caused by the large point product value in high-dimensional space. **The Necessity of Scaling**: When dimension d is large, the variance of the dot product increases, causing the softmax function to enter the saturation region and the gradient becomes small. By dividing by √d, the variance of the dot product can be kept stable. **Mathematical Derivation**: Assuming that the elements q and k are independent random variables, with a mean of 0 and a variance of 1, then: - q^T · The variance of k is d - The variance of (q^T · k) / √d is 1 **3. Additive Attention**: α_i = v^T · tanh(W_q · q + W_x · x_i) Queries and inputs are mapped to the same space through a learnable parameter matrix W_q and W_x, and then similarity is calculated. **Advantage Analysis**: - Flexibility: Can handle queries and keys in different dimensions - Learning Capabilities: Adapt to complex similarity relationships with learnable parameters - Expression Capabilities: Nonlinear transformations provide enhanced expression capabilities **Parameter Analysis**: - W_q ∈ R^{d_h×d_q}: Query the projection matrix - W_x ∈ R^{d_h×d_x}: Key projection matrix - v ∈ R^{d_h}: Attention weight vector - d_h: Hidden layer dimensions **4. MLP Attention**: α_i = MLP([q; x_i]) Use multilayer perceptrons to learn correlation functions between queries and inputs directly. **Network Structure**: MLPs typically contain 2-3 fully connected layers: - Input layer: splicing queries and key vectors - Hidden layer: Activate functions using ReLU or tanh - Output layer: Outputs scalar attention scores **Pros and Cons Analysis**: Pros: - Strongest expressive skills - Complex nonlinear relationships can be learned - No restrictions on input dimensions Cons: - Large number of parameters and easy overfitting - High computational complexity - Long training time ### Multiple Head Attention Mechanism Multi-Head Attention is a core component of the Transformer architecture, allowing models to pay attention to different types of information in parallel in different representation subspaces. **Mathematical Definition**: MultiHead(Q, K, V) = Concat(head₁, head₂, ..., headₕ) · W^O where each attention head is defined as: headᵢ = Attention(Q· W_i^Q, K· W_i^K, V·W_i^V) **Parameter Matrix**: - W_i^Q ∈ R^{d_model×d_k}: The query projection matrix of the ith header - W_i^K ∈ R^{d_model×d_k}: the key projection matrix of the ith header - W_i^V ∈ R^{d_model×d_v}: Value projection matrix for the ith head - W^O ∈ R^{h·d_v×d_model}: Output projection matrix **Advantages of Bull Attention**: 1. **Diversity**: Different heads can focus on different types of traits 2. **Parallelism**: Multiple heads can be computed in parallel, improving efficiency 3. **Expression Ability**: Enhanced the model's representation learning ability 4. **Stability**: The integration effect of multiple heads is more stable 5. **Specialization**: Each head can specialize in specific types of relationships **Considerations for Head Selection**: - Too few heads: May not capture enough information diversity - Excessive Head Count: Increases computational complexity, potentially leading to overfitting - Common options: 8 or 16 heads, adjusted according to model size and task complexity **Dimension Allocation Strategy**: Usually set d_k = d_v = d_model / h to ensure that the total amount of parameters is reasonable: - Keep the total computational volume relatively stable - Each head has sufficient representation capacity - Avoid information loss caused by too small dimensions ## Self-attention mechanism ### The concept of self-attention Self-attention is a special form of attention mechanism in which queries, keys, and values all come from the same input sequence. This mechanism allows each element in the sequence to focus on all other elements in the sequence. **Mathematical Representation**: For the input sequence X = {x₁, x₂, ..., xn}: - Query matrix: Q = X · W^Q - Key matrix: K = X · W^K - Value matrix: V = X · W^V Attention output: Attention(Q, K, V) = softmax(QK^T / √d_k) · V **Calculation Process of Self-Attention**: 1. **Linear Transformation**: The input sequence is obtained by three different linear transformations to obtain Q, K, and V 2. **Similarity Calculation**: Calculate the similarity matrix between all position pairs 3. **Weight Normalization**: Use the softmax function to normalize attention weights 4. **Weighted Summing**: Weighted summing of value vectors based on attention weights ### Advantages of self-attention **1. Long-Distance Dependency Modeling**: Self-attention can directly model the relationship between any two positions in a sequence, regardless of distance. This is especially important for OCR tasks, where character recognition often requires consideration of contextual information at a distance. **Time Complexity Analysis**: - RNN: O(n) sequence calculation, difficult to parallelize - CNN: O(log n) to cover the entire sequence - Self-Attention: The path length of O(1) directly connects to any location **2. Parallel Computation**: Unlike RNNs, the calculation of self-attention can be fully parallelized, greatly improving training efficiency. **Parallelization Advantages**: - Attention weights for all positions can be calculated simultaneously - Matrix operations can take full advantage of the parallel computing power of GPUs - Training time is significantly reduced compared to RNN **3. Interpretability**: The attention weight matrix provides a visual explanation of the model's decisions, making it easy to understand how the model works. **Visual Analysis**: - Attention heatmap: Shows how much attention each location pays to the others - Attention Patterns: Analyze patterns of attention from different heads - Hierarchical Analysis: Observe changes in attention patterns at different levels **4. Flexibility**: It can be easily extended to sequences of different lengths without modifying the model architecture. ### Position Coding Since the self-attention mechanism itself does not contain position information, it is necessary to provide the model with position information of elements in the sequence through position coding. **The Necessity of Position Coding**: The self-attention mechanism is immutable, i.e., changing the order of the input sequence does not affect the output. But in OCR tasks, the location information of the characters is crucial. **Sine Position Coding**: PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model)) Among them: - pos: Location index - i: Dimension index - d_model: Model dimension **Advantages of Sine Position Coding**: - Deterministic: No learning required, reducing the amount of parameters - Extrapolation: Can handle longer sequences than when trained - Periodicity: It has a good periodic nature, which is convenient for the model to learn relative position relationships **Learnable Position Coding**: The position coding is used as a learnable parameter, and the optimal position representation is automatically learned through the training process. **Implementation method**: - Assign a learnable vector to each position - Add up with the input embeddings to get the final input - Update the position code with backpropagation **Pros and Cons of Learnable Position Coding**: Pros: - Adaptable to learn task-specific positional representations - Performance is generally slightly better than fixed-position encoding Cons: - Increase the amount of parameters - Inability to process sequences beyond the training length - More training data is needed **Relative Position Coding**: It does not directly encode absolute position, but encodes relative position relationships. **Implementation Principle**: - Adding relative position bias to attention calculations - Focus only on the relative distance between elements, not their absolute position - Better generalization ability ## Attention Applications in OCR ### Sequence-to-sequence attention The most common application in OCR tasks is the use of attention mechanisms in sequence-to-sequence models. The encoder encodes the input image into a sequence of features, and the decoder focuses on the relevant part of the encoder through an attention mechanism as it generates each character. **Encoder-Decoder Architecture**: 1. **Encoder**: CNN extracts image features, RNN encodes as sequence representation 2. **Attention Module**: Calculate the attention weight of the decoder state and the encoder output 3. **Decoder**: Generate character sequences based on attention-weighted context vectors **Attention Calculation Process**: At the decoding moment t, the decoder state is s_t, and the encoder output is H = {h₁, h₂, ..., hn}: e_ti = a(s_t, h_i) # Attention score α_ti = softmax(e_ti) # Attention weight c_t = Σᵢ α_ti · h_i # Context vector **Selection of Attention Functions**: Commonly used attention functions include: - Accumulated attention: e_ti = s_t^T · h_i - Additive attention: e_ti = v^T · tanh(W_s · s_t + W_h · h_i) - Bilinear attention: e_ti = s_t^T · W · h_i ### Visual Attention Module Visual attention applies attention mechanisms directly on the image feature map, allowing the model to focus on important areas in the image. **Spatial Attention**: Calculate attention weights for each spatial position of the feature map: A(i,j) = σ(W_a · [F(i,j); g]) Among them: - F(i,j): eigenvector of position (i,j). - g: Global context information - W_a: Learnable weight matrix - σ: sigmoid activation function **Steps to Achieve Spatial Attention**: 1. **Feature Extraction**: Use CNN to extract image feature maps 2. **Global Information Aggregation**: Obtain global features through global average pooling or global maximum pooling 3. **Attention Calculation**: Calculate attention weights based on local and global features 4. **Feature Enhancement**: Enhance the original feature with attention weights **Channel Attention**: Attention weights are calculated for each channel of the feature graph: A_c = σ(W_c · GAP(F_c)) Among them: - GAP: Global average pooling - F_c: Feature map of channel c - W_c: The weight matrix of the channel's attention **Principles of Channel Attention**: - Different channels capture different types of features - Selection of important feature channels through attention mechanisms - Suppress irrelevant features and enhance useful ones **Mixed Attention**: Combine spatial attention and channel attention: F_output = F ⊙ A_spatial ⊙ A_channel where ⊙ represents element-level multiplication. **Advantages of Mixed Attention**: - Consider the importance of both spatial and passage dimensions - More refined feature selection capabilities - Better performance ### Multiscale attention The text in the OCR task has different scales, and the multi-scale attention mechanism can pay attention to relevant information at different resolutions. **Characteristic Pyramid Attention**: The attention mechanism is applied to the feature maps of different scales, and then the attention results of multiple scales are fused. **Implementation Architecture**: 1. **Multi-scale feature extraction**: Use feature pyramid networks to extract features at different scales 2. **Scale-Specific Attention**: Calculate attention weights independently on each scale 3. **Cross-scale fusion**: Integrate attention results from different scales 4. **Final Prediction**: Make a final prediction based on the fused features **Adaptive Scale Selection**: According to the needs of the current recognition task, the most suitable feature scale is dynamically selected. **Selection Strategy**: - Content-Based Selection: Automatically selects the appropriate scale based on the image content - Task-Based Selection: Select the scale based on the characteristics of the identified task - Dynamic Weight Allocation: Assign dynamic weights to different scales ## Variations of attention mechanisms ### Sparse attention The computational complexity of the standard self-attention mechanism is O(n²), which is computationally expensive for long sequences. Sparse attention reduces computational complexity by limiting the range of attention. **Local Attention**: Each location focuses only on the location within the fixed window around it. **Mathematical Representation**: For position i, only the attention weight within the range of position [i-w, i+w] is calculated, where w is the window size. **Pros and Cons Analysis**: Pros: - Computational complexity reduced to O(n·w) - Local context information is maintained - Suitable for handling long sequences Cons: - Unable to capture long-distance dependencies - Window size needs to be carefully tuned - Potential loss of important global information **Chunking Attention**: Divide the sequence into chunks, each focusing only on the rest within the same block. **Implementation method**: 1. Divide the sequence of length n into n/b blocks, each of which is a size b 2. Calculate complete attention within each block 3. No attention calculation between blocks Computational complexity: O(n·b), where b << n **Random Attention**: Each position randomly selects a part of the location for attention calculation. **Random Selection Strategy**: - Fixed Random: Predetermined random connection patterns - Dynamic Random: Dynamically select connections during training - Structured Random: Combines local and random connections ### Linear attention Linear attention reduces the complexity of attention calculations from O(n²) to O(n) through mathematical transformations. **Nucleated Attention**: Approximating softmax operations using kernel functions: Attention(Q, K, V) ≈ φ(Q) · (φ(K)^T · V) φ of these are feature mapping functions. **Common Kernel Functions**: - ReLU core: φ(x) = ReLU(x) - ELU Kernel: φ(x) = ELU(x) + 1 - Random feature kernels: Use random Fourier features **Advantages of Linear Attention**: - Computational complexity increases linearly - Memory requirements are significantly reduced - Suitable for handling very long sequences **Performance Trade-offs**: - Accuracy: Typically slightly below standard attention - Efficiency: Significantly improves computational efficiency - Applicability: Suitable for resource-constrained scenarios ### Cross attention In multimodal tasks, cross-attention allows for the interaction of information between different modalities. **Image-Text Cross Attention**: Text features are used as queries, and image features are used as keys and values to realize text's attention to images. **Mathematical Representation**: CrossAttention(Q_text, K_image, V_image) = softmax(Q_text · K_image^T / √d) · V_image **Application Scenarios**: - Image description generation - Visual Q&A - Multimodal document comprehension **Two-Way Cross Attention**: Calculate both image-to-text and text-to-image attention. **Implementation method**: 1. Image to Text: Attention (Q_image, K_text, V_text) 2. Text to Image: Attention (Q_text, K_image, V_image) 3. Feature fusion: Merge attention results in both directions ## Training Strategies and Optimization ### Attention Supervision Guide the model to learn the correct attention patterns by providing supervised signals for attention. **Attention Alignment Loss**: L_align = || A - A_gt|| ² Among them: - A: Predicted attention weight matrix - A_gt: Authentic attention tags **Supervised Signal Acquisition**: - Manual Annotation: Experts mark important areas - Heuristics: Generate attention labels based on rules - Weak supervision: Use coarse-grained supervisory signals **Attention regularization**: Encourage sparsity or smoothness of attention weights: L_reg = λ₁ · || A|| ₁ + λ₂ · || ∇A|| ² Among them: - || A|| ₁: L1 regularization to encourage sparsity - || ∇A|| ²: Smoothness regularization, encouraging similar attention weights in adjacent positions **Multitasking Learning**: Attention prediction is used as a secondary task and trained in conjunction with the main task. **Loss Function Design**: L_total = L_main + α · L_attention + β · L_reg where α and β are the hyperparameters that balance different loss terms. ### Attention Visualization Visualization of attention weights helps to understand how the model works and debug model problems. **Heat Map Visualization**: Map the attention weights as a heat map, overlaying them on the original image to show the area of interest of the model. **Implementation Steps**: 1. Extract the attention weight matrix 2. Map the weight values to the color space 3. Adjust the heat map size to match the original image 4. Overlay or side-by-side **Attention Trajectory**: Displays the movement trajectory of the focus of attention during decoding, aiding in understanding the model's recognition process. **Trajectory Analysis**: - The order in which attention moves - Attention span dwelling - Pattern of attention jumps - Identification of abnormal attention behavior **Multi-Head Attention Visualization**: The weight distribution of different attention heads is visualized separately, and the degree of specialization of each head is analyzed. **Analytical Dimensions**: - Head-to-Head Differences: Regional differences of concern for different heads - Head specialization: Some heads specialize in specific types of features - Importance of Heads: The contribution of different heads to the final result ### Computational Optimization **Memory Optimization**: - Gradient checkpoints: Use gradient checkpoints in long sequence training to reduce memory footprint - Mixed Precision: Reduces memory requirements with FP16 training - Attention Caching: Caches calculated attention weights **Computational Acceleration**: - Matrix chunking: Calculate large matrices in chunks to reduce memory peaks - Sparse Calculations: Accelerate calculations with the sparsity of attention weights - Hardware Optimization: Optimize attention calculations for specific hardware **Parallelization Strategy**: - Data Parallelism: Process different samples in parallel on multiple GPUs - Model parallelism: Distribute attention calculations across multiple devices - Pipeline parallelization: Pipeline different layers of compute ## Performance evaluation and analysis ### Attention Quality Assessment **Attention Accuracy**: Measure the alignment of attention weights with manual annotations. Calculation Formula: Accuracy = (Number of Positions Correctly Focused) / (Total Positions) **Concentration**: The concentration of the attention distribution is measured using entropy or the Gini coefficient. Entropy Calculation: H(A) = -Σᵢ αᵢ · log(αᵢ) where αi is the attention weight of the ith position. **Attention Stability**: Evaluate the consistency of attention patterns under similar inputs. Stability indicators: Stability = 1 - || A₁ - A₂|| ₂ / 2 where A₁ and A₂ are the attention weight matrices of similar inputs. ### Computational Efficiency Analysis **Time Complexity**: Analyze the computational complexity and actual running time of different attention mechanisms. Complexity comparison: - Standard attention: O(n²d) - Sparse attention: O(n·k·d), k<< n - Linear attention: O(n·d²) **Memory Usage**: Evaluate the demand for GPU memory for attention mechanisms. Memory Analysis: - Attention Weight Matrix: O(n²) - Intermediate calculation result: O(n·d) - Gradient Storage: O(n²d) **Energy Consumption Analysis**: Evaluate the energy consumption impact of attention mechanisms on mobile devices. Energy Consumption Factors: - Calculation Strength: Number of floating-point operations - Memory access: Data transfer overhead - Hardware Utilization: Efficient use of computing resources ## Real-World Application Cases ### Handwritten text recognition In handwritten text recognition, the attention mechanism helps the model focus on the character it is currently recognizing, ignoring other distracting information. **Application Effects**: - Recognition accuracy increased by 15-20% - Enhanced robustness for complex backgrounds - Improved ability to handle irregularly arranged text **Technical Implementation**: 1. **Spatial Attention**: Pay attention to the spatial area where the character is located 2. **Temporal Attention**: Utilize the temporal relationship between characters 3. **Multi-Scale Attention**: Handle characters of different sizes **Case Study**: In handwritten English word recognition tasks, attention mechanisms can: - Accurately locate the position of each character - Deal with the phenomenon of continuous strokes between characters - Utilize language model knowledge at the word level ### Scene text recognition In natural scenes, text is often embedded in complex backgrounds, and attention mechanisms can effectively separate text and background. **Technical Features**: - Multi-scale attention to work with text of different sizes - Spatial attention to locate text areas - Channel attention selection of useful features **Challenges and Solutions**: 1. **Background Distraction**: Filter out background noise with spatial attention 2. **Lighting Changes**: Adapt to different lighting conditions through channel attention 3. **Geometric Deformation**: Incorporates geometric correction and attention mechanisms **Performance Enhancements**: - 10-15% improvement in accuracy on ICDAR datasets - Significantly enhanced adaptability to complex scenarios - Reasoning speed is kept within acceptable limits ### Document Analysis In document analysis tasks, attention mechanisms help models understand the structure and hierarchical relationships of documents. **Application Scenarios**: - Table Identification: Focus on the column structure of the table - Layout Analysis: Identify elements such as headlines, body, images, and more - Information extraction: locate the location of key information **Technological Innovation**: 1. **Hierarchical Attention**: Apply attention at different levels 2. **Structured Attention**: Consider the document's structured information 3. **Multimodal Attention**: Blending text and visual information **Practical Results**: - Increase the accuracy of table recognition by more than 20% - Significantly increased processing power for complex layouts - The accuracy of information extraction has been greatly improved ## Future development trends ### Efficient attention mechanism As the length of the sequence increases, the computational cost of the attention mechanism becomes a bottleneck. Future research directions include: **Algorithm Optimization**: - More efficient sparse attention mode - Improvements in approximate calculation methods - Hardware-friendly attention design **Architectural Innovation**: - Hierarchical attention mechanism - Dynamic attention routing - Adaptive calculation charts **Theoretical Breakthrough**: - Theoretical analysis of the mechanism of attention - Mathematical proof of optimal attention patterns - Unified theory of attention and other mechanisms ### Multimodal attention Future OCR systems will integrate more information from multiple modalities: **Visual-Language Fusion**: - Joint attention of images and text - Information transmission across modalities - Unified multimodal representation **Temporal Information Fusion**: - Timing attention in video OCR - Text tracking for dynamic scenes - Joint modeling of space-time **Multi-Sensor Fusion**: - 3D attention combined with depth information - Attention mechanisms for multispectral images - Joint modeling of sensor data ### Interpretability Enhancement Improving the interpretability of attention mechanisms is an important research direction: **Attention Explanation**: - More intuitive visualization methods - Semantic explanation of attention patterns - Error analysis and debugging tools **Causal Reasoning**: - Causal analysis of attention - Counterfactual reasoning methods - Robustness verification technology **Human-Computer Interaction**: - Interactive attention adjustments - Incorporation of user feedback - Personalized attention mode ## Summary As an important part of deep learning, the attention mechanism plays an increasingly important role in the field of OCR. From basic sequence to sequence attention to complex multi-head self-attention, from spatial attention to multi-scale attention, the development of these technologies has greatly improved the performance of OCR systems. **Key Takeaways**: - The attention mechanism simulates the ability of human selective attention and solves the problem of information bottlenecks - Mathematical principles are based on weighted summing, enabling information selection by learning attention weights - Multi-head attention and self-attention are the core techniques of modern attention mechanisms - Applications in OCR include sequence modeling, visual attention, multi-scale processing, and more - Future development directions include efficiency optimization, multimodal fusion, interpretability enhancement, etc **Practical Advice**: - Choose the appropriate attention mechanism for the specific task - Pay attention to the balance between computational efficiency and performance - Make full use of the interpretability of attention for model debugging - Keep an eye on the latest research advancements and technological developments As technology continues to evolve, attention mechanisms will continue to evolve, providing even more powerful tools for OCR and other AI applications. Understanding and mastering the principles and applications of attention mechanisms is crucial for technicians engaged in OCR research and development.